login      my ps      about      contact  



home instructor-led training online training blogs


XML Nation » I finally get REST. Wow.

I finally get REST. Wow.

Yeah, I'm alive. And I remember the password to my blog. I've been away for a bit, working on something very cool involving the TV. If all goes well, you'll hear about it in a big way. Anyway, I'm still having a ball out here in reality. Building something real has a way of focusing your decisions about technology. My app is a distributed system, some of which runs in a cable plant head-end or telco office (whatever's on the other end of the wire in your living room), and some of which runs elsewhere. We also connect to some things on the Web. These connections have to be extremely flexible and the bar to adoption has to be low. The thing I finally realized (some of you will say “Duh!”) is that Web services are not a good way to do this.

It's depressing to think that SOAP started just about 10 years ago and that now that everything is said and done, we built RPC again. I know SOAP is really an XML messaging protocol, you can do oneway async stuff, etc, etc, but let's face it. The tools make the technology and the tools (and the examples and the advice you get) point at RPC. And we know what the problems with RPC are. If you want to build something that is genuinely loosely-coupled, RPC is a pretty hard path to take.

That realization would have gotten me down if not for the fact that something else jazzed me up an hour or so later. I was in the process of considering the alternatives when I finally understood REST. And wow, it was eye-opening. REST is often positioned as CRUD operations against entities identified by URIs. Then it is dismissed as to simplistic to be useful. You can't build with just CRUD, the reasoning goes, just think about why we write sprocs. I've been down that path any number of times and always ended up in the same place. But I had it all wrong.

I skimmed Fielding's thesis a while back, but it wasn't until I read Sam Ruby's recent posts that it really sank in. Here's what I came to understand. Every communication protocol has a state machine. For some protocols they are very simple, for others they are more complex. When you implement a protocol via RPC, you build methods that modify the state of the communication. That state is maintained as a black box at the endpoint. Because the protocol state is hidden, it is easy to get things wrong. For instance, you might call Process before calling Init. People have been looking for ways to avoid these problems by annotating interface type information for a long time, but I'm not aware of any mainstream solutions. The fact that the state of the protocol is encapsulated behind method invocations that modify that state in non-obvious ways also makes versioning interesting.

The essence of REST is to make the states of the protocol explicit and addressible by URIs. The current state of the protocol state machine is represented by the URI you just operated on and the state representation you retrieved. You change state by operating on the URI of the state you're moving to, making that your new state. A state's representation includes the links (arcs in the graph) to the other states that you can move to from the current state. This is exactly how browser based apps work, and there is no reason that your app's protocol can't work that way too. (The ATOM Publishing protocol is the canonical example, though its easy to think that its about entities, not a state machine.)

The “state machine as node graph traversed via URI” view of the world has really interesting implications for being able to suspend and resume a protocol. Because links to other states are embedded in a state's representation there are interesting ways to solve dynamic load-balancing, data-directed-routing, versioning and other problems using normal Web infrastructure. And because it's HTTP based, you get all the features that protocol supplies, including streaming and support for non-XML MIME types (a huge concern when you're doing TV stuff). The one thing that's really missing here is a simple framework for implementing a URI graph on top of an HTTP handler (similar to what Marc's been working on for Java). I'm building my own now.

The thing I love about this model is that, as Sam says, it is of the Web, not over the Web. That doesn't mean I'll use it for everything. I use TDS to get to SQL Server. I use WCF for RPC-style communication between distributed components within major systems. I'll use this model when I cross major system boundaries, especially when I don't own both sides. I'll let you know how it turns out.

 

 

Posted Apr 26 2007, 02:39 PM by tim-ewald

Comments

Chris Ortman wrote re: I finally get REST. Wow.
on 04-26-2007 3:40 PM
Great post. I've been toiling back and forth all week trying to choose between REST / Soap. I currently use soap, but the only reason for that is that's what makes ASMX happy, and I constantly get that queezy feeling because that framework either limits me or makes me get into all kinds of hairy details just to send someone some xml. You mentioned you're working on something like a URI graph on top of HTTP handler. I've done a little bit of experimenting with fitting this behaviour into MonoRail (http://www.castleproject.org/monorail/index.html) I just haven't had balls to try this approach full on due to lack of time but so far my results have been pleasing.
The 2 things I think it really offers at the moment are a convenient way to render XML using view engines and a solid architecture for pulling the data off the wire and putting it into my object model (I am not a big fan of xml serializer).
dave howard wrote re: I finally get REST. Wow.
on 04-26-2007 4:22 PM
sam ruby's link should be intertwingly.net instead of intertwingly.com
Tim wrote re: I finally get REST. Wow.
on 04-26-2007 4:40 PM
Doh! Fixed the link to Sam, thanks Dave!
Dennis wrote re: I finally get REST. Wow.
on 04-26-2007 4:51 PM
Woah....that is beautiful. You just changed the way I think about the entire Web.

I've been building webservices in .Net ever since .Net came out, never really paid much attention to REST.

Time to start googling.

Dennis wrote re: I finally get REST. Wow.
on 04-26-2007 5:03 PM
So, I've got Fielding's thesis, before I get started on all that I have one newbie question...how does that state transfer physically work? Ie., what is the REST equivalent of hyperlinks in HTML?
Brian Maso wrote re: I finally get REST. Wow.
on 04-26-2007 6:40 PM
I've been working on a SOAP-based protocol, PSP (www.powersportsstandardprotocol.com) for about half a year. This is a standard based on OAGiS. To sove the state problem, PSP uses a 2-part request, where the first part is a "Verb" and the second part is a "Noun". The noun, identified by a unique UUID, is a unit of knowledge owned by one side of the conversation (the "supplier"). PSP defines standardized verbs for requesting information or performing state transitions on nouns.

Examples of noun type include things like a "purchase order", a "warrantee claim" etc. For each noun type, the spec defines a state model. Using a noun.verb request, the client (a.k.a. "dealer" in PSP speak) makes requests of state changes in the original noun. There are also some simple query verbs a dealer uses to gather information about a noun.

So there's some definite similarities with what Tim is describing about REST I think: the idea of requests being commands to a finite state machine engine to move an addressable entity between states, or to gather information about current states.

The lack of a standard for endpoint descriptions in REST is a bit problematic for B2B applications. But the brain-dead use of WSDL/Schema as an RPC endpoint description makes SOAP a very annoying B2B electronic interlingua also. Use of SOAP almost always with HTTP(S) is brain-dead as well.
Brian Maso wrote re: I finally get REST. Wow.
on 04-26-2007 6:45 PM
Oops, got the URL wrong: www.powersportsstandardprotocol.org
Mark Baker wrote re: I finally get REST. Wow.
on 04-26-2007 7:31 PM
Dennis - state transfer refers to the exchange of data between agents, where the data represents the state of a resource at some point in time. Regarding links, links in HTML are RESTful; for other data formats, they'd also have to use links.
Mark Baker wrote re: I finally get REST. Wow.
on 04-26-2007 7:32 PM
Brian - the lack of "standard endpoint descriptions" in RESTful systems is a feature, not a bug. All resources expose the same interface, therefore there's no value in describing the differences because there aren't any!
Harold wrote re: I finally get REST. Wow.
on 04-26-2007 8:56 PM
Good stuff by Joe Gregorio here: http://bitworking.org/news/How_to_create_a_REST_Protocol.
Stefan Tilkov's Random Stuff wrote Tim Ewald Gets REST
on 04-26-2007 10:32 PM
Tim Ewald: I was in the process of considering the alternatives when I finally understood REST. And wow, it was eye-opening. REST is often positioned as CRUD operations against entities identified by URIs. Then it is dismissed as to simplistic to be useful. You can’t build with just CRUD, the reasoning goes, just think about why we write sprocs. I’ve...
Danny wrote re: I finally get REST. Wow.
on 04-27-2007 2:21 AM
Tim, nicely put. This might amuse you : http://dannyayers.com/engine-of-state/on
Ittay wrote Care to elaborate for a given example?
on 04-27-2007 5:13 AM
Say I want to model a chess opponent in REST. Now, in RPC, one major procedure would have been makeNextMove, which would have returning the move.

How do you do that in rest by modeling states? The obvious response is creating some URI like http://example.com/next_move that accepts GET. But isn't this just RPC?
Bill de hOra wrote re: I finally get REST. Wow.
on 04-27-2007 5:40 AM
"Say I want to model a chess opponent in REST. How do you do that in rest by modeling states?"

Post the move (or the entire board state) to the chess board. Plus, that's what distance chess players do more or less.
Ittay wrote re: I finally get REST. Wow.
on 04-27-2007 6:12 AM
sorry if i wasn't clear. makeNextMove was referring to a computer opponent. so i ask it, what is its next move.

thinking of it, probably the RESTful manner would have been just to inquire the state of the whole board. but that doesn't look easy to work with.

my question (and point) is, say i have an application with a function like 'int foo(string, int, bool)' (meaning, a function that takes many arguments and returns some value). how can you model that in REST?
Me wrote re: I finally get REST. Wow.
on 04-27-2007 6:25 AM
Ittay - If you don't have state in the endpoints, you would have to embed the complete state (chess position, e.g., in FEN notation) in the request.
BradK wrote re: I finally get REST. Wow.
on 04-27-2007 7:38 AM
Two things:

- I've not seen any good aiuthentication mechanisms for REST. How do you use this in the real world enterprise, where you have to authenticate and authorize ?

- I both hate and love soap headers. But I can't deny their usefulness in practice (correlation ID's for messages, for example). Does this metadata concept not apply in REST ?
Gordon Weakliem wrote re: I finally get REST. Wow.
on 04-27-2007 7:46 AM
Ittay - Tim's point is that it's really a different model. If you start by asking "how do I translate this RPC interface to REST?", you're approaching the problem the wrong way. It's more like the classic model for a filesystem, or device drivers.
I'm not very familiar with chess rules and I'm thinking about this model on the fly, but throwing out a guess at how I'd do it:

1) POST to the server to create a new game, saying which color the client is going to be. The server returns the URI of the "board"
2) GET the URI returned from the server at any time to get the current state of the board. You could optimize this by having POST return the current complete state of the board. If the server goes first, that state could be reflected in the first state, or you could have a separate operation on that URI to tell the server it's time to start play.
3) POST a move to the URI to ask the server for a state change (have the client make a move). Since chess is a synchronous protocol (players always alternate turns), you could have the server return its move in its response to the client. If the server will take a long time to think about its move, this obviously isn't a great idea; in that case you might poll the new board state with GET.

You said "probably the RESTful manner would have been just to inquire the state of the whole board" - how is that different from an RPC interface? My point is that you can do it either way - there's no rule that says you can't have the server respond to a client's state change request with its own declaration of state.
Chris Parker wrote re: I finally get REST. Wow.
on 04-27-2007 8:48 AM
At the end of the day, this is like xml-rpc, but without the protocol and all of the nice libraries. It IS nice, but at the end of the day you are still sending requests back and forth as XML. I like the current XML-based RPC mechanisms.
Andrew Wahbe wrote re: I finally get REST. Wow.
on 04-27-2007 8:59 AM
For what its worth, a conversation about this with a colleague yielded the following: if you think of the rules of chess to be the "protocol" then you can model the game states as documents identified by URIs. You GET a clean board with links for all possible moves (ya there's a lot but so what). You GET the appropriate link to make a move. The server applies both your move and the opponent's move and returns a new document with the board state and options. And so on.

The contrast with Joe's POST-based solution is very relevant to the article. With that solution the client has to know the rules of chess to construct a valid new board; with the link-based solution, it doesn't.
Andrew Wahbe wrote re: I finally get REST. Wow.
on 04-27-2007 9:06 AM
Sorry it was Bill not Joe... wrong REST guru ;-)
David Van Couvering 's Blog wrote REST as State Machine - Duh!
on 04-27-2007 9:26 AM
Tim Ewald get's the "aha" moment and shares it with the REST of us.
mamund wrote re: I finally get REST. Wow.
on 04-27-2007 10:21 AM
i'm with you!

i've been spending the last few weeks implementing a base class httphandler that i use to implement http methods ala REST. i still have lots to do, but now i can create a new class that already has all it's methods ready for overriding for GET, POST, etc.

it's taken me some time to get my head around it, but i spend my time creating my uri/representation/method tables and then popping into the code to implement each http method for the URIs.

i'm finding it much more direct than SOAP and i'm enjoying the process quite a bit.
Tim wrote re: I finally get REST. Wow.
on 04-27-2007 10:34 AM
Dennis,

Mark is right on the money. You want your state representation to include links. If you define your own XML format for the state, you can decide what syntax to use for representing them.

Tim-
Tim wrote re: I finally get REST. Wow.
on 04-27-2007 10:36 AM
Brian,

I agree with Mark about the lack of a standard for endpoint descriptions. The risk, I think, is in doing something that doesn't reflect the dynamicism that this solution allows, e.g., it makes assumptions about URI formats that you later want to change, or the introduction of new possible states as a protocol changes, or whatever.

If people feel we must have such a language, then at the very least we should wait for several years while we get experience building this way and come to understand what we really want. That way we have some hope of avoiding the fate of WSDL.

Tim-
Tim wrote re: I finally get REST. Wow.
on 04-27-2007 10:38 AM
Harold,

I looked at Joe's post at http://bitworking.org/news/How_to_create_a_REST_Protocol. I think it focuses too much on the entity-CRUD style. That is the protocol for some apps, including the original one for the Web. I worry that it heads people down a path where they think that's all REST is about.

Tim-
Ittay wrote re: I finally get REST. Wow.
on 04-27-2007 10:39 AM
the thing is, when you write software, you use an RPC model. what bothers me about REST is that it is not only an API. it enforces you to change your programming model.

that is not to say i don't like it. i do, for its simplicity and self documentation (e.g., provide all moves as links), but there is a price you pay.
Tim wrote re: I finally get REST. Wow.
on 04-27-2007 10:40 AM
Danny,

I like the EngineOfState. I have to avoid the semantic stuff you raise on the explanation page, though! :-)

Tim-
Tim wrote re: I finally get REST. Wow.
on 04-27-2007 10:43 AM
BradK,

There are plenty of authentication/authorization mechanisms: basic, integrated, certs, SSL, SAML, encrypted payloads instead of pipes via a range of technologies, etc. There are headers and you can add your own. You can use cookies or your own solution for correlation. In short, just look at the Web for answers. The goal is to work the same way.

Tim-

Tim wrote re: I finally get REST. Wow.
on 04-27-2007 10:49 AM
Ittay,

You can treat your Foo method as a very simple protocol and model it's state machine. You haven't started so your in no state. You move to a state where you are done and have the result. To move to that state, you operate on a URI, providing the data necessary to perform generate the new state, ie., the string, int and bool. Whether you do this with GET or POST is beside the point (which is not to say that it is not important!). What you get back is the completed state, the representation of which is an int.

Mechanically this may seem like an XML-RPC call, and in this simple protocol example derived from a method call, that shouldn't be a surprise. The big difference comes when the resultant state includes links to other states that you can then traverse. The RPC world doesn't typically do that, and that's really the difference in the models.
Tim wrote re: I finally get REST. Wow.
on 04-27-2007 10:51 AM
Chris,

With all do respect, I very much disagree. I spent a long, long time thinking the same thing: that REST is really just either CRUD with entities or RPC by another name. It really is about transitioning between states via links embedded in them. It's true that you could build something really RESTian with xml-rpc (or even SOAP), but in practice people don't.

Tim-
Ittay wrote re: I finally get REST. Wow.
on 04-27-2007 10:58 AM
Tim, if i understand you correctly, you say i have a URL like http://example.com/Foo and i POST to it with an xml containing the parameters, getting back either a URL whose GET returns the int value, or getting it back from the original POST. in any case, this is just a fancy wrapper for a function call.
Tim wrote re: I finally get REST. Wow.
on 04-27-2007 12:19 PM
Ittay,

Of course it is, because what you started with was a simple protocol derived from a single function call. The difference comes when the protocol is multiple steps. I'll post one of the examples I've been using in my own thinking about this.

Tim-
XML Nation wrote On programming models...
on 04-27-2007 1:04 PM
Alan Dean wrote re: I finally get REST. Wow.
on 04-27-2007 3:05 PM
To BradK

Re: http://pluralsight.com/blogs/tewald/archive/2007/04/26/46984.aspx#47009

"I've not seen any good aiuthentication mechanisms for REST. How do you use this in the real world enterprise, where you have to authenticate and authorize ?"

You can use plain or digest http auth

"I both hate and love soap headers. But I can't deny their usefulness in practice (correlation ID's for messages, for example). Does this metadata concept not apply in REST ?"

Use HTTP Headers to elaborate metadata.
Alan Dean wrote re: I finally get REST. Wow.
on 04-27-2007 3:08 PM
To Chris Parker

Re: http://pluralsight.com/blogs/tewald/archive/2007/04/26/46984.aspx#47012

"... but at the end of the day you are still sending requests back and forth as XML ..."

This is not correct. REST utilizes any MIME type, not just XML. It is your choice as publisher which MIME types that you choose to support.
Hovhannes Tumanyan wrote re: I finally get REST. Wow.
on 04-27-2007 6:44 PM
Gentlemen,
With all due respect, I disagree with your interpretation of the REST. My understanding is that rest is not about data formats (XML vs. MIME) or state encoding.
From what I learned on the net, the key characteristic of REST-ful system is its lack of state. In restful architecture the state information, I believe, is to be submitted with the request. That relieves the server from responsibility of state information storage and association with context or "connection". REST allows development of connectionless systems. HTTP is a good example of REST architecture (if we exclude the POST from it).

The chess game example brought in this thread appears not to be REST based. It would be REST based if the state of the chess game, i.e. the color of the board, positioning of figures etc. would be submitted with each and every request. Then the server would be stateless and restful.
Christopher Steen wrote Link Listing - April 27, 2007
on 04-27-2007 6:51 PM
When you think things are not possible: WCF duplex callbacks through NATs and firewalls - safe and secure...
Stefan Tilkov wrote re: I finally get REST. Wow.
on 04-27-2007 11:32 PM
@Hovhannes: No, you've misunderstood REST -- the communication is supposed to be stateless, but that does not at all mean the server can't store any state. It just should expose it as resources.
Chui wrote re: I finally get REST. Wow.
on 04-28-2007 2:43 AM
hmmm, any operations on REST resources can still happen in the wrong order anyway. REST wins when limiting verbs to commonly understood semantics beats reading up documentation on RPC signatures. REST is a nice subset of RPC, but when you need RPC you need RPC. There's no substitute.
Hovhannes Tumanyan wrote re: I finally get REST. Wow.
on 04-28-2007 10:05 AM
Stefan: I have to disagree. REST properties such as reliability and scalability could
not be achieved if the server was stateless.
Reliability here implies the ability to recover from server failures. Imagine a cluster cluster that consists of two independent servers. If one fails, the other picks up the request and finishes processing. In stateful model the spare server needs the state information stored on the failed server in order to process the request.
Stateless, i.e. restful design allows request processing without state information. That simplifies the implementation to a significant degree since there is no need to store the state and synchronize across servers in the cluster.
The same logic applies to scalability -stateless system allows any server can pick up and process any request at arbitrary processing phase.
Roy Fielding explained the steteless requirement very well in the section 5.1.3
of the original REST paper http://www.ics.uci.edu/~fielding/pubs/dissertation/rest_arch_style.htm#sec_5_1_3
Alex Blewitt wrote re: I finally get REST. Wow.
on 04-28-2007 2:06 PM
You should take a look at Restlet (www.restlet.org) if you want a Java API for dealing with REST that doesn't concern itself with specifics like Atom or HTTP.
Tim wrote re: I finally get REST. Wow.
on 04-28-2007 4:34 PM
Hovhannes,

REST does not imply or require that servers to not store state. If it did, how would Amazon maintain your shopping cart or your list of previous orders? Clearly there is state there, and it isn't always shipped by the client as part of the request. To achieve scalability and reliability, individual server processes are typically stateless and leverage a shared database or other storage system so that information location is not tied to client connection. None of this violates the notions of REST.

Tim-
Tim wrote re: I finally get REST. Wow.
on 04-28-2007 4:37 PM
Chui,

Yes, it's true, you can invoke state transitions in the wrong order if you set your mind to it. You can invoke non-existant transitions as well. But if you use transition links in your state payloads, it's definitely harder to get wrong *by accident*, which is the real problem with this facet of RPC. As to REST being a subset of RPC, I completely disagree.

Tim-
Greg Hoover wrote re: I finally get REST. Wow.
on 04-28-2007 7:04 PM
Tim,
In response to the discussion thread with Chui,

I was thinking along the same lines as Chui, and I also agree with your response: "But if you use transition links in your state payloads, it's definitely harder to get wrong *by accident*"

I was coming from the perspective of how REST holds up in a multi-user environment where data concurrency is an issue. If it is the communication that is stateless and not the resource (i.e. we are not passing the entire state of the resource on which we are operating), it seems that contention for the same data could result... and providing a locking mechanism hardly seems stateless to me.

I guess part of the link returned to the client could include a "timestamp" value (like used in Sql Server) or some important attributes of the resource so that when that link is requested to the server, the server could compare the attributes sent against the current values in the resource to determine if the state transition is indeed still valid to perform.

What are your thoughts / experiences in this area?
Bill de hOra wrote re: I finally get REST. Wow.
on 04-29-2007 5:04 AM
Ittay: "my question (and point) is, say i have an application with a function like 'int foo(string, int, bool)' (meaning, a function that takes many arguments and returns some value). how can you model that in REST?"

That's a heck of a lot different from a chessboard; you should at least acknowledge your first question was answered.

But let's assume you're not playing fetch me a rock. Then try to move past metasyntactic functions. What does foo do? is it a pure function? Or does it flip a bit in a database we don't know about? Does it log me in? Place an order? What?

You need to answer these questions because the most important thing is to figure out the actors and the state; the function interface is secondary. Once the state exchange is known and between who, you can choose a appropriate method.
Hugo wrote re: I finally get REST. Wow.
on 04-29-2007 9:40 AM