TPR6: HTTP 201 - Or, What Happens When Your User-Agent Isn't A Browser?

[Intro Music]

Announcer: You’re listening to one in a series of presentations from the 2008 HighEdWeb Conference in Springfield, Missouri.

Jason Woodward: Yes. So, as we said, I gave a presentation last year that took a look what actually the text that actually went over the wire on an HTTP transaction between a client and server; explains some of the terms that are used, talked about some of the headers, and how you could leverage them to have, say, search engines, know a little be bit more about your site by setting up your 404s are not so 404s; maybe a 410 or a 301 instead of 302.

So this time, since the other one got rejected, I’m doing this which is a little bit more--assuming that you understand all that and then you want to kind of take a look at using the HTTP protocol for something called REST interfaces, which you probably have all heard of. But maybe you haven’t.

So the first thing I’m going to start with is taking a look at an HTTP transaction in Wireshark so we can all take a look at what headers are for those of you who don’t know. So once you go to a web browser and go to--and now that that’s done--oh, wait, everybody’s sniffing on local host. That’s on purpose which you’ll see later. We’ll stop that. There we are.

So, I don’t know how many of you have seen this before but the red is what the client is sending to the server, the blue is what’s coming back from the server. You have things in there; the method. You have the path to the--what’s called the request--in the parlance. You have a status code coming back. Everybody has heard of 404 before. And you have these various bits of many information on the request side and the response side that give the client and that, sort of, respectively a little bit more information for about what went on.

And one thing to remember from that is that, as somebody I heard from last year, is that for each HHTP request that’s made, the entirety of what the server can use to decide about how to respond is contained within that request.

In other words, there’s no notion of two requests being attached to one another. You can achieve that concept with cookies and it makes it kind of seem that way. But at the very base level, at an HTTP specification, it’s what’s called stateless. Meaning that you can execute a request and the response to that request comes back based upon the entirety of the contents of that request. And that becomes important when we talk about REST.

So, what is it? It was a term coined in the paper in 2000. A system architecture academic paper, software system architecture academic paper, where the person who wrote it was talking about why systems like the Internet can scale so well. And he talked about the fact that HTTP and other articles are stateless as being one of the base building blocks of that.

They talked about some other things too that slipping my mind right now. But the base concept of REST, Representational State Transfer, is that you have these things called resources; jarred thing, a document, a webpage, an object that say we’re going to build an application that is a basic staff profile. So I might be a resource, you might be a resource, the chair might be a resource. And then actions are performed on those resources.

You can get something from the chair, you can put something on the chair, and that sort of thing. And each of those actions is stateless and well defined about what happens and what you’re doing.

This doesn’t still mean it is HTTP when he was talking about the RESTful system designs. But we’re going to talk about it on HTTP because that’s what most people are using as a transport these days.

REST in HTTP resources are identified by resolvable URIs. Those of you who are in my pre-session know what the difference between URIs to URL is. So they’re typically URLs but in principle they should be URIs when you think about the notion of an object, a thing, something that’s named that you’re actually manipulating.

The actions that can be performed on these objects are represented in an HTTP transport by the various methods, HTTP methods that you can execute, get post, put, delete, head, all the various nine or ten standardized ones.

We’re going to use three of them today. And then the state of whether or not that action was successful, or was allowed, or was authorized, or broke, or whatever is communicated using the HTTP response, the status mechanism in HTTP, as opposed to other more heavyweight RPC protocols like some other PC; whether it be a body of XML that might say have some sort of representation of this didn’t work in there but simply using the HTTP status.

Okay. So it’s all actually very pretty simple, the concept of REST. To illustrate it all, I built a small application here, and we’re going to walk through it and take a look at what’s going on and ask you some questions about what should be going on or what maybe shouldn’t be going on. Because actually, I think they have on purpose a design flaw in how I represented this. And so we’ll see if everyone agrees with me if that it’s not. Because, by the way, there’s arguments out there, people have arguments about whether a particular interface is RESTful or not because it’s sort of the definition of the term is not precise term.

You might call an interface RESTful even though it isn’t. For instance you might see, I think, Flickr is one of them that’s semi-RESTful, meaning that instead of using just URIs to identify resources, you end up having to get parameters on the URL. And so some people argue that that’s not a strictly RESTful interface.

So we want to build an application or restoring information about staff. We want to build a RESTful interface and we’ve decided, in other words, I’ve decided that we want this interface to be consumable in Ajax applications and other non-browser contexts.

I consider Ajax not really a browser context because it’s actually your software that’s reading the data back and then generating HTML or whatever, other UIQs. It’s not your browser rendering straight HTML coming from the server. And also a simple HTML human interface to it. So the web application that I built is running on the web server that’s on the Mac. Here are three resources that you might want to consider putting into a staff profile application.

You want a URI that identifies the profile of an individual, a URI that identifies one of the attribute, say, my name, a person’s name on there, and then there might be this resource which is a list of all the stored profiles.

So let’s take a look at one of them. This is the URI for the JDW 5 staff profile. What would you expect to happen if a Get Request was executed on this URI? We cannot feedback. I’m not going to be one of those just yakking for 45 minutes because it’s got to get in another 45 minutes.

On this one? Yeah, you see the profile. And since I said I wanted it to be consumable in Ajax, it’s actually the content of the response body is going to be encoded in JSON. Everybody know what JSON is? Okay, JSON stands for Java Script Object Notation. It’s a serialization method that ends up taking your data and representing arrays and strings in a format that would be cut and paste-able in the Java Script.

So effectively, you could eval one of these things and get an assignment out. Now, it’s bad practice but that’s the encoding methodology. But it would be very useful for a piece of software that was kind of interacts with this system as opposed to having to decode HTML or XML or whatever. It’s just a serialization method.

How about PUT? What would you expect to happen on a PUT? That’s right. How about the LIT? It’s hard. It’s really hard, isn’t it? But that’s actually the simplicity of it is that because you’re creating this interface with well-defined actions on well-defined named objects.

You end up being able to write random applications all over the place that interact with these things much like when you write object or in its software and put a public interface on it that you can’t manipulate what’s going on inside the object. You can have other people using this interface to do mashups or things like that. Alright, so that is the entirety of a valid HTTP 1.0 request that I am going to execute right now.

No, I guess I better go cut and paste it first and have a little helper tool as we get a little bit more into this. But right now, I just want to use our old friend Telnet. So, everybody read this?

So here I issued a request that is associated array encoded in JSON, its name, group and fallen attribute with my name, a random group name, and my phone number. Yes, that is my phone number. It’s on Facebook. I don’t care.

And here’s the response that comes back from the server. The server says, “Yup, that’s okay. Everything went okay.” It doesn’t need to say here’s where it is because you know where it is. You put it there. It doesn’t need to say anything. I mean, I suppose you could have extra meta-information here but that sort of goes along against the spirit of do something to a resource and tell me if it worked or not.

So we will now see what happens when I get this resource. It comes back. Now, you might ask where did it save it? That’s the function of the web application that I wrote. That’s completely transparent to just like all the web applications that you all typically write.

How that is stored, what persistence mechanism you’re using, is irrelevant to the users of your web interface, okay? So expect it. When you put some information that you can get back the same information, or maybe slightly different if that’s part of your RESTful interface. Let’s say you wanted a password updating, want a password updating mechanism in here, you might be able to put to a password URI, but you probably don’t want to get from it.

An interesting thing about this is because it’s all HTTP, I can go over to my trustee web browser and execute the same thing. So if you are writing a consumer of one of these interfaces in software, maybe it’s in a command line script that you’re writing to do maintenance on some site that you operate, maybe it’s a plug-in for your CMS that’s reading an events calendar for you.

Maybe that’s a bad example because that probably be our assess. But you can debug this by hand by using the web browser as a user-agent even though the end user argent or the target user agent is not going to be the web browser.

When I was at hotel school, because I recently switched jobs, they had a script that was running nightly on our windows domain controllers, wanting to update people’s room address, phone number, those sorts of things on our change server. And because of the mysteries of how Cornell operates politically, the system of record is not the hotel schools’ exchange server.

It is an Elda directory operated by central IT resources. So, instead of just going into exchange in however you update stuff in exchange, you probably have to click a bunch of places and maybe some okay dialogues or whatever, the HR or the self service will go to the central location that updates their phone number, and then the script on the exchange site will pull it in.

They might say, “Well, why didn’t the exchange folks just write an Elda connector?” Well, it turns out our exchange folks are more windows administrators and programmers and our software development team already had copies of the central data-marts inside our local databases. So they could make a connector to oracle and do selects out of it.

Well, but that means that they need the Oracle client library installed. That means they need to set up an ADBC thingy. It was much easier for them to use an HTTP client library which comes in just about every program environment you can get that hit our RESTful URL on our Internet.

That’s it. That was essentially “get me the phone address of this particular Met ID”. And it was really easy to teach them how to use it because I gave them a URL that in Outlook you clicked on and it showed to me information that’s coming back; much like this.

So, you might also, in this interface, want to have alternate views into the data that you’re storing. Your interface might identify this resource, be a list of group, say, like I have up there. So you might have another URI that points to that was the groups or a particular group. Or you might want, like I said before, the human readable versions of these things.

So, will you look at them. So if you go to Group, it will returns a JSON encoded array of the groups that are available. If you ask for that group, it gives you JSON encoded array of who’s a member of that group.

We can go into this. So I’d written a quick command line script that allows me to change the HTTP method. So what I did here was I posted back to me again same data. That doesn’t really make much sense because it’s not adding anybody to any of the groups so we’ll create another user.

Again, Response 200, everything’s okay. Now, there are more people in group. Now, this is all pretty straightforward. Gets slightly more interesting, I think, when you start thinking about what happens. I’ve only shown you URL so far that you’d think this must exist because I just put something there. But what should happen if there’s not something there?

Audience: Should have a URL message or something?

Jason Woodward: Yes? What kind? What do you suggest if you’re building this?

Audience: It does not exist.

Audience: 404.

Jason Woodward: 404, yeah. That’s what it does. How about this? What if with the interface that we’ve talked about so far, the interface being the specification that I talked about that we’re going to build this whole staff profile thing, what do you think should happen if somebody executes a PUT on one of these groups?

Audience: We create a group or create a POST in the view. Am not sure what.

Jason Woodward: Well, this says PUT into that group--so what is it? You could build it however you want it.

In my case, I decided that there is no notion in my physical representation of a group that didn’t have anybody in it. So if you just look PUT and you don’t give anything, it’s going to give you another error. Which should be what? What would you guess that be? It’s probably an HTTP code you’ve never seen before.

Audience: 401?

Jason Woodward: Who said 401? That would be not authorized or needs authentication. That’s 400. Green? So, why wouldn’t 401 work? I’ll tell you it doesn’t work. Can you think about why I might say that or can anybody think about why I might say that? Because we’re not actually putting access control on this resource.

There’s not really a resource there. Or rather there is but--actually wait. Is there a resource there from how I defined what a group was?

Audience: No.

Jason Woodward: No, there isn’t because I haven’t put anybody into that group. So this, if I wrote myself for a write which I didn’t test, should be a 404 because that resource is not found. Let’s see what happens. 404.

Now, if I try to put to the HEW or like this, because that’s really where I was going for when I started this line of questioning. So we know that group exists because we’ve already created me and put me in there. But you wouldn’t get a 401 because it’s not something that--or access to that messages is that we don’t know who you are or we know who you are and you haven’t been granted access to this resource.

Four hundred bad requests is violation of the HTTP protocol. In other words, the server has decided that whatever the request sent to it was not well formed. I believe it’s going to be 405. Oh, you’re right. Thank you. There we are. Much better.

Method not allowed. Where the service says you actually only allowed to do Gets on this resource. Other methods don’t make any sense for this particular resource. So a good RESTful design will actually communicate details to about the design of the resource to the consumer without actually having to have this everybody does get imposed sense that most web developers probably have because most of their sites, that’s all they use.

I built another interface which gives a list of a human readable list representation of this resource. I can click on there. I can see the values that I uploaded. Same here. And the others has actually filled that update count which I’ll come back to later.

Now, I think what I’m going to do now, is upload an image. Oh, yes. So that just shovel that J peg up to that URI. It’s a 200. Service says it’s happy. It should be there. We’ll see if it is.

The content lays at zero. These are the Response Headers. It’s okay. It’s a good question. Whoops. I’m not going to make it to the Information Science Department beer night tonight. That’s what you call an ice breaker.

Where was it? Oh, yes, we’re going to go see if this image made it up there successfully by looking at it in our trustee browser because if I tell them then I’ll just get garbage back. Where did I put that? Oh-oh.

Say what? Well, let me cheat then. No, this worked an hour ago. Oh, No need to bug it. I’ll show you the code. The code actually is commented and it goes straight this sense of writing your interface such that it’s going to behave responsibly when poked in ways that are unintended. Does it have curse words in it? No.

[Laughter]

Although it’s written in PHP which some of you might call curse words. That’s the client’s script. I couldn’t find an easy PHP library that allowed me to manipulate the request method. So I wrote that kind if line thing in pearl.
So I set up to build this. I set up an Apache rewritable that rewrites every URL. Most end users use that PHP framework or frameworks, I guess, in general with Apache to set up a rewritable that says send everything to this one script. So I did that.

But the top here I load in, this really ugly serialized PHP data structure that is the persistence mechanism. All the date that I got put in here is stored in a PHP structure and memory and then serialized out to disk.

So I have a big case statement here. If the command is profile and it doesn’t have an argument, then it will accept the Get method which will return these names in the system. Otherwise, it will return Method not Allowed. Similarly here, with the PUT, it will save the data if the user’s not there. Then, if it’s not a PUT, it checks to see if the user’s there. If it’s not there, then the 404s; If it is there, and it’s a Get, returns the information. If it’s a delete, it deletes to there. That’s what I’m talking about; using the last modified in there.

I guess we got to tie the last three a little bit and looking to see if the 304 is going to be simulated here; not simulated but used. And this is the 403 arguments. The group method and the brows method.

I always have a 405 at the bottom. That’s sort of the catch all. Except for down here where the de-fall is to 404. Because presumably, any URI that gets past into the system that was not caught up there is one that was unintended. So I’m going to capture the image upload.

We’ll see if it works this time because you know if you try things twice, it always works the second time when you haven’t changed any code. So this is the PUT request that was generated by the pearl client. Oh, that’s where the bug is. I’ve forgotten about that. My upload program wasn’t smart enough to set the mind type. And so I forgot that I had to set it on the command line which I saw here.

So that’s the J peg and the response here says, “Yeah, it got uploaded fine but it stored the wrong mind type in the database.” Now that I uploaded it again correctly, I should be able to refresh that and see my Facebook picture there.

In the view image, see the URL up here. And I can see that the browser actually issued a conditional Get Request. Let’s actually sniff this and watch it.

So the browser issued this Get Request for that URL and also added an If-Modified-Since header there because the last time that the browser requested this, the server had set a last modified header on the original response that sent the image down to the client. And so the browser can ask when’s the last time--don’t send it to me if it hasn’t been updated which is what happened here. I’ll make sure I got everything.

Audience: If you have to rely on the 304 handler, is that something that’s sent automatically to the server?

Jason Woodward: Well, that’s actually an interesting question. I don’t know if you asked it because you wanted to lead me or because you weren’t sure. In general--I don’t mean that to be whatever--but it’s very interesting. If you’re writing a piece of software that runs on the server and is generating a response, the server itself can’t know how old that content was or how old it was generated.

They’re running a CGI. It’s just going to send back whatever headers the CGI sends back. Now, you see, that’s probably has a bunch of stuff in there that knows the last time you updated the textbox or uploaded an image or whatever and can generate this Expires headers.

So the answer to your question is yes. I had to manage that in there because otherwise, the only response headers that come back are ones that the server can put in there on its own without any knowledge of the application such as the server header, what date the response was generated on, and these two connections and keep alive which are HTTP1.1 mechanisms for keeping the TCP connection open and pipelining or sending multiple requests down the same connection.

But here, my script didn’t set anything else. If we run this again, if we run this sniff again--by the way, the reason I said that it was interesting that not many people know it is that if your blog or whatever software is not actually generating expires headers, cash control headers, last modified headers, those sorts of things, all your clients are going to be making requests every time.

They’re not going to be able to take advantage of cashes in between cashes proxies or the caches on the local browser because, effectively, your web server is saying, “Hey, this content is always refreshed, it’s always ground up” which for many of the resources aren’t. Sometimes they are.

But if you wrote a script that generates mild assessment, for instance, you’d want to do that. It’s easy to forget. In fact, I would say that in most applications that I built, I only remember it when we’re starting to see a large bill from the folks who ran that network. So if I shift-refresh this, so now the response is a 200. Now we need to see that again.

We’ll come over here. All TCP stream. And you can see the extra stuff that now up here is one where there’s some more full response. The content type. I had to set that in the PHP. For most of you who have written web applications that spit out binary data, you know you have to execute header or whatever the equivalent is on your language of choice to tell the client what the actual mind type of the binary body is.

That s done, the transfer and coding is done by Apache. Last modified is something that the client has to set. And PHP does that because it’s really vain.

So that’s it. I mean, it’s really pretty simple and straightforward.

Those are two pretty famous ones out there. There’s a lot more. I didn’t want to put the Flickr one up because some people argue that it’s not RESTful because it doesn’t get parameters and the response. If you’re going to look at their main page is an extra melon coding. So you can’t use the HTTP transport to signal whether or not something work. You have to actually decode their response which is unintended-ly or it’s more complicated than it needs to be, I guess.

Audience: Well, I was wondering. I realize there’s probably certain amount of heresy in this question but if you wanted to tack on and off on occasions onto a RESTful interface, how would you do that?

Jason Woodward: You do it the same ways that you do it on a regular web service. So, first of all, you can answer that question, the context of HTTP or you can just do it in the context of general RESTful interfaces. The RESTful concept says we’re going to perform an action on a resource. Whether or not you’re allowed to do that is controlled by the resources because it doesn’t let URL now.

So in the HTTP world, you can do that several ways. You could do it with GSSAPI cover as headers, you could do it with HTTP basic or digest authentication headers, you could go over to cookie.

Audience: Okay, but given all that, is there a standardized way of doing it that works well across line incorporations.

Jason Woodward: Usually, what I’ve seen is the multi-request model. You have a resource which is the “I want to log in resource”. And it gives you back a session token, a cookie, of some sort that you on the client side must continue to send back along with the rest of your request. And the resource will then take the cookie and unpack it and see whether or not you’re authorized. And that will be HTPS, usually time-limited, that sort of thing. But you don’t have to do it that way. Sure. It’s really bad. Alright, that’s all I have.

[Applause]

Announcer: For more presentations from the 2008 HighEdWeb Conference visit HighEdWeb.org/2008 or sign up for our podcast and feed at HighEdWeb.org/podcast.xml

[End of Music]