The reason for the headers is so that you can issue a head request to see if the page is updated, and if not, serve it from a cache (on your machine, the ISP, or anywhere down stream)

Why? Well, things like google also only have to issue a head request to know if it’s worth re-crawling your page. 99.9% of pages on the net are old, stale, and never change anymore. This saves an enoromous amount of bandwidth and processing for them.

I don’t believe this would be “premature optimization”; I believe it would be sensible. If you are sending useless data between machines millions (billions?) of times per day, then it would make sense to remove the redundant information. Maybe it’s too late now, though.

Thank you for the link to that paper.

Thanks,
Brendon.

It’s not obvious that there would be a substantial improvement by a small reduction in header length; we need a quantitative measure for RTT effects and number of packets in real-world implementations to say. But there would definitely be a substantial interoperability cost; adjusting server and client implementations, and working out a usable migration path, takes more than 5 minutes.

HTTP is such a widely used protocol, that nobody would accept the flag day. To get adoption transition must be smooth.
In addition, one of the benefits of HTTP is the headers aren’t set in stone, new ones can be added, servers and clients can support custom headers, including core extensions like basic authentication.

Using single characters to designate header types basically limits the extensibility.

Your HTTP client has to somehow figure out if the server supports this new header compression extension, which means (most likely) an additional round trip, which takes additional latency (the page will take longer to load, since the client has to somehow figure out that the server supports this new header format).

Or choose to break interoperability, in violation of the robustness principle.

But interoperability and Latency are _everything_ in HTTP on the World wide web, the delay time to actually get a request received is what matters, not the precise size of the request; especially in the 21st century, the age of broadband, a few hundred extra bytes extra per request are almost negligible, as long as the request still fits in one frame.

The complexity avoided by having human-friendly headers may be well worth the small number of extra bytes.

As for the interoperability issues: if the web server doesn’t speak the same language as you, then you don’t get to see any web page, which is even worse than transferring more bytes. There is currently a HTTP standard. It would be essential for any improvements to ensure backwards compatibility, and I don’t see an obvious way to introduce a change like this without unacceptable costs.

In addition, the full extended header (even without the shortening) very likely fits into one packet, and has the advantage that humans can easily disseminate and construct headers.

Protocols that humans can enter messages in directly are easier to troubleshoot; it’s a great debugging aid to be able to simply telnet to a web server, and type in a HTTP request.

The real consumer of HTTP bandwidth is the file itself; typically HTML or XML, which is a text-based language. There is a standards-based compression option to use deflate/gzip, when requested by the client.

However, it might be of some benefit, if a new standard some form of ‘Binary XML/HTML’ were developed. In this case, you’d replace tags like the ‘body’ tag with a 16-bit code.

Instead of ‘img’ tags referring to external files, the img object would contain a binary pointer to the image, which would exist in the same binary stream as the text (probably at the end, so lower-bandwidth text could be rendered first).

In fact, you’d no longer need start and stop tags at all, just a tag-type bit length, and then binary pointers to objects in the document.

Optimizing the size of the actual payload has benefits, because the payload is normally a larger portion of the data being transferred.

In addition, an extra round trip shouldn’t be required, the client can declare it supports the special document type by using HTTP headers.

Because it isn’t worth it. Yes you could save a few bytes in each header, but that doesn’t make the internet noticeably faster.

Using human-readable text protocols simplifies development and debugging, and the modest bloat in data transfer size is barely noticeable on anything but the slowest connections.

Improvements like content caching (last-modified, etags, etc.), content compression, minification, CSS sprites, CDNs, and basically everything that YSlow tells you to do, can make a huge difference in the end-user performance of web sites. Compared to those things, cutting a few dozen bytes out of the headers is a waste of time.

Consider just the issue of latency. A ping from my house in San Francisco to a server in Dallas and back takes 50ms. In that time, on a 1.5MBit DSL line I could download 7500 bytes. So changing the size of that ping packet doesn’t matter much compared to trying to cut down on the number of round-trips to the server.

And when the content is much larger, the % of time spent on the header is minimal compared to the time spent on the content — which can be compressed, provided that you add yet another header telling the HTTP client so.

In this paper the authors do the hard work of substituting a manually-compressed binary protocol for HTTP traffic and compare the real world results.
ftp://ftp.computer.org/MAGS/MULTIMED/mms/webEng/112272.pdf

The easiest way to see that would be to sniff the traffic for lets say a few days and save all the HTTP request and see how much data it is and then compress the data in your way and see compare the data.

I’m very new to this programing and such stuff but I think I can manage to do it with Wireshark mabie.
Don’t know how i can compress/exchange all the data thou..
If I do succeed doing it I will post my results.