The much maligned User Agent header

This post is the first in a series of posts that will explore some piece of the HTTP specification with the objective of providing practical insights into the uses and abuses of the feature. Consider these posts my attempt to provide HTTP guidance in language that is slightly more digestible than the official IETF specifications.

The user agent header is one of those headers that most developers already know about, but pay little attention to.  It gets very little love. I’m sure most developers have wondered at some point if we really need to send all the bytes in the giant browser user agent strings?  And why does Internet Explorer claim that it is a Mozilla browser?

We know that our analytics tools use this header to give us statistics on what operating system our users are using and it lets us watch the ebb and flow of browser market share.  But we’ve heard mixed messages about user-agent sniffing, and we’ve heard concerns about advertisers using the uniqueness of user-agent strings for fingerprinting users.  It is interesting that many of the things user agents are used for today were not part of the original goals for this HTTP header. 

Lost its meaning

Many developers I talk to are not aware of the intent of the user agent header and even fewer know how to properly format one. Its not particularly surprising considering how many poorly formed instances exist in the wild.   In the world of browser development we don’t really have any control over the user-agent header, the browser defines it. However, for mobile applications and native client applications we have the opportunity to assign our own values and start getting the value from it that was originally intended.

How can it help me?

The HTTPbis specification says that the user agent header can be used to “identify the scope of reported interoperability problems”. This is the eloquent way of saying that we can find out which versions of client applications are broken. Sometimes clients get released out into the wild with bugs. Sometimes those bugs don’t become a problem until changes are made to the server. Asking all your users to update to a new client version is a good start, but its going to take time to get the clients updated. What should we do in the meanwhile? We could hold off on the server deployment, but that would suck. We could instead add a small piece of middleware that sniffs the incoming requests and looks for the broken clients and fixes the problem for them. It might be something in the request headers that needs to be fixed, or something in the response headers. Hopefully it is not in the body, and if it is, let’s hope the body is small. When the broken clients are gone, we throw away the middleware.

User agent headers should not be used as a primary mechanism for returning different content to different devices. Clients should be able to choose their content based on accept headers, media attributes, client hints and prefer headers. Using the user agent header as a feature or content selection mechanism is just encouraging client developers to lie to you, which defeats the point. This is why IE claims it is a Mozilla browser, and why Opera no longer identifies itself primarily as the Opera browser.  These browsers were forced to lie to provide a first class experience to its users because web site developers were only delivering features to browsers that they knew could support them.  This is not the same as providing temporary workarounds for clients until new versions can be deployed.

So what is it supposed to look like

The user agent header is made up of a list of whitespace delimited product names and versions. The HTTPbis specification defines the header to look like this,

     user-agent     = product * ( RWS ( product / comment ) ) 

This notation is what is called ABNF and you will find it throughout the IETF specs. It’s not the most wonderful of syntax descriptions, but once you get over a few initial hurdles it is fairly easy to understand. In this example the first product token is indicating that the user-agent header must consist of at least one product token. The *(…) syntax that follows says that there may be zero or more of the stuff that is inside the parentheses after the product. The RWS is a special predefined token that is defined in another spec that means ‘Required White Space’. The effect of putting the RWS token is that multiple products will be whitespace delimited. However, the user agent header can also contain comments. The comment syntax is not specific to the user agent header. There are a number of other http headers that allow it. A comment is differentiated from product token by being surrounded by parentheses.

The spec hints that the comments following the product are related to the preceding comment, although it doesn’t come right out and say it.

Why more than one product?

Most applications are built on top of libraries and frameworks. Identifying just the main product may not be helpful when trying to understand why a particular set of clients are sending an invalid request or failing on a particular response. By listing the core client application, followed by the components upon which the client is built, we can get a better picture of the client environment. Obviously you can take this idea too far. Enumerating every dll that your app has a dependency on is going to be a waste of bytes and effort processing the header. The idea is to capture the main components that impact the creation and sending of requests and the consumption of responses.

Syntax of a product token

The product identifier is described in the spec as

     product         = token["/' product-version]
     product-version = token

This is fairly self explanatory except for the fact that we have no idea what the syntax of a token is! If you dig around you will find that token is defined in httpbis part1 as,

        token     = 1*tchar 

Having fun yet?  So a token is one or more tchars.  A tchar is defined as,

     tchar     = "!" / "#" / "$" / "%" / "&" / "'" / "*"
                    / "+" / "-" / "." / "^" / "_" / "`" / "|" / "~"
                    / DIGIT / ALPHA
                    ; any VCHAR, except delimiters

The terms DIGIT, ALPHA and VCHAR are defined in yet another IETF spec here, but what are these delimiters that we are not allowed to use?  The delimiters that are not allowed to appear in a token are defined as:

 (DQUOTE and "(),/:;<=>?@[\]{}")

For those of you still paying attention, you may be saying, but I see semi-colons in user agent strings all the time.  That is because they are in the comment not in the product token.  The “comment” has a different set of syntax rules.  The following is valid example of user-agent that is full of special characters,

user-agent: foo&bar-product!/1.0a$*+ (a;comment,full=of/delimiters@{fun})

It’s not very surprising that there are many examples of invalid user-agent headers considering the torturous set of syntax rules that need to be observed.  It is made worse by the fact that the vast majority of HTTP client libraries simply allow a developer to assign an arbitrary string to the user-agent header.  It is exactly this type of situation that makes me appreciate Microsoft’s System.Net.Http HTTP library.  This library has created strong types for many of the HTTP headers to ensure that you don’t break any of these rules.  You get to offload the burden of knowing these rules and be confident that you are compliant.

[Fact]
public void TestValidCharactersInUserAgent()
{
    var request = new HttpRequestMessage();
    request.Headers.UserAgent.Add(new ProductInfoHeaderValue("foo&bar-product!","1.0a$*+"));
    request.Headers.UserAgent.Add(new ProductInfoHeaderValue("(a;comment,full=of/delimiters@{fun})"));

    Assert.Equal("foo&bar-product!/1.0a$*+ (a;comment,full=of/delimiters@{fun})", request.Headers.UserAgent.ToString());
}

The ProductInfoHeaderValue class allows you to create a product token by providing a product and a version. It also allows has a constructor overload that allows you to create a comment.  All the syntax rules are handled.

So what’s the problem?

As an real world example of how poorly the user-agent is treated, this is the user-agent of the IE11 web browser,

User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; Touch; rv:11.0) like Gecko

As I mentioned earlier, IE declares itself as Mozilla for compatibility with sites that do feature detection based on the browser.  The comment that follows provides additional information about the environment.  I’m not sure why the Trident/7.0 is in the comment and not listed as another product.  The addition of the Touch comment leads me to suspect that additional feature detection is being done on that value.  The final “like Gecko” tokens should be interpreted as two distinct products, “like” and “Gecko”, according to the syntax.  However, if you look at the Chrome user agent header you can see the origin of this weird string.

User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.131 Safari/537.36

I’m assuming that some sites must do feature detection on the string “like Gecko” which is a comment on the product AppleWebKit.  However, in copying this string the IE team have completely ignored the semantics of the elements of the user-agent header and assumed that anyone processing the header will doing substring searches based on the assumption that it is a simple string.

Maybe there is real-world internet nastiness that has forced IE to format their header in the way they have, but I would think something like this,

User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64;) Trident/7.0 (Touch; like Gecko) rv/11.0

would make more sense.

Now before you dismiss this as “it’s Microsoft who doesn’t follow standards”, you can be sure they not the only ones who are missing the intent of user-agent semantics.  The Go library built in HTTP client library uses the following user-agent header.

Go 1.1 package http

I suspect the following user-agent header would be more semantically correct,

Go/1.1 package-http

I’m quite sure if I spent any amount of time watching HTTP requests, I could find violations from all different products.

Why you should care

There is an argument that could me made to say, well if the big browser vendors don’t care about respecting the semantics, then why should I concern myself with it. 

The user-agent was added to HTTP to help web application developers deliver a better user experience.  By respecting the syntax and semantics of the header we make it easier and faster for header parsers to extract useful information from the headers that we can then act on.

Browser vendors are motivated to make web sites work no matter what specification violations are made.  When the developers building web applications don’t care about following the rules, the browser vendors work to accommodate that.  It is only by us application developers developing a healthy respect of the rules of the web, that the browser vendors will be able start tightening up their codebase knowing that they don’t need to account for non-conformances.

For client libraries that do not enforce the syntax rules, you run the risk of using invalid characters that many server side frameworks will not detect.  It is possible that only certain users, in certain environments would detect the syntax violation.  This can lead to difficult to track down bugs.

Hopefully you gained some additional insight from this rather wordy exploration of the the user-agent header.  I welcome suggestions regarding other dusty areas of the HTTP protocol that might be worth shining a light on.

Image Credit: Agent Perry https://flic.kr/p/dsSxFt