This chapter is for the web-developer course only
This chapter covers:
- Protocol basics
- Requests and responses
- HTTP request/response control flow
- Statelessness and idempotence
- Cache related headers
- Hyper-Text Transfer Protocol, HTTP, is at the core of the web
- Specified by the IETF, there are two main versions: HTTP/1.1 and HTTP/2
- Varnish 4.0 supports HTTP/1.1
HTTP is a networking protocol for distributed systems. It is the foundation of data communication for the web. The development of this standard is done by the IETF and the W3C. In 2014, RFCs 723X were published to clarify HTTP/1.1 and they obsolete RFC 2616.
A new version of HTTP called HTTP/2 has been released under RFC 7540. HTTP/2 is an alternative to HTTP/1.1 and does not obsolete the HTTP/1.1 message syntax. HTTP’s existing semantics remain unchanged.
The protocol allows multiple requests to be sent in serial mode over a single connection. If a client wants to fetch resources in parallel, it must open multiple connections.
Resources and Representations¶
- Resource: target of an HTTP request
- A resource may have different representations
- Representation: a particular instantiation of a resource
- A representation may have different states: past, current and desired
Each resource is identified by a Uniform Resource Identifier (URI), as described in Section 2.7 of [RFC7230].
A resource can be anything and such a thing can have different representations.
A representation is an instantiation of a resource.
An origin server, a.k.a. backend, produces this instantiation based on a list of request field headers, e.g.,
When an origin server produces different representations of one resource, it includes a
Vary response header field.
This response header field is used by Varnish to differentiate between resource variations.
More details on this are in the Vary subsection.
An origin server might include metadata to reflect the state of a representation.
This metadata is contained in the validator header fields
In order to construct a response for a client request, an algorithm is used to evaluate and select one representation with a particular state. This algorithm is implemented in Varnish and you can customize it in your VCL code. Once a representation is selected, the payload for a 200 (OK) or 304 (Not Modified) response can be constructed.
Requests and Responses¶
- A request is a message from a client to a server that includes the method to be applied to a requested resource, the identifier of the resource, the protocol version in use and an optional message body
- A method is a token that indicates the method to be performed on a URI
- Standard request methods are:
- Examples of URIs are
- Header fields are allowed in requests and responses
- Header fields allow client and servers to pass additional information
- A response is a message from a server to a client that consists of a response status, headers and an optional message body
The first line of a request message is called Request-Line, whereas the first line of a response message is called Status-Line. The Request-Line begins with a method token, followed by the requested resource (URI) and the protocol version.
A request method informs the web server what sort of request this is:
Is the client trying to fetch a resource (
GET), update some data (
POST) at the backend, or just get the headers of a resource (
Methods are case-sensitive.
After the Request-Line, request messages may have an arbitrary number of header fields.
Message bodies are optional but they must comply to the requested method.
For instance, a
GET request should not contain a request body, but a
POST request may contain one.
Similarly, a web server cannot attach a message body to the response of a
The Status-Line of a response message consists of the protocol version followed by a numeric status code and its associated textual phrase.
This associated textual phrase is also called reason.
Important is to know that the reason is intended for the human user.
That means that the client is not required to examine the reason, as it may change and it should not affect the protocol.
Examples of status codes with their reasons are:
404 File Not Found and
304 Not Modified.
Responses also include header fields after the Status-Line, which allow the server to pass additional information about the response.
Examples of response header fields are:
Requests and responses share the same syntax for headers and message body, but some headers are request- or response-specific.
GET / HTTP/1.1 Host: localhost User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; fr; rv:188.8.131.52) \ Gecko/20110319 Firefox/3.6.16 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: fr,fr-fr;q=0.8,en-us;q=0.5,en;q=0.3 Accept-Encoding: gzip,deflate Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Keep-Alive: 115 Connection: keep-alive Cache-Control: max-age=0
The above example is a typical HTTP request that includes a Request-Line, and headers, but no message body.
The Request-Line consists of the
GET request for the
/ resource and the
The request includes the header fields
Note that the
Host header contains the hostname as seen by the browser.
The above request was generated by entering http://localhost/ in the browser.
Most browsers automatically add a number of headers.
Some of the headers will vary depending on the configuration and state of the client. For example, language settings, cached content, forced refresh, etc. Whether the server honors these headers will depend on both the server in question and the specific header.
The following is an example of an HTTP request using the
POST method, which includes a message body:
POST /accounts/ServiceLoginAuth HTTP/1.1 Host: www.google.com User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; fr; rv:184.108.40.206) \ Gecko/20110319 Firefox/3.6.16 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: fr,fr-fr;q=0.8,en-us;q=0.5,en;q=0.3 Accept-Encoding: gzip,deflate Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Keep-Alive: 115 Connection: keep-alive Referer: https://www.google.com/accounts/ServiceLogin Cookie: GoogleAccountsLocale_session=en;[...] Content-Type: application/x-www-form-urlencoded Content-Length: 288 ltmpl=default[...]&signIn=Sign+in&asts=
HTTP/1.1 200 OK Server: Apache/2.2.14 (Ubuntu) X-Powered-By: PHP/5.3.2-1ubuntu4.7 Cache-Control: public, max-age=86400 Last-Modified: Mon, 04 Apr 2011 04:13:41 +0000 Expires: Sun, 11 Mar 1984 12:00:00 GMT Vary: Cookie,Accept-Encoding ETag: "1301890421" Content-Type: text/html; charset=utf-8 Content-Length: 23562 Date: Mon, 04 Apr 2011 09:02:26 GMT X-Varnish: 1886109724 1886107902 Age: 17324 Via: 1.1 varnish Connection: keep-alive [data]
The example above is an HTTP response that contains a Status-Line, headers and message body.
The Status-Line consists of the
HTTP/1.1 version, the status code
200 and the reason
The response status code informs the client (browser) whether the server understood the request and how it replied to it.
These codes are fully defined in https://tools.ietf.org/html/rfc2616#section-10, but here is an overview of them:
- 1xx: Informational – Request received, continuing process
- 2xx: Success – The action was successfully received, understood, and accepted
- 3xx: Redirection – Further action must be taken in order to complete the request
- 4xx: Client Error – The request contains bad syntax or cannot be fulfilled
- 5xx: Server Error – The server failed to fulfill an apparently valid request
- HTTP is a stateless protocol
- Common methods: safe, idempotent and cacheable
- Most common cacheable request methods are
HTTP is by definition a stateless protocol meaning that each request message can be understood in isolation. Hence, a server MUST NOT assume that two requests on the same connection are from the same user agent unless the connection is secured and specific to that agent.
HTTP/1.1 persists connections by default.
This is contrary to most implementations of HTTP/1.0, where each connection is established by the client prior to the request and closed by the server after sending the response.
Therefore, for compatibility reasons, persistent connections may be explicitly negotiated as they are not the default behavior in HTTP/1.0 [https://tools.ietf.org/html/rfc7230#appendix-A.1.2].
In practice, there is a header called
Keep-Alive you may use if you want to control the connection persistence between the client and the server.
A method is “safe” if it is read-only; i.e., the client request does not alter any state on the server.
TRACE methods are defined to be safe.
An idempotent method is such that multiple identical requests have the same effect as a single request.
DELETE and safe requests methods are idempotent.
Cacheable methods are those that allow to store their responses for future reuse.
POST as cacheable.
However, responses from
POST are very rarely treated as cacheable.
Constructing Responses from Caches¶
When to serve a cached object?
- The cached object is properly matched: cache-hit
- The requested method and its matched object allows it
- The freshness of the cached object is acceptable
When Varnish matches a request with a cached object (aka cache-hit), it evaluates whether the cache or origin server should be used to construct the response. There are many rules that should be taken into consideration when validating a cache. Most of those rules use caching header fields.
This subsection describes first the concept of cache-hit and cache-miss.
After that, it describes three header fields commonly used to effectively match caches.
These fields are
- Cache-hits are used to reuse, update or invalidate caches
- Objects may have variants (
Fig. 19 shows the flow diagram of a cache-hit. A cache-hit occurs when the requested object (URI) matches a stored HTTP response message (cache). If the matched stored message is valid to construct a response for the client, Varnish serves construct a response and serves it without contacting the origin server.
Fig. 20 shows the flow diagram of a cache-miss. A cache-miss happens when Varnish does not match a cache. In this case, Varnish forwards the request to the origin server.
- Selects a representation of a resource
- Be careful when using
- Wrong usage can create a very large number of cached objects and reduce efficiency
If the origin server sends
Vary in a response, Varnish does not use this response to satisfy a later request unless the later request has the same values for the listed fields in
Vary as the original request.
As a consequence,
Vary expands the cache key required to match a new request to the stored cache entry.
Vary is one of the trickiest headers to deal with when caching.
A caching server like Varnish does not necessarily understand the semantics of a header, or what part triggers different variants of a response.
In other words, an inappropriate use of
Vary might create a very large number of cached objects, and reduce the efficiency of your cache server.
Therefore, you must be extremely cautious when using
Caching objects taking into consideration all differences from requesters creates a very fine-grained caching policy.
This practice is not recommended, because those cached objects are most likely retrieved only by their original requester.
Thus, fine-grained caching strategies do not scale well.
This is a common mistake if
Vary is not used carefully.
An example of wrong usage of
Vary is setting
This tells Varnish that for absolutely any difference in
User-Agent, the response from the origin server might look different.
This is not optimal because there are probably thousands of User-Agent strings out there.
Another example of bad usage is when using
Vary: Cookie to differentiate a response.
Again, there could be a very large number of cookies and hence a very large number of cached objects, which are going to be retrieved most likely only by their original requesters.
The most common usage of
Vary: Accept-Encoding, which tells Varnish that the content might look different depending on the request
For example, a web page can be delivered compressed or uncompressed depending on the client.
For more details on how to use
Vary for compressions, refer to https://www.varnish-cache.org/docs/trunk/users-guide/compression.html.
One way to assist
Vary is by building the response body from cached and non-cached objects.
We will discuss this further in the Content Composition chapter.
Varnish Test Cases (VTC) in
varnishtest can also help you to understand and isolate the behavior of
For more information about it, refer to the subsection Understanding Vary in varnishtest.
Varnish can handle
Vary: Accept-Encoding, because Varnish has support for gzip compression.
- An Entity Tag (
ETag) is metadata to differentiate between multiple states of a resource’s representation
- A differentiator key of presentations in addition to
ETagis a validator header field
- Response header field
Origin servers normally add metadata to further describe the representation of a resource.
This metadata is used in conditional requests.
“Conditional requests are HTTP requests that include one or more header fields indicating a precondition to be tested before applying the method semantics to the target resource” [RFC7232].
ETag is a validator header field.
ETag header field provides a state identifier for the requested variant (resource representation).
ETags are used to differentiate between multiple states based on changes over time of a representation.
ETags are also used differentiate between multiple representations based on content negotiation regardless their state.
Example of an ETag header:
ETag field is validated against the request
If-None-Match header field.
We will see the details of
If-None-Match later in this subsection, but before, we learn about the other validator header field:
- Time-based state of presentations
- Validator header field
- Response header field
Last-Modified response header field is a timestamp that indicates when the variant was last modified.
This headers may be used in conjunction with
Example of a
Last-Modified: Wed, 01 Sep 2004 13:24:52 GMT
Last-Modified are validator header fields, which help to differentiate between representations.
Normally, origin servers send both fields in successful responses.
Whether you use one, another or both, depends on your use cases.
Please refer to Section 2.4 in https://tools.ietf.org/html/rfc7232#section-2.4 for a full description on when to use either of them.
- Precondition Header Field
- Request header field
- Validates local caches against
A client that has obtained a response message and stored it locally, may reuse the obtained
ETag value in future requests to validate its local cache against the selected cache in Varnish.
The obtained value from an
ETag is sent from the client to Varnish in the request
If-None-Match header field.
In fact, a client may have stored multiple resource representations and therefore a client may send an
If-None-Match field with multiple
ETag values to validate.
The purpose of this header field is to reuse local caches without compromising its validity. If the local cache is valid, Varnish replies with a 304 (Not Modified) response, which does not include a message body. In this case, the client reuses its local cache to construct the requested resource.
Example of an If-None-Match header:
- Validates local caches by modification date
- Precondition Header Field
- Request header field
A request containing an
If-Modified-Since header field indicates that the client wants to validate one or more of its local caches by modification date.
If the requested representation has not been modified since the time specified in this field, Varnish returns a 304 (not modified) response.
A 304 response does not contain message body.
This behavior is similar to as when using
Example of an If-Modified-Since header:
If-Modified-Since: Wed, 01 Sep 2004 13:24:52 GMT
The subsection Understanding Last-Modified and If-Modified-Since in varnishtest explains further these concepts with a practical VTC example.
- How to control which caches can be served?
Pragma(for backwards compatibility only)
no-cachedirective. This subsection reviews two common header fields,
Pragma, to check caching allowance.
Cache-Control header field specifies directives that must be applied by all caching mechanisms (from proxy cache to browser cache).
Table 12 summarizes the directives you may use for each context.
The most relevant directives of
public: The response may be cached by any cache.
no-store: The response body must not be stored by any cache mechanism.
no-cache: Authorizes a cache mechanism to store the response in its cache but it must not reuse it without validating it with the origin server first.
In order to avoid any confusion with this argument think of it as a “store-but-do-no-serve-from-cache-without-revalidation” instruction.
max-age: Specifies the period in seconds during which the cache is considered fresh.
max-agebut it applies only to public caches.
Example of a
Cache-Control: public, max-age=2592000
A more hands-on explanation as VTC can be found in the subsection Understanding Cache-Control in varnishtest.
Cache-Control always overrides Expires.
By default, Varnish does not care about the
Cache-Control request header.
If you want to let users update the cache via a force refresh you need to do it yourself.
- Only for legacy
Pragmarequest header is a legacy header and should no longer be used. Some applications still send headers like
Pragma: no-cachebut this is for backwards compatibility reasons only. Any proxy cache should treat
Cache-Control: no-cache, and should not be seen as a reliable header especially when used as a response header.
- Fresh object: age has not yet exceeded its freshness lifetime
- Stale object: has exceeded its freshness lifetime, i.e., expired object
When reusing a stored response (cached object), you should always check its freshness and evaluate whether to deliver expired objects or not. A response’s freshness lifetime is the length of time between its generation by the origin server and its expiration time. A stale (expired) object can also be reused, but only after further validation with the origin server.
As defined in RFC7234 [https://tools.ietf.org/html/rfc7234#section-4.2]:
A response’s age is the time that has passed since it was generated by, or successfully validated with, the origin server. The primary mechanism for determining freshness is for an origin server to provide an explicit expiration time in the future, using either the ``Expires`` header field or the ``max-age`` response directive.
- Response header field calculated at the cache server, i.e., Varnish
- Varnish sends an additional response header field,
Age, to indicate the age of the cache
- Clients (and Varnish) will use the
Ageheader field to determine the freshness of a cache
cache duration = max-age - Age
Agecan be used to disallow caches at the client side
Consider what happens if you let Varnish cache content for a week.
If Varnish does not calculate the age of a cached object, Varnish might happily inform clients that the content is fresh, but the cache could be older than the maximum allowed
By age we mean an estimate amount of time since the response was generated or successfully validated at the origin server.
Client browsers calculate a cache duration based on the
Age header field and the
max-age directive from
If this calculation results in a negative number, the browser does not cache the response locally.
Negative cache duration times, however, do not prevent browsers from using the received object.
Varnish does the same, if you put one Varnish server in front of another.
Exercise: Use article.php to test
- Send a request to article.php via Varnish, and see the response
Ageheader field in
varnishlog -g request -i ReqHeader,RespHeader
- Click the link several times and refresh your browser. Can you identify patterns?
- Analyze the output of
varnishstat -f MAIN.client_req -f MAIN.cache_hitin addition to
- Can you use the
Agefield to determine whether Varnish made a cache hit?
- What is the difference between caching time in Varnish and the client?
- Use different browsers and analyze whether their internal caches work different
You might encounter that different browsers have different behaviors. Some of them might cache content locally, and their behavior when refreshing might be different, which can be very confusing. This just highlights that Varnish is not the only part of your web-stack that parses and honors cache-related headers. There might also be other caches along the way which you do not control, like a company-wide proxy server.
Since browsers might interpret cache headers differently, it is a good idea to control them in your cache server. In the next chapters, you will learn how to modify the response headers Varnish sends. This also allows your origin server to emit response headers that should be seen and used by Varnish only, not in your browser.
When browsers decide to load a resource from their local cache, requests are never sent.
Therefore this exercise and these type of tests are not possible to be simulated in
- Used to stale objects
- Response header field only
Expires response header field gives the time after which the response is considered stale.
Normally, a stale cache item should not be returned by any cache (proxy cache or client cache).
The syntax for this header is:
Expires: GMT formatted date
It is recommended not to define
Expires too far in the future.
Setting it to 1 year is usually enough.
The use of
Expires does not prevent the cached object from being updated.
For example, if the name of the resource is updated.
Cache-Control do more or less the same job, but
Cache-Control gives you more control.
The most significant differences between these two headers is:
Cache-Controluses relative times in seconds
Cache-Controlis both a request and a response header field.
Expiresalways returns an absolute time
Expiresis a response header field only
To learn more about the behavior of
Expires, refer to the subsection Understanding Expires in varnishtest.
Availability of Header Fields¶
Exercise: Test Various Cache Headers Fields with a Real Browser¶
Against a real backend:
- Use httpheadersexample.php via your Varnish server to experiment and get a sense of what it is all about.
- Copy the PHP file from
varnishstat -f MAIN.client_req -f MAIN.cache_hitand
varnishlog -g request -i ReqHeader,RespHeaderto analyze the responses.
- Try every link several times by clicking on them and refreshing your browser.
- Analyze the response in your browser and the activity in your Varnish server.
- Discuss what happens when having the
Expiresfields in the third link.
- When testing
If-Modified-Since, does your browser issue a request to Varnish? If the item was in cache, does Varnish query the origin server?
- Try the
Varyheader field from two different browsers.
varnishstatand another browser, or use client mock-ups in
varnishtestinstead of browsers.