Introduction

Table of contents:

  • What is Varnish?
  • Benefits of Varnish
  • Open source / Free software
  • Varnish Software: The company
  • What is Varnish Plus?
  • Varnish: more than a cache server
  • History of Varnish
  • Varnish Governance Board (VGB)

What is Varnish?

Reverse Proxy

Fig. 1 Varnish is more than a reverse proxy

Varnish is a reverse HTTP proxy, sometimes referred to as an HTTP accelerator or a web accelerator. A reverse proxy is a proxy server that appears to clients as an ordinary server. Varnish stores (caches) files or fragments of files in memory that are used to reduce the response time and network bandwidth consumption on future, equivalent requests. Varnish is designed for modern hardware, modern operating systems and modern work loads.

Varnish is more than a reverse HTTP proxy that caches content to speed up your server. Depending on the installation, Varnish can also be used as:

  • web application firewall,
  • DDoS attack defender,
  • hotlinking protector,
  • load balancer,
  • integration point,
  • single sign-on gateway,
  • authentication and authorization policy mechanism,
  • quick fix for unstable backends, and
  • HTTP router.

Varnish is Flexible

Example of Varnish Configuration Language (VCL):

vcl 4.0;

backend default {
    .host = "127.0.0.1";
    .port = "8080";
}

sub vcl_recv {
    # Do request header transformations here.
    if (req.url ~ "^/admin") {
        return(pass);
    }
}
Varnish is flexible because you can configure it and write your own caching policies in its Varnish Configuration Language (VCL). VCL is a domain specific language based on C. VCL is then translated to C code and compiled, therefore Varnish executes lightning fast. Varnish has shown itself to work well both on large (and expensive) servers and tiny appliances.

Varnish Cache and Varnish Plus

Table 1 Topics Covered in This Book and Their Availability in Varnish Cache and Varnish Plus
Topic Varnish Cache Varnish Plus
VCL Yes Yes
varnishlog Yes Yes
varnishadm Yes Yes
varnishncsa Yes Yes
varnishstat Yes Yes
varnishhist Yes Yes
varnishtest Yes Yes
varnishtop Yes Yes
directors Yes Yes
purge Yes Yes
ban Yes Yes
force cache misses Yes Yes
Hashtwo/xkeys (Varnish Software Implementation of Surrogate Keys) Yes Yes
vagent2 Yes Yes
Massive Storage Engine (MSE) No Yes
Varnish Administration Console (VAC) No Yes
Varnish Tuner No Yes
Varnish Custom Statistics (VCS) No Yes
Varnish High Availability (VHA) No Yes
SSL/TLS frontend support with hitch Yes Yes
SSL/TLS backend support No Yes

Varnish Cache is an open source project, and free software. The development process is public and everyone can submit patches, or just take a peek at the code if there is some uncertainty on how does Varnish Cache work. There is a community of volunteers who help each other and newcomers. The BSD-like license used by Varnish Cache does not place significant restriction on re-use of the code, which makes it possible to integrate Varnish Cache in virtually any solution.

Varnish Cache is developed and tested on GNU/Linux and FreeBSD. The code-base is kept as self-contained as possible to avoid introducing out-side bugs and unneeded complexity. Therefore, Varnish uses very few external libraries.

Varnish Software is the company behind Varnish Cache. Varnish Software and the Varnish community maintain a package repository of Varnish Cache for several common GNU/Linux distributions.

Varnish Software also provides a commercial suite called Varnish Plus with software products for scalability, customization, monitoring and expert support services. The engine of the Varnish Plus commercial suite is the enhanced commercial edition of Varnish Cache. This edition is proprietary and it is called Varnish Cache Plus.

Table 1 shows the components covered in this book and their availability for Varnish Cache users and Varnish Plus customers. The covered components of Varnish Plus are described in the Varnish Plus Software Components chapter. For more information about the complete Varnish Plus offer, please visit https://www.varnish-software.com/what-is-varnish-plus.

At the moment of writing this book, Varnish Cache supports the operating systems and Linux distributions listed in Table 2.

Table 2 Varnish Cache and Varnish Plus supported platforms
  Varnish Cache Varnish Plus
RedHat Enterprise Linux 5 Deprecated Deprecated
RedHat Enterprise Linux 6 Yes Yes
RedHat Enterprise Linux 7 Yes Yes
Ubuntu Linux 12.04 LTS (precise) Yes Yes
Ubuntu Linux 14.04 LTS (trusty) Yes Yes
Debian Linux 7 (wheezy) Yes Yes
FreeBSD 9 Yes No
FreeBSD 10 Yes No

Varnish Cache and Varnish Plus support only 64-bit systems.

Note

Varnish Cache Plus should not be confused with Varnish Plus, a product offering by Varnish Software. Varnish Cache Plus is one of the software components available for Varnish Plus customers.

Varnish Cache and Varnish Software Timeline

  • 2005: Ideas! Verdens Gang (www.vg.no, Norway’s biggest newspaper) were looking for alternative cache solutions
  • 2006: Work began: Redpill Linpro was in charge of project management, infrastructure and supporting development. Poul-Henning Kamp did the majority of the actual development.
  • 2006: Varnish 1.0 is released
  • 2008: Varnish 2.0 is released
  • 2008: varnishtest is introduced
  • 2009: The first Varnish User Group Meeting is held in London Roughly a dozen people participate from all around the world
  • 2010: Varnish Software is born as a spin-off to Redpill Linpro AS
  • 2011: Varnish 3.0 is released
  • 2012: The fifth Varnish User Group Meeting is held in Paris Roughly 70 people participate on the User-day and around 30 on the developer-day!
  • 2012: The Varnish Book is published
  • 2013: Varnish Software chosen as a 2013 Red Herring Top 100 Europe company
  • 2013: BOSSIE award winner
  • 2013: Varnish Software receives World Summit on Innovation & Entrepreneurship Global Hot 100 award
  • 2014: Varnish Plus is launched
  • 2014: Varnish 4.0 is released
  • 2015: Varnish API Engine is released
  • 2015: Gartner names Varnish Software as a 2015 ‘Cool Vendor’ in Web-Scale Platforms
  • 2015: Varnish Plus supports SSL/TLS
  • 2016: Varnish in the cloud
  • 2016: Varnish 5.0 is released

VG, a large Norwegian newspaper, initiated the Varnish project in cooperation with Linpro. The lead developer of the Varnish project, Poul-Henning Kamp, is an experienced FreeBSD kernel hacker. Poul-Henning Kamp continues to bring his wisdom to Varnish in most areas where it counts.

From 2006 throughout 2008, most of the development was sponsored by VG, API, Escenic and Aftenposten, with project management, infrastructure and extra man-power provided by Redpill Linpro. At the time, Redpill Linpro had roughly 140 employees mostly centered around consulting services.

Today Varnish Software is able to fund the core development with income from service agreements, in addition to offering development of specific features on a case-by-case basis. The interest in Varnish continues to increase. An informal study based on the list of most popular web sites in Norway indicates that about 75% or more of the web traffic that originates in Norway is served through Varnish.

Varnish development is governed by the Varnish Governance Board (VGB), which thus far has not needed to intervene. The VGB consists of an architect, a community representative and a representative from Varnish Software.

As of November 2015, the VGB positions are filled by Poul-Henning Kamp (Architect), Rogier Mulhuijzen (Community) and Lasse Karstensen (Varnish Software). On a day-to-day basis, there is little need to interfere with the general flow of development.

What Is New in Varnish 4?

  • Version statement vcl 4.0;
  • req.request is now req.method
  • vcl_fetch is now vcl_backend_response
  • Directors have been moved to the vmod_directors
  • Hash directors as a client directors
  • vcl_error is now vcl_backend_error
  • error() is now synth(), and you must explicitly return it: return (synth(999, "Response"));
  • Synthetic responses in vcl_synth
  • Setting headers on synthetic response bodies made in vcl_synth are now done on resp.http instead of obj.http.
  • obj.* in vcl_error replaced by beresp.* in vcl_backend_error
  • hit_for_pass objects are created using beresp.uncacheable
  • req.* not available in vcl_backend_response
  • bereq.* in vcl_backend_response
  • vcl_* prefix reserved for builtin subroutines
  • req.backend.healthy replaced by std.healthy(req.backend_hint)
  • client.port and server.port replaced by std.port(client.ip) and std.port(server.ip)
  • Cache invalidation with purges is now done via return(purge) in vcl_recv
  • obj.* is now read-only
  • obj.last_use is retired
  • vcl_recv must now return hash instead of lookup
  • vcl_hash must now return lookup instead of hash
  • vcl_pass must now return fetch instead of pass
  • restart in the backend is now retry, this is now called return(retry), and jumps back up to vcl_backend_fetch
  • default VCL is now called builtin VCL
  • The builtin VCL now honors Cache-Control: no-cache (and friends) to indicate uncacheable content from the backend
  • remove keyword replaced by unset
  • X-Forwarded-For is now set before vcl_recv
  • session_linger has been renamed to timeout_linger and it is in seconds now (previously was milliseconds)
  • sess_timeout is renamed to timeout_idle
  • Increasing sess_workspace is not longer necessary, you may need to increase either workspace_backend or workspace_client
  • thread_pool_purge_delay is renamed to thread_pool_destroy_delay and it is in seconds now
  • thread_pool_add_delay and thread_pool_fail_delay are in seconds now
  • New parameter vcc_allow_inline_c to disable inline C in your VCL
  • New query language to filter logs: -m option replaced by -q

The above list tries to summarize the most important changes from Varnish Cache 3 to Varnish Cache 4. For more information, please visit: https://www.varnish-cache.org/docs/trunk/whats-new/upgrading.html

If you want to migrate your VCL code from Varnish 3 to Varnish 4, you may be interested in looking at the varnish3to4 script. See the VCL Migrator from Varnish 3 to Varnish 4 section for more information.