Appendix A: Varnish Programs

SHMLOG tools

  • varnishlog
  • varnishncsa
  • varnishstat
  • varnishhist
  • varnishtop
  • varnishsizes

Administration

  • varnishadm

Misc

  • varnishtest
  • varnishreplay

Varnish provides several tools to help monitor and control Varnish. varnishadm, used to access the management interface, is the only one that can affect a running instance of Varnish.

All the other tools operate exclusively on the shared memory log, often called shmlog in the context of Varnish. They take similar (but not identical) command line arguments, and use the same underlying API.

Among the log-parsing tools, varnishstat is so far unique in that it only looks at counters. The counters are easily found in the shmlog, and are typically polled at reasonable interval to give the impression of real-time updates. Counters, unlike the rest of the log, are not directly mapped to a single request, but represent how many times some specific action has occurred since Varnish started.

The rest of the tools work on the round robin part of the shmlog, which deals with specific requests. Since the shmlog provides large amounts of information, it is usually necessary to filter it. But that does not just mean “show me everything that matches X”. The most basic log tool, varnishlog, will do precisely that. The rest of the tools, however, can process the information further and display running statistical information.

If varnishlog is used to dump data to disk, varnishreplay can simulate a similar load. varnishtest is used for regression tests, mainly during development. Both are outside the scope of this course.

Note

There is a delay in the log process, though usually it is not noticeable. The shmlog is 80MB large by default, which gives some potential history, but that is not guaranteed and it depends heavily on when the last roll-around of the shmlog occurred.

varnishtop

varnishtop -i TxStatus

  list length 6                                                          hostname

  3864.45 TxStatus       200
  1001.33 TxStatus       304
    33.93 TxStatus       301
     3.99 TxStatus       302
     3.00 TxStatus       404
     1.00 TxStatus       403
  • Group tags and tag-content by frequency

varnishtop groups tags and the content of the tag together to generate a sorted list of the most frequently appearing tag/tag-content pair.

Because the usefulness is only visible once you start filtering, it is often overlooked. The above example lists status codes that Varnish returns.

Two of the perhaps most useful variants of varnishtop is:

  • varnishtop -i TxUrl creates a list of URLs requested from a web server. Use this this find out what is causing back-end traffic and start hitting items on the top of the list.
  • varnishtop -i TxStatus lists what status codes Varnish returns to clients. (As shown above)

Some other possibly useful examples are:

  • varnishtop -i RxUrl displays what URLs are most frequently requested from a client.
  • varnishtop -i RxHeader -I 'User-Agent:.*Linux.*' lists User-Agent headers with “Linux” in it (e.g: most used Linux web browsers, that report them self as Linux).
  • varnishtop -i RxStatus will list status codes received from a web server.
  • varnishtop -i VCL_call shows what VCL functions are used.
  • varnishtop -i RxHeader -I Referrer shows the most common referrer addresses.

varnishncsa

10.10.0.1 - - [24/Aug/2008:03:46:48 +0100] "GET \
http://www.example.com/images/foo.png HTTP/1.1" 200 5330 \
"http://www.example.com/" "Mozilla/5.0"

If you already have tools in place to analyze Apache-like logs (NCSA logs), varnishncsa can be used to print the shmlog as ncsa-styled log.

Filtering works similar to varnishlog.

varnishhist

                      #
                      #
                      #
                      #
                      ##
                     ###
                     ###
                     ###
                     ###
                     ###
                  |  ###
                  |  ###
                | |  ###
                |||| ###                    #
                |||| ####                   #
                |##|#####   #           #   #  # #
+-------+-------+-------+-------+-------+-------+-------+-------+-------
|1e-6   |1e-5   |1e-4   |1e-3   |1e-2   |1e-1   |1e0    |1e1    |1e2

Exercise: Try the tools

  • Send a few requests to Varnish using GET -e http://localhost:8000
  • verify you have some cached objects using varnishstat
  • look at the communication with the clients, using varnishlog. Try sending various headers and see them appear in varnishlog.
  • Install siege
  • Run siege against localhost while looking at varnishhist