Varnish Cache Migration from Version 3.x to 4.0

Varnish Cache Migration from Version 3.x to 4.0

Overview

Varnish Cache is a popular reverse proxy web application accelerator. It generally operates by caching generated pages in memory. Recently, version 4.0 was released which at a high level has the following enhancements:

  • Full support for streaming objects through from the backend on a cache miss. Bytes will be sent to 1..n requesting clients as they come in from the backend server.
  • Background (re)fetch of expired objects. On a cache miss where a stale copy is available, serve the client the stale copy while fetching an updated copy from the backend in the background.
  • New varnishlog query language, allowing automatic grouping of requests when debugging ESI or a failed backend request. (among much more)
  • Comprehensive request timestamp and byte counters.
  • Security improvements, including disabling of run-time changes to security sensitive parameters.

Source: https://www.varnish-cache.org/content/varnish-cache-400

Migration from Version 3.4 to Version 4.0

There were many changes to the VCL language and to some of the parameters that the “varnishd” binary accepts. This is not meant to be a comprehensive guide but lessons learned from converting a working “default.vcl” to the new VCL 4.0 syntax. I do not expect my previous “default.vcl” file to have been perfect either, but I did do some further optimization some while working on this conversion. Also I am running NGINX as my backend but none of this is specific to using that as the backend web server.

Topics Covered:

  • Syntax version declaration
  • Import of standard VMOD’s
  • Removal of deprecated syntax and functions
  • Modification of previous syntax and functions to VCL 4.0 format
  • Modification of subroutines to VCL 4.0 format
  • Additional Varnish Security Features
  • VarnishADM CLI Changes
  • Useful Links
  • Builtin.VCL

Please note that one big change is that “builtin.vcl” is automatically appended now to the user “default.vcl”. For reference, I am including the “builtin.vcl” at the end of this post of which I was only able to find by looking at the GitHub repository.

Syntax Version Declaration

For Varnish Cache to know that it is using the new VCL 4.0 format use the following at the head of your default.vcl configuration.

# Required for VCL 4.0
vcl 4.0;

Note: I do not believe you can use VCL 3.4 syntax in Varnish Cache 4.0 by declaring a different version, but I am not positive.

Import of Standard VMOD’s

This is not technically required but if you choose to use a probe and a check to see if the backend is healthy then you need to import the standard VMOD.

# Import VMod's
import std;

Removal of Deprecated Syntax and Functions

  • saintmode.threshold
  • session.linger

Modification of Previous Syntax and Functions to VCL 4.0 Format

[table] Old Syntax,New Syntax[attr style=”width:250px;”], Notes
error,return( synth(error_code\,message)),
remove,unset,
vcl_fetch,vcl_backend_response,
req.backend.healthy,std.healthy(resp.backend_hint), The current documentation is wrong\, use backend_hint and not backend
req.request,req.method,
session_linger,timeout_linger,Parameter for varnishd
client.port,std.port(client.ip),Requires Standard VMOD: import std;
server.port,std.port(server.ip),Requires Standard VMOD: import std;
[/table]

Additional Varnish Security Features

One of the two new features available are that of limiting commands in the Varnish Admin CLI and limiting the use of inline-C code in you VCL files.

To limit commands available for use in the Varnish Admin CLI, add the following argument to your “varnishd” startup script (following parameters after the “-r” are an example of several commands to limit, mostly recommended by the linked blog entry):

-r cc_command,listen_address,user,group,vcc_allow_inline_c

The other security feature available is that, by default, inline-C code is disabled in VCL. Therefore to re-enable you need to do the following:

-p vcc_allow_inline_c=on

The only issue with the above is that I cannot find a way to actually make it work. I can find it in the source code and in theory it looks like it will work but in practice it does not appear to. In addition, I have tried “true”, 1, “on” and nothing works so far. I chose to put “on” in the example as that was what was in the source code.

VarnishADM CLI Changes

One change that is not obvious is that the following will not work anymore:

ban url ~ EXPRESSION

The correct syntax is the following:

ban req.url ~ EXPRESSION

Useful Links

Builtin.VCL

Source: https://raw.githubusercontent.com/varnish/Varnish-Cache/master/bin/varnishd/builtin.vcl

/*-
 * Copyright (c) 2006 Verdens Gang AS
 * Copyright (c) 2006-2014 Varnish Software AS
 * All rights reserved.
 *
 * Poul-Henning Kamp <[email protected]>
 *
 * Redistribution and use in source and binary forms, with or without
 * modification, are permitted provided that the following conditions
 * are met:
 * 1. Redistributions of source code must retain the above copyright
 *    notice, this list of conditions and the following disclaimer.
 * 2. Redistributions in binary form must reproduce the above copyright
 *    notice, this list of conditions and the following disclaimer in the
 *    documentation and/or other materials provided with the distribution.
 *
 * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
 * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
 * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
 * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
 * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
 * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
 * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
 * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
 * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
 * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
 * SUCH DAMAGE.
 *

 *
 * The built-in (previously called default) VCL code.
 *
 * NB! You do NOT need to copy & paste all of these functions into your
 * own vcl code, if you do not provide a definition of one of these
 * functions, the compiler will automatically fall back to the default
 * code from this file.
 *
 * This code will be prefixed with a backend declaration built from the
 * -b argument.
 */

vcl 4.0;

#######################################################################
# Client side

sub vcl_recv {
    if (req.method != "GET" &&
      req.method != "HEAD" &&
      req.method != "PUT" &&
      req.method != "POST" &&
      req.method != "TRACE" &&
      req.method != "OPTIONS" &&
      req.method != "DELETE") {
        /* Non-RFC2616 or CONNECT which is weird. */
        return (pipe);
    }

    /* We don't support chunked uploads, except when piping. */
    if ((req.method == "POST" || req.method == "PUT") &&
      req.http.transfer-encoding ~ "chunked") {
        return(pipe);
    }

    if (req.method != "GET" && req.method != "HEAD") {
        /* We only deal with GET and HEAD by default */
        return (pass);
    }
    if (req.http.Authorization || req.http.Cookie) {
        /* Not cacheable by default */
        return (pass);
    }
    return (hash);
}

sub vcl_pipe {
    # By default Connection: close is set on all piped requests, to stop
    # connection reuse from sending future requests directly to the
    # (potentially) wrong backend. If you do want this to happen, you can undo
    # it here.
    # unset bereq.http.connection;
    return (pipe);
}

sub vcl_pass {
    return (fetch);
}

sub vcl_hash {
    hash_data(req.url);
    if (req.http.host) {
        hash_data(req.http.host);
    } else {
        hash_data(server.ip);
    }
    return (lookup);
}

sub vcl_purge {
    return (synth(200, "Purged"));
}

sub vcl_hit {
    if (obj.ttl >= 0s) {
        // A pure unadultered hit, deliver it
        return (deliver);
    }
    if (obj.ttl + obj.grace > 0s) {
        // Object is in grace, deliver it
        // Automatically triggers a background fetch
        return (deliver);
    }
    // fetch & deliver once we get the result
    return (fetch);
}

sub vcl_miss {
    return (fetch);
}

sub vcl_deliver {
    return (deliver);
}

/*
 * We can come here "invisibly" with the following errors: 413, 417 & 503
 */
sub vcl_synth {
    set resp.http.Content-Type = "text/html; charset=utf-8";
    set resp.http.Retry-After = "5";
    synthetic( {"
<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
  <head>
    <title>"} + resp.status + " " + resp.reason + {"</title>
  </head>
  <body>
    <h1>Error "} + resp.status + " " + resp.reason + {"</h1>
    <p>"} + resp.reason + {"</p>
    <h3>Guru Meditation:</h3>
    <p>XID: "} + req.xid + {"</p>
    <hr>
    <p>Varnish cache server</p>
  </body>
</html>
"} );
    return (deliver);
}

#######################################################################
# Backend Fetch

sub vcl_backend_fetch {
    return (fetch);
}

sub vcl_backend_response {
    if (beresp.ttl <= 0s ||
      beresp.http.Set-Cookie ||
      beresp.http.Surrogate-control ~ "no-store" ||
      (!beresp.http.Surrogate-Control &&
        beresp.http.Cache-Control ~ "no-cache|no-store|private") ||
      beresp.http.Vary == "*") {
        /*
        * Mark as "Hit-For-Pass" for the next 2 minutes
        */
        set beresp.ttl = 120s;
        set beresp.uncacheable = true;
    }
    return (deliver);
}

sub vcl_backend_error {
    set beresp.http.Content-Type = "text/html; charset=utf-8";
    set beresp.http.Retry-After = "5";
    synthetic( {"
<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
  <head>
    <title>"} + beresp.status + " " + beresp.reason + {"</title>
  </head>
  <body>
    <h1>Error "} + beresp.status + " " + beresp.reason + {"</h1>
    <p>"} + beresp.reason + {"</p>
    <h3>Guru Meditation:</h3>
    <p>XID: "} + bereq.xid + {"</p>
    <hr>
    <p>Varnish cache server</p>
  </body>
</html>
"} );
    return (deliver);
}

#######################################################################
# Housekeeping

sub vcl_init {
    return (ok);
}

sub vcl_fini {
    return (ok);
}

 

Tags