Varnish Cache Migrate from Version 3 to 4

Varnish Cache Migration from Version 3.x to 4.0

Overview

Varnish Cache is a popular reverse proxy web application accelerator. It generally operates by caching generated pages in memory. Recently, version 4.0 was released which at a high level has the following enhancements:

Full support for streaming objects through from the backend on a cache miss. Bytes will be sent to 1..n requesting clients as they come in from the backend server.
Background (re)fetch of expired objects. On a cache miss where a stale copy is available, serve the client the stale copy while fetching an updated copy from the backend in the background.
New varnishlog query language, allowing automatic grouping of requests when debugging ESI or a failed backend request. (among much more)
Comprehensive request timestamp and byte counters.
Security improvements, including disabling of run-time changes to security sensitive parameters.

Source: https://www.varnish-cache.org/content/varnish-cache-400

Migration from Version 3.4 to Version 4.0

There were many changes to the VCL language and to some of the parameters that the “varnishd” binary accepts. This is not meant to be a comprehensive guide but lessons learned from converting a working “default.vcl” to the new VCL 4.0 syntax. I do not expect my previous “default.vcl” file to have been perfect either, but I did do some further optimization some while working on this conversion. Also I am running NGINX as my backend but none of this is specific to using that as the backend web server.

Topics Covered:

Syntax version declaration
Import of standard VMOD’s
Removal of deprecated syntax and functions
Modification of previous syntax and functions to VCL 4.0 format
Modification of subroutines to VCL 4.0 format
Additional Varnish Security Features
VarnishADM CLI Changes
Useful Links
Builtin.VCL

Please note that one big change is that “builtin.vcl” is automatically appended now to the user “default.vcl”. For reference, I am including the “builtin.vcl” at the end of this post of which I was only able to find by looking at the GitHub repository.

Syntax Version Declaration

For Varnish Cache to know that it is using the new VCL 4.0 format use the following at the head of your default.vcl configuration.

# Required for VCL 4.0
vcl 4.0;

1 2	# Required for VCL 4.0 vcl 4.0;

Note: I do not believe you can use VCL 3.4 syntax in Varnish Cache 4.0 by declaring a different version, but I am not positive.

Import of Standard VMOD’s

This is not technically required but if you choose to use a probe and a check to see if the backend is healthy then you need to import the standard VMOD.

# Import VMod's
import std;

1 2	# Import VMod's import std;

Removal of Deprecated Syntax and Functions

saintmode.threshold
session.linger

Modification of Previous Syntax and Functions to VCL 4.0 Format

[table] Old Syntax,New Syntax[attr style=”width:250px;”], Notes
error,return( synth(error_code\,message)),
remove,unset,
vcl_fetch,vcl_backend_response,
req.backend.healthy,std.healthy(resp.backend_hint), The current documentation is wrong\, use backend_hint and not backend
req.request,req.method,
session_linger,timeout_linger,Parameter for varnishd
client.port,std.port(client.ip),Requires Standard VMOD: import std;
server.port,std.port(server.ip),Requires Standard VMOD: import std;
[/table]

Additional Varnish Security Features

One of the two new features available are that of limiting commands in the Varnish Admin CLI and limiting the use of inline-C code in you VCL files.

To limit commands available for use in the Varnish Admin CLI, add the following argument to your “varnishd” startup script (following parameters after the “-r” are an example of several commands to limit, mostly recommended by the linked blog entry):

-r cc_command,listen_address,user,group,vcc_allow_inline_c

1	-r cc_command,listen_address,user,group,vcc_allow_inline_c

The other security feature available is that, by default, inline-C code is disabled in VCL. Therefore to re-enable you need to do the following:

-p vcc_allow_inline_c=on

1	-p vcc_allow_inline_c=on

The only issue with the above is that I cannot find a way to actually make it work. I can find it in the source code and in theory it looks like it will work but in practice it does not appear to. In addition, I have tried “true”, 1, “on” and nothing works so far. I chose to put “on” in the example as that was what was in the source code.

VarnishADM CLI Changes

One change that is not obvious is that the following will not work anymore:

ban url ~ EXPRESSION

1	ban url ~ EXPRESSION

The correct syntax is the following:

ban req.url ~ EXPRESSION

1	ban req.url ~ EXPRESSION

Useful Links

Builtin.VCL

Source: https://raw.githubusercontent.com/varnish/Varnish-Cache/master/bin/varnishd/builtin.vcl

/*-
 * Copyright (c) 2006 Verdens Gang AS
 * Copyright (c) 2006-2014 Varnish Software AS
 * All rights reserved.
 *
 * Poul-Henning Kamp <phk@phk.freebsd.dk>
 *
 * Redistribution and use in source and binary forms, with or without
 * modification, are permitted provided that the following conditions
 * are met:
 * 1. Redistributions of source code must retain the above copyright
 *    notice, this list of conditions and the following disclaimer.
 * 2. Redistributions in binary form must reproduce the above copyright
 *    notice, this list of conditions and the following disclaimer in the
 *    documentation and/or other materials provided with the distribution.
 *
 * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
 * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
 * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
 * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
 * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
 * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
 * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
 * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
 * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
 * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
 * SUCH DAMAGE.
 *

 *
 * The built-in (previously called default) VCL code.
 *
 * NB! You do NOT need to copy & paste all of these functions into your
 * own vcl code, if you do not provide a definition of one of these
 * functions, the compiler will automatically fall back to the default
 * code from this file.
 *
 * This code will be prefixed with a backend declaration built from the
 * -b argument.
 */

vcl 4.0;

#######################################################################
# Client side

sub vcl_recv {
    if (req.method != "GET" &&
      req.method != "HEAD" &&
      req.method != "PUT" &&
      req.method != "POST" &&
      req.method != "TRACE" &&
      req.method != "OPTIONS" &&
      req.method != "DELETE") {
        /* Non-RFC2616 or CONNECT which is weird. */
        return (pipe);
    }

    /* We don't support chunked uploads, except when piping. */
    if ((req.method == "POST" || req.method == "PUT") &&
      req.http.transfer-encoding ~ "chunked") {
        return(pipe);
    }

    if (req.method != "GET" && req.method != "HEAD") {
        /* We only deal with GET and HEAD by default */
        return (pass);
    }
    if (req.http.Authorization || req.http.Cookie) {
        /* Not cacheable by default */
        return (pass);
    }
    return (hash);
}

sub vcl_pipe {
    # By default Connection: close is set on all piped requests, to stop
    # connection reuse from sending future requests directly to the
    # (potentially) wrong backend. If you do want this to happen, you can undo
    # it here.
    # unset bereq.http.connection;
    return (pipe);
}

sub vcl_pass {
    return (fetch);
}

sub vcl_hash {
    hash_data(req.url);
    if (req.http.host) {
        hash_data(req.http.host);
    } else {
        hash_data(server.ip);
    }
    return (lookup);
}

sub vcl_purge {
    return (synth(200, "Purged"));
}

sub vcl_hit {
    if (obj.ttl >= 0s) {
        // A pure unadultered hit, deliver it
        return (deliver);
    }
    if (obj.ttl + obj.grace > 0s) {
        // Object is in grace, deliver it
        // Automatically triggers a background fetch
        return (deliver);
    }
    // fetch & deliver once we get the result
    return (fetch);
}

sub vcl_miss {
    return (fetch);
}

sub vcl_deliver {
    return (deliver);
}

/*
 * We can come here "invisibly" with the following errors: 413, 417 & 503
 */
sub vcl_synth {
    set resp.http.Content-Type = "text/html; charset=utf-8";
    set resp.http.Retry-After = "5";
    synthetic( {"
<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
  <head>
    <title>"} + resp.status + " " + resp.reason + {"</title>
  </head>
  <body>
    <h1>Error "} + resp.status + " " + resp.reason + {"</h1>
    <p>"} + resp.reason + {"</p>
    <h3>Guru Meditation:</h3>
    <p>XID: "} + req.xid + {"</p>
    <hr>
    <p>Varnish cache server</p>
  </body>
</html>
"} );
    return (deliver);
}

#######################################################################
# Backend Fetch

sub vcl_backend_fetch {
    return (fetch);
}

sub vcl_backend_response {
    if (beresp.ttl <= 0s ||
      beresp.http.Set-Cookie ||
      beresp.http.Surrogate-control ~ "no-store" ||
      (!beresp.http.Surrogate-Control &&
        beresp.http.Cache-Control ~ "no-cache|no-store|private") ||
      beresp.http.Vary == "*") {
        /*
        * Mark as "Hit-For-Pass" for the next 2 minutes
        */
        set beresp.ttl = 120s;
        set beresp.uncacheable = true;
    }
    return (deliver);
}

sub vcl_backend_error {
    set beresp.http.Content-Type = "text/html; charset=utf-8";
    set beresp.http.Retry-After = "5";
    synthetic( {"
<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
  <head>
    <title>"} + beresp.status + " " + beresp.reason + {"</title>
  </head>
  <body>
    <h1>Error "} + beresp.status + " " + beresp.reason + {"</h1>
    <p>"} + beresp.reason + {"</p>
    <h3>Guru Meditation:</h3>
    <p>XID: "} + bereq.xid + {"</p>
    <hr>
    <p>Varnish cache server</p>
  </body>
</html>
"} );
    return (deliver);
}

#######################################################################
# Housekeeping

sub vcl_init {
    return (ok);
}

sub vcl_fini {
    return (ok);
}

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

/*-

* Poul-Henning Kamp <[email protected]>

* Redistribution and use in source and binary forms, with or without

* modification, are permitted provided that the following conditions

* are met:

* 1. Redistributions of source code must retain the above copyright

* notice, this list of conditions and the following disclaimer.

* 2. Redistributions in binary form must reproduce the above copyright

* notice, this list of conditions and the following disclaimer in the

* documentation and/or other materials provided with the distribution.

* THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND

* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE

* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE

* ARE DISCLAIMED. IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE

* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL

* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS

* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)

* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT

* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY

* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF

* SUCH DAMAGE.

* The built-in (previously called default) VCL code.

* NB! You do NOT need to copy & paste all of these functions into your

* own vcl code, if you do not provide a definition of one of these

* functions, the compiler will automatically fall back to the default

* code from this file.

* This code will be prefixed with a backend declaration built from the

* -b argument.

vcl 4.0;

#######################################################################

# Client side

sub vcl_recv {

if (req.method != "GET" &&

req.method != "HEAD" &&

req.method != "PUT" &&

req.method != "POST" &&

req.method != "TRACE" &&

req.method != "OPTIONS" &&

req.method != "DELETE") {

/* Non-RFC2616 or CONNECT which is weird. */

return (pipe);

}

/* We don't support chunked uploads, except when piping. */

if ((req.method == "POST" || req.method == "PUT") &&

req.http.transfer-encoding ~ "chunked") {

return(pipe);

}

if (req.method != "GET" && req.method != "HEAD") {

/* We only deal with GET and HEAD by default */

return (pass);

}

if (req.http.Authorization || req.http.Cookie) {

/* Not cacheable by default */

return (pass);

}

return (hash);

}

sub vcl_pipe {

# By default Connection: close is set on all piped requests, to stop

# connection reuse from sending future requests directly to the

# (potentially) wrong backend. If you do want this to happen, you can undo

# it here.

# unset bereq.http.connection;

return (pipe);

}

sub vcl_pass {

return (fetch);

}

sub vcl_hash {

hash_data(req.url);

if (req.http.host) {

hash_data(req.http.host);

} else {

hash_data(server.ip);

}

return (lookup);

}

sub vcl_purge {

return (synth(200, "Purged"));

}

sub vcl_hit {

if (obj.ttl >= 0s) {

// A pure unadultered hit, deliver it

return (deliver);

}

if (obj.ttl + obj.grace > 0s) {

// Object is in grace, deliver it

// Automatically triggers a background fetch

return (deliver);

}

// fetch & deliver once we get the result

return (fetch);

}

sub vcl_miss {

return (fetch);

}

sub vcl_deliver {

return (deliver);

}

* We can come here "invisibly" with the following errors: 413, 417 & 503

sub vcl_synth {

set resp.http.Content-Type = "text/html; charset=utf-8";

set resp.http.Retry-After = "5";

synthetic( {"

<?xml version="1.0" encoding="utf-8"?>

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"

"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">

<html>

<head>

<title>"} + resp.status + " " + resp.reason + {"</title>

</head>

<body>

<h1>Error "} + resp.status + " " + resp.reason + {"</h1>

<p>"} + resp.reason + {"</p>

<h3>Guru Meditation:</h3>

<hr>

<p>Varnish cache server</p>

</body>

</html>

"} );

return (deliver);

}

#######################################################################

# Backend Fetch

sub vcl_backend_fetch {

return (fetch);

}

sub vcl_backend_response {

if (beresp.ttl <= 0s ||

beresp.http.Set-Cookie ||

beresp.http.Surrogate-control ~ "no-store" ||

(!beresp.http.Surrogate-Control &&

beresp.http.Cache-Control ~ "no-cache|no-store|private") ||

beresp.http.Vary == "*") {

* Mark as "Hit-For-Pass" for the next 2 minutes

set beresp.ttl = 120s;

set beresp.uncacheable = true;

}

return (deliver);

}

sub vcl_backend_error {

set beresp.http.Content-Type = "text/html; charset=utf-8";

set beresp.http.Retry-After = "5";

synthetic( {"

<?xml version="1.0" encoding="utf-8"?>

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"

"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">

<html>

<head>

<title>"} + beresp.status + " " + beresp.reason + {"</title>

</head>

<body>

<h1>Error "} + beresp.status + " " + beresp.reason + {"</h1>

<p>"} + beresp.reason + {"</p>

<h3>Guru Meditation:</h3>

<p>XID: "} + bereq.xid + {"</p>

<hr>

<p>Varnish cache server</p>

</body>

</html>

"} );

return (deliver);

}

#######################################################################

# Housekeeping

sub vcl_init {

return (ok);

}

sub vcl_fini {

return (ok);

}

Listek Consulting

Varnish Cache Migration from Version 3.x to 4.0

Overview

Migration from Version 3.4 to Version 4.0

Syntax Version Declaration

Import of Standard VMOD’s

Removal of Deprecated Syntax and Functions

Modification of Previous Syntax and Functions to VCL 4.0 Format

Additional Varnish Security Features

VarnishADM CLI Changes

Useful Links

Builtin.VCL

Tags

Leave a Reply Cancel reply