doc/sphinx/whats-new/upgrading-6.1.rst
.. Copyright (c) 2018-2019 Varnish Software AS SPDX-License-Identifier: BSD-2-Clause See LICENSE file for full text of license
.. _whatsnew_upgrading_6.1:
%%%%%%%%%%%%%%%%%%%%%%%% Upgrading to Varnish 6.1 %%%%%%%%%%%%%%%%%%%%%%%%
A configuration for Varnish 6.0.x will run for version 6.1 without
changes. There has been a subtle change in the interpretation of the
VCL variable beresp.keep under specific circumstances, as
discussed below. Other than that, the changes in 6.1 are new features,
described in the following.
We have added the :ref:ref_param_max_vcl parameter to set a
threshold for the number of loaded VCL programs, since it is a common
error to let previous VCL instances accumulate without discarding
them. The remnants of undiscarded VCLs take the form of files in the
working directory of the management process. Over time, too many of
these may take up significant storage space, and administrative
operations such as vcl.list may become noticeably slow, or even
time out, when Varnish has to iterate over many files.
The default threshold in :ref:ref_param_max_vcl is 100, and VCL
labels are not counted against the total. The
:ref:ref_param_max_vcl_handling parameter controls what happens when
you reach the limit. By default you just get a warning from the VCL
compiler, but you can set it to refuse to load more VCLs, or to ignore
the threshold.
Added the :ref:ref_param_backend_local_error_holddown and
:ref:ref_param_backend_remote_error_holddown parameters. These define
delays for new attempts to connect to backends when certain classes of
errors have been encountered, for which immediate re-connect attempts
are likely to be counter-productive. See the parameter documentation
for details.
VCL variables
``req.ttl``, ``req.grace`` and keep
-----------------------------------
``req.grace`` had been previously removed, but was now reintroduced,
since there are use cases that cannot be solved without it. Similarly,
``req.ttl`` used to be deprecated and is now fully supported again.
``req.ttl`` and ``req.grace`` limit the ttl and grace times that are
permitted for the current request. If ``req.ttl`` is set, then cache
objects are considered fresh (and may be cache hits) only if their
remaining ttl is less than or equal to ``req.ttl``. Likewise,
``req.grace`` sets an upper bound on the time an object has spent in
grace to be considered eligible for grace mode (which is to deliver
this object and fetch a fresh copy in the background).
A common application is to set shorter TTLs when the backend is known
to be healthy, so that responses are fresher when all is well. But if
the backend is unhealthy, then use cached responses with longer TTLs
to relieve load on the troubled backend::
sub vcl_recv {
# ...
if (std.healthy(req.backend_hint)) {
# Get responses no older than 70s for healthy backends
set req.ttl = 60s;
set req.grace = 10s;
}
# If the backend is unhealthy, then permit cached responses
# that are older than 70s.
}
The evaluation of the ``beresp.keep`` timer has changed a
bit. ``keep`` sets a lifetime in the cache in addition to TTL for
objects that can be validated by a 304 "Not Modified" response from
the backend to a conditional request (with ``If-None-Match`` or
``If-Modified-Since``). If an expired object is also out of grace
time, then ``vcl_hit`` will no longer be called, so it is impossible
to deliver the "keep" object in this case.
Note that the headers ``If-None-Match`` and ``If-Modified-Since``,
together with the 304 behavior, are handled automatically by Varnish.
If you, for some reason, need to explicitly disable this for a backend
request, then you need do this by removing the headers in
``vcl_backend_fetch``.
The documentation in :ref:`users-guide-handling_misbehaving_servers`
has been expanded to discuss these matters in greater depth, look
there for more details.
``beresp.filters`` and support for backend response processing with VMODs
-------------------------------------------------------------------------
The ``beresp.filters`` variable is readable and writable in
``vcl_backend_response``. This is a space-separated list of modules
that we call VFPs, for "Varnish fetch processors", that may be applied
to a backend response body as it is being fetched. In default Varnish,
the list may include values such as ``gzip``, ``gunzip``, and ``esi``,
depending on how you have set the ``beresp.do_*`` variables.
This addition makes it possible for VMODs to define VFPs to filter or
manipulate backend response bodies, which can be added by changing the
list in ``beresp.filters``. VFPs are applied in the order given in
``beresp.filters``, and you may have to ensure that a VFP is
positioned correctly in the list, for example if it can only apply to
uncompressed response bodies.
This is a new capability, and at the time of release we only know of
test VFPs implemented in VMODs. Over time we hope that an "ecology" of
VFP code will develop that will enrich the features available to
Varnish deployments.
``obj.hits``
------------
Has been fixed to return the correct value in ``vcl_hit`` (it had been
0 in ``vcl_hit``).
Other changes to VCL
The Host header in client requests is mandatory for HTTP/1.1, as
proscribed by the HTTP standard. If it is missing, then
builtin.vcl causes a synthetic 400 "Bad request" response to be
returned.
You can now provide a string argument to return(fail("Foo!")),
which can be used in vcl_init to emit an error message if the
VCL load fails due to the return.
Additional import statements of an already imported vmod are now
ignored.
Added the :ref:std.fnmatch() function to :ref:vmod_std(3), which
you can use for shell-style wildcard matching. Wildcard patterns may
be a good fit for matching URLs, to match against a pattern like
/foo/*/bar/*. The patterns can be built at runtime, if you need to
do that, since they don't need the pre-compile step at VCL load time
that is required for regular expressions. And if you are simply more
comfortable with the wildcard syntax than with regular expressions,
you now have the option.
:ref:vmod_unix(3) is now supported for SunOS and descendants. This
entails changing the privilege set of the child process while the VMOD
is loaded, see the documentation.
varnishd(1):
Some VCL compile-time error messages have been improved, for example when a symbol is not found or arguments to VMOD calls are missing.
Varnish now won't rewrite the Content-Length header when
responding to any HEAD request, making it possible to cache
responses to HEAD requests independently from the GET responses
(previously a HEAD request had to be a pass to avoid this
rewriting).
If you have set .proxy_header=1 (to use the PROXYv1 protocol)
for a backend addressed as a Unix domain socket (with a .path
setting for the socket file), and have also defined a probe for
the backend, then the address family UNKNOWN is sent in
the proxy header for the probe request. If you have set
.proxy_header=2 (for PROXYv2) for a UDS backend with a probe,
then PROXY LOCAL is sent for the probe request.
varnishlog(1) and vsl(7):
The contents of FetchError log entries have been improved to
give better human-readable diagnostics for certain classes of
backend fetch failures.
In particular, http connection (HTC) errors are now reported symbolically in addition to the previous numerical value.
Log entries under the new SessError tag now give more
diagnostic information about session accept failures (failure to
accept a client connection). These must be viewed in raw grouping,
since accept failures are not part of any request/response
transaction.
When a backend is unhealthy, Backend_health now reports some
diagnostic information in addition to the HTTP response and timing
information.
The backend name logged for Backend_health is just the backend
name without the VCL prefix (as appears otherwise for backend
naming).
Added the log entry tag Filters, which gives a list of the
filters applied to a response body (see beresp.filters
discussed above).
varnishadm(1) and varnish-cli(7)
For a number of CLI commands, you can now use the -j argument
to get a JSON response, which may help in automation. These include:
ping -j
backend.list -j
help -j
A JSON response in the CLI always includes a timestamp (epoch time in seconds with millisecond precision), indicating the time at which the response was generated.
The backend.list command now lists both directors and
backends, with their health status. The command now has a -v
option for verbose output, in which detailed health states for
each backend/director are displayed.
varnishstat(1) and varnish-counters(7):
We have added a number of counters to the VBE.* group to help
better diagnose error conditions with backends:
VBE.*.unhealthy: the number of fetches that were not
attempted because the backend was unhealthy
.busy: number of fetches that were not attempted because the
.max_connections limit was reached
.fail: number of failed attempts to open a connection to the
backend. Detailed reasons for the failures are given in the
.fail_* counters (shown at DIAG level), and in the log entry
FetchError. .fail is the sum of the values in the
.fail_* counters.
.fail_eaccess, .fail_eaddrnotavail,
.fail_econnrefused, .fail_enetunreach and
.fail_etimedout: these are the number of attempted
connections to the backend that failed with the given value of
errno(3).
.fail_other: number of connections to the backend that
failed for reasons other than those given by the other
.fail_* counters. For such cases, details on the failure
can be extracted from the varnish log as described above for
FetchError.
.helddown: the number of connections not attempted because
the backend was in the period set by one of the parameters
:ref:ref_param_backend_local_error_holddown or
:ref:ref_param_backend_remote_error_holddown
Similarly, we have added a series of counters for better diagnostics
of session accept failures (failure to accept a connection from a
client). As before, the sess_fail counter gives the total number
of accept failures, and it is now augmented with the sess_fail_*
counters. sess_fail is the sum of the values in sess_fail_*.
sess_fail_econnaborted, sess_fail_eintr,
sess_fail_emfile, sess_fail_ebadf and
sess_fail_enomem: the number of accept failures with the
indicated value of errno(3). The :ref:varnish-counters(7)
man page, and the "long descriptions" shown by varnishstat,
give possible reasons why each of these may happen, and what
might be done to counter the problem.
sess_fail_other: number of accept failures for reasons
other than those given by the other sess_fail_* counters.
More details may appear in the SessError entry of the log
(:ref:varnish-counters(7) shows a varnishlog invocation
that may help).
In curses mode, the information in the header lines (uptimes and cache hit rates) is always reported, even if you have defined a filter that leaves them out of the stats table.
Ban statistics are now reported more accurately (they had been subject to inconsistencies due to race conditions).
varnishtest(1) and vtc(7):
varnishtest and the vtc test script language now support
testing for haproxy as well as Varnish. The haproxy directive
in a test can be used to define, configure, start and stop a
haproxy instance, and you can also script messages to send on the
haproxy CLI connection, and define expectations for the
responses. See the haproxy section in :ref:vtc(7) for
details.
Related to haproxy support, you can now define a syslog
instance in test scripts. This defines a syslog server, and allows
you to test expectations for syslog output from a haproxy
instance.
Added the -keepalive argument for client and server scripts to
be used with the -repeat directive, which causes all test
iterations to run on the same connection, rather than open a new
connection each time. This makes the test run faster and use fewer
ephemeral ports.
Added the -need-bytes argument for the process command,
see :ref:vtc(7).
varnishhist(1):
-P min:max command-line parameters are now optional,
see :ref:varnishhist(1).For all of the utilities that access the Varnish log --
varnishlog(1), varnishncsa(1), varnishtop(1) and
varnishhist(1) -- it was already possible to set multiple -I
and -X command-line arguments. It is now properly documented
that you can use multiple include and exclude filters that apply
regular expressions to selected log records.
Changes for developers:
As mentioned above, VMODs can now implement VFPs that can be added
to backend response processing by changing beresp.filters.
The interface for VFPs is defined in cache_filters.h, and the
debug VMOD included in the distribution shows an example of a
VFP for rot13.
The Varnish API soname version (for libvarnishapi.so) has been bumped to 2.0.0.
The VRT version has been bumped to 8.0. See vrt.h for details
on the changes since 7.0.
Space required by varnish for maintaining the PRIV_TASK and
PRIV_TOP parameters is now taken from the appropriate
workspace rather than from the heap as before. For a failing
allocation, a VCL failure is triggered.
The net effect of this change is that in cases of a workspace
shortage, the almost unavoidable failure will happen earlier. The
amount of workspace required is slightly increased and scales with
the number of vmods per PRIV_TASK and PRIV_TOP parameter.
The VCL compiler (VCC) guarantees that if a vmod function is
called with a PRIV_* argument, that argument value is set.
There is no change with respect to the API the PRIV_* vmod
function arguments provide.
VRT_priv_task(), the function implementing the allocation of
the PRIV_TASK and PRIV_TOP parameters as described above,
is now more likely to return NULL for allocation failures for
the same reason.
Notice that explicit use of this function from within VMODs is considered experimental as this interface may change.
We have improved support for the STRANDS data type, which you
may find easier to use than the varargs-based STRING_LIST. See
vrt.h for details. :ref:vmod_blob(3) has been refactored to
use STRANDS, so you can look there for an example.
We have fixed a bug that had limited the precision available for
the INT data type, so you now get the full 64 bits.
Portions of what had previously been declared in
cache_director.h have been moved into vrt.h, constituting
the public API for directors. The remainder in
cache_director.h is not public, and should not be used by a
VMOD intended for VRT ABI compatibility.
The director API in vrt.h differs from the previous
interface. :ref:ref-writing-a-director has been updated
accordingly. In short, the most important changes include:
struct director_methods is replaced by struct vdi_methodsVRT_AddDirector() and VRT_DelDirector() are to be used
for initialization and destruction.vdi_methods callbacks are not to be called from vmods any moreVRT_Healthy() replaces calls to the healthy functiondestroy callback.Python 3 is now preferred in builds, and will likely be required in future versions.
We believe builds are now reproducible, and intend to keep them that way.
eof