doc/releases/dumpling.rst
Dumpling is the 4th stable release of Ceph. It is named after the dumpling squid (Euprymna tasmanica).
This stable update for Dumpling fixes a few longstanding issues with backfill in the OSD that can lead to stalled IOs. There is also a fix for memory utilization for reads in librbd when caching is enabled, and then several other small fixes across the rest of the system.
Dumpling users who have encountered IO stalls during backfill and who do not expect to upgrade to Firefly soon should upgrade. Everyone else should upgrade to Firefly already. This is likely to be the last stable release for the 0.67.x Dumpling series.
This stable update for Dumpling fixes several important bugs that affect a small set of users.
We recommend that all Dumpling users upgrade at their convenience. If none of these issues are affecting your deployment there is no urgency.
This stable update release for Dumpling includes primarily fixes for RGW, including several issues with bucket listings and a potential data corruption problem when multiple multi-part uploads race. There is also some throttling capability added in the OSD for scrub that can mitigate the performance impact on production clusters.
We recommend that all Dumpling users upgrade at their convenience.
For more detailed information, see :download:the complete changelog <../changelog/v0.67.10.txt>.
This Dumpling point release fixes several minor bugs. The most prevalent in the field is one that occasionally prevents OSDs from starting on recently created clusters.
We recommend that all Dumpling users upgrade at their convenience.
For more detailed information, see :download:the complete changelog <../changelog/v0.67.9.txt>.
This Dumpling point release fixes several non-critical issues since v0.67.7. The most notable bug fixes are an auth fix in librbd (observed as an occasional crash from KVM), an improvement in the network failure detection with the monitor, and several hard to hit OSD crashes or hangs.
We recommend that all users upgrade at their convenience.
The 'rbd ls' function now returns success and returns an empty when a pool does not store any rbd images. Previously it would return an ENOENT error.
Ceph will now issue a health warning if the 'mon osd down out interval' config option is set to zero. This warning can be disabled by adding 'mon warn on osd down out interval zero = false' to ceph.conf.
For more detailed information, see :download:the complete changelog <../changelog/v0.67.8.txt>.
This Dumpling point release fixes a few critical issues in v0.67.6.
All v0.67.6 users are urgently encouraged to upgrade. We also recommend that all v0.67.5 (or older) users upgrade.
For more detailed information, see :download:the complete changelog <../changelog/v0.67.7.txt>.
.. note:: This release contains a librbd bug that is fixed in v0.67.7. Please upgrade to v0.67.7 and do not use v0.67.6.
This Dumpling point release contains a number of important fixed for the OSD, monitor, and radosgw. Most significantly, a change that forces large object attributes to spill over into leveldb has been backported that can prevent objects and the cluster from being damaged by large attributes (which can be induced via the radosgw). There is also a set of fixes that improves data safety and RADOS semantics when the cluster becomes full and then non-full.
We recommend that all 0.67.x Dumpling users skip this release and upgrade to v0.67.7.
The OSD has long contained a feature that allows large xattrs to spill over into the leveldb backing store in situations where not all local file systems are able to store them reliably. This option is now enabled unconditionally in order to avoid rare cases where storing large xattrs renders the object unreadable. This is known to be triggered by very large multipart objects, but could be caused by other workloads as well. Although there is some small risk that performance for certain workloads will degrade, it is more important that data be retrievable. Note that newer versions of Ceph (e.g., firefly) do some additional work to avoid the potential performance regression in this case, but that is current considered too complex for backport to the Dumpling stable series.
It is very dangerous to downgrade from v0.67.6 to a prior version of Dumpling. If the old version does not have 'filestore xattr use omap = true' it may not be able to read all xattrs for an object and can cause undefined behavior.
For more detailed information, see :download:the complete changelog <../changelog/v0.67.6.txt>.
This release includes a few critical bug fixes for the radosgw, including a fix for hanging operations on large objects. There are also several bug fixes for radosgw multi-site replications, and a few backported features. Also, notably, the 'osd perf' command (which dumps recent performance information about active OSDs) has been backported.
We recommend that all 0.67.x Dumpling users upgrade.
For more detailed information, see :download:the complete changelog <../changelog/v0.67.5.txt>.
This point release fixes an important performance issue with radosgw, keystone authentication token caching, and CORS. All users (especially those of rgw) are encouraged to upgrade.
For more detailed information, see :download:the complete changelog <../changelog/v0.67.4.txt>.
This point release fixes a few important performance regressions with the OSD (both with CPU and disk utilization), as well as several other important but less common problems. We recommend that all production users upgrade.
For more detailed information, see :download:the complete changelog <../changelog/v0.67.3.txt>.
This is an important point release for Dumpling. Most notably, it fixes a problem when upgrading directly from v0.56.x Bobtail to v0.67.x Dumpling (without stopping at v0.61.x Cuttlefish along the way). It also fixes a problem with the CLI parsing of the CEPH_ARGS environment variable, high CPU utilization by the ceph-osd daemons, and cleans up the radosgw shutdown sequence.
For more detailed information, see :download:the complete changelog <../changelog/v0.67.2.txt>.
This is a minor point release for Dumpling that fixes problems with OpenStack and librbd hangs when caching is disabled.
For more detailed information, see :download:the complete changelog <../changelog/v0.67.1.txt>.
This is the fourth major release of Ceph, code-named "Dumpling." The headline features for this release include:
Multi-site support for radosgw. This includes the ability to set up separate "regions" in the same or different Ceph clusters that share a single S3/Swift bucket/container namespace.
RESTful API endpoint for Ceph cluster administration. ceph-rest-api, a wrapper around ceph_rest_api.py, can be used to start up a test single-threaded HTTP server that provides access to cluster information and administration in very similar ways to the ceph commandline tool. ceph_rest_api.py can be used as a WSGI application for deployment in a more-capable web server. See ceph-rest-api.8 for more.
Object namespaces in librados.
.. _dumpling-upgrade:
It is possible to do a rolling upgrade from Cuttlefish to Dumpling.
#. Upgrade ceph-common on all nodes that will use the command line 'ceph' utility. #. Upgrade all monitors (upgrade ceph package, restart ceph-mon daemons). This can happen one daemon or host at a time. Note that because cuttlefish and dumpling monitors can't talk to each other, all monitors should be upgraded in relatively short succession to minimize the risk that an a untimely failure will reduce availability. #. Upgrade all osds (upgrade ceph package, restart ceph-osd daemons). This can happen one daemon or host at a time. #. Upgrade radosgw (upgrade radosgw package, restart radosgw daemons).
There is monitor internal protocol change, which means that v0.67 ceph-mon daemons cannot talk to v0.66 or older daemons. We recommend upgrading all monitors at once (or in relatively quick succession) to minimize the possibility of downtime.
The output of 'ceph status --format=json' or 'ceph -s --format=json' has changed to return status information in a more structured and usable format.
The 'ceph pg dump_stuck [threshold]' command used to require a --threshold or -t prefix to the threshold argument, but now does not.
Many more ceph commands now output formatted information; select with '--format=<format>', where <format> can be 'json', 'json-pretty', 'xml', or 'xml-pretty'.
The 'ceph pg <pgid> ...' commands (like 'ceph pg <pgid> query') are deprecated in favor of 'ceph tell <pgid> ...'. This makes the distinction between 'ceph pg <command> <pgid>' and 'ceph pg <pgid> <command>' less awkward by making it clearer that the 'tell' commands are talking to the OSD serving the placement group, not the monitor.
The 'ceph --admin-daemon <path> <command ...>' used to accept the command and arguments as either a single string or as separate arguments. It will now only accept the command spread across multiple arguments. This means that any script which does something like::
ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok 'config set debug_ms 1'
needs to remove the quotes. Also, note that the above can now be shortened to::
ceph daemon osd.0 config set debug_ms 1
The radosgw caps were inconsistently documented to be either 'mon = allow r' or 'mon = allow rw'. The 'mon = allow rw' is required for radosgw to create its own pools. All documentation has been updated accordingly.
The radosgw copy object operation may return extra progress info during the operation. At this point it will only happen when doing cross zone copy operations. The S3 response will now return extra <Progress> field under the <CopyResult> container. The Swift response will now send the progress as a json array.
In v0.66 and v0.65 the HASHPSPOOL pool flag was enabled by default for new pools, but has been disabled again until Linux kernel client support reaches more distributions and users.
ceph-osd now requires a max file descriptor limit (e.g., ulimit -n ...) of at least
filestore_wbthrottle_(xfs|btrfs)_inodes_hard_limit (5000 by default)
in order to accommodate the new write back throttle system. On
Ubuntu, upstart now sets the fd limit to 32k. On other platforms,
the sysvinit script will set it to 32k by default (still
overridable via max_open_files). If this field has been customized
in ceph.conf it should likely be adjusted upwards.
In addition to the above notes about upgrading from v0.66:
There has been a huge revamp of the 'ceph' command-line interface
implementation. The ceph-common client library needs to be
upgrade before ceph-mon is restarted in order to avoid problems
using the CLI (the old ceph client utility cannot talk to the
new ceph-mon).
The CLI is now very careful about sending the 'status' one-liner output to stderr and command output to stdout. Scripts relying on output should take care.
The 'ceph osd tell ...' and 'ceph mon tell ...' commands are no longer supported. Any callers should use::
ceph tell osd.<id or *> ... ceph tell mon.<id or name or *> ...
The 'ceph mds tell ...' command is still there, but will soon also transition to 'ceph tell mds.<id or name or *> ...'
The 'ceph osd crush add ...' command used to take one of two forms::
ceph osd crush add 123 osd.123 <weight> <location ...> ceph osd crush add osd.123 <weight> <location ...>
This is because the id and crush name are redundant. Now only the simple form is supported, where the osd name/id can either be a bare id (integer) or name (osd.<id>)::
ceph osd crush add osd.123 <weight> <location ...> ceph osd crush add 123 <weight> <location ...>
There is now a maximum RADOS object size, configurable via 'osd max object size', defaulting to 100 GB. Note that this has no effect on RBD, CephFS, or radosgw, which all stripe over objects. If you are using librados and storing objects larger than that, you will need to adjust 'osd max object size', and should consider using smaller objects instead.
The 'osd min down {reporters|reports}' config options have been
renamed to 'mon osd min down {reporters|reports}', and the
documentation has been updated to reflect that these options apply
to the monitors (who process failure reports) and not OSDs. If you
have adjusted these settings, please update your ceph.conf
accordingly.
Huge revamp of the 'ceph' command-line interface implementation.
The ceph-common client library needs to be upgrade before
ceph-mon is restarted in order to avoid problems using the CLI
(the old ceph client utility cannot talk to the new
ceph-mon).
The CLI is now very careful about sending the 'status' one-liner output to stderr and command output to stdout. Scripts relying on output should take care.
The 'ceph osd tell ...' and 'ceph mon tell ...' commands are no longer supported. Any callers should use::
ceph tell osd.<id or *> ... ceph tell mon.<id or name or *> ...
The 'ceph mds tell ...' command is still there, but will soon also transition to 'ceph tell mds.<id or name or *> ...'
The 'ceph osd crush add ...' command used to take one of two forms::
ceph osd crush add 123 osd.123 <weight> <location ...> ceph osd crush add osd.123 <weight> <location ...>
This is because the id and crush name are redundant. Now only the simple form is supported, where the osd name/id can either be a bare id (integer) or name (osd.<id>)::
ceph osd crush add osd.123 <weight> <location ...> ceph osd crush add 123 <weight> <location ...>
There is now a maximum RADOS object size, configurable via 'osd max object size', defaulting to 100 GB. Note that this has no effect on RBD, CephFS, or radosgw, which all stripe over objects.
ceph.conf
accordingly.