Development Report for June 23, 2017

Events Service

We added an events service for getting events across various services including containers, content, execution, images, namespaces, and snapshots. Additionally we added ctr events to view events emitted by the service.

$ ctr events
2017-06-23 23:21:30.271802153 +0000 UTC   /snapshot/prepare   key=registry parent=sha256:dc22a13eb565d14bfe2b16f6fa731a05da0eeff02a52059c7b59cdc2c232a2b2
2017-06-23 23:21:30.28045193 +0000 UTC   /containers/create   id=registry image=docker.io/library/registry:latest runtime=&ContainerCreate_Runtime{Name:io.containerd.runtime.v1.linux,Options:map[string]string{},}
2017-06-23 23:21:30.347842352 +0000 UTC   /runtime/task-create   id=registry type=CREATE pid=30411 status=0 exited=0001-01-01 00:00:00 +0000 UTC
2017-06-23 23:21:30.355290368 +0000 UTC   /runtime/create   id=registry bundle=/var/lib/containerd/io.containerd.runtime.v1.linux/default/registry rootfs=type=overlay:src=overlay checkpoint=
2017-06-23 23:21:30.362012776 +0000 UTC   /tasks/create   id=registry
2017-06-23 23:21:30.369742117 +0000 UTC   /runtime/task-start   id=registry type=START pid=30411 status=0 exited=0001-01-01 00:00:00 +0000 UTC
2017-06-23 23:21:30.369793151 +0000 UTC   /tasks/start   id=registry
2017-06-23 23:21:40.169884207 +0000 UTC   /runtime/task-exit   id=registry type=EXIT pid=30411 status=130 exited=2017-06-23 23:21:40.16962998 +0000 UTC
2017-06-23 23:21:40.185850194 +0000 UTC   /runtime/delete   id=registry runtime=io.containerd.runtime.v1.linux status=130 exited=2017-06-23 23:21:40.16962998 +0000 UTC
2017-06-23 23:21:40.225633455 +0000 UTC   /tasks/delete   id=registry pid=30411 status=130
2017-06-23 23:21:42.053154149 +0000 UTC   /snapshot/remove   key=registry
2017-06-23 23:21:42.061579495 +0000 UTC   /containers/delete   id=registry

#956: Events service

Filters

We added the syntax to use for filtration of items over the containerd API. The filter package defines a syntax and parser that can be used across types and use cases in a uniform manner. This will be used commonly across the API for images, containers, events, snapshots, etc.

The syntax is fairly familiar, if you've used container ecosystem projects. At the core, we base it on the concept of protobuf field paths, augmenting with the ability to quote portions of the field path to match arbitrary labels. These "selectors" come in the following syntax:

<fieldpath>[<operator><value>]

A basic example is as follows:

name==foo

This would match all objects that have a field name with the value foo. If we only want to test if the field is present, we can omit the operator. This is most useful for matching labels in containerd. The following will match objects that has the field labels and have the label "foo" defined:

labels.foo

We also allow for quoting of parts of the field path to allow matching of arbitrary items:

labels."very complex label"==something

We also define != and ~= as operators. The != operator will match all objects that don't match the value for a field and ~= will compile the target value as a regular expression and match the field value against that.

Selectors can be combined using a comma, such that the resulting selector will require all selectors are matched for the object to match. The following example will match objects that are named foo and have the label bar:

name==foo,labels.bar

#995: Add package for filters

#1050: Container list filters

#1069: Event filters

Support for pushing and pulling OCI indexes

We added support for pushing and pulling OCI indexes. Currently all content referenced by the list are pulled and further work on the client will be done to allow selection of the pulled manifest to extract and run.

#1022: Support pull for OCI indexes

Snapshot list

We added ctr snapshot list to snapshots from containerd. This will output all snapshots, not just the active snapshots used by containers.

$ ctr snapshot list
ID                                                                      Parent                                                                  State     Readonly
registry2                                                               sha256:dc22a13eb565d14bfe2b16f6fa731a05da0eeff02a52059c7b59cdc2c232a2b2 active    false
registry3                                                               sha256:dc22a13eb565d14bfe2b16f6fa731a05da0eeff02a52059c7b59cdc2c232a2b2 active    false
sha256:4ac69ce655ab8aa97362915793348d31361fb3c047e223c2b58be706e89c48fc sha256:ba2cc2690e31f63847e4bc0d266b354f8f11dc04474d45d44312ff70edae9c98 committed true
sha256:ba2cc2690e31f63847e4bc0d266b354f8f11dc04474d45d44312ff70edae9c98                                                                         committed true
sha256:bfe0b04fc169b94099b29dbf5a527f6a11db627cd0a6126803edf8f42bd7b4b3 sha256:4ac69ce655ab8aa97362915793348d31361fb3c047e223c2b58be706e89c48fc committed true
sha256:d959def87dadbb9ba85070c09e99b46d994967b12f5748f617c377073b8d1e39 sha256:bfe0b04fc169b94099b29dbf5a527f6a11db627cd0a6126803edf8f42bd7b4b3 committed true
sha256:dc22a13eb565d14bfe2b16f6fa731a05da0eeff02a52059c7b59cdc2c232a2b2 sha256:d959def87dadbb9ba85070c09e99b46d994967b12f5748f617c377073b8d1e39 committed true

#1058: Add list snapshot subcommand

API cleanup

As part of our API review process we have started implementing some changes to make the API clearer and more consistent.

#1040: Update GRPC APIs

#1047: Rename execution service to tasks

#1059: enforce a character set for namespaces

#1052: version protobuf service packages

#1072: move events types into service package

#1073: consolidate API types packages

#1074: replace whale with D

Containerd notes from Moby Summit

At the Moby summit on 06/19/2017 there was a containerd round table meeting. This was a good opportunity to discuss the upcoming API freeze and release of containerd with others working with it in the community. Always remember that these summit round tables are not the only opportunity to have these topics discussed and everyone is encouraged to open issues and engage the community on Slack.

• "What are the plans for a resolver-resolver and image reference namespacing?"

  • Maintainers are trying to figure out what everyone’s plans/needs are for a resolver-resolver.
  • A resolver-resolver allows configuring where push/pull happens. Could take in a configuration format which specifies how to push, pull, and authenticate. Needed in order to do discover of names to registry configurations.
  • Stephen confirms we are thinking of more configuration driven rather programmatic. The resolver-resolver and any configuration is always client side, allowing the integrator to design any interface or configuration suits their needs. But we are also looking to define something that could be consistently applied by containerd clients.
  • The resolver-resolver will be compatible with Docker, but could allow Docker to expand its resolution support.

• "What is the plan for garbage collection?"

  • Current design is oriented around being garbage collected. The data model is designed around references which are used to walk a root set. Everything unreferenced would be garbage collected.
  • Another more complicated aspect of garbage collection is around policy, allowing to clean up resources based on age or client specified policies.
  • Client side implementations would allow establishing policies.
  • Containerd will require a stop the world to do the garbage collection.
  • A heavy delete which does not garbage collect is an option and similar to to the interface today. The API does not guarantee that disk is cleaned up after a deletion, only that the resource is freed. Inline deletion would require reference counting to know when to delete. This would also require global locking to protect the references.
  • How to handle content which is active but unreferenced, leasing vs pre-allocate. This has not been decided on.

• "What will need to change in Docker in regards to graphdrivers for accommodating the containerd snapshotters?"

  • The goal is to have graphdrivers be able to be used as snapshot drivers.
  • To accomplish this graphdrivers need to be able to return mounts without needing to require action on unmount. Unmount will no longer contact the graphdriver since snapshotters to do not control the mount/unmount lifecycle.
  • For implementation, in the repository tree only overlayfs and btrfs are supported. Everything else will be out of tree and require special builds or proxying.

• "Version numbers on events and API objects"

  • Are objects mutable? Some objects are mutable but not necessarily relevant to clients.
  • Are containers mutable? It can have multiple tasks, can be started and stopped.
  • We may need to have task id separate from the container to differentiate between tasks within a container.
  • Tasks have a pid which could be used to identify the task. Someone had a customer that ran out of memory from running too many tasks, could be caused by repeated execs in the same container.
  • Getting state of a containers task involves requesting containers and tasks, or just tasks could be listed.
  • What are the API costs, is there measurements of the cost of API calls and comparisons with Docker? Calling docker ps can be expensive due to locks, containerd should be much cheaper and faster to call. This need to be verified and measurements added.

• "How can clients understand the capabilities of the containerd instance it is talking to?"

  • As an orchestrator, interested in understanding what can be done with the plugins. Example, docker log drivers change feature set but the log driver names do not change. Stuck on checking docker version.
  • There have been problems in the past with requiring multiple clients of docker to handle changes.
  • GRPC is adding introspection, plan is to wait for this work rather than making something ourselves.
  • The maintainers would like to understand what were the really bad changes in Docker that caused problems with version support for users. Problems around needing to bump the whole API to get a new feature. Containerd API versions each service and v1 interface will be stable and supported.