docs/generated/logformats.md
The supported log output formats are documented below.
crdb-v1This is a legacy file format used from CockroachDB v1.0.
Each log entry is emitted using a common prefix, described below, followed by:
[ and ], if any. It is possible
for this to be omitted if there were no context tags.Beware that the text of the log entry can span multiple lines. The following caveats apply:
The text of the log entry can start with text enclosed between [ and ].
If there were no logging tags to start with, it is not possible to distinguish between
logging context tag information and a [...] string in the main text of the
log entry. This means that this format is ambiguous.
To remove this ambiguity, you can use the option 'show-counter'.
The text of the log entry can embed arbitrary application-level strings, including strings that represent log entries. In particular, an accident of implementation can cause the common entry prefix (described below) to also appear on a line of its own, as part of the payload of a previous log entry. There is no automated way to recognize when this occurs. Care must be taken by a human observer to recognize these situations.
The log entry parser provided by CockroachDB to read log files is faulty and is unable to recognize the aforementioned pitfall; nor can it read entries larger than 64KiB successfully. Generally, use of this internal log entry parser is discouraged for entries written with this format.
See the newer format crdb-v2 for an alternative
without these limitations.
At the beginning of each file, a header is printed using a similar format as regular log entries. This header reports when the file was created, which parameters were used to start the server, the server identifiers if known, and other metadata about the running process.
INFO (with an I prefix
at the start of the line) even though it does not really have a severity.INFO level.Each line of output starts with the following prefix:
Lyymmdd hh:mm:ss.uuuuuu goid [chan@]file:line marker tags counter
Reminder, the tags may be omitted; and the counter is only printed if the option 'show-counter' is specified.
| Field | Description |
|---|---|
| L | A single character, representing the log level (e.g., I for INFO). |
| yy | The year (zero padded; i.e., 2016 is 16). |
| mm | The month (zero padded; i.e., May is 05). |
| dd | The day (zero padded). |
| hh:mm:ss.uuuuuu | Time in hours, minutes and fractional seconds. Timezone is UTC. |
| goid | The goroutine id (omitted if zero for use by tests). |
| chan | The channel number (omitted if zero for backward compatibility). |
| file | The file name where the entry originated. |
| line | The line number where the entry originated. |
| marker | Redactability marker + redactableIndicator + (see below for details). |
| tags | The logging tags, enclosed between [ and ]. May be absent. |
| counter | The entry counter. Only included if 'show-counter' is enabled. |
The redactability marker can be empty; in this case, its position in the common prefix is a double ASCII space character which can be used to reliably identify this situation.
If the marker + redactableIndicator + is present, the remainder of the log entry
contains delimiters (‹...›) around
fields that are considered sensitive. These markers are automatically recognized
by cockroach debug zip and cockroach debug merge-logs when log redaction is requested.
Additional options recognized via format-options:
| Option | Description |
|---|---|
show-counter | Whether to include the counter column in the line header. Without it, the format may be ambiguous due to the optionality of tags. |
colors | The color profile to use. Possible values: none, auto, ansi, 256color. Default is auto. |
timezone | The timezone to use for the timestamp column. The value can be any timezone name recognized by the Go standard library. Default is UTC |
crdb-v1-countThis format name is an alias for 'crdb-v1' with the following format option defaults:
show-counter: truecolors: nonecrdb-v1-ttyThis format name is an alias for 'crdb-v1' with the following format option defaults:
show-counter: falsecolors: autocrdb-v1-tty-countThis format name is an alias for 'crdb-v1' with the following format option defaults:
show-counter: truecolors: autocrdb-v2This is the main file format used from CockroachDB v21.1.
Each log entry is emitted using a common prefix, described below, followed by the text of the log entry.
Each line of output starts with the following prefix:
Lyymmdd hh:mm:ss.uuuuuu goid [chan@]file:line marker [tags...] counter cont
| Field | Description |
|---|---|
| L | A single character, representing the log level (e.g., I for INFO). |
| yy | The year (zero padded; i.e., 2016 is 16). |
| mm | The month (zero padded; i.e., May is 05). |
| dd | The day (zero padded). |
| hh:mm:ss.uuuuuu | Time in hours, minutes and fractional seconds. Timezone is UTC. |
| goid | The goroutine id (zero when cannot be determined). |
| chan | The channel number (omitted if zero for backward compatibility). |
| file | The file name where the entry originated. Also see below. |
| line | The line number where the entry originated. |
| marker | Redactability marker "⋮" (see below for details). |
| tags | The logging tags, enclosed between [ and ]. See below. |
| counter | The optional entry counter (see below for details). |
| cont | Continuation mark for structured and multi-line entries. See below. |
The chan@ prefix before the file name indicates the logging channel,
and is omitted if the channel is DEV.
The file name may be prefixed by the string (gostd) to indicate
that the log entry was produced inside the Go standard library, instead
of a CockroachDB component. Entry parsers must be configured to ignore this prefix
when present.
marker can be empty; in this case, its position in the common prefix is
a double ASCII space character which can be used to reliably identify this situation.
If the marker "⋮" is present, the remainder of the log entry
contains delimiters (‹...›)
around fields that are considered sensitive. These markers are automatically recognized
by cockroach debug zip and cockroach debug merge-logs
when log redaction is requested.
The logging tags are enclosed between square brackets [...],
and the syntax [-] is used when there are no logging tags
associated with the log entry.
counter is numeric, and is incremented for every
log entry emitted to this sink. (There is thus one counter sequence per
sink.) For entries that do not have a counter value
associated (e.g., header entries in file sinks), the counter position
in the common prefix is empty: tags is then
followed by two ASCII space characters, instead of one space; the counter,
and another space. The presence of the two ASCII spaces indicates
reliably that no counter was present.
cont is a format/continuation indicator:
| Continuation indicator | ASCII | Description |
|---|---|---|
| space | 0x32 | Start of an unstructured entry. |
| equal sign, "=" | 0x3d | Start of a structured entry. |
| exclamation mark, "!" | 0x21 | Start of an embedded stack trace. |
| plus sign, "+" | 0x2b | Continuation of a multi-line entry. The payload contains a newline character at this position. |
| vertical bar | 0x7c | Continuation of a large entry. |
Example single-line unstructured entry:
I210116 21:49:17.073282 14 server/node.go:464 ⋮ [-] 23 started with engine type ‹2›
Example multi-line unstructured entry:
I210116 21:49:17.083093 14 1@cli/start.go:690 ⋮ [-] 40 node startup completed:
I210116 21:49:17.083093 14 1@cli/start.go:690 ⋮ [-] 40 +CockroachDB node starting at 2021-01-16 21:49 (took 0.0s)
Example structured entry:
I210116 21:49:17.080713 14 1@util/log/event_log.go:32 ⋮ [-] 32 ={"Timestamp":1610833757080706620,"EventType":"node_restart"}
Example long entries broken up into multiple lines:
I210116 21:49:17.073282 14 server/node.go:464 ⋮ [-] 23 aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa....
I210116 21:49:17.073282 14 server/node.go:464 ⋮ [-] 23 |aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
I210116 21:49:17.080713 14 1@util/log/event_log.go:32 ⋮ [-] 32 ={"Timestamp":1610833757080706620,"EventTy...
I210116 21:49:17.080713 14 1@util/log/event_log.go:32 ⋮ [-] 32 |pe":"node_restart"}
Entries in this format can be read by most crdb-v1 log parsers,
in particular the one included in the DB console and
also the cockroach debug merge-logs
facility.
However, implementers of previous version parsers must
understand that the logging tags field is now always
included, and the lack of logging tags is included
by a tag string set to [-].
Likewise, the entry counter is now also always included,
and there is a special character after counter
to indicate whether the remainder of the line is a
structured entry, or a continuation of a previous entry.
Finally, in the previous format, structured entries
were prefixed with the string Structured entry: . In
the new format, they are prefixed by the = continuation
indicator.
Additional options recognized via format-options:
| Option | Description |
|---|---|
colors | The color profile to use. Possible values: none, auto, ansi, 256color. Default is auto. |
timezone | The timezone to use for the timestamp column. The value can be any timezone name recognized by the Go standard library. Default is UTC |
crdb-v2-ttyThis format name is an alias for 'crdb-v2' with the following format option defaults:
colors: autojsonThis format emits log entries as a JSON payload.
The JSON object is guaranteed to not contain unescaped newlines or other special characters, and the entry as a whole is followed by a newline character. This makes the format suitable for processing over a stream unambiguously.
Each entry contains at least the following fields:
Field name if tag-style: compact is specified | Field name if tag-style: verbose is specified | Description |
|---|---|---|
tag | tag | (Only if the option fluent-tag: true is given.) A Fluent tag for the event, formed by the process name and the logging channel. |
d | datetime | The pretty-printed date/time of the event timestamp, if enabled via options. |
f | file | The name of the source file where the event was emitted. |
g | goroutine | The identifier of the goroutine where the event was emitted. |
l | line | The line number where the event was emitted in the source. |
r | redactable | Whether the payload is redactable (see below for details). |
t | timestamp | The timestamp at which the event was emitted on the logging channel. |
v | version | The binary version with which the event was generated. |
After a couple of header entries written at the beginning of each log sink, all subsequent log entries also contain the following fields:
Field name if tag-style: compact is specified | Field name if tag-style: verbose is specified | Description |
|---|---|---|
C | channel | The name of the logging channel where the event was sent. |
sev | severity | The severity of the event. |
c | channel_numeric | The numeric identifier for the logging channel where the event was sent. |
n | entry_counter | The entry number on this logging sink, relative to the last process restart. |
s | severity_numeric | The numeric value of the severity of the event. |
Additionally, the following fields are conditionally present:
Field name if tag-style: compact is specified | Field name if tag-style: verbose is specified | Description |
|---|---|---|
N | node_id | The node ID where the event was generated, once known. Only reported for single-tenant or KV servers. |
x | cluster_id | The cluster ID where the event was generated, once known. Only reported for single-tenant of KV servers. |
q | instance_id | The SQL instance ID where the event was generated, once known. |
T | tenant_id | The SQL tenant ID where the event was generated, once known. |
V | tenant_name | The SQL virtual cluster where the event was generated, once known. |
tags | tags | The logging context tags for the entry, if there were context tags. |
message | message | For unstructured events, the flat text payload. |
event | event | The logging event, if structured (see below for details). |
stacks | stacks | Goroutine stacks, for fatal events. |
When an entry is structured, the event field maps to a dictionary
whose structure is one of the documented structured events. See the reference documentation
for structured events for a list of possible payloads.
When the entry is marked as redactable, the tags, message, and/or event payloads
contain delimiters (‹...›) around
fields that are considered sensitive. These markers are automatically recognized
by cockroach debug zip and cockroach debug merge-logs when log redaction is requested.
Additional options recognized via format-options:
| Option | Description |
|---|---|
datetime-format | The format to use for the datetime field. The value can be one of none, iso8601/rfc3339 (synonyms), or rfc1123. Default is none. |
datetime-timezone | The timezone to use for the datetime field. The value can be any timezone name recognized by the Go standard library. Default is UTC |
tag-style | The tags to include in the envelope. The value can be compact (one letter tags) or verbose (long-form tags). Default is verbose. |
fluent-tag | Whether to produce an additional field called tag for Fluent compatibility. Default is false. |
json-compactThis format name is an alias for 'json' with the following format option defaults:
fluent-tag: falsetag-style: compactjson-fluentThis format name is an alias for 'json' with the following format option defaults:
fluent-tag: truetag-style: verbosejson-fluent-compactThis format name is an alias for 'json' with the following format option defaults:
fluent-tag: truetag-style: compact