content/en/docs/troubleshooting/missing-fields.md
Many of the Supported Output Fields are derived from multiple events and mechanisms. To provide a more concrete explanation, for each spawned process, Falco extracts and derives fields from the clone*/*fork/execve* syscalls. Falco generates a struct in userspace, stores the relevant information within this struct, and then adds it to the process cache table in memory. If a process makes additional system calls during its lifetime, such as opening a file, in a Falco rule, you typically also export process fields — assuming we haven't missed the spawned process event and the information is available. These details extend to various use cases, and, in essence, dropped events can lead to missing fields as well as race conditions.
As a result, Falco logs can never be perfect, and null values can occur. We are constantly aiming to improve the robustness in this regard. We encourage you to contribute to the project if you encounter such cases or have improvement ideas. Also be aware that, unfortunately, missing fields can have different natures. Sometimes the field may be an empty string, or the string <NA>, or, if numeric, the default numeric value. These inconsistencies may be more difficult to address, as many Falco rules rely on legacy declarations.
Furthermore, sometimes Linux may not operate exactly as expected. One concrete example is that shell built-ins like echo do not cause a new spawned process, and the echo command does not get logged with Falco. Similarly, if a base64 encoded string gets interpreted during decoding, you do not have the original base64 blob in the command args unless the command was passed with the sh -c flag. Lastly, some fields only work for certain kernel versions or system configs (e.g. proc.is_exe_upper_layer requires a container overlayfs).
Check the basics:
HOST_ROOT prefix: /host/run/k3s/containerd/containerd.sock. See deploy-kubernetes example template.-o container_engines.cri.sockets[]=<socket_path> command line option when running Falco. The default paths include: /run/containerd/containerd.sock, /run/k3s/containerd/containerd.sock, /run/crio/crio.sock.-o container_engines.cri.disable_async=true command line option when running Falco.container.id is set to host, it indicates that the process is running on the host, and therefore, no container image is associated with it.{{% pageinfo color=info %}}
The k8s.* fields are extracted from the container runtime socket simultaneously as we look up the container.* fields from the CRI API calls responses.
{{% /pageinfo %}}
When using containerd as your container runtime, you should configure Falco to use the CRI engine to consume the containerd socket (/run/containerd/containerd.sock). This is important because:
If you are missing container.name or other container metadata fields while using containerd, ensure you are using the CRI engine configuration (not the containerd engine) in your Falco setup. For example, configure the container plugin with:
engines:
cri:
enabled: true
sockets:
- /run/containerd/containerd.sock
containerd:
enabled: false
Carefully read the field description documentation:
container.* retrieved from the container runtime socketk8s.* also retrieved from the container runtime socketThe container info enrichment, while robust, depends on the speed of making API requests against the container runtime socket.
Falco's metrics config (see also Falco Metrics) provides a range of useful metrics related to software functioning, now also featuring metrics around Falco's internal state:
state_counters_enabled: trueHere is an example metrics log snippet highlighting the fields crucial for this analysis.
{
"output_fields": {
"evt.source": "syscall",
"falco.n_containers": 50,
"falco.n_missing_container_images": 0,
},
"rule": "Falco internal: metrics snapshot"
}
falco.n_containers indicates how many containers are running at a given time, typically less than 100-300 at maximum. falco.n_missing_container_images is an updated snapshot of how many containers are internally stored in Falco without a container image at any given time.
To complicate matters, some processes in Kubernetes run in the pod sandbox container, which has no container image in the API responses. In such cases, the container.id is the same as the k8s.pod.sandbox_id. If the container image is consistently missing throughout the lifetime of the container, it's likely a process in a pod sandbox container in the majority of the cases. However, sandbox containers likely constitute less than 1% of the distinct containers in your overall Falco logs. Note that this comparison will be fully supported by Falco 0.38 and is a work in progress.
Additionally, the improvement of the overall efficiency of the container engine, especially for the -o container_engines.cri.disable_async=true option, is also a work in progress. A more performant implementation is expected to be available by Falco 0.38. This improvement aims to address missing images observed by adopters and resolve most cases, leaving only some edge cases of race conditions where the lookup hasn't happened yet.
/etc:/host/etc) when running Falco as a daemonset in Kubernetes, for example.kubectl exec), we must disappoint you. The Linux kernel lacks knowledge of the control plane. However, we are actively exploring ways to support this. Refer to this issue for more details.Let's consider another example: the fields related to the process tree lineage (e.g. proc.aname*).
spawned_process events is not equivalent to the current process tree on the system. Check out the Falco rules macro container_entrypoint for one such example and explore this resource.Falco's metrics config (see also Falco Metrics) provides a range of useful metrics related to software functioning, now also featuring metrics around Falco's internal state:
state_counters_enabled: trueHere is an example metrics log snippet highlighting the fields crucial for this analysis.
{
"output_fields": {
"evt.source": "syscall",
"falco.n_drops_full_threadtable": 0,
"falco.n_store_evts_drops": 0,
"falco.n_failed_fd_lookups": 0,
"falco.n_failed_thread_lookups": 0,
"falco.n_retrieve_evts_drops": 0
},
"rule": "Falco internal: metrics snapshot"
}
falco.n_drops_full_threadtable and falco.n_store_evts_drops reflect similar occurrences. They are monotonic counters indicating how often a spawned process event was dropped due to a full table (configurable by Falco 0.38 with a higher default value) and how frequently store actions to update the process structs in memory failed and were subsequently dropped. On the flip side, there are also counters keeping track of failed lookup or retrieve actions. Internally, Falco is granular and talks about threads, not processes.