src/collectors/systemd-journal.plugin/README.md
systemd journal pluginKEY FEATURES | PREREQUISITES | JOURNAL SOURCES | JOURNAL FIELDS | VISUALIZATION | PLAY MODE | FULL TEXT SEARCH | QUERY PERFORMANCE | PERFORMANCE AT SCALE | BEST PRACTICES | CONFIGURATION | FAQ | HOW TO TROUBLESHOOT COMMON ISSUES | HOW TO VERIFY SETUP
The systemd journal plugin provides an efficient way to view, explore, and analyze systemd journal logs directly from the Netdata dashboard. It combines powerful filtering, real-time updates, and visual analysis tools to help you troubleshoot system issues effectively.
| Requirement | Details |
|---|---|
| Netdata Agent v1.44+ | This plugin requires Netdata version 1.44 or newer |
| Netdata Cloud account | Required to use Netdata Functions, including this plugin |
| Not supported in: | Static builds (use Debian-based containers instead) |
:::tip
This plugin is a Netdata Function Plugin. A free Netdata Cloud account is required. See the Netdata Functions discussion.
:::
The plugin is designed for native package installations, source installations, and Docker installations (Debian-based). If using Docker, make sure you're using the Debian-based containers.
The plugin automatically detects available journal sources based on the journal files in /var/log/journal (persistent logs) and /run/log/journal (volatile logs).
By default, all sources merge into a unified view of log messages.
:::tip
Select a specific source before analyzing logs in depth to improve query performance.
:::
Default journals on all systemd-based systems. Includes:
kmsg)systemd-journaldremote journals on centralization serversLogNamespace= in systemd unit filessystemd-journal-remotesystemd journals support dynamic fields per log entry. All fields and values are indexed for fast querying.
Fields are enriched for readability:
| Field | Enrichment applied |
|---|---|
_BOOT_ID | Timestamp of first message for boot |
PRIORITY | Human-readable priority name |
SYSLOG_FACILITY | Named facility |
ERRNO | Error name |
| UID and GID fields | Resolved to user and group names |
_CAP_EFFECTIVE | Human-readable capabilities |
_SOURCE_REALTIME_TIMESTAMP | UTC timestamp |
MESSAGE_ID | Well-known event name if known |
:::tip
Enrichments are visual only and not searchable. UID/GID values are based on the system where the plugin runs.
:::
Use the ⚙️ icon above the table to select fields to display as columns.
The table view provides a powerful way to analyze logs with:
Click a log entry to open the right-hand info panel showing all fields for that entry.
The sidebar shows:
The plugin offers select fields as filters with counters. Field allowlists and blocklists protect performance.
:::tip
"Full data queries" mode enables negative/empty matches but may slow performance.
:::
Key filter features:
Histograms visualize log frequency per field value over time. Supports:
The plugin offers several visualization features to help you understand and navigate your logs effectively.
The timeline at the top of the interface shows:
Field-specific histograms provide:
The plugin uses color to enhance readability:
The interface offers several ways to navigate logs:
The plugin supports PLAY mode for real-time log streaming. Click the ▶️ button at the top of the dashboard to activate it.
:::tip
PLAY mode offers a similar experience to journalctl -f, but with visual enhancements.
:::
The plugin supports full-text search using flexible pattern matching:
| Feature | Description |
|---|---|
| Contains match | Default pattern style (e.g., error matches error, error_count) |
| Wildcards | * matches any characters (e.g., a*b matches acb, a_long_b) |
| Multiple patterns | Separate with | for OR logic (e.g., error|warning matches lines containing either "error" OR "warning") |
| Negation | Prefix with ! to exclude (e.g., !systemd|* excludes lines with systemd) |
:::tip
Full-text search applies across all fields. Combine with filters for precise results.
:::
The plugin reads journal files directly using libsystemd, supporting concurrent readers and one writer.
Query performance depends on several factors:
| Factor | Impact on Performance |
|---|---|
| Number of journal files queried | Fewer files lead to faster queries |
| Disk speed | Faster disks improve query times |
| Available memory | More memory allows better caching |
| Filters applied | Using fewer filters speeds up the query |
For best performance:
The plugin handles large datasets efficiently using a sampling algorithm, ensuring responsive queries even on busy log servers.
| Step | Description |
|---|---|
| 1 | Fully evaluates the latest 500,000 log entries |
| 2 | Distributes evaluation across journal files for up to 1 million entries |
| 3 | Marks additional entries as [unsampled] beyond the evaluation budget |
| 4 | Estimates counts as [estimated] once unsampled limits are hit |
| 5 | Uses sequence numbers (if available) for precise estimation |
| 6 | Continues responsive histogram generation while managing performance |
The plugin uses a sophisticated algorithm that prioritizes newer logs while maintaining reasonable estimates for historical data. This approach ensures that even with terabytes of journal data, the interface remains responsive and usable.
At scale, this plugin achieves up to 25–30x faster query performance compared to journalctl, especially on multi-journal queries.
:::tip
The sampling algorithm is designed to be resilient to large datasets. Even if you see [unsampled] or [estimated] indicators, the results remain statistically representative of the full dataset.
:::
Netdata's sampling budget evaluates up to 1,000,000 log entries before it ever marks rows as [unsampled]. The proportion of the dataset we examine is:
evaluated_entries = min(total_entries, 1_000_000)
evaluated_ratio = evaluated_entries / total_entries
Because the sampling set is so large, percentage breakdowns stay tight even on massive datasets. For a 10 M–entry window where 60 % of logs share a value, the 95 % confidence interval around that percentage is:
standard_error ≈ sqrt(p * (1 - p) / evaluated_entries)
CI95 ≈ 1.96 * standard_error = 1.96 * sqrt(0.6 * 0.4 / 1_000_000) ≈ ±0.9 %
By contrast, evaluating only 5,000 entries (a small-sample approach typical of many log explorers when speed is prioritized) would yield:
CI95 ≈ 1.96 * sqrt(0.6 * 0.4 / 5_000) ≈ ±8.7 %
The result is that even at extreme scale, mainly because Netdata samples 200x more data, it can provide significantly more accurate estimations on value distributions, at comparable performance.
systemd-journal is designed for reliability first and performance second. It uses deduplication, field linking, and compression to minimize disk footprint, but the structure of journal files can still result in higher disk I/O during queries.
| Recommendation | Benefit |
|---|---|
Use compressed filesystems (ext4, btrfs, zfs) | Reduces disk I/O by minimizing file size |
| Use SSD or NVMe storage | Speeds up journal file reads |
| Avoid small fragmented journal files | Prevents query slowdowns on busy centralization servers |
| Recommendation | Benefit |
|---|---|
| Allocate more RAM for the system | Improves OS caching of journal files |
| Query the same timeframe repeatedly | Benefits from cached journal data |
| Limit query timeframes on large datasets | Reduces memory overhead and improves speed |
:::tip
Journal data is cached by the operating system. The more RAM available for caching, the faster your queries will be.
:::
| Strategy | Why it helps |
|---|---|
| Narrow the timeframe of your queries | Minimizes data scanned per request |
| Use specific filters and source selections | Limits the scope of journal files being queried |
| Limit the number of rows returned in the UI | Keeps response times fast and manageable |
| Enable PLAY mode only when necessary | Reduces continuous query load on the system |
The Netdata systemd journal plugin is designed to work out of the box with minimal configuration.
| Requirement | Description |
|---|---|
| Netdata Agent | Installed on the node or centralization server |
| Journal files | Located in /var/log/journal (persistent) or /run/log/journal (volatile) |
| Netdata Cloud account | Required to access Netdata Functions, including this plugin |
:::tip
No additional configuration is required for this plugin to operate on supported systems.
:::
| Task | Purpose |
|---|---|
| Keep Netdata up to date | Ensures plugin compatibility and performance optimizations |
| Monitor disk usage of journal files | Prevents performance issues caused by excessive log volume |
| Verify journal file locations | Confirms the plugin can access the intended sources |
| Review source selections periodically | Adjusts scope as infrastructure changes |
Yes — you can centralize your logs using systemd-journal-remote and install Netdata on the centralization server to explore logs from your entire infrastructure.
The plugin provides multi-node views and allows you to combine logs from multiple servers.
:::tip
For details on configuring a journal centralization server, see the journal centralization setup guide.
:::
</details> <details> <summary><strong>Can I use this plugin from a parent Netdata node?</strong></summary>Yes — if your nodes are connected to a Netdata parent, all their functions are accessible via the parent's UI.
This includes access to the systemd journal plugin for each child node.
No — when accessing the Agent directly, no data is exposed to Netdata Cloud.
The Cloud account is only used for authentication. Data flows directly from your Netdata Agent to your web browser.
:::tip
When using https://app.netdata.cloud, communication is encrypted but data is not stored in Netdata Cloud.
See this discussion for more details.
:::
</details> <details> <summary><strong>What are `volatile` and `persistent` journals?</strong></summary>/var/log/journal/run/log/journal and cleared on reboot:::tip
For more, check man systemd-journald.
:::
</details> <details> <summary><strong>I centralize my logs with Loki. Why use Netdata for journals?</strong></summary>systemd journals support dynamic, high-cardinality labels with all fields indexed by default.
When sending logs to Loki, you must predefine which fields to include, reducing flexibility.
Netdata reads journals directly, providing:
| Feature | Netdata journal plugin | Loki |
|---|---|---|
| Indexed on all fields | ✔️ Yes | ❌ Only selected labels |
| Supports dynamic fields | ✔️ Yes | ❌ Assumes fixed schema |
| Requires configuration | ❌ No | ✔️ Yes (relabel rules, label selection) |
:::tip
Loki and systemd journals serve different use cases — they can complement, not replace, each other.
:::
</details> <details> <summary><strong>Is it worth setting up a `systemd` logs centralization server?</strong></summary>Yes — the tools required are included in modern Linux systems, and setup is straightforward.
Centralized logs provide high visibility with minimal overhead.
Two main strategies:
| Strategy | Description |
|---|---|
| Active sources | Central server fetches logs from each node |
| Passive sources | Nodes push logs to the central server |
:::tip
See the passive journal centralization without encryption guide
or the encrypted setup guide.
:::
</details> <details> <summary><strong>What are the limitations when using centralization?</strong></summary>| Limitation | Notes |
|---|---|
| Namespaces not supported by Docker | Related issue |
systemd-journal-upload does not handle namespaces automatically | Requires manual configuration per namespace |
If you encounter issues or have ideas for improvements:
The plugin is actively maintained, and feedback helps improve it for everyone.
</details> <details> <summary><strong>Can I customize the plugin's appearance or behavior?</strong></summary>Currently, customization options are limited to:
Additional customization features may be added in future releases based on user feedback.
</details>Use the following solutions to resolve common issues with the systemd journal plugin.
| Possible Cause | Solution |
|---|---|
| Running an older Netdata version (pre-1.44) | Update Netdata to version 1.44 or later |
| Using Alpine-based Docker container | Use the Debian-based Netdata container (from 1.44+) |
Using a static build of Netdata (/opt/netdata) | Switch to a package-based installation or source build |
Missing libsystemd or required dependencies | Make sure libsystemd is installed on the host |
| Possible Cause | Solution |
|---|---|
| Querying too many journal files at once | Select specific sources before running your query |
| Long timeframes selected | Narrow the timeframe to improve performance |
| Low disk speed or insufficient RAM | Use faster disks and increase memory for better caching |
| Too many active filters | Reduce the number of filters applied |
:::tip
Sampling ensures responsiveness at scale, but selecting sources and filters remains the best way to optimize performance.
:::
| Possible Cause | Solution |
|---|---|
| Journals stored outside default paths | Create a symlink to /var/log/journal or /run/log/journal |
| Journals not persistent across reboots | Configure systemd-journald to enable persistent logs with Storage=persistent in /etc/systemd/journald.conf |
| Possible Cause | Solution |
|---|---|
| Journals rotated or deleted | Ensure persistent storage is enabled |
| Misconfigured journal centralization | Check systemd-journal-remote and systemd-journal-upload settings |
| Namespace logs not forwarded | Manually configure forwarding for each namespace |
| Possible Cause | Solution |
|---|---|
| Outdated browser | Update to the latest version of Chrome, Firefox, Safari, or Edge |
| Zoom level affecting layout | Reset browser zoom to 100% |
| Ad blockers or script blockers | Temporarily disable to test if they're interfering |
| Network connectivity issues | Check network connections to Netdata server |
| Error Message | Meaning | Solution |
|---|---|---|
| "Plugin not available" | The systemd-journal plugin isn't loaded | Check your Netdata installation type (must not be Alpine or static) |
| "Unable to open journal" | Permission issues accessing journal files | Ensure Netdata has proper permissions for journal directories |
| "Timeout while querying" | Query is taking too long to complete | Reduce the query scope with filters or shorter timeframes |
| "No sources detected" | Cannot find valid journal files | Check journal file locations and setup |
| "Source selection failed" | Selected source cannot be accessed | Verify the source exists and permissions are correct |
sudo netdata -W plugins
Check that the systemd-journal plugin is listed as active.
system, user, remote, or specific namespaces):::tip
If sources are missing, check the journal file locations and verify symlinks if needed.
:::
PRIORITY=3) and confirm logs are returnederror) and verify results populate correctlyCheck the Netdata Agent logs for plugin startup messages:
sudo journalctl -u netdata | grep journal
Look for lines confirming the journal plugin started successfully and detected sources.