Overview

journald (the systemd Journal daemon) is a system service that collects and stores logging data. It was introduced with systemd and is the default logging utility on CentOS 7.x and other modern Linux distributions. journald creates structured, indexed, binary log entries, centrally managing logs for the operating system, kernel, and installed services.

These logs are then viewed and manipulated using the journalctl command-line utility. Understanding how to use journalctl is crucial for system administration, troubleshooting, and monitoring your server’s health and applications.

This guide will walk you through common journalctl commands and troubleshooting steps related to journald on CentOS 7.

Prerequisites

A server running CentOS 7.x.
Root or sudo access to the server.
Basic familiarity with the Linux command line.

Understanding journal storage (volatile vs. persistent)

By default, journald on CentOS 7 might store logs in a volatile way, meaning they are kept in RAM (specifically in /run/log/journal/) and are lost upon reboot. For long-term log retention and analysis, persistent storage is recommended.

journald uses a setting called Storage= in its configuration file (/etc/systemd/journald.conf), which defaults to auto. With auto:

If the directory /var/log/journal exists, logs will be stored persistently there.
If /var/log/journal does not exist, logs will be stored in /run/log/journal (volatile).

To enable persistent journal logging:

Create the journal directory if it doesn’t exist: sudo mkdir -p /var/log/journal
Set the correct permissions and ownership (systemd-journal usually handles this, but it’s good to verify or enforce): sudo systemd-tmpfiles --create --prefix /var/log/journal Alternatively, you can manually set ownership and permissions: sudo chown root:systemd-journal /var/log/journal sudo chmod 2755 /var/log/journal
Restart the systemd-journald service for the change to take effect: sudo systemctl restart systemd-journald Or, you can send a SIGUSR1 signal to systemd-journald to flush logs from /run/log/journal to /var/log/journal if the directory was just created: sudo systemctl kill --signal=SIGUSR1 systemd-journald

After enabling persistent storage, new log entries will be written to /var/log/journal/.

Basic log viewing with journalctl

Viewing all logs since last boot

To view all log messages recorded since the most recent system boot, use the -b flag. This is often one of the first commands you’ll use when troubleshooting boot-related issues.

sudo journalctl -b

The output will be paged (usually through less) and will look similar to this (timestamps and hostnames will vary):

Aug 24 09:30:02 webserver.domain.com systemd[1]: Reloaded Berkeley Internet Name Domain (DNS). Aug 24 09:30:02 webserver.domain.com NetworkManager[595]: <info> parsing /etc/sysconfig/network-scripts/ifcfg-lo ... Aug 24 09:30:02 webserver.domain.com network[2842]: Could not load file '/etc/sysconfig/network-scripts/ifcfg-lo' Aug 24 09:30:02 webserver.domain.com NetworkManager[595]: <info> (lo): link disconnected Aug 24 09:30:02 webserver.domain.com NetworkManager[595]: <info> parsing /etc/sysconfig/network-scripts/ifcfg-lo ... Aug 24 09:30:02 webserver.domain.com network[2842]: Could not load file '/etc/sysconfig/network-scripts/ifcfg-lo' Aug 24 09:30:02 webserver.domain.com network[2842]: [ OK ] Aug 24 09:30:02 webserver.domain.com systemd[1]: Starting LSB: Bring up/down networking... Aug 24 09:30:02 webserver.domain.com systemd-sysctl[3046]: Overwriting earlier assignment of kernel/sysrq in file '/etc/sysctl.d/99-sys Aug 24 09:30:02 webserver.domain.com systemd-sysctl[3046]: Overwriting earlier assignment of kernel/shmmax in file '/etc/sysctl.d/99-sy Aug 24 09:30:02 webserver.domain.com systemd-sysctl[3046]: Overwriting earlier assignment of kernel/shmall in file '/etc/sysctl.d/99-sy Aug 24 09:30:02 webserver.domain.com NetworkManager[595]: <info> parsing /etc/sysconfig/network-scripts/ifcfg-lo ... Aug 24 09:30:02 webserver.domain.com network[3023]: Bringing up loopback interface: Could not load file '/etc/sysconfig/network-script Aug 24 09:30:02 webserver.domain.com NetworkManager[595]: <info> parsing /etc/sysconfig/network-scripts/ifcfg-lo ... Aug 24 09:30:02 webserver.domain.com network[3023]: Could not load file '/etc/sysconfig/network-scripts/ifcfg-lo' Aug 24 09:30:02 webserver.domain.com NetworkManager[595]: <info> (lo): link connected Aug 24 09:30:02 webserver.domain.com NetworkManager[595]: <info> parsing /etc/sysconfig/network-scripts/ifcfg-lo ... Aug 24 09:30:02 webserver.domain.com network[3023]: Could not load file '/etc/sysconfig/network-scripts/ifcfg-lo' Aug 24 09:30:02 webserver.domain.com NetworkManager[595]: <info> parsing /etc/sysconfig/network-scripts/ifcfg-lo ... Aug 24 09:30:02 webserver.domain.com network[3023]: Could not load file '/etc/sysconfig/network-scripts/ifcfg-lo' Aug 24 09:30:02 webserver.domain.com network[3023]: [ OK ]

To view logs from previous boots, you can specify an offset. For example, journalctl -b -1 shows logs from the previous boot, journalctl -b -2 from the boot before that, and so on. To list available boots:

sudo journalctl --list-boots

Viewing kernel messages (like dmesg)

To display only kernel messages, similar to the output of the dmesg command, use the -k or --dmesg flag:

sudo journalctl -k

Output example:

journalctl -k
Sep 01 17:48:26 webserver.domain.com kernel: Firewall: *UDP_IN Blocked* IN=ens18 OUT= MAC=ff:ff:ff:ff:ff:ff:bc:ae:c5:29:25:87:08:00 SRC
Sep 01 17:48:51 webserver.domain.com kernel: Firewall: *TCP_IN Blocked* IN=ens18 OUT= MAC=d6:82:aa:58:cc:3d:00:0e:39:2f:b4:00:08:00 SRC
Sep 01 17:48:51 webserver.domain.com kernel: Firewall: *TCP_IN Blocked* IN=ens18 OUT= MAC=d6:82:aa:58:cc:3d:00:0e:39:2f:b4:00:08:00 SRC
Sep 01 17:48:56 webserver.domain.com kernel: Firewall: *UDP_IN Blocked* IN=ens18 OUT= MAC=ff:ff:ff:ff:ff:ff:bc:ae:c5:29:25:87:08:00 SRC
Sep 01 17:48:56 webserver.domain.com kernel: Firewall: *UDP_IN Blocked* IN=ens18 OUT= MAC=ff:ff:ff:ff:ff:ff:bc:ae:c5:29:25:87:08:00 SRC

Filtering logs

journalctl offers powerful filtering capabilities to help you find specific log entries.

By service unit

To view logs for a specific systemd service (e.g., httpd for Apache, sshd for SSH), use the -u or --unit flag. This shows system-level logs for the service itself (start, stop, errors), not application-specific logs like Apache access/error logs or domain logs.

To see logs for Apache since the last boot:

sudo journalctl -b -u httpd.service

Example output:

-- Logs begin at Sat 2015-08-22 21:30:03 EDT, end at Tue 2015-09-01 17:15:01 EDT. -- Aug 24 21:47:13 webserver.domain.com systemd[1]: Stopping Apache webserver managed by cPanel EasyApache... Aug 24 21:47:24 webserver.domain.com systemd[1]: Starting Apache webserver managed by cPanel EasyApache... Aug 24 21:47:27 webserver.domain.com restartsrv_httpd[16755]: AH00548: NameVirtualHost has no effect and will be removed in the next re Aug 24 21:47:28 webserver.domain.com systemd[1]: PID file /usr/local/apache/logs/httpd.pid not readable (yet?) after start. Aug 24 21:47:29 webserver.domain.com systemd[1]: Started Apache webserver managed by cPanel EasyApache. Aug 24 22:16:58 webserver.domain.com systemd[1]: Stopping Apache webserver managed by cPanel EasyApache... Aug 24 22:16:58 webserver.domain.com systemd[1]: Starting Apache webserver managed by cPanel EasyApache... Aug 24 22:16:58 webserver.domain.com restartsrv_httpd[19481]: AH00548: NameVirtualHost has no effect and will be removed in the next re Aug 24 22:16:59 webserver.domain.com systemd[1]: PID file /usr/local/apache/logs/httpd.pid not readable (yet?) after start. Aug 24 22:17:00 webserver.domain.com systemd[1]: Started Apache webserver managed by cPanel EasyApache. Aug 24 22:27:14 webserver.domain.com systemd[1]: Starting Apache webserver managed by cPanel EasyApache... Aug 24 22:27:15 webserver.domain.com systemd[1]: PID file /usr/local/apache/logs/httpd.pid not readable (yet?) after start. Aug 24 22:27:16 webserver.domain.com systemd[1]: Started Apache webserver managed by cPanel EasyApache.

Note: The service unit name might vary (e.g., apache2.service on Debian/Ubuntu systems). You can list all units systemd knows about with systemctl list-units --all.

By multiple service units

You can view logs for multiple services simultaneously by providing multiple -u flags:

sudo journalctl -b -u httpd.service -u sshd.service

This will interleave log entries from both httpd.service and sshd.service based on their timestamps.

By time frame

journalctl allows you to filter logs based on time using the --since and --until flags. You can use specific timestamps in the format YYYY-MM-DD HH:MM:SS or relative terms like “yesterday”, “today”, “now”, “1 hour ago”, “30 minutes ago”.

Example: View logs from 9:00 AM yesterday until 1 hour ago:

sudo journalctl --since "yesterday 09:00:00" --until "1 hour ago"

Example: View logs for a specific date:

sudo journalctl --since "2025-05-28" --until "2025-05-29"

If the time is omitted for --since, it defaults to 00:00:00. If omitted for --until, it defaults to now.

By Process ID (PID), User ID (UID), or Group ID (GID)

You can filter logs generated by a specific process, user, or group using field matches:

Process ID (PID): Bashsudo journalctl _PID=12345 (Replace 12345 with the actual PID)
User ID (UID): Bashsudo journalctl _UID=1000 (Replace 1000 with the actual UID; 0 is typically root)
Group ID (GID): Bashsudo journalctl _GID=100 (Replace 100 with the actual GID)

By message priority

journald logs messages with different priority levels, identical to standard syslog levels. You can filter messages to show only those of a certain priority or higher using the -p or --priority flag.

The available priority levels are (from highest to lowest severity):

0: emerg (emergency)
1: alert (alert)
2: crit (critical)
3: err (error)
4: warning (warning)
5: notice (notice)
6: info (informational)
7: debug (debug)

Example: View all error messages and higher:

sudo journalctl -p err -b

This is equivalent to sudo journalctl -p 3 -b.

Listing available filter fields

To see what fields journald has recorded and can be used for filtering (like _UID, _PID, _SYSTEMD_UNIT, etc.), use the -F flag followed by the field name.

To show which UIDs have entries in the journal:

sudo journalctl -F _UID

This outputs a simple list of UIDs, one per line.

To discover all available fields for filtering, consult the manual:

man systemd.journal-fields

Modifying log output

By default, journalctl pipes its output through a pager like less. To send output directly to standard output (stdout), which is useful for piping to other commands (like grep) or redirecting to a file, use the --no-pager flag:

sudo journalctl --no-pager | grep "Failed password"

Displaying full, untruncated messages

Sometimes, log lines can be very long and journalctl might truncate them. To see the full, untrimmed output, use the -a or --all flag. This is often combined with --no-pager:

sudo journalctl -p err -a --no-pager

Showing the last ‘n’ log entries

Similar to tail -n, you can view the last specified number of log entries using the -n or --lines flag. If no number is specified, it defaults to 10.

Example: Show the last 20 kernel messages:

sudo journalctl -n 20 -k

Viewing live logs (tailing)

To watch logs as they are being written (live tailing), similar to tail -f, use the -f or --follow flag:

sudo journalctl -f

To follow logs for a specific service:

sudo journalctl -f -u nginx.service

Press Ctrl+C to stop following the logs.

Changing output format

journalctl can display logs in various formats using the -o or --output flag. Some useful formats include:

short: The default, human-readable format.
short-iso: Includes ISO 8601 wallclock timestamps.
verbose: Shows all fields stored in the journal for each entry.
json: Outputs entries as JSON objects, useful for scripting.
json-pretty: Outputs entries as formatted JSON objects.

Example: Output logs in JSON format:

sudo journalctl -u sshd -o json-pretty --no-pager

Managing journal disk usage

If you have persistent logging enabled, the journal can grow over time. journalctl provides tools to check and manage its disk footprint.

Checking disk usage

To see how much disk space the journal files are currently occupying:

sudo journalctl --disk-usage

Output will be similar to:

<pre><code> Journals take up 368.0M on disk. </code></pre>

Reducing journal size (vacuuming)

Caution: The following commands will permanently delete log data. Use with care.

You can reduce the journal’s disk usage based on size or time.

Reduce by Size (--vacuum-size): This will remove older journal files until the total disk space used by the journal is less than the specified amount. Example: Reduce the journal to a maximum size of 200 Megabytes: sudo journalctl --vacuum-size=200M
Reduce by Time (--vacuum-time): This will remove journal files containing entries older than the specified time. Example: Keep only logs from the last year: sudo journalctl --vacuum-time=1years You can use units like s (seconds), m (minutes), h (hours), days, weeks, months, years.

You can also configure automatic limits for journal size in /etc/systemd/journald.conf (e.g., SystemMaxUse=, SystemKeepFree=).

Troubleshooting common journald and rsyslog issues on CentOS 7

On CentOS 7, rsyslog (a traditional syslog daemon) is often configured to read messages from the journal via the imjournal module. Issues can arise where traditional log files like /var/log/messages or /var/log/secure are blank or missing recent entries.

Blank log files (/var/log/messages, /var/log/secure, etc.)

If you find that standard log files in /var/log/ are not being populated:

a. Check rsyslog Configuration for Journal Input:

By default, CentOS 7’s rsyslog is configured to get its logs from journald. Open the main rsyslog configuration file:

sudo vi /etc/rsyslog.conf

Look for a line like:

$OmitLocalLogging on

This setting means rsyslog does not listen on the traditional local syslog socket because it expects to get local messages via imjournal. This is usually the correct default behavior if imjournal is working.

If you suspect imjournal is not the source and want rsyslog to listen on its traditional socket (less common for journald integration issues), you could change this to off, but first investigate imjournal.

Ensure the imjournal module is loaded. You should see a line like:

$ModLoad imjournal

b. Check listen.conf for rsyslog: Open the listen.conf file for rsyslog:

sudo vi /etc/rsyslog.d/listen.conf

Ensure the following line is present and not commented out, as this is the socket imjournal uses:

$SystemLogSocketName /run/systemd/journal/syslog

Restart rsyslog if you make changes:

sudo systemctl restart rsyslog.service

Corrupted imjournal.state file

rsyslog uses a state file (/var/lib/rsyslog/imjournal.state) to keep track of its position within the journal. If this file becomes corrupted, rsyslog might stop pulling new messages from journald.

You might see an error in /var/log/messages (if it’s partially working) or in journalctl itself, similar to:

Dec 26 20:30:01 your_server_hostname rsyslogd-2027: imjournal: fscanf on state file '/var/lib/rsyslog/imjournal.state' failed

To check for this error, you can also try restarting rsyslog and then immediately checking its logs:

sudo systemctl restart rsyslog.service
sudo journalctl -u rsyslog.service -n 20

Solution: If you see this error, you can resolve it by moving the corrupted state file and restarting rsyslog. journald and rsyslog will recreate this file.

Move the old state file: sudo mv /var/lib/rsyslog/imjournal.state /var/lib/rsyslog/imjournal.state.lwbak
Restart rsyslog: sudo systemctl restart rsyslog.service

After restarting, /var/log/messages and other traditional log files should start populating, possibly rapidly with backlogged information. You might see messages about rate-limiting, which is normal as rsyslog catches up. If the issue reoccurs (the file becomes corrupted again quickly), repeat the steps. Test by performing an action that should be logged (e.g., SSH into the server) and check if it appears in /var/log/secure.

imjournal.state not being created or logs still not populating

If rsyslog is running, imjournal is configured, but logs are still not appearing in /var/log/ and /var/lib/rsyslog/imjournal.state is not being created or updated, there might be an issue with the communication socket that journald provides for rsyslog.

Try restarting the systemd-journald.socket and then systemd-journald itself, followed by rsyslog:

sudo systemctl restart systemd-journald.socket
sudo systemctl restart systemd-journald.service
sudo systemctl restart rsyslog.service

This can sometimes resolve issues where the socket was not properly established or had become stale.

Advanced configuration (/etc/systemd/journald.conf)

For more advanced control over journald‘s behavior, you can edit its configuration file: /etc/systemd/journald.conf. Some common parameters you might adjust include:

Storage=: (e.g., persistent, volatile, auto) – Already discussed.
SystemMaxUse=: Maximum disk space the journal can use (e.g., 1G).
SystemKeepFree=: Amount of disk space journald should leave free (e.g., 500M).
SystemMaxFileSize=: Maximum size for individual journal files before rotation (e.g., 100M).
RateLimitIntervalSec=, RateLimitBurst=: Controls rate limiting of log messages to prevent log flooding.
ForwardToSyslog=: (yes or no) Whether to forward messages to a traditional syslog daemon. Default is no if /run/systemd/journal/syslog is used by rsyslog.
MaxRetentionSec=: How long to keep journal entries.

After making changes to /etc/systemd/journald.conf, restart the systemd-journald service:

sudo systemctl restart systemd-journald.service

Conclusion

journald and journalctl provide a robust and centralized logging system for CentOS 7. By mastering the journalctl command and understanding how journald integrates with services like rsyslog, you can efficiently monitor your server, diagnose issues, and manage log data effectively. Remember to consult the man pages (man journalctl, man journald.conf, man systemd.journal-fields) for the most detailed information.

Was this article helpful?

Thank you for your input.

Thank you for your feedback.