Maintaining a healthy and productive Linux environment requires effective system log management. The journalctl command has evolved into an essential resource for accessing and analyzing system logs with the introduction of systemd, the system and service manager for recent Linux distributions.
The journalctl command's adaptability and utility on Linux-based systems will be examined thoroughly in this article through a variety of instances.
Introduction to journalctl
'journalctl' is a command-line tool that provides access to the systemd journal, an integrated logging system built into Linux distributions that use the systemd logging mechanism.
It gathers logs into a single, binary format rather than traditional log files, which frequently split across several files and directories. Consequently, log management becomes easier and efficiency increases.
This combination makes better system monitoring possible, which also allows you to get logs from a variety of sources, such as kernel messages, services, applications, and more.
Basic Usage
Use the journalctl command to see the logs that the journald daemon has collected.
When used alone, every journal entry that is in the system will be displayed within a pager (usually less) for you to browse. The oldest entries will be at the top.
journalctl
Sample output
Jun 14 12:22:29 groot kernel: Linux version 5.15.0-73-generic (buildd@bos03-amd64-060) (gcc Jun 14 12:22:29 groot kernel: Command line: BOOT_IMAGE=/vmlinuz-5.15.0-73-generic root=/dev/mapper/> Jun 14 12:22:29 groot kernel: KERNEL supported cpus: Jun 14 12:22:29 groot kernel: Intel GenuineIntel Jun 14 12:22:29 groot kernel: AMD AuthenticAMD Jun 14 12:22:29 groot kernel: Hygon HygonGenuine Jun 14 12:22:29 groot kernel: Centaur CentaurHauls Jun 14 12:22:29 groot kernel: zhaoxin Shanghai Jun 14 12:22:29 groot kernel: [Firmware Bug]: TSC doesn't count with P0 frequency! Jun 14 12:22:29 groot kernel: x86/fpu: x87 FPU will use FXSAVE Jun 14 12:22:29 groot kernel: signal: max sigframe size: 1440 = Jun 14 12:22:29 groot kernel: BIOS-provided physical RAM map: Jun 14 12:22:29 groot kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable Jun 14 12:22:29 groot kernel: BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved Jun 14 12:22:29 groot kernel: BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved Jun 14 12:22:29 groot kernel: BIOS-e820: [mem 0x0000000000100000-0x00000000dffeffff] usable Jun 14 12:22:29 groot kernel: BIOS-e820: [mem 0x00000000dfff0000-0x00000000dfffffff] ACPI data Jun 14 12:22:29 groot kernel: BIOS-e820: [mem 0x00000000fec00000-0x00000000fec00fff] reserved Jun 14 12:22:29 groot kernel: BIOS-e820: [mem 0x00000000fee00000-0x00000000fee00fff] reserved Jun 14 12:22:29 groot kernel: BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved Jun 14 12:22:29 groot kernel: BIOS-e820: [mem 0x0000000100000000-0x00000002145fffff] usable Jun 14 12:22:29 groot kernel: NX (Execute Disable) protection: active Jun 14 12:22:29 groot kernel: SMBIOS 2.5 present. Jun 14 12:22:29 groot kernel: DMI: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006 Jun 14 12:22:29 groot kernel: Hypervisor detected: KVM Jun 14 12:22:29 groot kernel: kvm-clock: Using msrs 4b564d01 and 4b564d00 Jun 14 12:22:29 groot kernel: kvm-clock: cpu 0, msr 1d7401001, primary cpu clock Jun 14 12:22:29 groot kernel: kvm-clock: using sched offset of 924714554074 cycles Jun 14 12:22:29 groot kernel: clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e> Jun 14 12:22:29 groot kernel: tsc: Detected 3293.726 MHz processor Jun 14 12:22:29 groot kernel: e820: update [mem 0x00000000-0x00000fff] usable ==> reserved Jun 14 12:22:29 groot kernel: e820: remove [mem 0x000a0000-0x000fffff] usable Jun 14 12:22:29 groot kernel: last_pfn = 0x214600 max_arch_pfn = 0x400000000 Jun 14 12:22:29 groot kernel: Disabled Jun 14 12:22:29 groot kernel: x86/PAT: MTRRs disabled, skipping PAT initialization too.
You will likely have pages and pages of data to scroll through, which can be tens or hundreds of thousands of lines long if systemd has been on your system for a long while. This demonstrates how much data is available in the journal database.
The format will be familiar to those used for standard syslog logging. However, this actually collects data from more sources than traditional syslog implementations are capable of. It includes logs from the early boot process, the kernel, the initrd, and application standard error and out. These are all available in the journal.
You may notice that all of the timestamps being displayed are local time. This is available for every log entry now that we have our local time set correctly on our system. All of the logs are displayed using this new information.
If you want the timestamps to display in UTC, you can use the '--utc' flag:
journalctl –utc
Printing in reverse order
To print the log entries in reverse order, or the most recent entry first, use the -r option. Here is an example that prints the sshd service logs in reverse order and prints only 10 entries.
journalctl -u sshd.service -r -n 10
Live Console Output
If you want to view the journal entries in realtime in console as they are being created, use the follow switch "-f".
journalctl -f
The output would then print the log entries in realtime.
$ journalctl -f Aug 24 11:02:48 enlightened NetworkManager[1250]: <warn> [1692855168.5237] platform-linux: do-add-ip6-address[2: fe80::16ed:72a8:1882:57a3]: failure 95 (Operation not supported) Aug 24 11:02:50 enlightened NetworkManager[1250]: <warn> [1692855170.5245] platform-linux: do-add-ip6-address[2: fe80::43d8:e74c:c1c3:467]: failure 95 (Operation not supported) Aug 24 11:02:52 enlightened NetworkManager[1250]: <warn> [1692855172.5248] ipv6ll[ccef0847d9ed30c9,ifindex=2]: changed: no IPv6 link local address to retry after Duplicate Address Detection failures (back off) Aug 24 11:03:02 enlightened NetworkManager[1250]: <warn> [1692855182.5254] platform-linux: do-add-ip6-address[2: fe80::de70:176b:722b:ec56]: failure 95 (Operation not supported) Aug 24 11:03:03 enlightened kernel: [UFW BLOCK] IN=enp1s0 OUT= MAC=01:00:5e:00:00:01:80:26:89:c3:6a:ca:08:00 SRC=192.168.1.7 DST=224.0.0.1 LEN=36 TOS=0x00 PREC=0x00 TTL=1 ID=37338 DF PROTO=2 Aug 24 11:03:04 enlightened NetworkManager[1250]: <warn> [1692855184.5264] platform-linux: do-add-ip6-address[2: fe80::963a:8c7b:4b91:4b82]: failure 95 (Operation not supported) Aug 24 11:03:06 enlightened NetworkManager[1250]: <warn> [1692855186.5268] platform-linux: do-add-ip6-address[2: fe80::3128:3058:2ba5:ab9]: failure 95 (Operation not supported)
Journal Filtering
While having access to such a large collection of data is definitely useful, it can be difficult or impossible to inspect and process manually. Because of this, one of the most important features of journalctl is its filtering options.
Getting Logs from the Current Boot
The most basic of these, which you might use daily, is the '-b' flag. This will show you all of the journal entries that have been collected since the most recent reboot.
journalctl -b
In cases where you aren't using this feature and are displaying more than one day of boots, you will see that journalctl has inserted a line that looks like this whenever the system was down.
Output . . . -- Reboot -- . . .
This can be used to help you logically separate the information into boot sessions.
Previous Boots
In the above example, we have gone through the detailed logs of current boot. Now, let us go through the logs of previous boots. There are certainly times when past boots would be helpful as well. The journal can save information from many previous boots; journalctl can be used to display information easily.
Some distributions enable saving previous boot information by default, while others do not. To enable persistent boot information, you can create the directory to store the journal by typing:
sudo mkdir -p /var/log/journal
Or, you can edit the journal configuration file.
sudo nano /etc/systemd/journald.conf
Under the [Journal] section, set the Storage= option to 'persistent' to enable persistent logging.
[journal] Storage = persistent
When saving previous boots is enabled on your server, journalctl provides some commands to help you work with boots as a unit of division. To see the boots that journald knows about, use the '--list-boots' option with journalctl.
journalctl --list-boots
Sample output
-5 75dd1dfd1c12443181d0c77c428d6557 Wed 2023-06-14 12:22:29 UTC—Wed 2023-06-14 12:37:21 UTC -4 a8ce1690897a42ddbf88dd9e95f0a38b Wed 2023-06-14 12:38:10 UTC—Wed 2023-06-14 13:01:54 UTC -3 9fe10a2b03bc4d1d930bcbe2feb24970 Wed 2023-06-14 13:02:13 UTC—Wed 2023-06-14 13:14:24 UTC -2 ef89772662e84596a89466adc7d5ce8b Wed 2023-06-14 13:14:43 UTC—Wed 2023-06-14 13:16:10 UTC -1 780a22258bf64c8e989579e3b879ceb0 Tue 2023-08-15 02:48:22 UTC—Tue 2023-08-15 02:50:06 UTC 0 ab466baee590498cb90a027753db9984 Tue 2023-08-15 03:49:31 UTC—Tue 2023-08-15 04:04:35 UTC
This will display a line for each boot. The first column is the offset for the boot that can be used to easily reference the boot with journalctl.
If you need an absolute reference, the boot ID is in the second column. You can tell the time that the boot session refers to with the two time specifications listed towards the end.
To display information from these boots, you can use data from either the first or second column.
For instance, to see the journal from the previous boot, use the '-1' relative pointer with the '-b' flag.
journalctl -b -1
You can also use the boot ID to call back the data from a boot.
journalctl -b caf0524a1d394ce0bdbcff75b94444fe
Timed Logs
We have seen log entries by boot are more useful. But, often, we may want to view timed logs. This may be especially true when dealing with long-running servers having significant uptime.
You can filter time limits by using the '- - since' and '- - until' options, which help to display entries before and after the given time, respectively.
The time values can come in a variety of formats. For absolute time values, you should use the following format:
YYYY-MM-DD HH:MM:SS
For instance, we can see all of the entries since June 14th to August 15th:
journalctl --since "2023-06-14" --until "2023-08-15"
Sample output
Jun 14 12:22:29 groot kernel: Linux version 5.15.0-73-generic (buildd@bos03-amd64-060) (gcc (Ubuntu> Jun 14 12:22:29 groot kernel: Command line: BOOT_IMAGE=/vmlinuz-5.15.0-73-generic root=/dev/mapper/> Jun 14 12:22:29 groot kernel: KERNEL supported cpus: Jun 14 12:22:29 groot kernel: Intel GenuineIntel Jun 14 12:22:29 groot kernel: AMD AuthenticAMD Jun 14 12:22:29 groot kernel: Hygon HygonGenuine Jun 14 12:22:29 groot kernel: Centaur CentaurHauls Jun 14 12:22:29 groot kernel: zhaoxin Shanghai Jun 14 12:22:29 groot kernel: [Firmware Bug]: TSC doesn't count with P0 frequency! Jun 14 12:22:29 groot kernel: x86/fpu: x87 FPU will use FXSAVE Jun 14 12:22:29 groot kernel: signal: max sigframe size: 1440 Jun 14 12:22:29 groot kernel: BIOS-provided physical RAM map: Jun 14 12:22:29 groot kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable Jun 14 12:22:29 groot kernel: BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved Jun 14 12:22:29 groot kernel: BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved Jun 14 12:22:29 groot kernel: BIOS-e820: [mem 0x0000000000100000-0x00000000dffeffff] usable Jun 14 12:22:29 groot kernel: BIOS-e820: [mem 0x00000000dfff0000-0x00000000dfffffff] ACPI data Jun 14 12:22:29 groot kernel: BIOS-e820: [mem 0x00000000fec00000-0x00000000fec00fff] reserved Jun 14 12:22:29 groot kernel: BIOS-e820: [mem 0x00000000fee00000-0x00000000fee00fff] reserved Jun 14 12:22:29 groot kernel: BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved Jun 14 12:22:29 groot kernel: BIOS-e820: [mem 0x0000000100000000-0x00000002145fffff] usable Jun 14 12:22:29 groot kernel: NX (Execute Disable) protection: active Jun 14 12:22:29 groot kernel: SMBIOS 2.5 present. Jun 14 12:22:29 groot kernel: DMI: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006 Jun 14 12:22:29 groot kernel: Hypervisor detected: KVM Jun 14 12:22:29 groot kernel: kvm-clock: Using msrs 4b564d01 and 4b564d00 Jun 14 12:22:29 groot kernel: kvm-clock: cpu 0, msr 1d7401001, primary cpu clock Jun 14 12:22:29 groot kernel: kvm-clock: using sched offset of 924714554074 cycles Jun 14 12:22:29 groot kernel: clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e> Jun 14 12:22:29 groot kernel: tsc: Detected 3293.726 MHz processor Jun 14 12:22:29 groot kernel: e820: update [mem 0x00000000-0x00000fff] usable ==> reserved Jun 14 12:22:29 groot kernel: e820: remove [mem 0x000a0000-0x000fffff] usable Jun 14 12:22:29 groot kernel: last_pfn = 0x214600 max_arch_pfn = 0x400000000 Jun 14 12:22:29 groot kernel: Disabled Jun 14 12:22:29 groot kernel: x86/PAT: MTRRs disabled, skipping PAT initialization too.
To get the data from yesterday, you could type:
journalctl --since yesterday
If you received reports of a service interruption starting at 9:00 AM and continuing until an hour ago, you could type:
journalctl --since 09:00 --until "1 hour ago"
As you can see, it is very easy to define flexible time to filter the entries you wish to see.
Filter By service or component
Previously, we have learned some ways to filter the journal data using time constraints. In this section, we'll discuss how to filter based on what service or component you are interested in. The systemd journal provides different ways to do this.
The most useful way of filtering is by the unit you are interested in. We can use the '-u' option in this way.
For instance, to see all of the logs from an Nginx unit on our system, run the following command.
journalctl -u nginx.service
Furthermore, we can filter the logs of nginx or any other service according to the time.
journalctl -u nginx.service --since today
journalctl -u nginx.service -u php-fpm.service --since today
To monitor or follow the mysql logs we can use the following command
journalctl -u mysql.service -f
$ journalctl -u mysql.service -f Aug 19 07:48:26 enlightened systemd[1]: Starting mysql.service - MySQL Community Server... Aug 19 07:48:29 enlightened systemd[1]: Started mysql.service - MySQL Community Server.
Filter by process, user, or group id
In the above example, we have gone through the filtering of logs according to the service or component. Now, let us dig into filtering logs by proves, user, or group id.
To do this, we can filter by specifying the '_PID' field. If the PID we're interested in is 8088, we could type:
journalctl _PID=8088
Sometimes, you may wish to show all of the entries logged from a specific user or group. This can be done with the '_UID' or '_GID' filters.
If your web server runs under the www-data user, you can find the user ID by typing:
id -u www-data
Sample output
32
Afterwards, you can use the ID that was returned to filter the journal results.
journalctl _UID=33 --since today
The systemd journal has many fields that can be used for filtering. Some of those are passed from the process being logged, and some are applied by journald using information it gathers from the system at the time of the log.
To see which group IDs the systemd journal has entries for, you can enter:
journalctl -F _GID
Sample output
119 124 104 113 127 103 105 102 4 0 133 131 1 1000
This will show you all of the values that the journal has stored for the group ID field. It can help you construct your filters.
Kernel Messages
Kernel messages, those usually found in dmesg output, can be retrieved from the journal as well.
To display only these messages, we can add the '-k' or '--dmesg' flags to our command.
journalctl -k
By default, this will display the kernel messages from the current boot. You can specify an alternative boot using the normal boot selection flags discussed previously. For instance, to get the messages from five boots ago, you could type:
journalctl -k -b -5
Active Process Monitoring
To display a set amount of records, you can use the '-n' option, which works exactly as tail -n.
By default, it will display the most recent 10 entries.
journalctl -n
You can specify the number of entries you'd like to see with a number after the -n.
Journalctl -n 12
Maintenance and Cleanup
Apart from retrieving logs, 'journalctl' also offers the functionality for log management and cleanup.
Rotate Logs
To rotate archived logs and start anew, use the '--rotate' flag, followed by the '--vacuum-time' flag to set the retention period.
sudo journalctl --rotate sudo journalctl --vacuum-time=1d
Current Disk Usage
We can find current disk usage by using '- -disk -usage' flag.
journalctl --disk-usage
Sample output
Archived and active journals take up 72.0M in the file system.
Deleting Old Logs
If you wish to shrink your journal, you can do that in two different ways (available with systemd version 218 and later).
Use the '--vacuum-size' option to shrink your journal by indicating a size. This will remove old entries until the total journal space taken up on the disk is at the requested size.
sudo journalctl --vacuum-size=1G
Another way that you can shrink the journal is providing a cutoff time with the '--vacuum-time' option. Any entries beyond that time are deleted. This allows you to keep the entries that have been created after a specific time.
To keep entries from the last year, run the following command:
sudo journalctl --vacuum-time=1years
Journald configuration file
On a typical ubuntu system, the journald configuration file is located at "/etc/systemd/journald.conf".
$ cat /etc/systemd/journald.conf # This file is part of systemd. # # systemd is free software; you can redistribute it and/or modify it under the # terms of the GNU Lesser General Public License as published by the Free # Software Foundation; either version 2.1 of the License, or (at your option) # any later version. # # Entries in this file show the compile time defaults. Local configuration # should be created by either modifying this file, or by creating "drop-ins" in # the journald.conf.d/ subdirectory. The latter is generally recommended. # Defaults can be restored by simply deleting this file and all drop-ins. # # Use 'systemd-analyze cat-config systemd/journald.conf' to display the full config. # # See journald.conf(5) for details. [Journal] #Storage=auto #Compress=yes #Seal=yes #SplitMode=uid #SyncIntervalSec=5m #RateLimitIntervalSec=30s #RateLimitBurst=10000 #SystemMaxUse= #SystemKeepFree= #SystemMaxFileSize= #SystemMaxFiles=100 #RuntimeMaxUse= #RuntimeKeepFree= #RuntimeMaxFileSize= #RuntimeMaxFiles=100 #MaxRetentionSec= #MaxFileSec=1month #ForwardToSyslog=yes #ForwardToKMsg=no #ForwardToConsole=no #ForwardToWall=yes #TTYPath=/dev/console #MaxLevelStore=debug #MaxLevelSyslog=debug #MaxLevelKMsg=notice #MaxLevelConsole=info #MaxLevelWall=emerg #LineMax=48K #ReadKMsg=yes #Audit=no $
To learn more about each configuration parameter in the above file, simply read the man pages for journald.conf with the following command:
man journald.conf
Conclusion
Effective log management is crucial for Linux system administration since it helps with problem diagnosis, performance monitoring, and infrastructure maintenance. A powerful tool that makes log analysis and administration simpler is called "journalctl."
It provides an array of choices for filtering, searching, and extracting data from the systemd journal.
'journalctl' shows how it is a useful tool for both experienced administrators and newbies to the Linux environment because of its user-friendly interface and adaptability. 'journalctl' enables you to dig thoroughly into your system's logs and gain useful information, whether you're debugging an error, monitoring system health, or analyzing service performance.
