Common Linux Log Files and How to View Them

By | March 31, 2023

What are Linux logs?

Linux logs are a critical component of the OS that gives a snapshot of everything that has been happening in the system. It helps in tracking events within the system such as process initiation/failure, errors in running applications, login failures, boot-up process etc.

They are essential for debugging running applications, tracking security events, such as a brute force attack etc.

The log files also provide important diagnostic information in the even of some kind of failure, so that system administrators and gather information and fix the system.

For example a server application like apache or nginx or postfix failed to start or suddenly stopped working, then it is always best to first check the log files to see what went wrong. The log files will most of the time contain accurate information about the cause of the error which will help in fixing the problem.

In this article we take a quick look at some of the common log files found on linux systems. They would be present on both server and desktop linux installations.

Types of log files

On linux there are lots of log files and some common ones include:

  • /var/log/syslog
  • /var/log/auth.log
  • /var/log/daemon.log
  • /var/log/kern.log

Logs like syslog, kern.log and auth.log are found on almost all linux distros.

Before looking at the types of log files, let's see the location of Linux logs under "/var/log".
Navigate to the directory and list the files.

cd /var/log/

$ ls
alternatives.log  btmp          dmesg.1.gz      gdm3             kern.log  speech-dispatcher           vmware-network.1.log  vmware-vmsvc-root.1.log   wtmp
apt               cups          dmesg.2.gz      gpu-manager.log  lastlog   syslog                      vmware-network.2.log  vmware-vmsvc-root.2.log
auth.log          dist-upgrade  dpkg.log        hp               openvpn   ubuntu-advantage.log        vmware-network.3.log  vmware-vmsvc-root.3.log
boot.log          dmesg         faillog         installer        private   ubuntu-advantage-timer.log  vmware-network.4.log  vmware-vmsvc-root.log
bootstrap.log     dmesg.0       fontconfig.log  journal          README    unattended-upgrades         vmware-network.log    vmware-vmtoolsd-root.log

In the above screenshot, we can see the log files present in the system. For this tutorial, I will be using 'Ubuntu 22.10'.

On a different system the contents of /var/log looks like this:

enlightened@enlightened:~$ ls /var/log/
alternatives.log        apache2          boot.log       dmesg.4.gz      dpkg.log.9.gz    kern.log.3.gz  mail.log.4.gz        syslog.6.gz                   ubuntu-advantage-timer.log.2.gz  vbox-setup.log.1
alternatives.log.1      apport.log       boot.log.1     dpkg.log        faillog          kern.log.4.gz  mysql                syslog.7.gz                   ubuntu-advantage-timer.log.3.gz  vbox-setup.log.2
alternatives.log.10.gz  apport.log.1     bootstrap.log  dpkg.log.1      firebird         lastlog        prime-offload.log    sysstat                       ubuntu-advantage-timer.log.4.gz  vbox-setup.log.3
alternatives.log.11.gz  apport.log.2.gz  btmp           dpkg.log.10.gz  firewalld        lxc            prime-supported.log  tallylog                      ubuntu-advantage-timer.log.5.gz  vbox-setup.log.4
alternatives.log.12.gz  apport.log.3.gz  btmp.1         dpkg.log.11.gz  fontconfig.log   mail.err       private              ubuntu-advantage.log          ubuntu-advantage-timer.log.6.gz  wpslog
alternatives.log.2.gz   apt              cups           dpkg.log.12.gz  gpu-manager.log  mail.err.1     README               ubuntu-advantage.log.1        ufw.log                          wtmp
alternatives.log.3.gz   aptitude         distccd.log    dpkg.log.2.gz   gufw.log         mail.err.2.gz  samba                ubuntu-advantage.log.2.gz     ufw.log.1                        wtmp.1
alternatives.log.4.gz   aptitude.1.gz    dist-upgrade   dpkg.log.3.gz   hp               mail.err.3.gz  sddm.log             ubuntu-advantage.log.3.gz     ufw.log.2.gz                     Xorg.0.log
alternatives.log.5.gz   auth.log         dmesg          dpkg.log.4.gz   installer        mail.err.4.gz  syslog               ubuntu-advantage.log.4.gz     ufw.log.3.gz                     Xorg.0.log.old
alternatives.log.6.gz   auth.log.1       dmesg.0        dpkg.log.5.gz   journal          mail.log       syslog.1             ubuntu-advantage.log.5.gz     ufw.log.4.gz                     Xorg.1.log
alternatives.log.7.gz   auth.log.2.gz    dmesg.1.gz     dpkg.log.6.gz   kern.log         mail.log.1     syslog.2.gz          ubuntu-advantage.log.6.gz     unattended-upgrades              Xorg.pid-12656.log
alternatives.log.8.gz   auth.log.3.gz    dmesg.2.gz     dpkg.log.7.gz   kern.log.1       mail.log.2.gz  syslog.3.gz          ubuntu-advantage-timer.log    upgrade
alternatives.log.9.gz   auth.log.4.gz    dmesg.3.gz     dpkg.log.8.gz   kern.log.2.gz    mail.log.3.gz  syslog.4.gz          ubuntu-advantage-timer.log.1  vbox-setup.log
enlightened@enlightened:~$

Following are the types of log files that will be discussed in this section:

  • System logs
  • Application logs
  • Non-human readable logs
  • Syslog

Even though there are several logs files in linux, we can classify them into 4 major categories:

1. Syslog (/var/log/syslog)

Syslog is a standard network-based logging protocol which can be configured to store logs for various devices and applications to a centralized server.

Ubuntu uses rsyslog for syslog configuration which is discussed in the later section. It stores all kinds of messages except "auth" related messages.

The Syslog daemon listens for logs and writes them to a specific location. The location(s) is defined in the configuration file for the daemon. rsyslog is the Syslog daemon shipped with most of the distros.
You can view the local syslog on your system under "/var/log/syslog".

tail -2f /var/log/syslog
Mar 19 13:06:18 overfittedropout-virtual-machine NetworkManager[853]:   [1679211378.7842] manager: NetworkManager state is now CONNECTED_GLOBAL
Mar 19 13:06:28 overfittedropout-virtual-machine systemd[1]: NetworkManager-dispatcher.service: Deactivated successfully.
Mar 19 13:08:03 overfittedropout-virtual-machine dbus-daemon[1144]: [session uid=1000 pid=1144] Activating service name='org.gnome.Nautilus' requested by ':1.24' (uid=1000 pid=1292 comm="/usr/bin/gnome-shell" label="unconfined")
Mar 19 13:08:03 overfittedropout-virtual-machine dbus-daemon[1144]: [session uid=1000 pid=1144] Successfully activated service 'org.gnome.Nautilus'

In the above excerpt, we can see Nautilus service is loaded, which indicates that the file system is accessed/interacted with.

2. dmesg log (/var/log/dmesg)

The /var/log/dmesg log file is used to write the kernel messages retrieved from the kernel ring buffer. The kernel ring buffer is a circular buffer that is the first data structure storing log messages when the system boots up.

This particular log file contains a lot of information about the hardware detection process. For example whenever you plug a usb device into the system an entry shall appear in the dmesg log file.

One way of viewing the dmesg log file is by simply opening the /var/log/dmesg file. Another way is to use the dmesg command which presents the log file in a much easier to read format.

You can view more options to be used with dmesg –help. Here is a quick and simple example:

$ dmesg
[    0.000000] Linux version 5.19.0-31-generic (buildd@lcy02-amd64-087) (x86_64-linux-gnu-gcc-12 (Ubuntu 12.2.0-3ubuntu1) 12.2.0, GNU ld (GNU Binutils for Ubuntu) 2.39) #32-Ubuntu SMP PREEMPT_DYNAMIC Fri Jan 20 15:20:08 UTC 2023 (Ubuntu 5.19.0-31.32-generic 5.19.17)
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.19.0-31-generic root=UUID=01276451-a735-47ce-96df-1d22f6b2654e ro find_preseed=/preseed.cfg auto noprompt priority=critical locale=en_US quiet splash
[    0.000000] KERNEL supported cpus:
[    0.000000]   Intel GenuineIntel
[    0.000000]   AMD AuthenticAMD
[    0.000000]   Hygon HygonGenuine
[    0.000000]   Centaur CentaurHauls
[    0.000000]   zhaoxin   Shanghai

In the above excerpt, we can see the boot-up logs where Linux is loading the boot image and also printing the supported CPU preference for the kernel. We have written a full article on how to use the dmesg command to view kernel logs over here:

https://www.binarytides.com/dmesg-command-examples-in-linux/

3. Authorization log (/var/log/auth.log)

– Keeps track of login for users. It also logs privilege escalation events such as sudo.

$ sudo head -n6 /var/log/auth.log
Feb 22 20:47:35 <strong>overfittedropout-virtual-machine</strong> systemd-logind[749]: New seat seat0.
Feb 22 20:47:35 overfittedropout-virtual-machine systemd-logind[749]: Watching system buttons on /dev/input/event0 (Power Button)
Feb 22 20:47:35 overfittedropout-virtual-machine systemd-logind[749]: Watching system buttons on /dev/input/event1 (AT Translated Set 2 keyboard)
Feb 22 20:47:47 overfittedropout-virtual-machine gdm-autologin]: gkr-pam: no password is available for user
<strong>Feb 22 20:47:47 overfittedropout-virtual-machine gdm-autologin]: pam_env(gdm-autologin:session): deprecated reading of user environment enabled
Feb 22 20:47:47 overfittedropout-virtual-machine gdm-autologin]: pam_unix(gdm-autologin:session): session opened for user overfittedropout(uid=1000) by (uid=0)</strong>

In the above excerpt, we can see logon events for user overfittedropout. We can also see that the user has enabled auto-login.

4. Daemon Log (/var/log/daemon.log)

- Daemons are programs that run in the background, usually without user interaction. Daemon log stores information on running processes.

$ tail -n 20 /var/log/daemon.log 
Feb 23 23:03:34 overfittedropout-virtual-machine systemd[1014]: Started Application launched by gnome-session-binary.
Feb 23 23:04:01 overfittedropout-virtual-machine dbus-daemon[1057]: [session uid=1000 pid=1057] Activating service name='org.gnome.TextEditor' requested by ':1.73' (uid=1000 pid=1494 comm="/usr/bin/nautilus --gapplication-service" label="unconfined")

In the above log excerpt, we can see the gnome text editor was opened.

5. Kernel log (/var/log/kern.log)

Logs information from the linux kernel.

$ tail -n20 kern.log 
Feb 23 23:22:26 overfittedropout-virtual-machine kernel: [  213.623243] audit: type=1400 audit(1677174746.121:56): apparmor="ALLOWED" operation="connect" class="file" profile="libreoffice-soffice" name="/run/user/1000/at-spi/bus" pid=2192 comm="soffice.bin" requested_mask="wr" denied_mask="wr" fsuid=1000 ouid=1000
Feb 23 23:22:26 overfittedropout-virtual-machine kernel: [  213.623489] audit: type=1400 audit(1677174746.121:57): apparmor="ALLOWED" operation="file_perm" class="file" profile="libreoffice-soffice" name="/run/user/1000/at-spi/bus" pid=2192 comm="soffice.bin" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000
Feb 23 23:22:26 overfittedropout-virtual-machine kernel: [  213.623492] audit: type=1400 audit(1677174746.121:58): apparmor="ALLOWED" operation="file_perm" class="file" profile="libreoffice-soffice" name="/run/user/1000/at-spi/bus" pid=2192 comm="soffice.bin" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000
Feb 23 23:22:29 overfittedropout-virtual-machine kernel: [  217.180515] audit: type=1400 audit(1677174749.677:59): apparmor="ALLOWED" operation="connect" class="file" profile="libreoffice-soffice" name="/run/user/1000/at-spi/bus" pid=2213 comm="soffice.bin" requested_mask="wr" denied_mask="wr" fsuid=1000 ouid=1000
Feb 23 23:22:29 overfittedropout-virtual-machine kernel: [  217.180730] audit: type=1400 audit(1677174749.677:60): apparmor="ALLOWED" operation="file_perm" class="file" profile="libreoffice-soffice" name="/run/user/1000/at-spi/bus" pid=2213 comm="soffice.bin" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000
Feb 23 23:22:29 overfittedropout-virtual-machine kernel: [217.180733] audit: type=1400 audit(1677174749.677:61): apparmor="ALLOWED" operation="file_perm" class="file" profile="libreoffice-soffice" name="/run/user/1000/at-spi/bus" pid=2213 comm="soffice.bin" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000

In the above excerpt, we can see libre-office writers getting loaded.

6. Login failure logs(var/log/faillog)

Contains login failure events of the system. The "faillog" command is used to view the log file contents. This log is stored in binary format and cannot be read as plain text.

$ faillog -a
Login       Failures Maximum Latest                   On

root            0        0   01/01/70 05:30:00 +0530

You can view additional options and syntax using "man faillog" and "faillog –help".

7. Current login logs(/var/log/wtmp)

Contains login info used by other utilities to find out who's logged in. To view currently logged in users, use the "who" command.

who
overfittedropout tty2         2023-02-24 01:04 (tty2)

In the above excerpt, we can see the user "overfittedropout" is logged into the system since 01:04.

Note:
Most Linux files have time stamps which are quintessential to establish timelines of incident/errors and helps us filter the logs. It is a good practice to keep a consistent time zone configuration for all the servers.

Here is an exhaustive list of log files in Linux:-
LinuxLogFiles - Community Help Wiki (ubuntu.com)

Application Specific Logs

Different applications like apache, nginx, postfix also create their own specific log files that a lot contain runtime information useful for debugging issues.

For example if mail delivery has failed you might want to check the postfix log files or if some webpage is showing 404 not found or 500 internal server error then you have to check the error logs of the http server application whether apache or nginx.

1. Apache Error Log

$ cat /var/log/apache2/error.log
[Fri Mar 31 08:45:40.112067 2023] [mpm_prefork:notice] [pid 1294] AH00163: Apache/2.4.54 (Ubuntu) configured -- resuming normal operations
[Fri Mar 31 08:45:40.112092 2023] [core:notice] [pid 1294] AH00094: Command line: '/usr/sbin/apache2'
$

2. Apache Access Log

The access log contains an entry for every http request made to the server from any client. It can be viewed like this:

$ cat /var/log/apache2/access.log

3. Apache tomcat logs(apache-tomcat-10.1.5/logs/catalina.out)

These are logs from apache tomcat server installation.

$ head -n 2 catalina.out 
./catalina.sh: 1: eval: /home/overfittedropout/Downloads/openlogic-openjdk-8u362-b09-linux-x32/bin/bin/java: not found

In the above log, we can see 'java' is not found which can be used to debug any issues related to the tomcat server.

Commands to view log files

Log files on linux are mostly in plain text format with each entry as a newline. Some commonly used commands to view log files are:

less – Lets you page through a text file, displaying a screenful of text each time
grep – Find all occurrences of a search term in a file or filter a log file
tail/head - Output the last few lines of files

Syntax and usage

1. less

The less command pipes the output and makes it scrollable in the terminal. It can be used for viewing any log file in text format. For example we can view the syslog file like this:

$ cat /var/log/syslog | less
Feb 22 20:47:35 overfittedropout-virtual-machine systemd[1]: Starting Flush Journal to Persistent Storage...
Feb 22 20:47:35 overfittedropout-virtual-machine systemd[1]: Finished Set the console keyboard layout.
Feb 22 20:47:35 overfittedropout-virtual-machine systemd[1]: Finished Load/Save Random Seed.
Feb 22 20:47:35 overfittedropout-virtual-machine systemd[1]: First Boot Complete was skipped because of a failed condition check (ConditionFirstBoot=yes).
Feb 22 20:47:35 overfittedropout-virtual-machine systemd[1]: modprobe@pstore_zone.service: Deactivated successfully.

The cat command prints the whole file to "less" which in turn makes it scrollable and easy to view.

Searching terms using 'less'

For searching specific terms in a file, press the forward (/) button while you are viewing the file, write the search term and hit enter.
Syntax: /

For example - /2022-10-13 will search for logs which are dated on 13th October and highlight it.
You can view the options of less using "less --help" and "man less".

Note: To search backward from your current position in the file toward the start of the file, press the "?" key and type your search term. To find the next matching item, press "n". To search for the previous matching item, press "N".

Press 'q' to exit less.

2. grep

The Linux grep command is a string and pattern matching utility that displays matching lines from specified files. We can also use it in integration with other commands by piping the output.

Syntax – | grep

$ cat /etc/passwd | grep overfittedropout
overfittedropout:x:1000:1000:overfittedropout,,,:/home/overfittedropout:/bin/bash

In the above output, we can see grep has filtered the output of "cat /etc/passwd/" and displayed the lines which contains the word "overfittedropout".

The same output can also be obtained by "grep overfittedropout /etc/passed".

Syntax – grep

$ grep overfittedropout /etc/passwd
overfittedropout:x:1000:1000:overfittedropout,,,:/home/overfittedropout:/bin/bash

Note:
The above command is case-sensitive.

In case we forget the proper case of a keyword we are searching, we can view all the lines using the following command.

Syntax – grep -i
The "-i" flag denotes ignore case.

$ cat /etc/passwd |grep -i OverfittedRopouT
overfittedropout:x:1000:1000:overfittedropout,,,:/home/overfittedropout:/bin/bash

By default, grep searches for the target that appears anywhere in that line, including inside another string. We can force grep to search for a keyword as a whole using the -w flag.

Syntax – grep -w

Invert can be performed using "-v" flag searches using grep in the following way.
Syntax – grep -v
The -v represents inverse search.
Let's say we want to search a line in '/etc/passwd/', where the keyword "no login" is not present.

$ cat /etc/passwd | grep -v nologin
root:x:0:0:root:/root:/bin/bash
sync:x:4:65534:sync:/bin:/bin/sync
tss:x:106:113:TPM software stack,,,:/var/lib/tpm:/bin/false
whoopsie:x:117:124::/nonexistent:/bin/false
speech-dispatcher:x:119:29:Speech Dispatcher,,,:/run/speech-dispatcher:/bin/false
gdm:x:125:133:Gnome Display Manager:/var/lib/gdm3:/bin/false
hplip:x:126:7:HPLIP system user,,,:/run/hplip:/bin/false
gnome-initial-setup:x:127:65534::/run/gnome-initial-setup/:/bin/false
overfittedropout:x:1000:1000:overfittedropout,,,:/home/overfittedropout:/bin/bash

You can view additional options and syntax using "grep --help" and "man grep".

3. tail

As the name suggests, the tail command prints out lines beginning from the end of the file. It is complementary to the "head" command which prints text from the beginning of the file.

cat /var/log/syslog | head
Feb 22 20:47:35 overfittedropout-virtual-machine systemd[1]: Starting Flush Journal to Persistent Storage...
Feb 22 20:47:35 overfittedropout-virtual-machine systemd[1]: Finished Set the console keyboard layout.
Feb 22 20:47:35 overfittedropout-virtual-machine systemd[1]: Finished Load/Save Random Seed.
Feb 22 20:47:35 overfittedropout-virtual-machine systemd[1]: First Boot Complete was skipped because of a failed condition check (ConditionFirstBoot=yes).
Feb 22 20:47:35 overfittedropout-virtual-machine systemd[1]: modprobe@pstore_zone.service: Deactivated successfully.
Feb 22 20:47:35 overfittedropout-virtual-machine systemd[1]: Finished Load Kernel Module pstore_zone.
Feb 22 20:47:35 overfittedropout-virtual-machine systemd[1]: modprobe@pstore_blk.service: Deactivated successfully.
Feb 22 20:47:35 overfittedropout-virtual-machine systemd[1]: Finished Load Kernel Module pstore_blk.
Feb 22 20:47:35 overfittedropout-virtual-machine systemd[1]: [email protected]: Deactivated successfully.
Feb 22 20:47:35 overfittedropout-virtual-machine systemd[1]: Finished Load Kernel Module drm.

You can specify the number of lines that need to be printed either from top or bottom with "head" and "tail" commands respectively with the -n flag.

$ cat /var/log/syslog|head -n 4
Feb 22 20:47:35 overfittedropout-virtual-machine systemd[1]: Starting Flush Journal to Persistent Storage...
Feb 22 20:47:35 overfittedropout-virtual-machine systemd[1]: Finished Set the console keyboard layout.
Feb 22 20:47:35 overfittedropout-virtual-machine systemd[1]: Finished Load/Save Random Seed.
Feb 22 20:47:35 overfittedropout-virtual-machine systemd[1]: First Boot Complete was skipped because of a failed condition check (ConditionFirstBoot=yes).
$ cat /var/log/syslog|tail -n 4 
Feb 25 20:13:38 overfittedropout-virtual-machine snapd[806]: storehelpers.go:769: cannot refresh: snap has no updates available: "bare", "core20", "core22", "firefox", "gnome-3-38-2004", "gnome-42-2204", "gtk-common-themes", "snap-store", "snapd", "snapd-desktop-integration"
Feb 25 20:13:38 overfittedropout-virtual-machine snapd[806]: autorefresh.go:551: auto-refresh: all snaps are up-to-date
Feb 25 20:14:05 overfittedropout-virtual-machine systemd[1]: systemd-timedated.service: Deactivated successfully.
Feb 25 20:17:01 overfittedropout-virtual-machine CRON[2213]: (root) CMD (   cd / &amp;&amp; run-parts --report /etc/cron.hourly)

In the above example, we printed the first and last 4 lines of the file. You can view additional options and syntax using "man tail" and "man head".

For viewing logs which are continuously appended in real-time (e.g. catalina.out), we can view the latest content by adding "-f" flag to tail command:

tail -2f /var/log/syslog
Mar 19 13:06:18 overfittedropout-virtual-machine NetworkManager[853]:   [1679211378.7842] manager: NetworkManager state is now CONNECTED_GLOBAL
Mar 19 13:06:28 overfittedropout-virtual-machine systemd[1]: NetworkManager-dispatcher.service: Deactivated successfully.
Mar 19 13:08:03 overfittedropout-virtual-machine dbus-daemon[1144]: [session uid=1000 pid=1144] Activating service name='org.gnome.Nautilus' requested by ':1.24' (uid=1000 pid=1292 comm="/usr/bin/gnome-shell" label="unconfined")
Mar 19 13:08:03 overfittedropout-virtual-machine dbus-daemon[1144]: [session uid=1000 pid=1144] Successfully activated service 'org.gnome.Nautilus'

The above command starts printing logs from the last two lines of the file and prints further as the file is appended with new information.

Enabling/Disabling system logs through rsyslog

When you start exploring log files, you might see that there are several log files which might not be present under "/var/log/". You can enable the log files in the rsyslog configuration file.

a. Navigate to "/etc/rsyslog.d".

$ cd /etc/rsyslog.d
$ ls -ltr
total 12
-rw-r--r-- 1 root root  314 Sep 19  2021 20-ufw.conf
-rw-r--r-- 1 root root 1121 Feb 23 22:41 50-default.conf.save
-rw-r--r-- 1 root root 1114 Feb 23 23:18 50-default.con

b. In the above screenshot, we can see the configuration files. We will be working with '50-default.conf'

c. Please ensure to open the file in root user mode to edit it.

sudo nano 50-default.conf 
#  Default rules for rsyslog.
#
# For more information see rsyslog.conf(5) and /etc/rsysl&gt;
#
# First some standard log files.  Log by facility.
#
auth,authpriv.*                 /var/log/auth.log
*.*;auth,authpriv.none          -/var/log/syslog
cron.*                          /var/log/cron.log
#daemon.*                        -/var/log/daemon.log
kern.*                          -/var/log/kern.log
#lpr.*                          -/var/log/lpr.log
mail.*                          -/var/log/mail.log
user.*                          -/var/log/user.log

In the above excerpt, we can see the daemon logs are commented out with '#'. To enable daemon logs, we can remove the # and save the file. We can also see the path where the logs will be stored and this can be changed as well.

Please ensure to restart the system for the configurations to reflect. Similarly, we can also disable any logs by commenting it out with '#'.

Further readings

Logs are a very important component for any Linux user. After mastering the basics, you can go ahead and explore more about log analysis.

In depth explanation of log generation – Linux Logging Complete Guide – devconnected
Linux logs + SIEM solution – Monitoring Linux Server with Splunk - Patrick Bareiss (patrick-bareiss.com)

About Silver Moon

A Tech Enthusiast, Blogger, Linux Fan and a Software Developer. Writes about Computer hardware, Linux and Open Source software and coding in Python, Php and Javascript. He can be reached at [email protected].

Leave a Reply

Your email address will not be published. Required fields are marked *