Logging

Journald

Most programs write their log output to the syslog service. The syslogd daemon performs this service on traditional systems by listening for these messages and sending them to the appropriate channel (file, database, email, etc.) when received. On modern systems, journald typically does this work. journalctl can be used to work with journald.

You can determine if your system is using journald by typing journalctl in a shell. If the system is using journald, you will see a paged output. Unless you have a system that is using a traditional syslog daemon such as syslogd or rsyslogd, you will use the journal. To get the full output from journalctl, you need to run the command as root or as a user of the adm or systemd-journal groups.

• Some examples of using journalctl:

  • To search for logs from a process using the PID: journalctl _PID=555 (where 555 is the PID)
  • To search for messages from the past 4 hours: journalctl -S -4h
  • To filter by unit: journalctl -u sshd.service
  • To search by a given field: journalctl -F _SYSTEMD_UNIT
  • If you do not know what fields are available, use: journalctl -N
  • To view the logs from this boot: journalctl -b
  • To view the logs from the previous boot: journalctl -b -1
  • To list all boots by ID: journalctl --list-boots
  • To view Kernel messages: journalctl -k
  • To filter by severity level: journalctl -p 3 (where 3 is the severity level. Values range from 0 (most important) to 7 (least important))

• Journal maintenance

  • the journal files stored in /var/log/journal do not need to be rotated. journald handles the maintenance of these files.

Syslogd

Syslogd first appeared with the sendmail email server back in the 1980’s. Developers of other services readily adopted it, and RFC3164 was ratified to define it. The syslog mechanism is simple. It listens on a Unix domain socket, /dev/log. Though, it can also listen on a network socket, enabling any device on the network to send logs to it. This makes rsyslogd act as a log server.

• Facility, severity, and priority
  • Syslog sends messages of various types from different services to different destinations. Becuase of this, it needs a way to classify each message.
  • The facility is a general category of service, used to identify the service that sent the message. The available facilities in the syslog protocol are hardwired and there is no way to add your own. However, you can use a general local0 through local7 value.
  • The severity is the urgency of the log messages. This can be a value from 0 (most urgent) to 7 (least urgent)
    • 0. emerg
    • 1. alert
    • 2. crit
    • 3. err
    • 4. warn
    • 5. notice
    • 6. info
    • 7. debug
  • The facility and the severity together make up the priority, packaged as a single value in the syslog protocol. You can read more about this in RFC 5424

Logfile Rotation

When you are using a syslog daemon, log messages get put into files somewhere on the system. These files need to be rotated on a schedule to prevent the files from consuming too much storage space. logrotate performs this task.

• How logrotate works:
  1. Remove the oldest file, auth.log.3
  2. Renames auth.log.2 to auth.log.3
  3. Renames auth.log.1 to auth.log.2
  4. Renames auth.log to auth.log.1