Getting started with RSYSLOG in Linux

The Rocket-fast System for log processing (rsyslog) is a system utility provided in Linux which provides support for message logging. It offers high performance and comes with excellent security and has a modular design. This is a very useful utility that can accept input from a wide variety of sources and transform them and store them in multiple and diverse destinations.

 

Rsyslog is a GPL-ed, enhanced syslogd. Among others, it offers support for reliable Syslog over TCP, writing to MySQL databases, and fully configurable output formats (including great timestamps). Rsyslog was initiated by Rainer Gerhards, GroBrinderfeld.

The earlier world was using either one of the following

• The stock syslogd, e.g sysklogd
• Syslog-ng
• Another-solution

So basically, Rainer Gerhard’s club everything in another solution and created a new Syslog called rsyslog. 

Let’s start with the rsyslog

We are using Cent-OS 7 for the demo. You can use any distro you want. 

Step 1: Check if you have rsyslog installed.

[root@centos ~]# systemctl status rsyslog.service

rsyslog status

If it is not installed, please install using yum, dnf or package manager that you have in your distro and enable it.

1.1 To install rsyslog on centos use the following command.

[root@centos ~]# sudo yum install rsyslog

1.2 Enable rsyslog utility.

[root@centos ~]# systemctl enable rsyslog.service

1.3 Start the service.

[root@centos ~]# systemctl start rsyslog.service

1.4 Check the status of the service.

[root@centos ~]# systemctl status rsyslog.service

Here is one thing you should note if you are not able to get the status active and running please check the error with the “-l” option in the above command to check the failures. 

Step 2: Check the configuration file and default configurations.

The default configuration for rsyslog you can check in “/etc/rsyslog.conf”. With every configuration, you can see the comments which are much explanatory themselves. Here let’s focus only on setting up rsyslog and getting to know about rsyslog more.

[root@centos ~]# cat /etc/rsyslog.conf

# rsyslog configuration file

# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html

# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html

#### MODULES ####

# The imjournal module below is now used as a message source instead of imuxsock.

$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)

$ModLoad imjournal # provides access to the systemd journal

#$ModLoad imklog # reads kernel messages (the same are read from journald)

#$ModLoad immark  # provides –MARK– message capability

# Provides UDP syslog reception

#$ModLoad imudp

#$UDPServerRun 514

# Provides TCP syslog reception

#$ModLoad imtcp

#$InputTCPServerRun 514

#### GLOBAL DIRECTIVES ####

# Where to place auxiliary files

$WorkDirectory /var/lib/rsyslog

# Use default timestamp format

$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# File syncing capability is disabled by default. This feature is usually not required,

# not useful and an extreme performance hit

#$ActionFileEnableSync on

# Include all config files in /etc/rsyslog.d/

$IncludeConfig /etc/rsyslog.d/*.conf

# Turn off message reception via local log socket;

# local messages are retrieved through imjournal now.

$OmitLocalLogging on

# File to store the position in the journal

$IMJournalStateFile imjournal.state

#### RULES ####

# Log all kernel messages to the console.

# Logging much else clutters up the screen.

#kern.*                                                 /dev/console

# Log anything (except mail) of level info or higher.

# Don’t log private authentication messages!

*.info;mail.none;authpriv.none;cron.none                /var/log/messages

# The authpriv file has restricted access.

authpriv.*                                              /var/log/secure

# Log all the mail messages in one place.

mail.*                                                  -/var/log/maillog

# Log cron stuff

cron.*                                                  /var/log/cron

# Everybody gets emergency messages

*.emerg                                                 :omusrmsg:*

# Save news errors of level crit and higher in a special file.

uucp,news.crit                                          /var/log/spooler

# Save boot messages also to boot.log

local7.*                                                /var/log/boot.log

# ### begin forwarding rule ###

# The statement between the begin … end define a SINGLE forwarding

# rule. They belong together, do NOT split them. If you create multiple

# forwarding rules, duplicate the whole block!

# Remote Logging (we use TCP for reliable delivery)

#

# An on-disk queue is created for this action. If the remote host is

# down, messages are spooled to disk and sent when it is up again.

#$ActionQueueFileName fwdRule1 # unique name prefix for spool files

#$ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as possible)

#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown

#$ActionQueueType LinkedList   # run asynchronously

#$ActionResumeRetryCount -1    # infinite retries if host is down

# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional

#*.* @@remote-host:514

# ### end of the forwarding rule ###

To configure the rsyslog for our project it is better to add project-specific configurations in the following directory rather than adding those configurations in the rsyslog.conf. This will avoid the mess in config files.  /etc/rsyslog.conf is the default configuration for the rsyslog and to add any user configuration rsyslog.d directory is given and it is added in the default configuration to load all config files from this directory

/etc/rsyslog.d

As you can see I have two separate config files related to two different projects. This is how you can manage multiple project configurations to a single rsyslog daemon.

Step 3: Test rsyslog with CLI

Rsyslog not only supports most of the programming languages it also supports the CLI command to log messages. Let’s test whether our rsyslog is up and running.

logger is the cli command to log the message’s using rsyslog.

Logger

Logger supports local rsyslog logging as well as remote rsyslog server logging as well. Not only if we are listening to any specific UNIX socket in rsyslog that can also be used with this command. Overall we can test almost everything with this rsyslog command.

3.1 Let’s send a message to rsyslog .

We are going to send a simple rsyslog message and will check in /var/log/message whether it is logged or not.

Simple message

We have sent a message to rsyslog and rsyslog log it in the /var/log/message log file as this is the default configuration. The log file contains thousands of logs so it’s better to use “grep” to check our message.

3.2 Send messages with priority.

The priority is the most important in logging. We need to set the priority of every message according to its severity. 

rsyslog has the following severity and priorities is high to low

• Emergency
• Alert
• Critical
• Error
• Warning
• Notice
• Information
• DEbug

Will talk more about these severities and priorities in detail in an upcoming article. 

There are almost 8 main priorities in rsyslog but for this testing will going to use only emergency. Emergency priority is configured such that all logs with emergency priority get rerouted to console, terminals, and ssh sessions.

[root@centos ~]# logger -p emerg “Hi, This is a test message”

Logger with priority

3.3 Send messages with tags.

The thing you have noticed above is that messages are coming as general messages with the user name and the process id of the logger command. We need to make these messages more readable also we should be able to find which process is logging that message. This will help you when you have let’s say 2 or more microservices and you are maintaining the central logging for all services in which all logs get logged to the same logfile. rsyslog and its interface to all programming languages provide the liberty to add tags to messages. from CLI you can check with the following command.

[root@centos ~]# logger -t myapp -p emerg “Hi, This is a test message”

Logger with tag

Let’s send message to logfile.

logger with tag in logfile

Noticed that We have tagged the message with “my_App” which stands this message out in the logfile when there are thousands of logs present. If we have multiple services logging to the same file we can configure this tag to process the name.