There are different ways to view and track how your systems are operating. Logging is a basic method of keeping records of actions taken by a network device. Most operating systems have some type of logging function. In linux, these log files are mostly processed with syslog. Syslog supports logging to remote hosts, which is useful if a system fails and local logs aren’t available. IOS and Windows take some work to get them to use the same feature. By and large, syslog is a useful tool to have. It provides a view into the health and operation of your systems.

In this post, I’ll be covering the use of syslog on linux systems. It will be used as a central collection point for remote clients that send their syslog data. Several devices and operating systems support syslog, so it isn’t limited to just computers.

The first linux syslog service I’ll cover is rsyslog. This is a default daemon loaded on some distros. Syslog messages are processed by rsyslog and sent to the local file system for storage, or to a remote syslog server. The structure of the message follows a convention, here is an example

Date Time Source Facility Message
Jan  1 12:01:01 192.168.0.101 webmin[987]: Webmin starting

Based on the convention above, I can tell who the message applies to based on the Source. On these remote hosts, I specify that I want to send my syslog messages to my central host. There many ways to do this, I won’t cover them all here. For linux systems, editing the /etc/rsyslog.conf file and entering in the following line will do it.

*.*;auth,authpriv.none    @192.168.0.102    # Send all syslog messages except auth or authpriv to remote host at IP address

After making that addition and saving the file, restart the rsyslogd on the client (the computer you are sending syslog messages from). The format of that command uses this format.

(facilities);(priorities)

These are really basic filters on what to choose to send to a remote syslog server. To filter messages beyond this, we will need to use another service called syslog-ng. Some distros do not have this installed by default, to do that on Ubuntu, run this command.

[bash]
sudo apt-get install syslog-ng
[/bash]

At this point, I would like to use Webmin for my configuration. Before going further, I’d like to back track to rsyslog filtering setup in Webmin. Open Webmin and go to System / System Logs. Click the “Add a new system log” link. Under “Log to”, select the Syslog server on radio button and enter the FQDN or IP of the syslog server to send messages to. Next set your Facilities and Priorities. In the example above, I have All selected for Facilities and Priorities. The next line below it I have Facilities Many auth authpriv selected and Priorities None chosen. I click Save and on the main page I click Apply Changes. Now I’ll cover the same thing in syslog-ng.

From Webmin, go to System / System Logs NG. Lets specify a Log Destination, click that link. Click add a new log destination. In the Log to section, select the syslog server radio button and enter in the IP address in the server address field. Set the network protocol to UDP. The last thing to do before saving is entering in the log destination name, I like to use d_remote_IP_of_server. Now we want to set our filtering rule, similar to what we did with rsyslog. Click the log filters link from the main syslog-ng page. We are in luck, these are already defined. Notice that the names of the filters start with f_something. This is a good convention to follow as your destinations, filters, and sources grow. Now lets go back to the main syslog-ng page. Click log targets. Lets add a new log target. Select the s_src from the Log sources list. Under Target options, check the box process all sources. In Filters to apply, choose the f_syslog3 filter. Finally, in the destination files choose our newly created d_remote_IP_of_server from the list. Click save and then from the main syslog-ng page click the Start Syslog-NG link. Sometimes things don’t take and you may need to reboot. If you still have issues, take a look at this video for a more detailed look using the CLI to configure syslog-ng.

Windows hosts handle logging using Event Logs. To get these to pass to our linux syslog server, we’ll use a service called syslogwin. You can find more details from the site. The site had additional information regarding linux as well.

http://troy.jdmz.net/syslogwin/

Some prefer not to syslog messages on demand from Windows hosts. There is a way to dump event logs using Windows tool called Log Parser. This tool can be used to look for specific events and process them accordingly. The detail about Log Parser are far too great to fit in this post, see Microsoft’s weblink or buy the book from Amazon. This command will dump an event log to a syslog host.

[bash]
logparser.exe file:EventLogToSyslog.sql -i:EVT -o:SYSLOG
[/bash]

The file reference above is a query string that is contained in a plain text file. In this example the file contents is as such.

SELECT
     *
INTO 
     @FQDN_or_IP_of_SyslogServer:514 (514 is the default UDP Port)
FROM 
     system,
     application,
     security

Back to syslog-ng, I have a messages that I want to filter based on the content of the message. Here are the messages that I want.

May 10 11:17:51 Firewall_IP Wed, 2017-05-10 11:17:49 - TCP packet - Source: Source_IP - Destination: Destination_IP - [Service access request successful Src 44627 Dst 80 from WAN]
May 10 13:32:39 Firewall_IP Wed, 2017-05-10 13:32:38 - TCP packet - Source: Source_IP - Destination: Destination_IP - [Service access request successful Src 35824 Dst 80 from WAN]
May 10 14:39:57 Firewall_IP Wed, 2017-05-10 14:39:56 - TCP packet - Source: Source_IP - Destination: Destination_IP - [Service access request successful Src 44396 Dst 80 from WAN]
May 10 15:37:41 Firewall_IP Wed, 2017-05-10 15:37:40 - TCP packet - Source: Source_IP - Destination: Destination_IP - [Service access request successful Src 33263 Dst 80 from WAN]
May 10 19:55:39 Firewall_IP Wed, 2017-05-10 19:55:38 - TCP packet - Source: Source_IP - Destination: Destination_IP - [Service access request successful Src 49448 Dst 80 from WAN]
May 10 19:55:59 Firewall_IP Wed, 2017-05-10 19:55:58 - TCP packet - Source: Source_IP - Destination: Destination_IP - [Service access request successful Src 49499 Dst 80 from WAN]

The filter I want to use is “Service access request successful”. I would have difficulty in rsyslog, but syslog-ng can do it much easier, more so with webmin. From System / System Logs NG, I’ll start with defining a filter in log filters. Here I’ll create a new filter named f_message_firewall1. I’ll then use simple conditions and check to box next to match expression and place “Service access request successful” there. After clicking save, my filter policy is in place.

Next, I’ll want to create a unique log file for these messages. To do that I’ll click log destinations and create a new destination named d_firewall1. Here I’ll set the radio button to file and I’ll enter in the path of my new log file. I like to keep it in the same folder as my other logs. Note the selection Program input, we’ll be returning to this later. I’ll click save to finish the new destination policy.

Since our source policy works here, no need to change, remove, or add anything there. So the last thing to do is create a new log target. Here I select my default source. Next I’ll check the box for Process all sources under target options. The key items now will be selecting my new filter policy f_message_firewall1 and destination policy d_firewall1. I’ll click save and return to the System Logs NG screen to click Apply Configuration. I like to stop and start the Syslog-NG service just to make sure nothing fails. If it does, recheck your work.

Now I can view my new log file once those messages arrive. I did a sudo nano filepath/logfile to create my log file, just to be sure it was there. If your log file fails to create, try creating the file first.

As mentioned earlier, syslog-ng log destinations has an option to select program input. I’ll be using this in later posts to run programs to process the messages and take some form of action. This falls into the scope of alerts and notices, which I’ll cover in another post. It can also be used as an intrusion detection mechanism, which I may cover at a later time.

If you have stuck with me this far, good for you. Some of you might have thought, “Why not just use something like Nagios Log Server?!” The point of this post was to get into the nit and grit of syslog. Understanding its fundamentals can reveal much of its potential. If you prefer to cut to the chase, please see this presentation from SysAdmGirl. I won’t cover the setup or configuration of Nagios Log Server here. I will note that if you have Syslog-NG installed, it supersedes RSyslog, which NLS relies on. If you perform a default install of NLS on a Syslog-NG system, it will fail. Just keep that in mind.

Either way, syslog and the processing of syslog messages is a powerful toolset. It gives a global view of the landscape and can provide high level detail when processed correctly. I hope you have enjoyed this post and hope you join me again for later posts.