Centralized Syslog

For quite some time, I've wanted to get all of my logs in one place. Not just for easier post-disaster analysis, or forensics, but also so I could use some good log analysis, like Splunk or Swatch or something.

Syslog-ng Configuration

Server

source s_remote_logs {
        udp();
        tcp(ip(192.168.0.8) port(5000) max-connections(300));
};
destination d_remote_logs {
   file ( "/var/log/HOSTS/$HOST/$YEAR/$MONTH/$FACILITY-$HOST-$YEAR-$MONTH-$DAY.log"
   owner( root ) group( root ) perm ( 0600 ) dir_perm( 0700 ) create_dirs (yes )
   template ( "$DATE $HOST $PROGRAM $TAG [$FACILITY.$LEVEL] $MESSAGE\n" ));
};
# log them
log {
    source(s_remote_logs);
    destination(d_remote_logs);
};

This should accept logs from any host via UDP or TDP port 5000, and dump them in a directory and file named like '/var/log/HOSTS/$HOST/$YEAR/$MONTH/$FACILITY-$HOST-$YEAR-$MONTH-$DAY.log', so we have separation not only of each host on a month-by-month basis in directories, but separate files for each facility's logs within a month, on a daily basis. Remember to punch a hole in the firewall on the server for port 5000 and UDP port 514 (for syslog (not -ng) compatibility).

Client

I added a rule like:

filter f_NOTiptables   { not filter(f_iptables); };
destination r_dest    { tcp ("192.168.0.8", port(5000)); };
log { source(src); filter(f_NOTiptables); destination(r_dest); };

This logs everything *but* iptables, as the firewall tends to generate a *lot* of information. And since the box is on a trusted LAN, I didn't see much reason to need the firewall logs saved remotely.

Remote Clients

As per the article here, I'm using SSH to allow syslog from remote boxes. I did, however, switch around his logic and open the firewall hole on the server side, with the remote box opening the SSH tunnel. This was just to minimize the amount of configuration needed for adding remote hosts.

First, add something like the following to /etc/inittab:

log1:3:respawn:/usr/bin/ssh -nNTx -i /root/.ssh/id_dsa_nopass -L 5000:loghost:5000 loguser@sshGateway.example.com >/dev/null 2>&1

Then add the same directives to /etc/syslog-ng/syslog-ng.conf as for the regular client, substituting 127.0.0.1 for the destination IP, as we'e using an SSH tunnel connecting port 5000 on localhost to 5000 on the remote loghost.