How to Centrailize Apache Web Server Logs with Ryslog

Centralizing Apache Logs

Table of Contents

By default, Apache stores all logs on the local disk. This works well for development environments and small deployments but can become unsustainable once you have more than one server. Not only is it frustrating having to open each log file on each server, but trying to trace requests across multiple servers can quickly become error prone and time-consuming.

Log centralization services help prevent this by allowing you to store logs from your Apache servers in a single location, making it possible to view all of your web logs without having to open each log file individually.

Syslog vs Syslog-NG vs Rsyslog

Although functionally they are all the same, as all of them permit the logging of data from different types of systems in a central store/repository. But as software components they are three different projects, each project trying to improve the previous one with more reliability and functionalities.

Syslog

The syslog project was the very first project. It started in 1980. It is the root project to syslog protocol. At this time syslog is a very simple protocol. At the beginning it only supports UDP for transport, so that it does not guarantee the delivery of the messages.

System Logging Protocol (syslog) is a logging service commonly found on Linux, Unix, and Mac systems. Syslog handles logs from various sources, including applications, system services, daemons, and hardware. Syslog is reliable, standardized, and can forward your logs to another syslog server.

Syslog-NG

Next came syslog-ng in 1998. It extends basic syslog protocol with new features like:

Content-based filtering
Logging directly into a database
TCP for transport
TLS encryption

Rsyslog

Next came rsyslog in 2004. It extends syslog protocol with new features like:

RELP protocol support.
Buffered operation support.

Let’s say that today they are three concurrent projects that have grown separately upon versions, but also grown in parallel regarding what the neighbors was doing.

Note: In this post I will refer to all of these generally as syslog.

Different way of Configuring Apache and Syslog

Apache can be configured to write to a file, syslog and to a pipe. I will examine some of these options below.

Option 1: Configuring Apache Web Server to Log using Syslog

I am only describing this option to ensure you setup is valid and the Apache domain you are working with is setup correctly. If you are sure everything is working as expected you can skip this option.

In order to start logging to syslog, you need to change the settings of the ErrorLog directive in the configuration file. Change the following:

ErrorLog ${APACHE_LOG_DIR}/error.log

to:

ErrorLog syslog

This will start sending all error log entries to local syslog.

To consolidate Apache logs, I actually need to send these log entries to a remote syslog server for centralized server. I will be doing this in the section below.

Option 2: Configure Apache Web Server to Log to Remote Syslog with File Monitoring

A common approach to reading Apache logs is to configure syslog with file monitoring. With file monitoring enabled, syslog periodically scans a file on the system for changes, then imports those changes into its own log file. The benefit is you get the complete original log message wrapped in the standard syslog message format without modifying the original file.

The rsyslog text file input module, imfile, provides the ability to convert any standard text file into a syslog message. This module can read a log file line by line while passing each read line to rsyslog engine rules, applying filter conditions and executing required actions.

Note: Empty lines are ignored and thus not processed.

I need to do two tasks to make it work.

Create syslog (or rsyslog) config for Apache.
Update the Apache host config and change ErrorLog to use our variables.

Step 1: Create rsyslog Configuration for Apache

In order to use Rsyslog text file input module to read Apache log file and forward it to remove Rsyslog server, you would create a configuration file like /etc/rsyslog.d/02-apache2.conf, with the content below:

module(load="imfile" PollingInterval="10" statefile.directory="/var/spool/rsyslog")
input(type="imfile"
      File="/var/log/apache2/error.log"
      Tag="http_error"
      Severity="error"
      Facility="local6")
local6.error        @10.0.0.155:514

module

This specifies the module rsyslog is to load. In this case it is imfile set with attribute load.

Second attribute PollingInterval is set to 10 which specifies how often the error log file is read. Setting it lower means a lot more resource consumption, such as busy CPU while setting it higher may lead to higher network activity and potential loss of data. This value should not be set to 0 which will usually take CPU usage to 100% depending on the load on your server.

Final option for the module is statefile.directory, specifying syslog location.

input

The input option similarly comes with its own attributes which you need to set, first of which is the type. I have set it to imfile.

File attribute sets the location of the local Apache error log file to be polled

Tag attribute is used to assign a tag to messages read from the error log file.

Severity attribute is set to assign a severity for error messages.

Facility attribute is a syslog facility assigned for error log messages read from the file.

local6.error

This specifies the location of the rsyslog server where the error log messages will be sent. In my case my rsyslog server is running on IP 10.0.0.155.

Validate Rsyslog Configuration

With the new .conf file in place for Apache the first thing I do is validate the config and then restart the rsyslog server.

### Validate rsyslog Apache configuration

root@d1:~# rsyslogd -N1 -f /etc/rsyslog.d/02-apache2.conf
rsyslogd: version 8.2112.0, config validation run (level 1), master config /etc/rsyslog.d/02-apache2.conf
rsyslogd: End of config validation run. Bye.

### Restart rsyslog service

root@d1:~# service rsyslog restart
root@d1:~#

You can also restart rsyslog using the following command. Although I personally use the service option above.

root@d1:~# systemctl restart rsyslog

Let’s now move to configuring Apache host configuration to use our rsyslog setup.

Step 2: Update Host Configuration for Apache Domain to use Remote rsyslog

To forward Apache logs to remote syslog server I need to set the ErrorLog directive to pipe the logs to logger which can then write to a specific syslog facility, with the appropriate severity.

In this example I will be forwarding Apache logs to a remote server which I configured in step 1.

Change the line from your Apache virtual host config file from:

ErrorLog ${APACHE_LOG_DIR}/error.log           # This is the default. Maybe different for you if changes were made.

Errorlog "| /usr/bin/logger -thttp_error: -plocal6.err"      # ErrorLog is now using our new rsyslog configuration.

What our Error log entry does is to pipe (|) the logs to with the command logger and send to the tag https_error (with option -t) to the syslog priority, set with -p, of local6.err.

Once the changes are made to the Apache virtual host configuration, test the files for errors and restart the Apache web server. Use the command below:

### Test Apache Configuration

root@d1:~/# apachectl -t
Syntax OK

### Restart Apache web server to use new configuration

root@d1:~# sudo service apache2 restart

You should now be able to receive the logs at the remote central log server, assuming it is up and running.

Do note that default server configuration for rsyslog does not listen on TCP and UDP ports 514. You will need to enable this feature for receiving logs on remote server. Check the FAQ at end of this post on how to enable remote server rsyslog message acceptance.

One thing to note with this setting, ErrorLog “| /usr/bin/logger -thttp_error: -plocal6.err”, it doesn’t to logs the local system log file and hence, there is a high possibility that some logs may be lost.

To overcome this and log the Apache logs to both a local file and a remote server, use the tee command to pipe logs to logger and to the local log file as shown below:

ErrorLog "|/bin/sh -c '/usr/bin/tee -a /var/log/apache2/error.log | /usr/bin/logger -thttp_error: -plocal6.err'"
CustomLog ${APACHE_LOG_DIR}/access.log combined

Configure Rsyslog to sent logs on priority local6.err to remote log server.

Restart both Rsyslog and Apache:

root@d1:~/# systemctl restart rsyslog apache2

You should now be able log to the local file and to the remote rsyslog server.

Validating Reception of Apache Logs on Remote Server

There are multiple ways to validate that the logs are being synced.

Validate Remote Logging Using TcpDump Command

One of those is by using tcpdump to see if data is being sent over the network interface on the specific port.

### First use ip command to find out your network interface.

root@d1:~# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 00:0c:29:a7:b8:4a brd ff:ff:ff:ff:ff:ff
    altname enp2s1

### In my case it is ens33.
### Now use tcpdump to check traffic on the port

tcpdump -i ens33 src host 10.0.0.118 and udp port 514 -nn -vv

### Now send a request through curl to generate an error.

root@d1:~# curl http://localhost:90/index.php

In my case the output of tcpdump is as shown below:

root@d1:~# tcpdump -i ens33 src host 10.0.0.118 and udp port 514 -nn -vv

tcpdump: listening on ens33, link-type EN10MB (Ethernet), snapshot length 262144 bytes
14:41:03.670850 IP (tos 0x0, ttl 64, id 64780, offset 0, flags [DF], proto UDP (17), length 224)
    10.0.0.118.55423 > 10.0.0.155.514: [bad udp cksum 0x15ee -> 0xc227!] SYSLOG, length: 196
	Facility local6 (22), Severity error (3)
	Msg: Nov  9 14:41:03 d1 http_error:: [Wed Nov 09 14:41:03.669604 2022] [php:error] [pid 43159] [client 127.0.0.1:54624] PHP Fatal error:  Cannot divide by zero in /var/www/html/index.php on line 2
	0x0000:  3c31 3739 3e4e 6f76 2020 3920 3134 3a34
	0x0010:  313a 3033 2064 3120 6874 7470 5f65 7272
.........

Monitor Remote Rsyslog Server for Apache ErrorLog Messages

Logon to the remote server setup to centralize and consolidate Apache logs and type the following command to tail the syslog file.

### Tail the log file.

root@d1:~# tail -f /var/log/syslog

Then issue a curl to access a resource that results in an error.

oot@d1:~# curl http://localhost:90/index.php

### My index.php has the following two lines to generate an error ###
### <?php
### trigger_error("Cannot divide by zero", E_USER_ERROR);

View error log entry in the log tailing window. I get the following:

Nov 9 14:24:39 d1 http_error: : [Wed Nov 09 14:24:39.612605 2022] [php:error] [pid 43156] [client 10.0.0.118] PHP Fatal error: Cannot divide by zero in /var/www/html/index.php on line 2

There you have it. Apache web server sending error log messages to a remote rsyslog server setup for Apache log centralization.

More info on Piping Apache Logs to Syslog Through Logger

Apache httpd is capable of writing both error and access log files through a pipe to another process, rather than directly to a file. This capability dramatically increases the flexibility of logging, without adding code to the main server.

In order to write logs to a pipe, simply replace the filename with the pipe character “|“, followed by the name of the executable which should accept log entries on its standard input. The server will start the log process when it starts, and will restart it if it crashes. This technique is what some call to as reliable piped logging.

Piped log processes are spawned by the parent Apache httpd process, and inherit the userid of that process. This means that piped log programs usually run as root. It is therefore very important to keep the programs simple and secure.

One important use of piped logs is to allow log rotation without having to restart the server. The Apache HTTP Server includes a simple program called rotatelogs for this purpose. To rotate the logs every 24 hours, you can use:

CustomLog "|/usr/local/apache/bin/rotatelogs /var/log/access_log 86400" common

Note: Quotes, ", are used to enclose the entire command that will be called for the pipe. This technique can be used for either access or error log.

By default the piped log process is spawned without invoking a shell. Use “|$” instead of “|” to spawn using a shell, as with /bin/sh -c:

# Invoke "rotatelogs" using a shell
CustomLog "|$/usr/local/apache/bin/rotatelogs   /var/log/access_log 86400" common

Using this technique I earlier showed you how to send logs to remote rsyslog servers. This can have advantages on slower storage devices but comes with the risk that you can lose messages you send to the log consolidation server.

Although I suggest you use local file copies as well, but in case you chose not then then use the sizeparameter option to optimize the maximum message size for log syncing.

Now your logs will no longer be written to the access.log and error.log files but will instead go straight to syslog. If you want to continue logging to file as well as syslog, you can use the following configuration instead—this uses the tee command first to pipe the log message to file, then pipes the output from that command to logger:

ErrorLog  "|/usr/bin/tee -a /var/log/apach2/error.log  | /usr/bin/logger -thttpd_error -plocal6.err"
CustomLog "|/usr/bin/tee -a /var/log/apache2/access.log | /usr/bin/logger -thttpd_error -plocal6.notice" combined

Conclusion

With multiple servers serving a single domain it becomes very important to consolidate those logs for easier analysis, debugging and storage. Using ErrorLog directive options with rsyslog facility you are able to send log file entries to remote servers.

Let me know in comments below if you have any questions.

FAQ

Remote Rsyslog Server not Accepting Apache Log Messages

On Ubuntu and other Linux based distributions it maybe the case that remote message port 514 is not enabled by default. You will need to change your rsyslog configuration.

Follow the steps below.

### Edit the rsyslog file

root@s1:~# vi /etc/rsyslog.conf    # This is on Ubuntu

Under the MODULES section of the file, uncomment and enable imudp and imtcp.

# provides UDP syslog reception
module(load="imudp")
input(type="imudp" port="514")

# provides TCP syslog reception
module(load="imtcp")
input(type="imtcp" port="514")

Then restart the rsyslog process.

root@d1:~# service rsyslog restart

How to Send Apache Logs to Centralized rsyslog Server