1      Automated Event Handling With Logtrigger

1.1                About Logtrigger

Logtrigger is a programmable syslog (system log) parser that monitors logged events, triggering on (detecting) user defined patterns and executing a user defined script when the event happens.

Logtrigger is a lightweight replacement for the Linux Fail2Ban log scanner, the primary purpose of which is to detect invalid login attempts and block (ban) any IP address that is attempting invalid logins. Another use is to detect hacker Denial of Service (DOS) attacks by banning their IP address.

Logtrigger parses every system log event, looking for user configurable patterns, extracts user defined variables and passes them as environment variables to a user defined shell script, for further action, in real-time, as events occur. It is also possible to configure logtrigger to parse other log files.

Possible Logtrigger Event handlers (scripts) are:

• Block offending IP address using firewall commands. Included with SecureOffice.
• Log any syslog event class to a dedicated logfile. Included with SecureOffice
• Send an email for events, containing event parameters (write your own)
• Custom scripts can be created to handle any event in syslog such as security device alarms (motion detectors, IP cameras), IOT and home automation device events.
• In general, any syslog event can be detected by logtrigger and any arbitrary event handler script can be called to perform any action.

For SecureOffice, logtrigger is installed as a dependency of SecurePBX and performs the following functions:

Detecting and blocking IP addresses from:

• Five failed SIP registrations (invalid login attempts from a particular user) in a row, within ten minutes.
• Five failed SIP invites from a particular user in a row, within ten minutes.
• Five failed login attempts in a row, within ten minutes.

Logging important system events into dedicated log files:

• All DDNS (Dynamic Domain Name Service) failure and success messages to "/var/log/ddns.log". Useful for diagnosing DDNS problems, where DNS lookup is not returning the proper IP address for your domain.
• All failed login, SIP registration and SIP invite attempts to "/var/log/failed_logins.log". Useful to know who attempts to log in, when and their IP address. Any entry in this logfile has had their IP address blocked.
• All successful logins and SIP registrations to "/var/log/logins.log". Useful to know who logs in, when and their IP address.
• All blocked IP addresses are logged to "/var/log/blockedIPs.log".
• All dropbear events to "/var/log/dropbear.log". Dropbear events are security related, for example when users logout.

If, for some reason, a valid IP address ends up blocked (for example, during SIP client configuration, incorrect password), a firewall restart is required ("/etc/init.d/firewall restart") to unblock previously blocked IP addresses.

1.2                Variable Extraction From Log Events

Logtrigger, in addition to exact pattern matching (triggering) on strings is able to match and extract variables using format specifiers within the strings.

Logtrigger format specifiers are a subset of the standard format specifiers of the scanf() system interface (C function). Logtrigger format specifiers are used as wild cards for pattern matching (match if format matches) and, optionally to extract the variable values represented by the format field for use as environment variables to be passed to logtrigger action scripts, which can conditionally execute, based on variable values.

The format specifiers implemented by logtrigger are:

"%d, %i, %u, %f, %o, %x. %X, %c, %s, %b"

The meaning of the format specifiers is documented by the scanf() reference here. Note that logtrigger implements format specifiers only, with no optional modifiers.

An example pattern string is: "The %s ate my homework and, I know this is the %d th time this has happened."

If this string is logged: "The dog ate my homework and, I know this is the 10 th time this has happened.", logtrigger will match on the pattern. Optionally, if field variables (what, times) are defined, the following environment variables will be passed to the action script:

• LT_what (dog)
• LT_times (10)

Defining and using field variables is discussed in the next section.

1.3                Trigger Configuration

The logtrigger configuration file "/etc/config/logtrigger" specifies rules defining trigger conditions and actions (scripts to run) that occur when the trigger conditions are met. There is no OpenWrt GUI for logtrigger, it must be configured by command line and an editor. The configuration file is liberally commented and contains many examples.

Logtrigger optionally performs variable extraction from the pattern, when it occurs.

An example trigger rule is:

This rule instructs logtrigger to parse every syslog event for the pattern and, when detected, extract variables (fields) and, if supplementary conditions are met, call the script with the following environment variables:

• LT_message (the trigger message)
• LT_name (the name of the matching trigger rule)
• LT_user (whatever string is in place of the first %s)
• LT_ip (whatever string is in place of the second %s)

The LT_user and LT_ip environment variables are a consequence of the (optional) "fields" option. For every format code in the trigger string, field variables are assigned one to one, the contents of the first format code to the first field variable and so on.

The supplementary conditions are:

• maxfail: The event must have happened maxfail times before triggering.
• time (optional): All trigger events must have occurred within time seconds before triggering.
• enabled (0/1): enabled must be "1" before triggering.

There is an optional logtrigger rule option that can be used to monitor events from another logfile. If this option is omitted, logtrigger defaults to using syslog:

• option file "full path of file to monitor"

The "pattern" option (log entry to trigger on) does not have to specify from the beginning of the log entry, so long as the log entry contains the pattern. The pattern must completely specify to the end of the trigger string, or the pattern will not trigger. For example:

pattern: "The %s ate my homework and, I know this is the %d th time this has happened."

Syslog event #1: "The dog ate my homework and, I know this is the 10 th time this has happened."

Syslog event #2: "The dog ate my homework and, I know this is the 10 th time this has happened. Sorry, won't happen again (heh heh)."

Logtrigger will trigger on syslog event #1 because the event matches the pattern to end of line.

Logtrigger will NOT trigger on syslog event #2 because the event does NOT match the pattern to end of line.

To detect both events, change the pattern to the following:

pattern: "The %s ate my homework and, I know this is the %d th time %s"

This will match any event containing the pattern followed by arbitrary text.

1.4                Trigger Scripts

Trigger scripts are shell scripts containing commands to execute. An example trigger script is "/etc/logtrigger/logit.sh" (log individual event classes to dedicated logfiles), shown below.

This script is passed the $LT_name (name of trigger event from config file) environment variable which it uses to determine which log file to log the event to. If additional trigger events and log files are desired, the triggers must be defined in the logtrigger config file and an additional "elif" clause added to the action script (logit.sh).

1.5                Recovering from Banned IP Addresses

If the situation occurs where your IP address has been banned due to incorrect password attempts and it is desired to undo this, a firewall restart is required.

From a SSH or console shell, enter: "/etc/init.d/firewall restart"

This will remove all firewall IP blocking rules.

Currently IP blocking happens under the following conditions:

• Five failed SIP registrations (invalid login attempts from a particular user) in a row, within ten minutes.
• Five failed SIP invites from a particular user in a row, within ten minutes.
• Five failed login attempts in a row, within ten minutes.

1.6                Testing and Debugging Triggers

It is tedious to test triggers, especially for rare events or events which require complex setup to make them occur. To avoid this work, it is best to have a copy of the syslog string you wish to trigger on and, use the logger utility to send it to syslog, allowing logtrigger to parse it.

For example, you wish to monitor, trigger on and log DHCP success events. An example syslog entry is:

"daemon.info dnsmasq-dhcp[4273]: DHCPACK(br-lan) 192.168.2.153 'mac address' 'computer name'"

In the trigger, the pattern would be defined as follows, replacing all variable (changing) fields of the trigger message with format codes:

option pattern "dnsmasq-dhcp[%d]: DHCPACK(%s) %s"

Once you have defined your trigger event and action script and are ready to test it, enter the following command:

logger "the message that should trigger"

If you are fortunate, the action script should have executed. If not, you will have to determine why. Logtrigger has a debug mode that will greatly assist in trouble shooting.

Disable logtrigger background execution: "/etc/init.d/logtrigger stop"

Start logtrigger in the foreground, in debug mode: "logtrigger -D N", where "N" is the debug verbosity desired. A value of "2" will show you what triggered and associated information such as environment variables passed to the trigger script. A value of 999 will give you information overload.

From another SSH console, repeat the above logger step, consider the debug information and solve any problems.

Note: Any changes to trigger configuration require logtrigger to be restarted:

From another console enter "killall logtrigger", or from within the logtrigger foreground console "CTL+C" (together). Then, start logtrigger again.

Once the triggers and action scripts are working properly, stop the foreground instance of logtrigger and restart logtrigger in the background:

"/etc/init.d/logtrigger start"