ProFTPD module mod_redis

## ProFTPD module `mod_redis`

The `mod_redis` module enables ProFTPD support for caching data in Redis servers, using the hiredis client library.

## Directives

- RedisEngine
- RedisLog
- RedisLogOnCommand
- RedisLogOnEvent
- RedisOptions
- RedisSentinel
- RedisServer
- RedisTimeouts

### RedisEngine

Syntax: RedisEngine on|off
Default: RedisEngine off
Context: server config, `<VirtualHost>`, `<Global>`
Module: mod_redis
Compatibility: 1.3.6rc5 and later

The `RedisEngine` directive enables or disables the `mod_redis` module, and thus the configuration of Redis support for the `proftpd` daemon.

### RedisLog

Syntax: RedisLog path|"none"
Default: None
Context: server config, `<VirtualHost>`, `<Global>`
Module: mod_redis
Compatibility: 1.3.6rc5 and later

The `RedisLog` directive is used to specify a log file for `mod_redis`'s reporting on a per-server basis. The file parameter given must be the full path to the file to use for logging.

Note that this path must not be to a world-writable directory and, unless `AllowLogSymlinks` is explicitly set to on (generally a bad idea), the path must not be a symbolic link.

### RedisLogOnCommand

Syntax: RedisLogOnCommand "none"|commands format-name [key]
Default: None
Context: server config, `<VirtualHost>`, `<Global>`, `<Anonymous>`, `<Directory>`
Module: mod_redis
Compatibility: 1.3.6rc5 and later

The `RedisLogOnCommand` directive configures the use of Redis for logging. Whenever one of the comma-separated list of commands occurs, `mod_redis` will compose a JSON object, using the `LogFormat` named by format-name as a template for the fields to include in the JSON object. The JSON object of that event will then be appended to a list stored in Redis, using format-name as the default key name. Multiple `RedisLogOnCommand` directives can be used, for different log formats for different events.

The optional key parameter, if present, specifies the value to use as the Redis key. This key parameter supports all of the `LogFormat` variables, thus you can use e.g.:

      RedisLogOnCommand RETR,STOR xferlog %m:ftpxfer:%u

More on the use of Redis logging, including a table showing how `LogFormat` variables are mapped to JSON object keys can be found here.

Example:

###

###

###

###

###

##

##

      LogFormat file-transfers "%h %l %u %t \"%r\" %s %b"

      # Only log to Redis for uploads in this directory

        RedisLogOnCommand APPE,STOR,STOU file-transfers

      # Only log to Redis for downloads from this directory

        RedisLogOnCommand RETR file-transfers

      # ...but prevent Redis logging in this subdirectory

        RedisLogOnCommand none

    Note that RedisLogOnCommand does not currently
    support the logging classes that the ExtendedLog directive
    supports.

    RedisLogOnEvent
    Syntax: RedisLogOnEvent "none"|events format-name [key]

    Default: None

    Context: server config, <VirtualHost>, <Global>, <Anonymous>, <Directory>

    Module: mod_redis

    Compatibility: 1.3.7rc1 and later

    The RedisLogOnEvent directive configures the use of Redis for
    logging.  Whenever one of the comma-separated list of events
    occurs, mod_redis will compose a JSON object, using the
    LogFormat named by
    format-name as a template for the fields to include in the
    JSON object.  The JSON object of that event will then be appended to a list
    stored in Redis, using format-name as the default key name.  Multiple
    RedisLogOnEvent directives can be used, for different log formats
    for different events.

    The optional key parameter, if present, specifies the value to use
    as the Redis key.  This key parameter supports all of the
    LogFormat variables, thus you can use e.g.:

      RedisLogOnEvent READ ftp.download.%u.%m:%f

    More on the use of Redis logging, including a table showing how
    LogFormat variables are mapped to JSON object keys can be found
    here.

    Example:

      LogFormat sessions "%{iso8601} %a"
      RedisLogOnEvent CONNECT,DISCONNECT sessions

    In addition to specific FTP commands, the events list can specify
    "ALL", for logging on all commands.  Or it can include the
    "CONNECT" and "DISCONNECT" events, which can be useful for logging the
    start and end times of a session.  Note that
    RedisLogOnEvent does support the logging classes
    that the ExtendedLog directive supports.

    RedisOptions
    Syntax: RedisOptions opt1 ...

    Default: None

    Context: server config, <VirtualHost>, <Global>

    Module: mod_redis

    Compatibility: 1.3.7rc1 and later

    The RedisOptions directive is used to configure various optional
    behavior of mod_redis.

    Example:

      RedisOptions NoReconnect

    The currently implemented options are:

      NoReconnect

        If the connection to Redis breaks unexpectedly, mod_redis
        will attempt to reconnect automatically.  Use this option to
        disable the automatic reconnection.

    RedisSentinel
    Syntax: RedisSentinel host1 ... ["master" name]

    Default: None

    Context: server config, <VirtualHost>, <Global>

    Module: mod_redis

    Compatibility: 1.3.7rc1 and later

    The RedisSentinel directive is used to configure a list of IP
    addresses/ports of Redis Sentinels that the mod_redis module is
    to use, for discovering the location of named master database.  For
    example:

      # Configure two Sentinels; the first master discovered will be used
      RedisSentinel 1.2.3.4:26379 5.6.7.8:36379

      # Configure two Sentinels on the default Sentinel port, and
      # look for the location of the master named "proftpd".
      RedisSentinel 1.2.3.4 5.6.7.8 master proftpd

      # Configure three Sentinels including an IPv6 address, with explicit ports,
      # and look for the location of the "proftpd" master.
      RedisSentinel redis1.example.com:26379 1.2.3.4:26379 [::1]:26379 master proftpd

    The RedisSentinel directive can be used instead of the
    RedisServer directive, for discovering the Redis server to use.
    However, if your Redis server requires authentication, or
    supports multiple databases, then you will need to use
    RedisServer as well:

      # Use this to configure our password and database
      RedisServer 127.0.0.1:6379 redisr0cks 2

      # And use Sentinels to discover the true address/port
      RedisSentine 1.2.3.4 5.6.7.8 9.10.11.12

    RedisServer
    Syntax: RedisServer server [password] [db-index]

    Default: None

    Context: server config, <VirtualHost>, <Global>

    Module: mod_redis

    Compatibility: 1.3.6rc5 and later

    The RedisServer directive is used to configure the IP address/port
    of the Redis server that the mod_redis module is to use.  For
    example:

      RedisServer 1.2.3.4:6379

    or, for an IPv6 address, make sure the IPv6 address is enclosed in square
    brackets:

      RedisServer [::ffff:1.2.3.4]:6379

    Alternatively, you can configure a Unix domain socket path using e.g.:

      RedisServer /var/run/redis.sock

    An optional password parameter can be provided, for Redis servers
    which are password protected.

    As of ProFTPD 1.3.7rc1, the optional db-index parameter can be
    provided, for selecting the server-side Redis database by index:

      RedisServer 1.2.3.4:6379 passwd 2

    Connecting to a Redis server without password authentication but still
    selecting the database would be done using the empty string for the password:

      RedisServer 1.2.3.4:6379 "" 2

    RedisTimeouts
    Syntax: RedisTimeouts connect-millis io-millis

    Default: RedisTimeouts 500 500

    Context: server config, <VirtualHost>, <Global>

    Module: mod_redis

    Compatibility: 1.3.6rc5 and later

    The RedisTimeouts directive configures timeouts to be used
    when communicating with the Redis server.  The connect-millis
    parameter specifies a timeout, in milliseconds, to use when first
    connecting to the Redis server.  The io-millis parameter specifies
    a timeout, in milliseconds, to use both when sending commands to Redis, and
    when reading responses.

    The default is 500 milliseconds for both timeouts:

      RedisTimeouts 500 500

    Installation
    The mod_redis module is distributed with ProFTPD.  To enable
    support and use of the Redis protocol in your proftpd daemon,
    use the --enable-redis configure option:

      $ ./configure --enable-redis ...
      $ make
      $ make install

    This option causes the mod_redis module to be compiled into
    proftpd.

    You may also need to tell configure how to find the
    hiredis header and library files:

      $ ./configure --enable-redis \
        --with-includes=/path/to/hiredis/include \
        --with-libraries=/path/to/hiredis/lib

    Usage

    Configuring Redis for use by other modules, e.g. mod_ban
    or mod_tls_redis:

      <IfModule mod_redis.c>
        RedisEngine on
        RedisLog /var/log/ftpd/redis.log
        RedisServer 127.0.0.1:6379
      </IfModule>

    This example shows the use of Redis logging for all commands:

      <IfModule mod_redis.c>
        RedisEngine on
        RedisLog /var/log/ftpd/redis.log
        RedisServer 127.0.0.1:6379

        LogFormat redis "%h %l %u %t \"%r\" %s %b"
        RedisLogOnCommand ALL redis
      </IfModule>

    Redis Logging

    When using Redis logging, the following table shows how mod_redis
    converts a LogFormat variable into the key names in the JSON
    logging objects:

        LogFormat Variable
        Key

         %A
        anon_password

         %a
        remote_ip

         %b
        bytes_sent

         %c
        connection_class

         %D
        dir_path

         %d
        dir_name

         %E
        session_end_reason

         %{epoch}
        Unix timestamp, in seconds since Jan 1 1970.

         %{name}e
        ENV:name

         %F
        transfer_path

         %f
        file

         %{file-modified}
        file_modified

         %g
        group

         %{gid}
        gid

         %H
        server_ip

         %h
        remote_dns

         %I
        session_bytes_rcvd

         %{iso8601}
        timestamp

         %J
        command_params

         %L
        local_ip

         %l
        identd_user

         %m
        command

         %{microsecs}
        microsecs

         %{millisecs}
        millisecs

         %{note:name}
        NOTE:name

         %O
        session_bytes_sent

         %P
        pid

         %p
        local_port

         %{protocol}
        protocol

         %r
        raw_command

         %S
        response_msg

         %s
        response_code

         %T
        transfer_secs

         %t
        local_time

         %{transfer-failure}
        transfer_failure

         %{transfer-status}
        transfer_status

         %U
        original_user

         %u
        user

         %{uid}
        uid

         %V
        server_dns

         %v
        server_name

         %{version}
        server_version

         %w
        rename_from

    In addition to the standard LogFormat variables, the
    mod_redis module also adds a "connecting" key for events
    generated when a client first connects, and a "disconnecting" key for events
    generated when a client disconnects.  These keys can be used for determining
    the start/finish events for a given session.

    Here is an example of the JSON-formatted records generated, using the above
    example configuration:

      {"connecting":true,"timestamp":"2013-08-21 23:08:22,171"}
      {"command":"USER","timestamp":"2013-08-21 23:08:22,278"}
      {"user":"proftpd","command":"PASS","timestamp":"2013-08-21 23:08:22,305"}
      {"user":"proftpd","command":"PASV","timestamp":"2013-08-21 23:08:22,317"}
      {"user":"proftpd","command":"LIST","bytes_sent":432,"transfer_secs":4.211,"timestamp":"2013-08-21 23:08:22,329"}
      {"user":"proftpd","command":"QUIT","timestamp":"2013-08-21 23:08:22,336"}
      {"disconnecting":true,"user":"proftpd","timestamp":"2013-08-21 23:08:22,348"}

    Notice that for a given event, not all of the LogFormat
    variables are filled in.  If mod_redis determines that a given
    LogFormat variable has no value for the logged event, it will
    simply omit that variable from the JSON object.

    Another thing to notice is that the generated JSON object ignores the textual
    delimiters configured by the LogFormat directive; all that
    matters are the LogFormat variables which appear in the directive.

    Frequently Asked Questions

    Question: How can I convert this SQL logging into the
    equivalent Redis logging?

      SQLNamedQuery upload FREEFORM "INSERT INTO ftplogs ('userid', 'server_ip', 'transfer_date', 'operation', 'protocol', 'client_ip', 'transfer_time', 'bytes_transfer', 'file_hash_type', 'file_hash', 'file_path', 'transfer_status') VALUES ('%u', '%H', NOW(), '%r', '%{protocol}', '%a', '%T', '%b', '%{note:mod_digest.algo}', '%{note:mod_digest.digest}', '%f', '%{transfer-status}')"
      SQLLog STOR upload

    Answer: Since the JSON object key names are hardcoded
    in mod_redis, converting the above SQLNamedQuery
    into a suitable/matching LogFormat is the necessary step.  Thus
    for example it might become:

      LogFormat upload "%u %H %{YYYY-MM-DD HH:MM:SS}t %r %{protocol} %a %T %b %{note:mod_digest.algo} %{note:mod_digest.digest} %f %{transfer-status}"
      RedisLogOnCommand STOR upload

    Note that LogFormat does not provide a NOW()
    function, unlike many SQL databases, thus the %t variable is
    needed to provide/fill in that timestamp.

    © Copyright 2017 The ProFTPD Project

     All Rights Reserved