Syslog

Real time reporting of execution state is mandatory functionality for most of Hat’s components implementing continuously running service. Primary way of logging is based on Syslog logging protocol as defined by:

Each component which reports log messages should implement Syslog TCP client. In this way, all logging messages can be aggregated in a single Syslog concentrator which can provide archiving, searching and real time monitoring functionalities.

Care must be taken for Syslog TCP client logging facility implementation not to interfere with other component’s functionalities. Logging should be considered best effort and not critical activity of each component.

SysLogHandler

SysLogHandler provides implementation of Python’s standard library logging.Handler interface. This implementation is part of hat-syslog python package.

Each instance of hat.syslog.handler.SysLogHandler starts new background thread responsible for sending syslog messages over TCP, UDP or TCP+SSL socket. If connection with remote syslog server could not be established or current connection is closed, new connect is called after 5 second timeout.

All log messages provided to SysLogHandler by hat.syslog.handler.SysLogHandler.emit are queued in queue with maximum size limited by configuration parameter. During new log message registration, if queue is full, oldest messages are discarded and counter of discarded messages is incremented. Once new log messages can be sent to server, new log message containing information about number of discarded messages will be sent.

Syslog Generator

Syslog generator is application used as simple testing tool responsible for generating syslog messages that can be sent to Syslog Server.

Implemented as python hat.syslog.generator module which can be run with hat-syslog-generator script with additional command line arguments:

Usage: generator.py [OPTIONS]

  Syslog test message generator

Options:
  --host HOST     syslog server host name  [default: 127.0.0.1]
  --port PORT     syslog server port  [default: 6514]
  --count N       number of log messages  [default: 1]
  --text TEXT     log message text  [default: syslog generator test]
  --queue-size N  client's log message queue size  [default: 1024]
  --msg-delay T   time delay between message generation in seconds  [default:
                  0.01]

  --end-delay T   time delay affter all messages have been generated in
                  seconds  [default: 0.5]

  --help          Show this message and exit.

This application is part of hat-syslog python package.

Syslog server

Syslog server is central concentration for syslog messages. Additionally, it provides web interface for real time monitoring and filtering of log messages.

Syslog listening socket can be configured as TCP, UDP or TCP+SSL socket. Communication is based on RFC 5425, RFC 5426, RFC 6587. Once message is received, server stores message in predefined database.

Warning

Current implementation of Syslog server doesn’t support UDP communication

skinparam linetype ortho

folder Client {
    component SysLogHandler
}

folder "Syslog Server" {
    component SysLogServer
    component Backend
    component WebServer

    interface add
    interface query
    interface notify
}

folder Browser {
    component JS
    component GUI
}

database Sqlite

interface syslog
interface http
interface websocket

syslog - SysLogServer
Backend -- Sqlite
add - Backend
Backend - query
Backend - notify
http - WebServer
WebServer -- websocket

SysLogHandler --> syslog
SysLogServer -> add
query <- WebServer
notify -> WebServer

http <-- Browser
websocket <-> JS
JS -> GUI

Running

Syslog Server is implemented as python hat.syslog.server package which can be run with hat-syslog script with additional command line arguments:

Usage: __main__.py [OPTIONS]

  Syslog Server main

Options:
  --conf PATH           configuration defined by hat://syslog.yaml# (default
                        $XDG_CONFIG_HOME/hat/syslog.{yaml|yml|json})

  --log LEVEL           console log level
  --syslog-addr ADDR    syslog listening address (default tcp://0.0.0.0:6514)
  --syslog-pem PATH     pem file path - mandatory for ssl communication
  --ui-addr ADDR        UI listening address (default http://0.0.0.0:23020)
  --ui-pem PATH         pem file path - mandatory for https communication
  --ui-path PATH        override web ui directory path (development argument)
  --db-path PATH        sqlite database file path (default
                        $XDG_DATA_HOME/hat/syslog.db)

  --db-low-size N       number of messages kept in database after database
                        cleanup (default 1000000)

  --db-high-size N      number of messages that will trigger database cleanup
                        (default 10000000)

  --db-enable-archive   should messages, deleted during database cleanup, be
                        kept in archive files

  --db-disable-journal  disable sqlite jurnaling
  --help                Show this message and exit.

This application is part of hat-syslog python package.

Configuration

Syslog Server configuration written in form of single YAML or JSON file with structure defined by JSON Schema hat://syslog/server.yaml#. Path to configuration file is provided as command line argument during process startup. Additionally, configuration parameters provided in configuration file can be overridden by command line arguments. If configuration file could not be found, default values of configuration parameters are used.

Example of configuration:

---
log:
    version: 1
syslog_addr: 'tcp://0.0.0.0:6514'
ui_addr: 'http://0.0.0.0:23020'
dp_path: 'syslog.db'
db_low_size: 1_000_000
db_high_size: 10_000_000
db_enable_archive: false
db_disable_journal: false
...

Data backend

All incoming syslog messages are stored in single sqlite database. Maximum number of syslog messages stored in this database can be configured by configuration parameter db_high_size (value 0 represents unlimited number of messages). Once number of messages exceed configured limit, database cleanup procedure is triggered. During cleanup procedure, oldest messages are removed from database until number of messages reaches configuration parameter db_low_size when cleanup procedure stops. Prior to message deletion, if configuration parameter db_enable_archive is set, new database with unique file name is created and all messages scheduled for removal are inserted into newly created database. Archive database has got same structure as original database and can be used in place of original database for accessing archived syslog messages.

Web UI

Together with acquiring and storing syslog messages, Syslog Server provides web-based user interface for querying messages from database and observing changes in real time. Communication between web server and browser is based on juggler communication.

Once juggler connection is established, server and client should change initial null local state to theirs current valid value. Server’s local state is defined by #/definitions/server and client’s local state is defined by #/definitions/client from JSON schema:

"$schema": "http://json-schema.org/schema#"
definitions:
    client:
        "$ref": "#/definitions/filter"
    server:
        type: object
        required:
            - filter
            - entries
            - first_id
            - last_id
        properties:
            filter:
                "$ref": "#/definitions/filter"
            entries:
                type: array
                items:
                    "$ref": "#/definitions/entry"
            first_id:
                type:
                    - 'null'
                    - integer
            last_id:
                type:
                    - 'null'
                    - integer
    filter:
        type: object
        required:
            - max_results
            - last_id
            - entry_timestamp_from
            - entry_timestamp_to
            - facility
            - severity
            - hostname
            - app_name
            - procid
            - msgid
            - msg
        properties:
            max_results:
                type:
                    - 'null'
                    - integer
            last_id:
                type:
                    - 'null'
                    - integer
            entry_timestamp_from:
                type:
                    - 'null'
                    - number
            entry_timestamp_to:
                type:
                    - 'null'
                    - number
            facility:
                oneOf:
                    - type: 'null'
                    - "$ref": "#/definitions/facility"
            severity:
                oneOf:
                    - type: 'null'
                    - "$ref": "#/definitions/severity"
            hostname:
                type:
                    - 'null'
                    - string
            app_name:
                type:
                    - 'null'
                    - string
            procid:
                type:
                    - 'null'
                    - string
            msgid:
                type:
                    - 'null'
                    - string
            msg:
                type:
                    - 'null'
                    - string
    entry:
        type: object
        required:
            - id
            - timestamp
            - msg
        properties:
            id:
                type: integer
            timestamp:
                type: number
            msg:
                "$ref": "#/definitions/msg"
    msg:
        type: object
        required:
            - facility
            - severity
            - version
            - timestamp
            - hostname
            - app_name
            - procid
            - msgid
            - data
            - msg
        properties:
            facility:
                oneOf:
                    - type: 'null'
                    - "$ref": "#/definitions/facility"
            severity:
                oneOf:
                    - type: 'null'
                    - "$ref": "#/definitions/severity"
            version:
                type: integer
            timestamp:
                type:
                    - 'null'
                    - number
            hostname:
                type:
                    - 'null'
                    - string
            app_name:
                type:
                    - 'null'
                    - string
            procid:
                type:
                    - 'null'
                    - string
            msgid:
                type:
                    - 'null'
                    - string
            data:
                type:
                    - 'null'
                    - string
            msg:
                type:
                    - 'null'
                    - string
    facility:
        enum:
            - KERNEL
            - USER
            - MAIL
            - SYSTEM
            - AUTHORIZATION1
            - INTERNAL
            - PRINTER
            - NETWORK
            - UUCP
            - CLOCK1
            - AUTHORIZATION2
            - FTP
            - NTP
            - AUDIT
            - ALERT
            - CLOCK2
            - LOCAL0
            - LOCAL1
            - LOCAL2
            - LOCAL3
            - LOCAL4
            - LOCAL5
            - LOCAL6
            - LOCAL7
    severity:
        enum:
            - EMERGENCY
            - ALERT
            - CRITICAL
            - ERROR
            - WARNING
            - NOTICE
            - INFORMATIONAL
            - DEBUG

Juggler MESSAGE messages are not used in communication.

When server detected change of client’s local data, it should update its local data to match filter from client’s local data.

Implementation

Documentation is available as part of generated API reference: