Syslog

Real time reporting of execution state is mandatory functionality for most of Hat’s components implementing continuously running service. Primary way of logging is based on Syslog logging protocol as defined by:

Each component which reports log messages should implement Syslog TCP client. In this way, all logging messages can be aggregated in a single Syslog concentrator which can provide archiving, searching and real time monitoring functionalities.

Care must be taken for Syslog TCP client logging facility implementation not to interfere with other component’s functionalities. Logging should be considered best effort and not critical activity of each component.

SysLogHandler

SysLogHandler provides implementation of Python’s standard library logging.Handler interface. This implementation is part of hat-syslog python package.

Each instance of hat.syslog.handler.SysLogHandler starts new background thread responsible for sending syslog messages over TCP, UDP or TCP+SSL socket. If connection with remote syslog server could not be established or current connection is closed, new connect is called after 5 second timeout.

All log messages provided to SysLogHandler by hat.syslog.handler.SysLogHandler.emit() are queued in queue with maximum size limited by configuration parameter. During new log message registration, if queue is full, oldest messages are discarded and counter of discarded messages is incremented. Once new log messages can be sent to server, new log message containing information about number of discarded messages will be sent.

Syslog Generator

Syslog generator is application used as simple testing tool responsible for generating syslog messages that can be sent to Syslog Server.

Implemented as python hat.syslog.generator module which can be run with hat-syslog-generator script with additional command line arguments:

usage: hat-syslog-generator [-h] [--count N] [--text text] [--queue-size N]
                            [--msg-delay t] [--end-delay t]

optional arguments:
  -h, --help      show this help message and exit
  --count N       number of log messages (default 1)
  --text text     log message text (default 'syslog generator test')
  --queue-size N  client's log message queue size (default 1024)
  --msg-delay t   time delay between message generation (seconds) (default
                  0.01)
  --end-delay t   time delay affter all messages have been generated (default
                  0.5)

This application is part of hat-syslog python package.

Syslog server

Syslog server is central concentration for syslog messages. Additionally, it provides web interface for real time monitoring and filtering of log messages.

Syslog listening socket can be configured as TCP, UDP or TCP+SSL socket. Communication is based on RFC 5425, RFC 5426, RFC 6587. Once message is received, server stores message in predefined database.

Warning

Current implementation of Syslog server doesn’t support UDP communication

skinparam linetype ortho

folder Client {
    component SysLogHandler
}

folder "Syslog Server" {
    component SysLogServer
    component Backend
    component WebServer

    interface add
    interface query
    interface notify
}

folder Browser {
    component JS
    component GUI
}

database Sqlite

interface syslog
interface http
interface websocket

syslog - SysLogServer
Backend -- Sqlite
add - Backend
Backend - query
Backend - notify
http - WebServer
WebServer -- websocket

SysLogHandler --> syslog
SysLogServer -> add
query <- WebServer
notify -> WebServer

http <-- Browser
websocket <-> JS
JS -> GUI

Running

Syslog Server is implemented as python hat.syslog.server package which can be run with hat-syslog script with additional command line arguments:

usage: hat-syslog [-h] [--conf path] [--log level] [--syslog-addr address]
                  [--syslog-pem path] [--ui-addr address] [--ui-pem path]
                  [--db-path path] [--db-low-size count]
                  [--db-high-size count] [--db-enable-archive]
                  [--db-disable-journal] [--ui-path path]

optional arguments:
  -h, --help            show this help message and exit
  --conf path           configuration defined by hat://syslog/server.yaml#
                        (default $XDG_CONFIG_HOME/hat/syslog.yaml)

configuration arguments:
  --log level           console log level
  --syslog-addr address
                        syslog listening address (default: tcp://0.0.0.0:6514)
  --syslog-pem path     pem file path - mandatory for ssl communication
  --ui-addr address     ui listening address (default: http://0.0.0.0:23020)
  --ui-pem path         pem file path - mandatory for https communication
  --db-path path        sqlite database file path (default:
                        $HATPATH/syslog.db)
  --db-low-size count   number of messages kept in database after database
                        cleanup (default: 1000000)
  --db-high-size count  number of messages that will trigger database cleanup
                        (default: 10000000)
  --db-enable-archive   should messages, deleted during database cleanup, be
                        kept in archive files
  --db-disable-journal  disable sqlite jurnaling

development arguments:
  --ui-path path        override web ui directory path

This application is part of hat-syslog python package.

Configuration

Syslog Server configuration written in form of single YAML file with structure defined by JSON Schema hat://syslog/server.yaml#. Path to configuration file is provided as command line argument during process startup. Additionally, configuration parameters provided in configuration file can be overridden by command line arguments. If configuration file could not be found, default values of configuration parameters are used.

Example of configuration:

---
log:
    version: 1
syslog_addr: 'tcp://0.0.0.0:6514'
ui_addr: 'http://0.0.0.0:23020'
dp_path: '$HATPATH/syslog.db'
db_low_size: 1_000_000
db_high_size: 10_000_000
db_enable_archive: false
db_disable_journal: false
...

Data backend

All incoming syslog messages are stored in single sqlite database. Maximum number of syslog messages stored in this database can be configured by configuration parameter db_high_size (value 0 represents unlimited number of messages). Once number of messages exceed configured limit, database cleanup procedure is triggered. During cleanup procedure, oldest messages are removed from database until number of messages reaches configuration parameter db_low_size when cleanup procedure stops. Prior to message deletion, if configuration parameter db_enable_archive is set, new database with unique file name is created and all messages scheduled for removal are inserted into newly created database. Archive database has got same structure as original database and can be used in place of original database for accessing archived syslog messages.

Web UI

Together with acquiring and storing syslog messages, Syslog Server provides web-based user interface for querying messages from database and observing changes in real time. Communication between web server and browser is based on juggler communication.

Once juggler connection is established, server and client should change initial null local state to theirs current valid value. Server’s local state is defined by #/definitions/server and client’s local state is defined by #/definitions/client from JSON schema:

"$schema": "http://json-schema.org/schema#"
definitions:
    client:
        "$ref": "#/definitions/filter"
    server:
        type: object
        required:
            - filter
            - entries
            - first_id
            - last_id
        properties:
            filter:
                "$ref": "#/definitions/filter"
            entries:
                type: array
                items:
                    "$ref": "#/definitions/entry"
            first_id:
                type:
                    - 'null'
                    - integer
            last_id:
                type:
                    - 'null'
                    - integer
    filter:
        type: object
        required:
            - max_results
            - last_id
            - entry_timestamp_from
            - entry_timestamp_to
            - facility
            - severity
            - hostname
            - app_name
            - procid
            - msgid
            - msg
        properties:
            max_results:
                type:
                    - 'null'
                    - integer
            last_id:
                type:
                    - 'null'
                    - integer
            entry_timestamp_from:
                type:
                    - 'null'
                    - number
            entry_timestamp_to:
                type:
                    - 'null'
                    - number
            facility:
                oneOf:
                    - type: 'null'
                    - "$ref": "#/definitions/facility"
            severity:
                oneOf:
                    - type: 'null'
                    - "$ref": "#/definitions/severity"
            hostname:
                type:
                    - 'null'
                    - string
            app_name:
                type:
                    - 'null'
                    - string
            procid:
                type:
                    - 'null'
                    - string
            msgid:
                type:
                    - 'null'
                    - string
            msg:
                type:
                    - 'null'
                    - string
    entry:
        type: object
        required:
            - id
            - timestamp
            - msg
        properties:
            id:
                type: integer
            timestamp:
                type: number
            msg:
                "$ref": "#/definitions/msg"
    msg:
        type: object
        required:
            - facility
            - severity
            - version
            - timestamp
            - hostname
            - app_name
            - procid
            - msgid
            - data
            - msg
        properties:
            facility:
                oneOf:
                    - type: 'null'
                    - "$ref": "#/definitions/facility"
            severity:
                oneOf:
                    - type: 'null'
                    - "$ref": "#/definitions/severity"
            version:
                type: integer
            timestamp:
                type:
                    - 'null'
                    - number
            hostname:
                type:
                    - 'null'
                    - string
            app_name:
                type:
                    - 'null'
                    - string
            procid:
                type:
                    - 'null'
                    - string
            msgid:
                type:
                    - 'null'
                    - string
            data:
                type:
                    - 'null'
                    - string
            msg:
                type:
                    - 'null'
                    - string
    facility:
        enum:
            - KERNEL
            - USER
            - MAIL
            - SYSTEM
            - AUTHORIZATION1
            - INTERNAL
            - PRINTER
            - NETWORK
            - UUCP
            - CLOCK1
            - AUTHORIZATION2
            - FTP
            - NTP
            - AUDIT
            - ALERT
            - CLOCK2
            - LOCAL0
            - LOCAL1
            - LOCAL2
            - LOCAL3
            - LOCAL4
            - LOCAL5
            - LOCAL6
            - LOCAL7
    severity:
        enum:
            - EMERGENCY
            - ALERT
            - CRITICAL
            - ERROR
            - WARNING
            - NOTICE
            - INFORMATIONAL
            - DEBUG

Juggler MESSAGE messages are not used in communication.

When server detected change of client’s local data, it should update its local data to match filter from client’s local data.

Python implementation

Common syslog functionality and data structures

class hat.syslog.common.Facility(value)

Bases: enum.Enum

An enumeration.

  • KERNEL

  • USER

  • MAIL

  • SYSTEM

  • AUTHORIZATION1

  • INTERNAL

  • PRINTER

  • NETWORK

  • UUCP

  • CLOCK1

  • AUTHORIZATION2

  • FTP

  • NTP

  • AUDIT

  • ALERT

  • CLOCK2

  • LOCAL0

  • LOCAL1

  • LOCAL2

  • LOCAL3

  • LOCAL4

  • LOCAL5

  • LOCAL6

  • LOCAL7

KERNEL = 0
USER = 1
MAIL = 2
SYSTEM = 3
AUTHORIZATION1 = 4
INTERNAL = 5
PRINTER = 6
NETWORK = 7
UUCP = 8
CLOCK1 = 9
AUTHORIZATION2 = 10
FTP = 11
NTP = 12
AUDIT = 13
ALERT = 14
CLOCK2 = 15
LOCAL0 = 16
LOCAL1 = 17
LOCAL2 = 18
LOCAL3 = 19
LOCAL4 = 20
LOCAL5 = 21
LOCAL6 = 22
LOCAL7 = 23
class hat.syslog.common.Severity(value)

Bases: enum.Enum

An enumeration.

  • EMERGENCY

  • ALERT

  • CRITICAL

  • ERROR

  • WARNING

  • NOTICE

  • INFORMATIONAL

  • DEBUG

EMERGENCY = 0
ALERT = 1
CRITICAL = 2
ERROR = 3
WARNING = 4
NOTICE = 5
INFORMATIONAL = 6
DEBUG = 7
class hat.syslog.common.Msg(facility, severity, version, timestamp, hostname, app_name, procid, msgid, data, msg)

Bases: tuple

Create new instance of Msg(facility, severity, version, timestamp, hostname, app_name, procid, msgid, data, msg)

app_name

application name

Type

Optional[str]

data

json serialized Dict[str,Dict[str,str]] data

Type

Optional[str]

facility

facility

Type

Facility

hostname

hostname

Type

Optional[str]

msg

message content

Type

Optional[str]

msgid

message id

Type

Optional[str]

procid

process id

Type

Optional[str]

severity

severity

Type

Severity

timestamp

timestamp

Type

Optional[float]

version

version

Type

int

hat.syslog.common.msg_to_str(msg)

Create string representation of message according to RFC 5424

Parameters

msg (Msg) – message

Returns

str

hat.syslog.common.msg_from_str(msg_str)

Parse message string formatted according to RFC 5424

Parameters

msg_str (str) – message string

Returns

Msg

hat.syslog.common.msg_to_json(msg)

Convert message to json serializable data

Parameters

msg (Msg) – message

Returns

json.Data

hat.syslog.common.msg_from_json(data)

Convert json serializable data to message

Parameters

data (json.Data) – data

Returns

Msg

Syslog hadler

Implementation of logging.Handler for syslog logging protocol.

hat.syslog.handler.reconnect_delay

delay in seconds before retrying connection with remote syslog server

Type

int

class hat.syslog.handler.CommType(value)

Bases: enum.Enum

Syslog communication type

  • UDP

  • TCP

  • SSL

UDP = 0
TCP = 1
SSL = 2
class hat.syslog.handler.SysLogHandler(host, port, comm_type, queue_size=1024)

Bases: logging.Handler

Syslog handler

Parameters
  • host (str) – hostname

  • port (int) – port number

  • comm_type (Union[CommType,str]) – communication type

  • queue_size (int) – queue size (number of log entries)

Initializes the instance - basically setting the formatter to None and the filter list to empty.

close()

“See logging.Handler.close()

emit(record)

“See logging.Handler.emit()

Common data structures and functions

class hat.syslog.server.common.Facility(value)

Bases: enum.Enum

An enumeration.

  • KERNEL

  • USER

  • MAIL

  • SYSTEM

  • AUTHORIZATION1

  • INTERNAL

  • PRINTER

  • NETWORK

  • UUCP

  • CLOCK1

  • AUTHORIZATION2

  • FTP

  • NTP

  • AUDIT

  • ALERT

  • CLOCK2

  • LOCAL0

  • LOCAL1

  • LOCAL2

  • LOCAL3

  • LOCAL4

  • LOCAL5

  • LOCAL6

  • LOCAL7

KERNEL = 0
USER = 1
MAIL = 2
SYSTEM = 3
AUTHORIZATION1 = 4
INTERNAL = 5
PRINTER = 6
NETWORK = 7
UUCP = 8
CLOCK1 = 9
AUTHORIZATION2 = 10
FTP = 11
NTP = 12
AUDIT = 13
ALERT = 14
CLOCK2 = 15
LOCAL0 = 16
LOCAL1 = 17
LOCAL2 = 18
LOCAL3 = 19
LOCAL4 = 20
LOCAL5 = 21
LOCAL6 = 22
LOCAL7 = 23
class hat.syslog.server.common.Severity(value)

Bases: enum.Enum

An enumeration.

  • EMERGENCY

  • ALERT

  • CRITICAL

  • ERROR

  • WARNING

  • NOTICE

  • INFORMATIONAL

  • DEBUG

EMERGENCY = 0
ALERT = 1
CRITICAL = 2
ERROR = 3
WARNING = 4
NOTICE = 5
INFORMATIONAL = 6
DEBUG = 7
class hat.syslog.server.common.Msg(facility, severity, version, timestamp, hostname, app_name, procid, msgid, data, msg)

Bases: tuple

Create new instance of Msg(facility, severity, version, timestamp, hostname, app_name, procid, msgid, data, msg)

app_name

application name

Type

Optional[str]

data

json serialized Dict[str,Dict[str,str]] data

Type

Optional[str]

facility

facility

Type

Facility

hostname

hostname

Type

Optional[str]

msg

message content

Type

Optional[str]

msgid

message id

Type

Optional[str]

procid

process id

Type

Optional[str]

severity

severity

Type

Severity

timestamp

timestamp

Type

Optional[float]

version

version

Type

int

hat.syslog.server.common.msg_from_str(msg_str)

Parse message string formatted according to RFC 5424

Parameters

msg_str (str) – message string

Returns

Msg

hat.syslog.server.common.msg_to_json(msg)

Convert message to json serializable data

Parameters

msg (Msg) – message

Returns

json.Data

class hat.syslog.server.common.Entry(id, timestamp, msg)

Bases: tuple

Create new instance of Entry(id, timestamp, msg)

id

int

msg

Msg

timestamp

float

class hat.syslog.server.common.Filter(max_results, last_id, entry_timestamp_from, entry_timestamp_to, facility, severity, hostname, app_name, procid, msgid, msg)

Bases: tuple

Create new instance of Filter(max_results, last_id, entry_timestamp_from, entry_timestamp_to, facility, severity, hostname, app_name, procid, msgid, msg)

app_name

Optional[str]

entry_timestamp_from

Optional[float]

entry_timestamp_to

Optional[float]

facility

Optional[Facility]

hostname

Optional[str]

last_id

Optional[int]

max_results

Optional[int]

msg

Optional[str]

msgid

Optional[str]

procid

Optional[str]

severity

Optional[Severity]

hat.syslog.server.common.filter_to_json(filter)

Convert filter to json data

Parameters

filter (Filter) – filter

Returns

Any

hat.syslog.server.common.filter_from_json(json_filter)

Create filter from json data

Parameters

json_filter (Any) – json data

Returns

Filter

hat.syslog.server.common.entry_to_json(entry)

Convert entry to json data

Parameters

entry (Entry) – entry

Returns

Any

Configuration parser

hat.syslog.server.conf.mlog

module logger

Type

logging.Logger

hat.syslog.server.conf.package_path

hat.syslog.server package directory

Type

pathlib.Path

hat.syslog.server.conf.user_data_dir

XDG user data directory

Type

pathlib.Path

hat.syslog.server.conf.user_conf_dir

XDG user conf directory

Type

pathlib.Path

hat.syslog.server.conf.default_ui_path

default ui directory

Type

pathlib.Path

hat.syslog.server.conf.default_conf_path

default conf path

Type

pathlib.Path

class hat.syslog.server.conf.SysLogServerConf(addr, pem)

Bases: tuple

Create new instance of SysLogServerConf(addr, pem)

addr

listening address

Type

str

pem

pem file path

Type

Optional[Path]

class hat.syslog.server.conf.WebServerConf(addr, pem, path)

Bases: tuple

Create new instance of WebServerConf(addr, pem, path)

addr

listening address

Type

str

path

static ui files path

Type

Path

pem

pem file path

Type

Optional[Path]

class hat.syslog.server.conf.BackendConf(path, low_size, high_size, enable_archive, disable_journal)

Bases: tuple

Create new instance of BackendConf(path, low_size, high_size, enable_archive, disable_journal)

disable_journal

disable journal flag

Type

bool

enable_archive

enable archive flag

Type

bool

high_size

high size count

Type

int

low_size

low size count

Type

int

path

database file path

Type

Path

class hat.syslog.server.conf.Conf(log, syslog, ui, db)

Bases: tuple

Create new instance of Conf(log, syslog, ui, db)

db

BackendConf

log

logging configuration

Type

Any

syslog

SysLogServerConf

ui

WebServerConf

hat.syslog.server.conf.get_conf()

Get configuration data

Returns

Conf

Backend implementation

hat.syslog.server.backend.mlog

module logger

Type

logging.Logger

hat.syslog.server.backend.register_delay

registration delay in seconds

Type

float

hat.syslog.server.backend.register_queue_size

registration queue size

Type

int

hat.syslog.server.backend.register_queue_treshold

registration queue threshold

Type

int

async hat.syslog.server.backend.create_backend(conf)

Create backend

Parameters

conf (hat.syslog.server.conf.BackendConf) – configuration

Returns

Backend

class hat.syslog.server.backend.Backend

Bases: object

property first_id

first entry id

Type

Optional[int]

property last_id

last entry id

Type

Optional[int]

register_change_cb(cb)

Register change callback

Callback is called if first_id changes and/or lats_id changes and/or new entries are available (passed as argument to registered callback).

Parameters

cb (Callable[[List[common.Entry]],None]) – callback

Returns

util.RegisterCallbackHandle

property closed

closed future

Type

asyncio.Future

async async_close()

Async close

async register(timestamp, msg)

Register message

Registration adds msg to registration queue. If queue is full, wait until message can be successfully added.

When message is added to empty queue, registration delay timer is started. Once delay timer expires or if number of messages in queue is greater than threshold, all messages are removed from queue and inserted into sqlite database.

Parameters
  • timestamp (float) – timestamp

  • msg (common.Msg) – message

async query(filter)

Query entries

Parameters

filter (common.Filter) – filter

Returns

List[common.Entry]

Interface to SQLite database

hat.syslog.server.database.mlog

module logger

Type

logging.Logger

async hat.syslog.server.database.create_database(path, disable_journal)

Create database

Parameters
  • path (pathlib.Path) – database file path

  • disable_journal (bool) – disable journal flag

Returns

Database

class hat.syslog.server.database.Database

Bases: object

property closed

closed future

Type

asyncio.Future

async async_close()

Async close

async get_first_id()

Get first entry id

Returns

Optional[int]

async get_last_id()

Get last entry id

Returns

Optional[int]

async add_msgs(msgs)

Add messages

Parameters

msgs (List[Tuple[float,hat.syslog.common.Msg]) – timestamp messages

Returns

List[Entry]

async add_entries(entries)

Add entries

Parameters

entries (List[Entry]) – entries

async query(filter)

Query entries that satisfy filter

Parameters

filter (Filter) – filter

Returns

List[Entry]

async delete(first_id)

Delete entries prior to first_id

Parameters

first_id (int) – first entry id

Syslog server implementation

hat.syslog.server.syslog.mlog

module logger

Type

logging.Logger

async hat.syslog.server.syslog.create_syslog_server(conf, backend)

Create syslog server

Parameters
  • conf (hat.syslog.server.conf.SysLogServer) – configuration

  • backend (hat.syslog.server.backend.Backend) – backend

Returns

SysLogServer

class hat.syslog.server.syslog.SysLogServer

Bases: object

Syslog server

For creating new instance see create_syslog_server().

property closed

closed future

Type

asyncio.Future

async async_close()

Async close

Web server implementation

hat.syslog.server.ui.mlog

module logger

Type

logging.Logger

hat.syslog.server.ui.max_results_limit

max results limit

Type

int

hat.syslog.server.ui.autoflush_delay

juggler autoflush delay

Type

float

async hat.syslog.server.ui.create_web_server(conf, backend)

Create web server

Parameters
  • conf (hat.syslog.server.conf.WebServerConf) – configuration

  • backend (hat.syslog.server.backend.Backend) – backend

Returns

WebServer

class hat.syslog.server.ui.WebServer

Bases: object

property closed

closed future

Type

asyncio.Future

async async_close()

Async close