Syslog¶

Real time reporting of execution state is mandatory functionality for most of Hat’s components implementing continuously running service. Primary way of logging is based on Syslog logging protocol as defined by:

RFC 5425

RFC 5426

RFC 6587

Each component which reports log messages should implement Syslog TCP client. In this way, all logging messages can be aggregated in a single Syslog concentrator which can provide archiving, searching and real time monitoring functionalities.

Care must be taken for Syslog TCP client logging facility implementation not to interfere with other component’s functionalities. Logging should be considered best effort and not critical activity of each component.

SysLogHandler¶

SysLogHandler provides implementation of Python’s standard library logging.Handler interface. This implementation is part of hat-syslog python package.

Each instance of hat.syslog.handler.SysLogHandler starts new background thread responsible for sending syslog messages over TCP, UDP or TCP+SSL socket. If connection with remote syslog server could not be established or current connection is closed, new connect is called after 5 second timeout.

All log messages provided to SysLogHandler by hat.syslog.handler.SysLogHandler.emit() are queued in queue with maximum size limited by configuration parameter. During new log message registration, if queue is full, oldest messages are discarded and counter of discarded messages is incremented. Once new log messages can be sent to server, new log message containing information about number of discarded messages will be sent.

Syslog Generator¶

Syslog generator is application used as simple testing tool responsible for generating syslog messages that can be sent to Syslog Server.

Implemented as python hat.syslog.generator module which can be run with hat-syslog-generator script with additional command line arguments:

usage: hat-syslog-generator [-h] [--count N] [--text text] [--queue-size N]
                            [--msg-delay t] [--end-delay t]

optional arguments:
  -h, --help      show this help message and exit
  --count N       number of log messages (default 1)
  --text text     log message text (default 'syslog generator test')
  --queue-size N  client's log message queue size (default 1024)
  --msg-delay t   time delay between message generation (seconds) (default
                  0.01)
  --end-delay t   time delay affter all messages have been generated (default
                  0.5)

This application is part of hat-syslog python package.

Syslog server¶

Syslog server is central concentration for syslog messages. Additionally, it provides web interface for real time monitoring and filtering of log messages.

Syslog listening socket can be configured as TCP, UDP or TCP+SSL socket. Communication is based on RFC 5425, RFC 5426, RFC 6587. Once message is received, server stores message in predefined database.

Warning

Current implementation of Syslog server doesn’t support UDP communication

Running¶

Syslog Server is implemented as python hat.syslog.server package which can be run with hat-syslog script with additional command line arguments:

usage: hat-syslog [-h] [--conf path] [--log level] [--syslog-addr address]
                  [--syslog-pem path] [--ui-addr address] [--ui-pem path]
                  [--db-path path] [--db-low-size count]
                  [--db-high-size count] [--db-enable-archive]
                  [--db-disable-journal] [--ui-path path]

optional arguments:
  -h, --help            show this help message and exit
  --conf path           configuration defined by hat://syslog/server.yaml#
                        (default $XDG_CONFIG_HOME/hat/syslog.yaml)

configuration arguments:
  --log level           console log level
  --syslog-addr address
                        syslog listening address (default: tcp://0.0.0.0:6514)
  --syslog-pem path     pem file path - mandatory for ssl communication
  --ui-addr address     ui listening address (default: http://0.0.0.0:23020)
  --ui-pem path         pem file path - mandatory for https communication
  --db-path path        sqlite database file path (default:
                        $HATPATH/syslog.db)
  --db-low-size count   number of messages kept in database after database
                        cleanup (default: 1000000)
  --db-high-size count  number of messages that will trigger database cleanup
                        (default: 10000000)
  --db-enable-archive   should messages, deleted during database cleanup, be
                        kept in archive files
  --db-disable-journal  disable sqlite jurnaling

development arguments:
  --ui-path path        override web ui directory path

This application is part of hat-syslog python package.

Configuration¶

Syslog Server configuration written in form of single YAML file with structure defined by JSON Schema hat://syslog/server.yaml#. Path to configuration file is provided as command line argument during process startup. Additionally, configuration parameters provided in configuration file can be overridden by command line arguments. If configuration file could not be found, default values of configuration parameters are used.

Example of configuration:

---
log:
    version: 1
syslog_addr: 'tcp://0.0.0.0:6514'
ui_addr: 'http://0.0.0.0:23020'
dp_path: '$HATPATH/syslog.db'
db_low_size: 1_000_000
db_high_size: 10_000_000
db_enable_archive: false
db_disable_journal: false
...

Data backend¶

All incoming syslog messages are stored in single sqlite database. Maximum number of syslog messages stored in this database can be configured by configuration parameter db_high_size (value 0 represents unlimited number of messages). Once number of messages exceed configured limit, database cleanup procedure is triggered. During cleanup procedure, oldest messages are removed from database until number of messages reaches configuration parameter db_low_size when cleanup procedure stops. Prior to message deletion, if configuration parameter db_enable_archive is set, new database with unique file name is created and all messages scheduled for removal are inserted into newly created database. Archive database has got same structure as original database and can be used in place of original database for accessing archived syslog messages.

Web UI¶

Together with acquiring and storing syslog messages, Syslog Server provides web-based user interface for querying messages from database and observing changes in real time. Communication between web server and browser is based on juggler communication.

Once juggler connection is established, server and client should change initial null local state to theirs current valid value. Server’s local state is defined by #/definitions/server and client’s local state is defined by #/definitions/client from JSON schema:

"$schema": "http://json-schema.org/schema#"
definitions:
    client:
        "$ref": "#/definitions/filter"
    server:
        type: object
        required:
            - filter
            - entries
            - first_id
            - last_id
        properties:
            filter:
                "$ref": "#/definitions/filter"
            entries:
                type: array
                items:
                    "$ref": "#/definitions/entry"
            first_id:
                type:
                    - 'null'
                    - integer
            last_id:
                type:
                    - 'null'
                    - integer
    filter:
        type: object
        required:
            - max_results
            - last_id
            - entry_timestamp_from
            - entry_timestamp_to
            - facility
            - severity
            - hostname
            - app_name
            - procid
            - msgid
            - msg
        properties:
            max_results:
                type:
                    - 'null'
                    - integer
            last_id:
                type:
                    - 'null'
                    - integer
            entry_timestamp_from:
                type:
                    - 'null'
                    - number
            entry_timestamp_to:
                type:
                    - 'null'
                    - number
            facility:
                oneOf:
                    - type: 'null'
                    - "$ref": "#/definitions/facility"
            severity:
                oneOf:
                    - type: 'null'
                    - "$ref": "#/definitions/severity"
            hostname:
                type:
                    - 'null'
                    - string
            app_name:
                type:
                    - 'null'
                    - string
            procid:
                type:
                    - 'null'
                    - string
            msgid:
                type:
                    - 'null'
                    - string
            msg:
                type:
                    - 'null'
                    - string
    entry:
        type: object
        required:
            - id
            - timestamp
            - msg
        properties:
            id:
                type: integer
            timestamp:
                type: number
            msg:
                "$ref": "#/definitions/msg"
    msg:
        type: object
        required:
            - facility
            - severity
            - version
            - timestamp
            - hostname
            - app_name
            - procid
            - msgid
            - data
            - msg
        properties:
            facility:
                oneOf:
                    - type: 'null'
                    - "$ref": "#/definitions/facility"
            severity:
                oneOf:
                    - type: 'null'
                    - "$ref": "#/definitions/severity"
            version:
                type: integer
            timestamp:
                type:
                    - 'null'
                    - number
            hostname:
                type:
                    - 'null'
                    - string
            app_name:
                type:
                    - 'null'
                    - string
            procid:
                type:
                    - 'null'
                    - string
            msgid:
                type:
                    - 'null'
                    - string
            data:
                type:
                    - 'null'
                    - string
            msg:
                type:
                    - 'null'
                    - string
    facility:
        enum:
            - KERNEL
            - USER
            - MAIL
            - SYSTEM
            - AUTHORIZATION1
            - INTERNAL
            - PRINTER
            - NETWORK
            - UUCP
            - CLOCK1
            - AUTHORIZATION2
            - FTP
            - NTP
            - AUDIT
            - ALERT
            - CLOCK2
            - LOCAL0
            - LOCAL1
            - LOCAL2
            - LOCAL3
            - LOCAL4
            - LOCAL5
            - LOCAL6
            - LOCAL7
    severity:
        enum:
            - EMERGENCY
            - ALERT
            - CRITICAL
            - ERROR
            - WARNING
            - NOTICE
            - INFORMATIONAL
            - DEBUG

Juggler MESSAGE messages are not used in communication.

When server detected change of client’s local data, it should update its local data to match filter from client’s local data.

Python implementation¶

Common syslog functionality and data structures

class hat.syslog.common.Facility(value)¶

Bases: enum.Enum

An enumeration.

KERNEL
USER
MAIL
SYSTEM
AUTHORIZATION1
INTERNAL
PRINTER
NETWORK
UUCP
CLOCK1
AUTHORIZATION2
FTP
NTP
AUDIT
ALERT
CLOCK2
LOCAL0
LOCAL1
LOCAL2
LOCAL3
LOCAL4
LOCAL5
LOCAL6
LOCAL7

KERNEL = 0¶

USER = 1¶

MAIL = 2¶

SYSTEM = 3¶

AUTHORIZATION1 = 4¶

INTERNAL = 5¶

PRINTER = 6¶

NETWORK = 7¶

UUCP = 8¶

CLOCK1 = 9¶

AUTHORIZATION2 = 10¶

FTP = 11¶

NTP = 12¶

AUDIT = 13¶

ALERT = 14¶

CLOCK2 = 15¶

LOCAL0 = 16¶

LOCAL1 = 17¶

LOCAL2 = 18¶

LOCAL3 = 19¶

LOCAL4 = 20¶

LOCAL5 = 21¶

LOCAL6 = 22¶

LOCAL7 = 23¶

class hat.syslog.common.Severity(value)¶

Bases: enum.Enum

An enumeration.

EMERGENCY
ALERT
CRITICAL
ERROR
WARNING
NOTICE
INFORMATIONAL
DEBUG

EMERGENCY = 0¶

ALERT = 1¶

CRITICAL = 2¶

ERROR = 3¶

WARNING = 4¶

NOTICE = 5¶

INFORMATIONAL = 6¶

DEBUG = 7¶

class hat.syslog.common.Msg(facility, severity, version, timestamp, hostname, app_name, procid, msgid, data, msg)¶

Bases: tuple

Create new instance of Msg(facility, severity, version, timestamp, hostname, app_name, procid, msgid, data, msg)

app_name¶

application name

Type: Optional[str]

data¶

json serialized Dict[str,Dict[str,str]] data

Type: Optional[str]

facility¶

facility

Type: Facility

hostname¶

hostname

Type: Optional[str]

msg¶

message content

Type: Optional[str]

msgid¶

message id

Type: Optional[str]

procid¶

process id

Type: Optional[str]

severity¶

severity

Type: Severity

timestamp¶

timestamp

Type: Optional[float]

version¶

version

Type: int

hat.syslog.common.msg_to_str(msg)¶

Create string representation of message according to RFC 5424

Parameters: msg (Msg) – message
Returns: str

hat.syslog.common.msg_from_str(msg_str)¶

Parse message string formatted according to RFC 5424

Parameters: msg_str (str) – message string
Returns: Msg

hat.syslog.common.msg_to_json(msg)¶

Convert message to json serializable data

Parameters: msg (Msg) – message
Returns: json.Data

hat.syslog.common.msg_from_json(data)¶

Convert json serializable data to message

Parameters: data (json.Data) – data
Returns: Msg

Syslog hadler

Implementation of logging.Handler for syslog logging protocol.

hat.syslog.handler.reconnect_delay¶

delay in seconds before retrying connection with remote syslog server

Type: int

class hat.syslog.handler.CommType(value)¶

Bases: enum.Enum

Syslog communication type

UDP
TCP
SSL

UDP = 0¶

TCP = 1¶

SSL = 2¶

class hat.syslog.handler.SysLogHandler(host, port, comm_type, queue_size=1024)¶

Bases: logging.Handler

Syslog handler

Parameters

host (str) – hostname
port (int) – port number
comm_type (Union[CommType,str]) – communication type
queue_size (int) – queue size (number of log entries)

Initializes the instance - basically setting the formatter to None and the filter list to empty.

close()¶: “See logging.Handler.close()

emit(record)¶: “See logging.Handler.emit()

Common data structures and functions

class hat.syslog.server.common.Facility(value)¶

Bases: enum.Enum

An enumeration.

KERNEL
USER
MAIL
SYSTEM
AUTHORIZATION1
INTERNAL
PRINTER
NETWORK
UUCP
CLOCK1
AUTHORIZATION2
FTP
NTP
AUDIT
ALERT
CLOCK2
LOCAL0
LOCAL1
LOCAL2
LOCAL3
LOCAL4
LOCAL5
LOCAL6
LOCAL7

KERNEL = 0¶

USER = 1¶

MAIL = 2¶

SYSTEM = 3¶

AUTHORIZATION1 = 4¶

INTERNAL = 5¶

PRINTER = 6¶

NETWORK = 7¶

UUCP = 8¶

CLOCK1 = 9¶

AUTHORIZATION2 = 10¶

FTP = 11¶

NTP = 12¶

AUDIT = 13¶

ALERT = 14¶

CLOCK2 = 15¶

LOCAL0 = 16¶

LOCAL1 = 17¶

LOCAL2 = 18¶

LOCAL3 = 19¶

LOCAL4 = 20¶

LOCAL5 = 21¶

LOCAL6 = 22¶

LOCAL7 = 23¶

class hat.syslog.server.common.Severity(value)¶

Bases: enum.Enum

An enumeration.

EMERGENCY
ALERT
CRITICAL
ERROR
WARNING
NOTICE
INFORMATIONAL
DEBUG

EMERGENCY = 0¶

ALERT = 1¶

CRITICAL = 2¶

ERROR = 3¶

WARNING = 4¶

NOTICE = 5¶

INFORMATIONAL = 6¶

DEBUG = 7¶

class hat.syslog.server.common.Msg(facility, severity, version, timestamp, hostname, app_name, procid, msgid, data, msg)¶

Bases: tuple

Create new instance of Msg(facility, severity, version, timestamp, hostname, app_name, procid, msgid, data, msg)

app_name¶

application name

Type: Optional[str]

data¶

json serialized Dict[str,Dict[str,str]] data

Type: Optional[str]

facility¶

facility

Type: Facility

hostname¶

hostname

Type: Optional[str]

msg¶

message content

Type: Optional[str]

msgid¶

message id

Type: Optional[str]

procid¶

process id

Type: Optional[str]

severity¶

severity

Type: Severity

timestamp¶

timestamp

Type: Optional[float]

version¶

version

Type: int

hat.syslog.server.common.msg_from_str(msg_str)¶

Parse message string formatted according to RFC 5424

Parameters: msg_str (str) – message string
Returns: Msg

hat.syslog.server.common.msg_to_json(msg)¶

Convert message to json serializable data

Parameters: msg (Msg) – message
Returns: json.Data

class hat.syslog.server.common.Entry(id, timestamp, msg)¶

Bases: tuple

Create new instance of Entry(id, timestamp, msg)

id¶: int

msg¶: Msg

timestamp¶: float

class hat.syslog.server.common.Filter(max_results, last_id, entry_timestamp_from, entry_timestamp_to, facility, severity, hostname, app_name, procid, msgid, msg)¶

Bases: tuple

Create new instance of Filter(max_results, last_id, entry_timestamp_from, entry_timestamp_to, facility, severity, hostname, app_name, procid, msgid, msg)

app_name¶: Optional[str]

entry_timestamp_from¶: Optional[float]

entry_timestamp_to¶: Optional[float]

facility¶: Optional[Facility]

hostname¶: Optional[str]

last_id¶: Optional[int]

max_results¶: Optional[int]

msg¶: Optional[str]

msgid¶: Optional[str]

procid¶: Optional[str]

severity¶: Optional[Severity]

hat.syslog.server.common.filter_to_json(filter)¶

Convert filter to json data

Parameters: filter (Filter) – filter
Returns: Any

hat.syslog.server.common.filter_from_json(json_filter)¶

Create filter from json data

Parameters: json_filter (Any) – json data
Returns: Filter

hat.syslog.server.common.entry_to_json(entry)¶

Convert entry to json data

Parameters: entry (Entry) – entry
Returns: Any

Configuration parser

hat.syslog.server.conf.mlog¶

module logger

Type: logging.Logger

hat.syslog.server.conf.package_path¶

hat.syslog.server package directory

Type: pathlib.Path

hat.syslog.server.conf.user_data_dir¶

XDG user data directory

Type: pathlib.Path

hat.syslog.server.conf.user_conf_dir¶

XDG user conf directory

Type: pathlib.Path

hat.syslog.server.conf.default_ui_path¶

default ui directory

Type: pathlib.Path

hat.syslog.server.conf.default_conf_path¶

default conf path

Type: pathlib.Path

class hat.syslog.server.conf.SysLogServerConf(addr, pem)¶

Bases: tuple

Create new instance of SysLogServerConf(addr, pem)

addr¶

listening address

Type: str

pem¶

pem file path

Type: Optional[Path]

class hat.syslog.server.conf.WebServerConf(addr, pem, path)¶

Bases: tuple

Create new instance of WebServerConf(addr, pem, path)

addr¶

listening address

Type: str

path¶

static ui files path

Type: Path

pem¶

pem file path

Type: Optional[Path]

class hat.syslog.server.conf.BackendConf(path, low_size, high_size, enable_archive, disable_journal)¶

Bases: tuple

Create new instance of BackendConf(path, low_size, high_size, enable_archive, disable_journal)

disable_journal¶

disable journal flag

Type: bool

enable_archive¶

enable archive flag

Type: bool

high_size¶

high size count

Type: int

low_size¶

low size count

Type: int

path¶

database file path

Type: Path

class hat.syslog.server.conf.Conf(log, syslog, ui, db)¶

Bases: tuple

Create new instance of Conf(log, syslog, ui, db)

db¶: BackendConf

log¶

logging configuration

Type: Any

syslog¶: SysLogServerConf

ui¶: WebServerConf

hat.syslog.server.conf.get_conf()¶

Get configuration data

Returns: Conf

Backend implementation

hat.syslog.server.backend.mlog¶

module logger

Type: logging.Logger

hat.syslog.server.backend.register_delay¶

registration delay in seconds

Type: float

hat.syslog.server.backend.register_queue_size¶

registration queue size

Type: int

hat.syslog.server.backend.register_queue_treshold¶

registration queue threshold

Type: int

async hat.syslog.server.backend.create_backend(conf)¶

Create backend

Parameters: conf (hat.syslog.server.conf.BackendConf) – configuration
Returns: Backend

class hat.syslog.server.backend.Backend¶

Bases: object

property first_id¶

first entry id

Type: Optional[int]

property last_id¶

last entry id

Type: Optional[int]

register_change_cb(cb)¶

Register change callback

Callback is called if first_id changes and/or lats_id changes and/or new entries are available (passed as argument to registered callback).

Parameters: cb (Callable[[List[common.Entry]],None]) – callback
Returns: util.RegisterCallbackHandle

property closed¶

closed future

Type: asyncio.Future

async async_close()¶: Async close

async register(timestamp, msg)¶

Register message

Registration adds msg to registration queue. If queue is full, wait until message can be successfully added.

When message is added to empty queue, registration delay timer is started. Once delay timer expires or if number of messages in queue is greater than threshold, all messages are removed from queue and inserted into sqlite database.

Parameters

timestamp (float) – timestamp
msg (common.Msg) – message

async query(filter)¶

Query entries

Parameters: filter (common.Filter) – filter
Returns: List[common.Entry]

Interface to SQLite database

hat.syslog.server.database.mlog¶

module logger

Type: logging.Logger

async hat.syslog.server.database.create_database(path, disable_journal)¶

Create database

Parameters

path (pathlib.Path) – database file path
disable_journal (bool) – disable journal flag

Returns

Database

class hat.syslog.server.database.Database¶

Bases: object

property closed¶

closed future

Type: asyncio.Future

async async_close()¶: Async close

async get_first_id()¶

Get first entry id

Returns: Optional[int]

async get_last_id()¶

Get last entry id

Returns: Optional[int]

async add_msgs(msgs)¶

Add messages

Parameters: msgs (List[Tuple[float,hat.syslog.common.Msg]) – timestamp messages
Returns: List[Entry]

async add_entries(entries)¶

Add entries

Parameters: entries (List[Entry]) – entries

async query(filter)¶

Query entries that satisfy filter

Parameters: filter (Filter) – filter
Returns: List[Entry]

async delete(first_id)¶

Delete entries prior to first_id

Parameters: first_id (int) – first entry id

Syslog server implementation

hat.syslog.server.syslog.mlog¶

module logger

Type: logging.Logger

async hat.syslog.server.syslog.create_syslog_server(conf, backend)¶

Create syslog server

Parameters

conf (hat.syslog.server.conf.SysLogServer) – configuration
backend (hat.syslog.server.backend.Backend) – backend

Returns

SysLogServer

class hat.syslog.server.syslog.SysLogServer¶

Bases: object

Syslog server

For creating new instance see create_syslog_server().

property closed¶

closed future

Type: asyncio.Future

async async_close()¶: Async close

Web server implementation

hat.syslog.server.ui.mlog¶

module logger

Type: logging.Logger

hat.syslog.server.ui.max_results_limit¶

max results limit

Type: int

hat.syslog.server.ui.autoflush_delay¶

juggler autoflush delay

Type: float

async hat.syslog.server.ui.create_web_server(conf, backend)¶

Create web server

Parameters

conf (hat.syslog.server.conf.WebServerConf) – configuration
backend (hat.syslog.server.backend.Backend) – backend

Returns

WebServer

class hat.syslog.server.ui.WebServer¶

Bases: object

property closed¶

closed future

Type: asyncio.Future

async async_close()¶: Async close

Syslog¶

SysLogHandler¶

Syslog Generator¶

Syslog server¶

Running¶

Configuration¶

Data backend¶

Web UI¶

Python implementation¶

Table of Contents

Previous topic

Next topic