Syslog¶
Real time reporting of execution state is mandatory functionality for most of Hat’s components implementing continuously running service. Primary way of logging is based on Syslog logging protocol as defined by:
Each component which reports log messages should implement Syslog TCP client. In this way, all logging messages can be aggregated in a single Syslog concentrator which can provide archiving, searching and real time monitoring functionalities.
Care must be taken for Syslog TCP client logging facility implementation not to interfere with other component’s functionalities. Logging should be considered best effort and not critical activity of each component.
SysLogHandler¶
SysLogHandler provides implementation of Python’s standard library logging.Handler interface. This implementation is part of hat-syslog python package.
Each instance of hat.syslog.handler.SysLogHandler starts new background thread responsible for sending syslog messages over TCP, UDP or TCP+SSL socket. If connection with remote syslog server could not be established or current connection is closed, new connect is called after 5 second timeout.
All log messages provided to SysLogHandler by hat.syslog.handler.SysLogHandler.emit are queued in queue with maximum size limited by configuration parameter. During new log message registration, if queue is full, oldest messages are discarded and counter of discarded messages is incremented. Once new log messages can be sent to server, new log message containing information about number of discarded messages will be sent.
Syslog Generator¶
Syslog generator is application used as simple testing tool responsible for generating syslog messages that can be sent to Syslog Server.
Implemented as python hat.syslog.generator module which can be run with
hat-syslog-generator script with additional command line arguments:
Usage: generator.py [OPTIONS] Syslog test message generator Options: --host HOST syslog server host name [default: 127.0.0.1] --port PORT syslog server port [default: 6514] --count N number of log messages [default: 1] --text TEXT log message text [default: syslog generator test] --queue-size N client's log message queue size [default: 1024] --msg-delay T time delay between message generation in seconds [default: 0.01] --end-delay T time delay affter all messages have been generated in seconds [default: 0.5] --help Show this message and exit.
This application is part of hat-syslog python package.
Syslog server¶
Syslog server is central concentration for syslog messages. Additionally, it provides web interface for real time monitoring and filtering of log messages.
Syslog listening socket can be configured as TCP, UDP or TCP+SSL socket. Communication is based on RFC 5425, RFC 5426, RFC 6587. Once message is received, server stores message in predefined database.
Warning
Current implementation of Syslog server doesn’t support UDP communication
Running¶
Syslog Server is implemented as python hat.syslog.server package which
can be run with hat-syslog script with additional command line
arguments:
Usage: __main__.py [OPTIONS] Syslog Server main Options: --conf PATH configuration defined by hat://syslog.yaml# (default $XDG_CONFIG_HOME/hat/syslog.{yaml|yml|json}) --log LEVEL console log level --syslog-addr ADDR syslog listening address (default tcp://0.0.0.0:6514) --syslog-pem PATH pem file path - mandatory for ssl communication --ui-addr ADDR UI listening address (default http://0.0.0.0:23020) --ui-pem PATH pem file path - mandatory for https communication --ui-path PATH override web ui directory path (development argument) --db-path PATH sqlite database file path (default $XDG_DATA_HOME/hat/syslog.db) --db-low-size N number of messages kept in database after database cleanup (default 1000000) --db-high-size N number of messages that will trigger database cleanup (default 10000000) --db-enable-archive should messages, deleted during database cleanup, be kept in archive files --db-disable-journal disable sqlite jurnaling --help Show this message and exit.
This application is part of hat-syslog python package.
Configuration¶
Syslog Server configuration written in form of single YAML or JSON file with
structure defined by JSON Schema hat://syslog/server.yaml#. Path to
configuration file is provided as command line argument during process startup.
Additionally, configuration parameters provided in configuration file can be
overridden by command line arguments. If configuration file could not be found,
default values of configuration parameters are used.
Example of configuration:
---
log:
version: 1
syslog_addr: 'tcp://0.0.0.0:6514'
ui_addr: 'http://0.0.0.0:23020'
dp_path: 'syslog.db'
db_low_size: 1_000_000
db_high_size: 10_000_000
db_enable_archive: false
db_disable_journal: false
...
Data backend¶
All incoming syslog messages are stored in single sqlite database. Maximum
number of syslog messages stored in this database can be configured by
configuration parameter db_high_size (value 0 represents unlimited
number of messages). Once number of messages exceed configured limit,
database cleanup procedure is triggered. During cleanup procedure, oldest
messages are removed from database until number of messages reaches
configuration parameter db_low_size when cleanup procedure stops. Prior
to message deletion, if configuration parameter db_enable_archive
is set, new database with unique file name is created and all messages
scheduled for removal are inserted into newly created database. Archive
database has got same structure as original database and can be used in place
of original database for accessing archived syslog messages.
Web UI¶
Together with acquiring and storing syslog messages, Syslog Server provides web-based user interface for querying messages from database and observing changes in real time. Communication between web server and browser is based on juggler communication.
Once juggler connection is established, server and client should change
initial null local state to theirs current valid value. Server’s local state
is defined by #/definitions/server and client’s local state is defined by
#/definitions/client from JSON schema:
"$schema": "http://json-schema.org/schema#"
definitions:
client:
"$ref": "#/definitions/filter"
server:
type: object
required:
- filter
- entries
- first_id
- last_id
properties:
filter:
"$ref": "#/definitions/filter"
entries:
type: array
items:
"$ref": "#/definitions/entry"
first_id:
type:
- 'null'
- integer
last_id:
type:
- 'null'
- integer
filter:
type: object
required:
- max_results
- last_id
- entry_timestamp_from
- entry_timestamp_to
- facility
- severity
- hostname
- app_name
- procid
- msgid
- msg
properties:
max_results:
type:
- 'null'
- integer
last_id:
type:
- 'null'
- integer
entry_timestamp_from:
type:
- 'null'
- number
entry_timestamp_to:
type:
- 'null'
- number
facility:
oneOf:
- type: 'null'
- "$ref": "#/definitions/facility"
severity:
oneOf:
- type: 'null'
- "$ref": "#/definitions/severity"
hostname:
type:
- 'null'
- string
app_name:
type:
- 'null'
- string
procid:
type:
- 'null'
- string
msgid:
type:
- 'null'
- string
msg:
type:
- 'null'
- string
entry:
type: object
required:
- id
- timestamp
- msg
properties:
id:
type: integer
timestamp:
type: number
msg:
"$ref": "#/definitions/msg"
msg:
type: object
required:
- facility
- severity
- version
- timestamp
- hostname
- app_name
- procid
- msgid
- data
- msg
properties:
facility:
oneOf:
- type: 'null'
- "$ref": "#/definitions/facility"
severity:
oneOf:
- type: 'null'
- "$ref": "#/definitions/severity"
version:
type: integer
timestamp:
type:
- 'null'
- number
hostname:
type:
- 'null'
- string
app_name:
type:
- 'null'
- string
procid:
type:
- 'null'
- string
msgid:
type:
- 'null'
- string
data:
type:
- 'null'
- string
msg:
type:
- 'null'
- string
facility:
enum:
- KERNEL
- USER
- MAIL
- SYSTEM
- AUTHORIZATION1
- INTERNAL
- PRINTER
- NETWORK
- UUCP
- CLOCK1
- AUTHORIZATION2
- FTP
- NTP
- AUDIT
- ALERT
- CLOCK2
- LOCAL0
- LOCAL1
- LOCAL2
- LOCAL3
- LOCAL4
- LOCAL5
- LOCAL6
- LOCAL7
severity:
enum:
- EMERGENCY
- ALERT
- CRITICAL
- ERROR
- WARNING
- NOTICE
- INFORMATIONAL
- DEBUG
Juggler MESSAGE messages are not used in communication.
When server detected change of client’s local data, it should update its local data to match filter from client’s local data.