Syslog¶
Real time reporting of execution state is mandatory functionality for most of Hat’s components implementing continuously running service. Primary way of logging is based on Syslog logging protocol as defined by:
Each component which reports log messages should implement Syslog TCP client. In this way, all logging messages can be aggregated in a single Syslog concentrator which can provide archiving, searching and real time monitoring functionalities.
Care must be taken for Syslog TCP client logging facility implementation not to interfere with other component’s functionalities. Logging should be considered best effort and not critical activity of each component.
SysLogHandler¶
SysLogHandler provides implementation of Python’s standard library logging.Handler interface. This implementation is part of hat-syslog python package.
Each instance of hat.syslog.handler.SysLogHandler starts new
background thread responsible for sending syslog messages over TCP, UDP or
TCP+SSL socket. If connection with remote syslog server could not be
established or current connection is closed, new connect is called after 5
second timeout.
All log messages provided to SysLogHandler by
hat.syslog.handler.SysLogHandler.emit() are queued in queue with
maximum size limited by configuration parameter. During new log message
registration, if queue is full, oldest messages are discarded and counter
of discarded messages is incremented. Once new log messages can be sent
to server, new log message containing information about number of discarded
messages will be sent.
Syslog Generator¶
Syslog generator is application used as simple testing tool responsible for generating syslog messages that can be sent to Syslog Server.
Implemented as python hat.syslog.generator module which can be run with
hat-syslog-generator script with additional command line arguments:
usage: hat-syslog-generator [-h] [--count N] [--text text] [--queue-size N] [--msg-delay t] [--end-delay t] optional arguments: -h, --help show this help message and exit --count N number of log messages (default 1) --text text log message text (default 'syslog generator test') --queue-size N client's log message queue size (default 1024) --msg-delay t time delay between message generation (seconds) (default 0.01) --end-delay t time delay affter all messages have been generated (default 0.5)
This application is part of hat-syslog python package.
Syslog server¶
Syslog server is central concentration for syslog messages. Additionally, it provides web interface for real time monitoring and filtering of log messages.
Syslog listening socket can be configured as TCP, UDP or TCP+SSL socket. Communication is based on RFC 5425, RFC 5426, RFC 6587. Once message is received, server stores message in predefined database.
Warning
Current implementation of Syslog server doesn’t support UDP communication
Running¶
Syslog Server is implemented as python hat.syslog.server package which
can be run with hat-syslog script with additional command line
arguments:
usage: hat-syslog [-h] [--conf path] [--log level] [--syslog-addr address] [--syslog-pem path] [--ui-addr address] [--ui-pem path] [--db-path path] [--db-low-size count] [--db-high-size count] [--db-enable-archive] [--db-disable-journal] [--ui-path path] optional arguments: -h, --help show this help message and exit --conf path configuration defined by hat://syslog/server.yaml# (default $XDG_CONFIG_HOME/hat/syslog.yaml) configuration arguments: --log level console log level --syslog-addr address syslog listening address (default: tcp://0.0.0.0:6514) --syslog-pem path pem file path - mandatory for ssl communication --ui-addr address ui listening address (default: http://0.0.0.0:23020) --ui-pem path pem file path - mandatory for https communication --db-path path sqlite database file path (default: $HATPATH/syslog.db) --db-low-size count number of messages kept in database after database cleanup (default: 1000000) --db-high-size count number of messages that will trigger database cleanup (default: 10000000) --db-enable-archive should messages, deleted during database cleanup, be kept in archive files --db-disable-journal disable sqlite jurnaling development arguments: --ui-path path override web ui directory path
This application is part of hat-syslog python package.
Configuration¶
Syslog Server configuration written in form of single YAML file with structure
defined by JSON Schema hat://syslog/server.yaml#. Path to configuration
file is provided as command line argument during process startup. Additionally,
configuration parameters provided in configuration file can be overridden
by command line arguments. If configuration file could not be found,
default values of configuration parameters are used.
Example of configuration:
---
log:
version: 1
syslog_addr: 'tcp://0.0.0.0:6514'
ui_addr: 'http://0.0.0.0:23020'
dp_path: '$HATPATH/syslog.db'
db_low_size: 1_000_000
db_high_size: 10_000_000
db_enable_archive: false
db_disable_journal: false
...
Data backend¶
All incoming syslog messages are stored in single sqlite database. Maximum
number of syslog messages stored in this database can be configured by
configuration parameter db_high_size (value 0 represents unlimited
number of messages). Once number of messages exceed configured limit,
database cleanup procedure is triggered. During cleanup procedure, oldest
messages are removed from database until number of messages reaches
configuration parameter db_low_size when cleanup procedure stops. Prior
to message deletion, if configuration parameter db_enable_archive
is set, new database with unique file name is created and all messages
scheduled for removal are inserted into newly created database. Archive
database has got same structure as original database and can be used in place
of original database for accessing archived syslog messages.
Web UI¶
Together with acquiring and storing syslog messages, Syslog Server provides web-based user interface for querying messages from database and observing changes in real time. Communication between web server and browser is based on juggler communication.
Once juggler connection is established, server and client should change
initial null local state to theirs current valid value. Server’s local state
is defined by #/definitions/server and client’s local state is defined by
#/definitions/client from JSON schema:
"$schema": "http://json-schema.org/schema#"
definitions:
client:
"$ref": "#/definitions/filter"
server:
type: object
required:
- filter
- entries
- first_id
- last_id
properties:
filter:
"$ref": "#/definitions/filter"
entries:
type: array
items:
"$ref": "#/definitions/entry"
first_id:
type:
- 'null'
- integer
last_id:
type:
- 'null'
- integer
filter:
type: object
required:
- max_results
- last_id
- entry_timestamp_from
- entry_timestamp_to
- facility
- severity
- hostname
- app_name
- procid
- msgid
- msg
properties:
max_results:
type:
- 'null'
- integer
last_id:
type:
- 'null'
- integer
entry_timestamp_from:
type:
- 'null'
- number
entry_timestamp_to:
type:
- 'null'
- number
facility:
oneOf:
- type: 'null'
- "$ref": "#/definitions/facility"
severity:
oneOf:
- type: 'null'
- "$ref": "#/definitions/severity"
hostname:
type:
- 'null'
- string
app_name:
type:
- 'null'
- string
procid:
type:
- 'null'
- string
msgid:
type:
- 'null'
- string
msg:
type:
- 'null'
- string
entry:
type: object
required:
- id
- timestamp
- msg
properties:
id:
type: integer
timestamp:
type: number
msg:
"$ref": "#/definitions/msg"
msg:
type: object
required:
- facility
- severity
- version
- timestamp
- hostname
- app_name
- procid
- msgid
- data
- msg
properties:
facility:
oneOf:
- type: 'null'
- "$ref": "#/definitions/facility"
severity:
oneOf:
- type: 'null'
- "$ref": "#/definitions/severity"
version:
type: integer
timestamp:
type:
- 'null'
- number
hostname:
type:
- 'null'
- string
app_name:
type:
- 'null'
- string
procid:
type:
- 'null'
- string
msgid:
type:
- 'null'
- string
data:
type:
- 'null'
- string
msg:
type:
- 'null'
- string
facility:
enum:
- KERNEL
- USER
- MAIL
- SYSTEM
- AUTHORIZATION1
- INTERNAL
- PRINTER
- NETWORK
- UUCP
- CLOCK1
- AUTHORIZATION2
- FTP
- NTP
- AUDIT
- ALERT
- CLOCK2
- LOCAL0
- LOCAL1
- LOCAL2
- LOCAL3
- LOCAL4
- LOCAL5
- LOCAL6
- LOCAL7
severity:
enum:
- EMERGENCY
- ALERT
- CRITICAL
- ERROR
- WARNING
- NOTICE
- INFORMATIONAL
- DEBUG
Juggler MESSAGE messages are not used in communication.
When server detected change of client’s local data, it should update its local data to match filter from client’s local data.