Event Server¶
Event Server is a central component responsible for registering, processing, storing and providing access to events.
Running¶
By installing Event Server from hat-event package, executable hat-event becomes available and can be used for starting this component.
Usage: __main__.py [OPTIONS] Main entry point Options: --conf PATH configuration defined by hat://event/main.yaml# (default $XDG_CONFIG_HOME/hat/event.{yaml|yml|json}) --help Show this message and exit.
Event¶
Event is a generic data structure used for communication and storage of relevant changes happening through time in a Hat system. For example, event can represent a data change triggered by a device outside of the system, or an inner notification created by a component inside the system. Each event is immutable and uniquely identified by its event id. Event Server is the only component responsible for creating events - all other components shall request Event Server to create a new event.
Event data structure:
id
A unique Event identifier containing Event Server identifier and event instance identifier. This property is always set by the server.
type
Type is a user (client) defined list of strings. Semantics of the list’s elements and their position in the list is determined by the user and is not predefined by the Event server. This list should be used as the main identifier of the occurred event’s type. Each component registering an event should have its own naming convention defined which does not collide with other components’ naming conventions. This property is set by the user while registering an event. Subtypes
?and*are not allowed as parts of event type.When used in querying and subscription, this property has additional semantics. Any string in the list can be replaced with
?while the last string can also be replaced with*. Replacements must substitute an entire string in the list. The semantics of these replacements are:
The string
?is matched with a single arbitrary string.
The string
*is matched with any number (zero or more) of arbitrary strings.
timestamp
This property determines the moment when the event was registered on the server. It is always set by the server.
source timestamp
This property is optional. It represents the moment the Event occurred as detected by the source of the Event. It is always set by the client.
payload
This property is optional. It can be used to provide additional data bound to an event. The payload property is always set by the client registering the event. Its contents are determined by the Event’s type and can be decoded by clients who understand Events of that type. Payload can be encoded as binary data, JSON data or SBS encoded data. Event server doesn’t decode payload while receiving event requests, storing events or providing query results. Payload can be optionally decoded by Event Server’s modules.
Server - Client communication¶
Communication between server and client is based on chatter communication utilizing following messages:
Message
Conversation
Direction
First
Last
Token
MsgSubscribe
T
T
T
c > s
MsgNotify
T
T
T
s > c
MsgRegisterReq
T
T/F
T
c > s
MsgRegisterRes
F
T
T
s > c
MsgQueryReq
T
F
T
c > s
MsgQueryRes
F
T
T
s > c
Actions available to clients which are directly mapped to exchange of communication messages:
subscribe
A client can, at any time, subscribe to Events of certain types by sending a subscribe message (MsgSubscribe) to the server. After server receives subscribe message, it will spontaneously notify the client (by sending MsgNotify) whenever an Event occurs with the type matched to any type in the subscription message. Matching is done as described in event’s type property including the
?and*options. A client can send as many subscribe messages as it wants, each new subscription message implicitly invalidates previous subscriptions. Initially, after new connection between server and client is established, client isn’t subscribed to any events. Both subscribe and notify messages can be sent at any time independently of other communication messages. Events that are notified by single MsgNotify are mutually unrelated.register event
A client can, at any time, send new request for event registration. Those register requests are sent as part of MsgRegisterReq message. Single MsgRegisterReq may contain an arbitrary number of registration requests which are all registered at the same time. Single register event contains event type; and optional source timestamp and payload. Upon receiving MsgRegisterReq, it is responsibility of a server to create new event for each register event. All events created based on a single MsgRegisterReq have the same timestamp. If a client doesn’t end chatter conversation (MsgRegisterReq last flag is false), once associated events are created server will respond with MsgRegisterRes and end conversation. For each register event in MsgRegisterReq, associated MsgRegisterRes contains newly created event, or information about event registration failure.
query events
At any time, client can initiate new event query by sending MsgQueryReq message. Upon receiving query request, server will provide all available events that match query criteria as part of single MsgQueryRes. Single query request can contain multiple filter conditions which ALL must be met for all events provided to client as query result. Query request contains:
ids - optional filter condition
If set, only events with ids which are defined as part of filter condition are matched.
types - optional filter condition
List of event types. If set, event type has to match at least one type from the list. Matching is done as defined in event’s type property description - including the
?and*options.from timestamp - optional filter condition
If set, only events with timestamp greater than or equal are matched.
to timestamp - optional filter condition
If set, only events with timestamp lower than or equal are matched.
from source timestamp - optional filter condition
If set, only events with source timestamp defined, and greater than or equal, are matched.
to source timestamp - optional filter condition
If set, only events with source timestamp defined, and lower than or equal, are matched.
payload - optional filter condition
If set, only events with payload defined and whose payload is the same as the query’s payload are matched.
order
Can be set to ‘ascending’ or ‘descending’. If set to ‘ascending’, matched Events will be returned ordered from the earliest to the latest dependent on their timestamp or source timestamp (this choice is determined by the order by property of the query). Earliest meaning lower timestamp, latest meaning greater timestamp. If set to descending the same logic applies, but the order is reversed.
order by
Can be set to ‘timestamp’ or ‘source timestamp’. Ordering Events by ‘source timestamp’ has events with ‘source timestamp’ undefined returned last in an arbitrary order.
unique type
If set to
true, it determines whether the matched Events will contain only one event instance of the same type. With the query ‘order’ set to ‘descending’, only one Event with the greatest timestamp or source timestamp will be matched. Setting the ‘order’ to ‘ascending’ will match the Event with the lowest timestamp or source timestamp.max results
If set, limits the number of matched Events to this number. Matched Events are dependent on the query ‘order’ the same way as in ‘unique type’.
Server - Server communication¶
Todo
define backend engine sync messages
Components¶
Event Server functionality can be defined by using the following components:
Client¶
Event client is any component that provides client functionality in Server - Client communication. Package hat-event provides python implementation of hat.event.client module which can be used as a basis for communication with Event Server. This module provides low-level and high-level communication API. For more detail see documentation of hat.event.client module.
Communication¶
Event Server’s communication module is responsible for providing implementation of server side Server - Client communication. This component translates client requests to module engine’s method calls. At the same time, it observes all new event notifications made by module engine and notifies clients with appropriate messages.
RegisterEvent objects obtained from client’s register requests must be converted to ProcessEvent before they can be passed for further processing to module engine. This conversion is done by module engine, as it is the only entity responsible for creating new ProcessEvent objects.
A unique identifier is assigned to each chatter connection established with communication (unique for the single execution lifetime of Event Server process). This identifier is associated with all ProcessEvent objects obtained from corresponding connection.
Communication associates connection with information received as part of connection’s last subscribe message. This subscription is used as a filter for selecting subset of event notifications which are sent to associated connection.
Communication module is responsible for registering events each time new chatter connection is established and existing chatter connection is closed:
‘event’, ‘communication’, ‘connected’
source timestamp - None
payload - None
‘event’, ‘communication’, ‘disconnected’
source timestamp - None
payload - None
Module engine¶
Module engine is responsible for creating modules and coordinating event registration, processing and querying between communication, modules and backend engine.
Module engine provides method for creating process events utilized by communication and modules. By creating process events, register events are enhanced with unique identifier and source identification. Identifier assigned to process event is the same one that is assigned to corresponding event. Information regarding source identifier is available only during processing of process event and is discarded once event is created.
Process of creating events based on a single set of process events is called session. Module engine starts new session each time communication or module requests new registration. Session ends once backend engine returns result of event registration and all modules are notified with this result. Start and end of each session is notified to each module by creating and closing module session. Each module instantiates its own module session.
During session processing, each module session is notified with a list of new and deleted process events which are not previously presented to that module. Processing these process events by module session can result in new process events which are to be added to current session or list of previously added process events which are to be deleted from session. All other module sessions, except the one that produced list of new and deleted process events, are notified with those process event lists. This process continues iteratively until all module sessions return empty lists for both new and deleted process events. Processing process events by single module session is always sequential - module session is notified with session changes after its previous notification processing is finished. Different module sessions may be processed concurrently. Module engine keeps order of new process events added to session, but it is allowed to aggregate processing results from multiple module sessions into a single session change notification.
Care should be taken by module implementation not to cause self recursive or mutually recursive endless processing loop.
Each module can define its event type filter condition which is used for filtering new and deleted process events that will get notified to module session. When session finishes, module session is closed by calling its async_close method, which receives a list of all newly created events resulting from the session processing. These events are not filtered by module subscription. By calling async_close with list of all newly registered events, module engine provides each module session with opportunity to post-process single session as a whole.
Todo
do we want to filter resulting events passed to module session close with module’s subscription?
Modules¶
Warning
Event server does not provide sandbox environment for loading end executing modules. Modules have full access to Event Server functionality which is controlled with module execution. Module implementation and configuration should be written in accordance to other modules and Event Server as a whole, keeping in mind processing execution time overhead and possible interference between modules.
Each module represents predefined and configurable closely related functions that can modify the process of registering new events or initiate new event registration sessions. When created, module is provided with reference to module engine which can be used for creating new process events, registering process events and querying events. Responsibility of each module, upon creation, is to create its own source identifier which will be unique for single Event Server process execution.
Modules available as part of hat-event package:
Backend engine¶
Backend engine is responsible for actions involving persisting process events and querying events. During initialization, backend engine creates a single instance of backend which is used for storage.
During registration of events, backend engine converts process events to backend events. This conversion involves setting event’s timestamp (which is the same for all events registered in a single session) and replacing event type with event type identifier. To enable this, backend engine has to maintain association reference between event types and corresponding event type identifiers. By replacing event types with identifiers, list of strings which represent event type is replaced with a single numeric identifier.
Query request which is passed to backend engine contains event types as optional filter condition. Responsibility of backend engine is to find all matching event type identifiers and query backend based on a list of identifiers.
Todo
can we associate type identifiers based on subtypes and utilize identifier ranges instead of explicitly passing each identifier to backend
Todo
sync between backend engines
Backends¶
Backends are simple wrappers for storing and retrieving events from specialized storage engines.
Backends available as part of hat-event package: