## panko.storage package

updated: 2020-10-17 18:35

# panko.storage package

## Subpackages

- panko.storage.mongo package
  - Submodules
  - panko.storage.mongo.utils module
  - Module contents
- panko.storage.sqlalchemy package
  - Submodules
  - panko.storage.sqlalchemy.models module
  - Module contents

## Submodules

## panko.storage.base module

Base classes for storage engines

class `panko.storage.base.``Connection`(conf)

  Bases: `object`

  Base class for event storage system connections.

  `CAPABILITIES` = {'events': {'query': {'simple': False}}}
  `STORAGE_CAPABILITIES` = {'storage': {'production_ready': False}}
  static `clear`()

    Clear database.

  static `clear_expired_data`(ttl, max_count=None)

    Clear expired data from the backend storage system.

    Clearing occurs according to the time-to-live. :param ttl: Number of seconds to keep records for. :param max_count: Number of records to delete.

  classmethod `get_capabilities`()

    Return an dictionary with the capabilities of each driver.

  static `get_event_types`()

    Return all event types as an iterable of strings.

  static `get_events`(event_filter, pagination=None)

    Return an iterable of model.Event objects.

  classmethod `get_storage_capabilities`()

    Return a dictionary representing the performance capabilities.

    This is needed to evaluate the performance of each driver.

  static `get_trait_types`(event_type)

    Return a dictionary containing the name and data type of the trait.

    Only trait types for the provided event_type are returned. :param event_type: the type of the Event

  static `get_traits`(event_type, trait_type=None)

    Return all trait instances associated with an event_type.

    If trait_type is specified, only return instances of that trait type. :param event_type: the type of the Event to filter by :param trait_type: the name of the Trait to filter by

  static `record_events`(events)

    Write the events to the backend storage system.

    Parameters

      events – a list of model.Event objects.

  static `upgrade`()

    Migrate the database to version or the most recent version.

class `panko.storage.base.``Model`(**kwds)

  Bases: `object`

  Base class for storage API models.

  `as_dict`()

## panko.storage.impl_elasticsearch module

class `panko.storage.impl_elasticsearch.``Connection`(url, conf)

  Bases: `panko.storage.base.Connection`

  Put the event data into an ElasticSearch db.

  Events in ElasticSearch are indexed by day and stored by event_type. An example document:

      {"_index":"events_2014-10-21",
       "_type":"event_type0",
       "_id":"dc90e464-65ab-4a5d-bf66-ecb956b5d779",
       "_score":1.0,
       "_source":{"timestamp": "2014-10-21T20:02:09.274797"
                  "traits": {"id4_0": "2014-10-21T20:02:09.274797",
                             "id3_0": 0.7510790937279408,
                             "id2_0": 5,
                             "id1_0": "18c97ba1-3b74-441a-b948-a702a30cbce2"}
                 }
      }

  `CAPABILITIES` = {'events': {'query': {'simple': True}}}
  `STORAGE_CAPABILITIES` = {'storage': {'production_ready': True}}
  `get_event_types`()

    Return all event types as an iterable of strings.

  `get_events`(event_filter, pagination=None)

    Return an iterable of model.Event objects.

  `get_trait_types`(event_type)

    Return a dictionary containing the name and data type of the trait.

    Only trait types for the provided event_type are returned. :param event_type: the type of the Event

  `get_traits`(event_type, trait_type=None)

    Return all trait instances associated with an event_type.

    If trait_type is specified, only return instances of that trait type. :param event_type: the type of the Event to filter by :param trait_type: the name of the Trait to filter by

  `record_events`(events)

    Write the events to the backend storage system.

    Parameters

      events – a list of model.Event objects.

  `upgrade`()

    Migrate the database to version or the most recent version.

## panko.storage.impl_log module

class `panko.storage.impl_log.``Connection`(conf)

  Bases: `panko.storage.base.Connection`

  Log event data.

  static `clear_expired_data`(ttl, max_count)

    Clear expired data from the backend storage system.

    Clearing occurs according to the time-to-live.

    Parameters

      - ttl – Number of seconds to keep records for.

      - max_count – Number of records to delete.

## panko.storage.impl_mongodb module

MongoDB storage backend

class `panko.storage.impl_mongodb.``Connection`(url, conf)

  Bases: `panko.storage.pymongo_base.Connection`

  Put the event data into a MongoDB database.

  `CONNECTION_POOL` = <panko.storage.mongo.utils.ConnectionPool object>
  `clear`()

    Clear database.

  `clear_expired_data`(ttl, max_count=None)

    Clear expired data from the backend storage system.

    Clearing occurs according to the time-to-live.

    Parameters

      - ttl – Number of seconds to keep records for.

      - max_count – Number of records to delete (not used for MongoDB).

  static `update_ttl`(ttl, ttl_index_name, index_field, coll)

    Update or create time_to_live indexes.

    Parameters

      - ttl – time to live in seconds.

      - ttl_index_name – name of the index we want to update or create.

      - index_field – field with the index that we need to update.

      - coll – collection which indexes need to be updated.

  `upgrade`()

    Migrate the database to version or the most recent version.

## panko.storage.impl_sqlalchemy module

SQLAlchemy storage backend.

class `panko.storage.impl_sqlalchemy.``Connection`(url, conf)

  Bases: `panko.storage.base.Connection`

  Put the event data into a SQLAlchemy database.

  Tables:

      - EventType
        - event definition
        - { id: event type id
            desc: description of event
            }
      - Event
        - event data
        - { id: event id
            message_id: message id
            generated = timestamp of event
            event_type_id = event type -> eventtype.id
            }
      - TraitInt
        - int trait value
        - { event_id: event -> event.id
            key: trait name
            value: integer value
            }
      - TraitDatetime
        - datetime trait value
        - { event_id: event -> event.id
            key: trait name
            value: datetime value
            }
      - TraitText
        - text trait value
        - { event_id: event -> event.id
            key: trait name
            value: text value
            }
      - TraitFloat
        - float trait value
        - { event_id: event -> event.id
            key: trait name
            value: float value
            }

  `CAPABILITIES` = {'events': {'query': {'simple': True}}}
  `STORAGE_CAPABILITIES` = {'storage': {'production_ready': True}}
  `clear`()

    Clear database.

  `clear_expired_data`(ttl, max_count)

    Clear expired data from the backend storage system.

    Clearing occurs according to the time-to-live.

    Parameters

      - ttl – Number of seconds to keep records for.

      - max_count – Number of records to delete.

  static `dress_url`(url)
  `get_event_types`()

    Return all event types as an iterable of strings.

  `get_events`(event_filter, pagination=None)

    Return an iterable of model.Event objects.

    Parameters

      - event_filter – EventFilter instance

      - pagination – Pagination parameters.

  `get_trait_types`(event_type)

    Return a dictionary containing the name and data type of the trait.

    Only trait types for the provided event_type are returned. :param event_type: the type of the Event

  `get_traits`(event_type, trait_type=None)

    Return all trait instances associated with an event_type.

    If trait_type is specified, only return instances of that trait type. :param event_type: the type of the Event to filter by :param trait_type: the name of the Trait to filter by

  `record_events`(event_models)

    Write the events to SQL database via sqlalchemy.

    Parameters

      event_models – a list of model.Event objects.

  `upgrade`()

    Migrate the database to version or the most recent version.

## panko.storage.models module

Model classes for use in the events storage API.

class `panko.storage.models.``Event`(message_id, event_type, generated, traits, raw)

  Bases: `panko.storage.base.Model`

  A raw event from the source system. Events have Traits.

  Metrics will be derived from one or more Events.

  `DEFAULT_DIR` = 'asc'
  `DEFAULT_SORT` = [('generated', 'asc'), ('message_id', 'asc')]
  `DUPLICATE` = 1
  `INCOMPATIBLE_TRAIT` = 3
  `PRIMARY_KEY` = 'message_id'
  `SUPPORT_DIRS` = ('asc', 'desc')
  `SUPPORT_SORT_KEYS` = ('message_id', 'generated')
  `UNKNOWN_PROBLEM` = 2
  `append_trait`(trait_model)
  `serialize`()

class `panko.storage.models.``Trait`(name, dtype, value)

  Bases: `panko.storage.base.Model`

  A Trait is a key/value pair of data on an Event.

  The value is variant record of basic data types (int, date, float, etc).

  `DATETIME_TYPE` = 4
  `FLOAT_TYPE` = 3
  `INT_TYPE` = 2
  `NONE_TYPE` = 0
  `TEXT_TYPE` = 1
  classmethod `convert_value`(trait_type, value)
  classmethod `get_name_by_type`(type_id)
  classmethod `get_type_by_name`(type_name)
  `get_type_name`()
  classmethod `get_type_names`()
  `serialize`()
  `type_names` = {0: 'none', 1: 'string', 2: 'integer', 3: 'float', 4: 'datetime'}

`panko.storage.models.``serialize_dt`(value)

  Serializes parameter if it is datetime.

## panko.storage.pymongo_base module

Common functions for MongoDB backend

class `panko.storage.pymongo_base.``Connection`(conf)

  Bases: `panko.storage.base.Connection`

  Base event Connection class for MongoDB driver.

  `CAPABILITIES` = {'events': {'query': {'simple': True}}}
  `STORAGE_CAPABILITIES` = {'storage': {'production_ready': True}}
  `get_event_types`()

    Return all event types as an iter of strings.

  `get_events`(event_filter, pagination=None)

    Return an iter of models.Event objects.

    Parameters

      - event_filter – storage.EventFilter object, consists of filters for events that are stored in database.

      - pagination – Pagination parameters.

  `get_trait_types`(event_type)

    Return a dictionary containing the name and data type of the trait.

    Only trait types for the provided event_type are returned.

    Parameters

      event_type – the type of the Event.

  `get_traits`(event_type, trait_name=None)

    Return all trait instances associated with an event_type.

    If trait_type is specified, only return instances of that trait type.

    Parameters

      - event_type – the type of the Event to filter by

      - trait_name – the name of the Trait to filter by

  `record_events`(event_models)

    Write the events to database.

    Parameters

      event_models – a list of models.Event objects.

## Module contents

Storage backend management

class `panko.storage.``EventFilter`(start_timestamp=None, end_timestamp=None, event_type=None, message_id=None, traits_filter=None, admin_proj=None)

  Bases: `object`

  Properties for building an Event query.

  Parameters

    - start_timestamp – UTC start datetime (mandatory)

    - end_timestamp – UTC end datetime (mandatory)

    - event_type – the name of the event. None for all.

    - message_id – the message_id of the event. None for all.

    - admin_proj – the project_id of admin role. None if non-admin user.

    - traits_filter – the trait filter dicts, all of which are optional. This parameter is a list of dictionaries that specify trait values:

      {'key': <key>,
      'string': <value>,
      'integer': <value>,
      'datetime': <value>,
      'float': <value>,
      'op': <eq, lt, le, ne, gt or ge> }

exception `panko.storage.``InvalidMarker`

  Bases: `Exception`

  Invalid pagination marker parameters

exception `panko.storage.``StorageBadAggregate`

  Bases: `Exception`

  Error raised when an aggregate is unacceptable to storage backend.

  `code` = 400

exception `panko.storage.``StorageBadVersion`

  Bases: `Exception`

  Error raised when the storage backend version is not good enough.

exception `panko.storage.``StorageUnknownWriteError`

  Bases: `Exception`

  Error raised when an unknown error occurs while recording.

`panko.storage.``get_connection`(url, conf)

  Return an open connection to the database.

`panko.storage.``get_connection_from_config`(conf)

updated: 2020-10-17 18:35

Except where otherwise noted, this document is licensed under Creative Commons Attribution 3.0 License. See all OpenStack Legal Documents.

questions?