Using QuantumLeap

First you need to have QuantumLeap and its complementary services running (of course). Refer to the Installation Manual for instructions.

Then, you need to connect Orion Context Broker to QuantumLeap through an NGSIv2 subscription for each Entity Type whose historical data you are interested in.

Not an Orion expert yet? No problem, you can create the subscription through QuantumLeap's API. However, you will still need to understand the basics of how subscriptions and notifications work, so take a time to read the docs.

Historical data for each entity type will be persisted as long as the subscription is active, correctly configured and the entity in the notification is NGSI compliant.

So, summing up, the usage flow is therefore the following...

  • Create an Orion subscription for each entity type of your interest
  • Insert/Update your data in Orion as usual
  • Your historical data will be persisted in QuantumLeap's database

Let's take a closer look at each step.

Orion Subscription

As stated before, the link between Orion and QuantumLeap is established through a subscription you need to create. It is therefore important that you understand well how the NGSIv2 Subscription mechanism works. This is carefully explained in the corresponding section of Orion docs.

To create the subscription, QuantumLeap offers an API endpoint documented here at /subscribe.

Alternatively, you can directly talk to Orion and create the subscription as you prefer. Here's an example of the payload of the subscription you need to create in Orion to establish the link Orion-QuantumLeap.

    {
        "description": "Test subscription",
        "subject": {
            "entities": [
            {
                "idPattern": ".*",
                "type": "Room"
            }
            ],
            "condition": {
                "attrs": [
                "temperature"
                ]
            }
        },
        "notification": {
            "http": {
                "url": "http://quantumleap:8668/v2/notify"
            },
            "attrs": [
            "temperature"
            ],
            "metadata": ["dateCreated", "dateModified"]
        },
        "throttling": 5
    }

Important things to notice from the subscription.

  • Notifications must come in complete NGSI JSON Entity Representation form. Other forms, such as the simplified entity representation are NOT supported by QL because they lack information on attribute types, required by QL to make proper translations. This means, DO NOT use options like "attrsFormat": "keyValues"

  • The "url" field of the subscription specifies where Orion will send the notifications. I.e, this must be QuantumLeap's /v2/notify endpoint. By default, QuantumLeap listens at port 8668. This url must be resolvable from Orion's container, so avoid using localhost or something that will not translate either by Docker, your /etc/hosts or DNS to the endpoint where QuantumLeap is running.

  • Though not compulsory, it is highly recommended to include the "metadata": ["dateCreated", "dateModified"] part in the notification part of the subscription. This instructs Orion to include the modification time of the attributes in the notification. This timestamp will be used as the time index in the database. If this is somehow missing, QuantumLeap will use its current system time at which the notification arrived, which might not be exactly what you want.

  • You can create any valid NGSI subscription for your entities respecting the previous rules.

Data Insertion

By now, it should be clear you don't typically insert data directly into QuantumLeap. You insert into Orion and Orion notifies QuantumLeap. For inserts and updates in Orion, refer to the docs.

It's not a problem if inserts were done before you created the subscription. Of course, you will only get historical records of the updates happening after the subscription was created.

Here's an example of an insert payload to Orion that will generate a notification to QuantumLeap based on the "Test subscription" example shown before.

{
    "id": "Room1",
    "type": "Room",
    "temperature": {
        "value": 24.2,
        "type": "Number",
        "metadata": {}
    },
    "pressure": {
        "value": 720,
        "type": "Number",
        "metadata": {}
    }
    "colour": {
        "value": "white",
        "type": "Text",
        "metadata": {}
    }
}

If you have a custom scenario and still want to directly insert to QuantumLeap, it is not impossible. However, you will have to use the same payload Orion uses in the Notification. Also, you will need attention to some other details you will have to discover on your own. This is not fully documented because it's not considered a common workflow.

Attributes DataType Translation

You need to make sure your NGSI entities are using the valid NGSI types for the attributes, which are documented in the Specification section of the NGSI API. See the first column of the translation table below, and mind its capitalisation.

The table below shows which attribute types will be translated to which CrateDB Data Types.

CrateDB Translation Table

NGSI Type CrateDB Type Observation
Array array(string) Issue 36: Support arrays of other types
Boolean boolean -
DateTime timestamp 'ISO8601' can be used as equivalent of 'DateTime'.
Integer long -
geo:point geo_point Attention! NGSI uses "lat, long" order whereas CrateDB stores points in [long, lat] order.
geo:json geo_shape -
Number float -
Text string This is the default type if the provided NGSI Type is unsupported or wrong.
StructuredValue object -

If the type of any of the received attributes is not present in the column NGSI Type of the previous table, the value of such attribute will be treated internally as a string. So, if you use Float for your attribute type (not valid), your attribute will be stored as a string.

Restrictions and Limitations

  • You cannot have two entity types with the same name but different capitalisation. E.g: Preprocessor and preProcessor. The same applies for attribute names of a given entity. I.e, attributes hotSpot and hotspot will be treated as the same. These are rare corner-cases, but it is worth keeping in mind this. Ultimately, the correct naming of types and attributes should respect the naming guidelines explained here.

  • Attributes metadata are still not being persisted. See Issue 12

Data Retrieval

To retrieve historical data from QuantumLeap, you can use the API endpoints documented here. Note there are a lot of possibilities, but not all of them are fully implemented yet.

If you want to, you can interact directly with the database. For more details refer to the CrateDB section of the docs. What you need to know in this case is that QuantumLeap will create one table per each entity type. Table names are formed with a prefix (et) plus the lowercase version of the entity type. I.e, if your entity type is AirQualityObserved, the corresponding table name will be etairqualityobserved. Table names should be prefixed also with the schema where they are defined. See the Multi-tenancy section below.

Finally, you can interact with your data visually using Grafana. See the Grafana section of the docs to see how.

Time Index

A fundamental element in the time-series database is the time index. You may be wondering... where is it stored?

QuantumLeap will persist the time index for each notification in a special column called time_index. If you payed attention in the Orion Subscription section, you know at least which value is used as the time index. This is, the "dateModified" value notified by Orion or, if you missed that option in the subscription, the notification arrival time.

In the future, this could be more flexible and allow users to define any DateTime attribute to be used as the time index.

Data Removal

You can remove historical data from QuantumLeap in two different ways.

To remove all records of a given entity, use this /entities delete API endpoint.

To remove all records of all entities of a given type, use this /types delete API endpoint.

Use the filters to delete only records in certain intervals of time.

Multi-tenancy

QuantumLeap supports the use of different tenants, just like Orion does with the usage FIWARE headers documented here.

Recall the use of tenancy headers (Fiware-Service and Fiware-ServicePath) is optional. Data insertion and retrieval will work by default without those. However, if you use headers for the insertion, you need to specify the same ones when querying data.

Note in the case of QuantumLeap, the headers at the time of "insertion" should actually be used by the client at the time of creating the Subscription to Orion and also by the device when pushing data to Orion. As mentioned, the same headers will have to be used in order to retrieve such data.

In case you are interacting directly with the database, you need to know that QuantumLeap will use the FIWARE-Service as the database scheme for crate, with a specific prefix. This way, if you insert an entity of type Room in the using the Fiware-Service: magic header, you should expect to find your table at mtmagic.etroom. This information is also useful for example if you are configuring the Grafana datasource, as explained in the Grafana section of the docs.

GeoCoding

This is an optional and experimental feature of QuantumLeap, which helps harmonising the way location information is stored in the historical records.

The idea is that if entities arrive in QuantumLeap with an attribute of type StructuredValue and named address, QuantumLeap interprets this as an address field typically found in the FIWARE Data Models. It then adds to the entity an attribute called location of the corresponding geo-type. This means, if the address is a complete address with city, street name and postal number it maps to a point and hence the generated attribute will be of type geo:point. Without a postal number, the address will represent the street (if any) or the city boundaries (if any) or even the country boundaries. In these cases the generated location will be of the geo:json form and will contain the values of such shape.

WARNING: This feature uses OpenStreetMap and its Nominatim service. As such, you need to be aware of its copyright notes and most importantly of their Usage Policies (API Usage Policy , Nominatim Usage Policy You should not abuse of this free service and you should cache your requests. This is why you need to specify a cache in order to enable the geocoding. QuantumLeap uses Redis.

So, to enable this feature, you need to pass (at initialisation time) to the QuantumLeap container the environment variable USE_GEOCODING set to True and the environment variables REDIS_HOST and REDIS_PORT respectively set to the location of your REDIS instance and its access port. See the docker-compose-dev.yml for example.