Data Migration
A few tools are available to assist with migrating data to QuantumLeap.
Migrating from QuantumLeap Crate to Timescale
QuantumLeap provides a self-contained Python script to help with
migrating tables from a QuantumLeap CrateDB database to a QuantumLeap
Timescale database. The script is located in the timescale-container
directory and is called crate-exporter.py
.
It exports rows in a given Crate table and generates, on stdout
,
all the SQL statements needed to import that data into Timescale.
These include creating a corresponding schema, table and hypertable
in PostgreSQL as needed. Note that the script generates DDL statements
that, when executed, will result in the exact same table structures
the QuantumLeap Timescale back end would have generated on seeing
NGSI entities corresponding to the rows stored in the Crate table.
Here's an example usage
$ python crate-exporter.py --schema mtyoutenant --table etdevice \
> mtyoutenant.etdevice-import.sql
where we export all the rows in the Crate table mtyoutenant.etdevice
.
The generated file contains all the SQL statements to recreate the
table and insert the data in Timescale. You may want to put this file
in the quantumleap-db-setup
script's init directory so that data
are migrated automatically for you when you bootstrap the QuantumLeap
DB on Timescale as explained in the Timescale section.
By default the script exports all the rows in the Crate table, but
you can also use the --query
argument to specify a query to select
only a subset of interest as shown below:
$ python crate-exporter.py --schema mtyoutenant --table etdevice --query \
"SELECT * FROM mtyoutenant.etdevice where time_index > '2019-04-15';"