Link Search Menu Expand Document
Libertas Things Documentation

Data API

We made every effort to ensure every detail of the Libertas data infrastructure aligns with our mission, independence, freedom, and privacy for end-users!

Thing-App Tasks can still collect data and store data on the SD card on Libertas Hub.

Libertas Hub will protect data from being harvested by any third party because any network destination hostname from every Thing-App Task has to be granted explicitly by the end-user on an allow-list basis.

As the data owner, the users still have total control over their data, including downloading data into various formats, visualizing data in many different ways, or using third-party Apps to analyze the data.

Libertas will promote collaboration on data in any way possible without sacrificing users’ privacy.

Data Access Model

Each “Task” has its data store. A task has exclusive access to its data store. The data store is not shared among tasks.

Data Model

There are only two data types supported in Libertas-OS.

  • Standalone - Binary blob data with a name; data is encoded with MessagePack format.
  • Timeseries - Time series data with a data structure definition schema. The schema is mandatory. A Thing-App must provide a schema when creating a time-series data store. We use Apache Avro schema.

Apache Avro Data Schema (Timeseries Data)

Apache Avro provides:

  • Rich data structures.
  • A compact, fast, binary data format.

By forcing the application to provide a data schema, the user can easily import any application Data into the JSON file. Third parties can develop algorithms to process the data based on schema without actually accessing the actual data.

Furthermore, in the future, we can provide a GUI tool that runs on a web page or smartphone to visualize any task data with a customizable template (for example, JSON-path or aggregation of JSON-path).

NOTE: Only ONE Data Schema Per-App Function

It is not a problem because the developer can stuff unlimited definitions into the schema.

The developer shall keep old schema records intact when developing new versions of a Thing-App function. If the schema needs to be upgraded, always append new records into an old schema (note the order shall also be kept). In this case, backward compatibility is guaranteed.

Note if the schema of an upgraded Thing-App function is not “compatible” with the old one, the Thing-App engine can detect it and throw a runtime error when Libertas_DataInitSchema API is called.

Restrictions

As of now, only the following data types are supported.

  • null
  • boolean
  • int
  • long
  • float
  • double
  • bytes
  • string
  • record
  • enum
  • array
  • map
  • fixed

Standalone Data Access API

The standalone access API deals with the actual data encoding/decoding. Note again, standalone data use MessagePack format while time-series data use Apache Avro format.

Note: Remember that cyclic reference is not allowed in the data; otherwise, an error will be raised!*

Libertas_DataWriteStandalone(name: string, value: any)

Libertas_DataReadStandalone(name: string): any

Libertas_DataEraseStandalone(name: string)

  • name is a string representing a name.
  • value is any valid Lua value, including nil.

When writing, data will be serialized internally into MessagePack format (a binary blob). Data will be deserialized from the MessagePack encoded binary blob into a Lua value when reading.

Note Libertas_DataReadStandalone(name) may return nil if the data is not found.

Time Series Data Access API

Data Types 

type LibertasDataTable      = number;
type LibertasDataTimestamp  = number;
type LibertasDataStat       = [count: number, start: number];
type LibertasTimeSeriesIndex= [number, LibreDataTimestamp];

declare interface LibertasTimeSeriesRecord {
    /** The index */
    i:      number;
    /** The record value in the predefined format. */
    v:      any;
}

APIs 

function Libertas_DataInitSchema(schema: LibertasAvroSchema[]): void

Parameters:

  • schema is a single schema object or an array of schema objects; see the example below.

Returns: None.

Note: The developer is responsible for maintaining schema compatibility among versions; otherwise, this function may raise an error. Read THIS NOTE for more details!

function Libertas_DataOpenTimeSeries(name: string, create: boolean): LibertasDataTable|undefined

Open a time series data table.

Parameters:

  • name - String, data table name.
  • create - Boolean, should we create the table if the table doesn’t exist?

Returns:

A table ID, or nil, if there is no table and “create” = false.

function Libertas_DataDropTimeSeries(name: string): void

Drop a time series data table.

Parameters:

  • name - String, data table name.
function Libertas_DataStatTimeSeries(table: LibertasDataTable): LuaMultiReturn<LibertasDataStat>

Get the time series table information.

Parameters:

  • table - The data table

Returns:

Multiple values:

  • count - Record count
  • start - Start index
function Libertas_DataReadTimeSeries(table: LibertasDataTable, indexLo: number, indexHi: number): LibertasTimeSeriesRecord[]

Read records from the table with the index within the specified range.

Parameters:

  • table - The data table
  • indexLo - Index >= indexLo
  • indexHi - Index <= indexHi

Returns:

An array of record objects with ascending index order.

  • Object
    • i - index
    • v - the actual record value (could be any Lua value)
function Libertas_DataWriteTimeSeries(table: LibertasDataTable, data: any, schema: string|number): number

Parameters:

  • table - The data table
  • data - A value with the format that conforms to the schema specified (the next argument).
  • schema - A name of schema record (string) or an index of schema record in the initial schema list.

Returns:

  • index - Record index

Note: An error will be raised if data is incompatible with the schema.

Data Encryption

Data is stored on the external SD card. As of now, data is not encrypted for performance reasons. We will introduce a separate secure data storage in the future.

Rate Limit

Data APIs are subject to rate limit. If API is called too frequently, the task will be terminated and marked as “disabled” for safety reasons.

Example

The example below demonstrates the API. We demonstrated the use of time series API. The data schema is below:

Note we defined three types in an array,

  1. “TestRecord”,
  2. “weekdays” and
  3. “treenode”.

Please also note that there are type references. “TestRecord” is a structure that contains “weekdays” (enumeration type) and a tree (defined in “treenode”).

Data Schema 

[
    {
        "type": "record",
        "name": "TestRecord",
        "fields" : [
            {"name": "a", "type": "long"},
            {"name": "b", "type": "string"},
            {"name": "dayofweek", "type": "weekdays"},
            {"name": "temperatures", "type": "array", "items": "float"},
            {"name": "tree", "type": ["null", "treenode"]}
        ]
    },
    {
        "type": "enum",
        "name": "weekdays",
        "symbols": ["Sun", "Mon", "Tue", "Wed", "Thu", "Fri", "Sat"]
    },
    {
        "type": "record",
        "name": "treenode",
        "fields" : [
            {"name": "name", "type": "string"},
            {"name": "children", "type": ["null", {"type": "array", "items": "treenode"}]}
        ]
    }
]

Sample Data

For time-series data, the sample data structure must conform to the data schema. Note the schema and sample data are complex data structures, yet the API can correctly encode and decode data.

Thing-App Code 

local qw_avro = qw_avro
local Libertas_Log = Libertas_Log
local json_encode = json.encode
local json_decode = json.decode
local Libertas_DataInitSchema = Libertas_DataInitSchema
local Libertas_DataOpenTimeSeries = Libertas_DataOpenTimeSeries
local Libertas_DataDropTimeSeries = Libertas_DataDropTimeSeries
local Libertas_DataReadTimeSeries = Libertas_DataReadTimeSeries
local Libertas_DataWriteTimeSeries = Libertas_DataWriteTimeSeries
local Libertas_DataStatTimeSeries = Libertas_DataStatTimeSeries
local Libertas_DataEraseTimeSeries = Libertas_DataEraseTimeSeries

local Libertas_DataWriteStandalone = Libertas_DataWriteStandalone
local Libertas_DataReadStandalone = Libertas_DataReadStandalone
local Libertas_DataEraseStandalone = Libertas_DataEraseStandalone

local function test_1()
    local schema_json = [[
        [
            {
                "type": "record",
                "name": "TestRecord",
                "fields" : [
                    {"name": "a", "type": "long"},
                    {"name": "b", "type": "string"},
                    {"name": "dayofweek", "type": "weekdays"},
                    {"name": "temperatures", "type": "array", "items": "double"},
                    {"name": "tree", "type": ["null", "treenode"]}
                ]
            },
            {
                "type": "enum",
                "name": "weekdays",
                "symbols": ["Sun", "Mon", "Tue", "Wed", "Thu", "Fri", "Sat"]
            },
            {
                "type": "record",
                "name": "treenode",
                "fields" : [
                    {"name": "name", "type": "string"},
                    {"name": "children", "type": ["null", {"type": "array", "items": "treenode"}]}
                ]
            }
        ]
    ]]

    local sample_data = {
        a = 1234,
        b = 'Hello',
        dayofweek = 'Wed',
        temperatures = {21, 21.3, 21.6, 22, 22.5, 23, 22.7, 21},
        tree = {name = 'T0', children = {
            {name = 'T1-1', children = {
                {name = 'T1-1-1'},
                {name = 'T1-1-2'},
            }},
            {name = 'T1-2'},
            {name = 'T1-3'},
        }}
    }

    local schema_raw = json_decode(schema_json)
    Libertas_DataInitSchema(schema_raw)
    -- Open a table named 'table1'; create if table doesn't exist
    local table = Libertas_DataOpenTimeSeries("table1", true);
    -- Get information about the time series
    local count, min, max = Libertas_DataStatTimeSeries(table)
    if count > 0 then   -- Read latest record
        Libertas_Log(0, "TimeSeries Count=" .. count)
        local latest = Libertas_DataReadTimeSeries(table, max, max)[1]
        Libertas_Log(0, json_encode(latest))
    end
    -- Test erase all (index <= max)
    Libertas_DataEraseTimeSeries(table, max)
    count, min, max = Libertas_DataStatTimeSeries(table)
    if count ~= 0 then
        Libertas_Log(0, "Libertas_DataEraseTimeSeries failure")
    end
    -- Write a new one; the format is the first in the schema list "TestRecord"
    for i=1,100 do
        -- Libertas_DataWriteTimeSeries(table, sample_data, "TestRecord") does the same thing with an explicit schema type name.
        Libertas_DataWriteTimeSeries(table, sample_data, 1)
        Libertas_Wait(0)
    end

    local standAloneName = 'Test'
    -- Test read
    local standAlone = Libertas_DataReadStandalone(standAloneName)
    if standAlone ~= nil then
        Libertas_Log(0, "standalone=" .. json_encode(standAlone))
    end
    -- Test erase
    Libertas_DataEraseStandalone(standAloneName)
    standAlone = Libertas_DataReadStandalone(standAloneName)
    if standAlone ~= nil then
        Libertas_Log(0, "Libertas_DataReadStandalone failure")
    end
    -- Test write. Will be read when the task is relaunched
    Libertas_DataWriteStandalone(standAloneName, sample_data)
end

function qa_test_avro()
    test_1()
end

Design Transparency

Our data model will allow anyone to inspect the data model of any Thing-App. Transparency helps in promoting collaboration in any way possible.

Future Plans

  1. Browser and download data from Web interface or Rest API.
  2. A framework for “uniform data transformation (and aggregation) and presentation (visualization with a uniform template).”