Data API

App Tasks can still collect data and store data on SD card on Liberty IoT Hub.

Liberty IoT Hub will protect data from being harvested by any third party, because App Engine will block any connection request to a destination outside local network by default.

On the local network however, users still have total control over their data, including downloading data into various format, visualizing data in many different ways, or using third party Apps to analyze the data.

Future version of Liberty IoT Hub will enable automatic data sharing with explicit user consent to specific App Task, i.e. user has to explicitly agree to share specific data with specific third party on explicitly specific terms.

Data Access Model

Each Task has it’s own data store. Task has exclusive access to its data store. Data store is not shared among tasks.

Data Types

There are only two data types supported in Liberty-IoT-OS.

  • Standalone - Binary blob data with a name, data is encoded with MessagePack format.
  • Timeseries - Time series data with a data structure definition schema. The schema is mandatory. Application must provide a schema when creating time series data store. We use Apache Avro schema.

Apache Avro Data Schema

Apache Avro provides:

  • Rich data structures.
  • A compact, fast, binary data format.

By forcing application providing a data schema, user can easily import any application Data into JSON file. Third party can develop algorithms to process the data based on schema without actually access the data itself.

Further more, in the future we can provide GUI tool that runs on web page and Smartphone that can visual any Task data with customized template (that supports various aggregations).

Restrictions

In schema, only the following data types are supported.

  • null
  • boolean
  • int
  • long
  • float
  • double
  • bytes
  • string
  • record
  • enum
  • array
  • map
  • fixed

Standalone Data Access API

The data access API deals with the actual encoding/decoding. Note again, standalone data uses MessagePack format while time series data uses Apache Avro format.

Note: Keep in mind that cyclic reference is not allowed in the data, otherwise an error will be raised!*

Libre_DataWriteStandalone(name, value)

Libre_DataReadStandalone(name)

Libre_DataEraseStandalone(name)

  • name is a string representing a name.
  • value is any valid Lua value, including nil.

Data will be serialized into MessagePack format (a binary blob) internally when writing. When reading, data will be deserialized from the MessagePack encoded binary blob into a Lua value.

Note Libre_DataReadStandalone(name) may return nil if the data is not found.

Time Series Data Access API

Libre_DataInitSchema(schema)

Parameters:

  • schema is single schema object, or an array of schema objects, see example below.

Returns: None.

Note: Task is responsible for maintaining the compatibility of schema among versions; otherwise this function may raise error.

Libre_DataOpenTimeSeries(name, create)

Open a time series data table.

Parameters:

  • name - String, data table name.
  • create - Booolean, if table doesn’t exist, shauld we create the table?

Returns:

A table ID, or nil if no table and create = false.

Libre_DataDropTimeSeries(name)

Drop a time series data table.

Parameters:

  • name - String, data table name.

Libre_DataStatTimeSeries(table)

Get the time series table information.

Parameters:

  • table - The data table

Returns:

Multiple values:

  • count - Record count
  • min - Min index
  • max - Max index

Libre_DataReadTimeSeries(table, indexLo, indexHi)

Read records from the table with the index within the specified range.

Parameters:

  • table - The data table
  • indexLo - Index >= indexLo
  • indexHi - Index <= indexHi

Returns:

An array of record objects with ascending index order.

  • Object
    • i - index
    • t - timestamp, seconds from Unix epoch.
    • v - the actual record value (could be any Lua value)

Libre_DataWriteTimeSeries(table, data, schema)

Parameters:

  • table - The data table
  • data - A Lua value with format that conforms to the schema specified (the next argument).
  • schema - A name of schema record (string), or an index of schema record in the initial schema list.

Returns:

Multiple values:

  • index - Record index
  • timestamp - Timestamp associated with data (seconds from Unix epoch)

Note: If data is not compatible with schema, an error will be raised.

Example

Example below demonstrated the API. We demonstrated the use of time series API. The data schema is below:

Note, we defined 3 types in an array, “TestRecord”, “weekdays” and “treenode”.

Please also note that there are type references. “TestRecord” is a structure that contains “weekdays” (enumeration type) and a tree (defined in “treenode”).

Data Schema

[
    {
        "type": "record",
        "name": "TestRecord",
        "fields" : [
            {"name": "a", "type": "long"},
            {"name": "b", "type": "string"},
            {"name": "dayofweek", "type": "weekdays"},
            {"name": "temperatures", "type": "array", "items": "float"},
            {"name": "tree", "type": ["null", "treenode"]}
        ]
    },
    {
        "type": "enum",
        "name": "weekdays",
        "symbols": ["Sun", "Mon", "Tue", "Wed", "Thu", "Fri", "Sat"]
    },
    {
        "type": "record",
        "name": "treenode",
        "fields" : [
            {"name": "name", "type": "string"},
            {"name": "children", "type": ["null", {"type": "array", "items": "treenode"}]}
        ]
    }
]

Sample Data

For time series data, structure of sample data has to conform to the data schema. Note the schema and sample data are pretty complex data structure, yet the API can correctly encode and decode data.

App Code

local qw_avro = qw_avro
local Libre_Log = Libre_Log
local json_encode = json.encode
local json_decode = json.decode
local Libre_DataInitSchema = Libre_DataInitSchema
local Libre_DataOpenTimeSeries = Libre_DataOpenTimeSeries
local Libre_DataDropTimeSeries = Libre_DataDropTimeSeries
local Libre_DataReadTimeSeries = Libre_DataReadTimeSeries
local Libre_DataWriteTimeSeries = Libre_DataWriteTimeSeries
local Libre_DataStatTimeSeries = Libre_DataStatTimeSeries
local Libre_DataEraseTimeSeries = Libre_DataEraseTimeSeries

local function test_1()
	local schema_json = [[
		[
			{
				"type": "record",
				"name": "TestRecord",
				"fields" : [
					{"name": "a", "type": "long"},
					{"name": "b", "type": "string"},
					{"name": "dayofweek", "type": "weekdays"},
					{"name": "temperatures", "type": "array", "items": "float"},
					{"name": "tree", "type": ["null", "treenode"]}
				]
			},
			{
				"type": "enum",
				"name": "weekdays",
				"symbols": ["Sun", "Mon", "Tue", "Wed", "Thu", "Fri", "Sat"]
			},
			{
				"type": "record",
				"name": "treenode",
				"fields" : [
					{"name": "name", "type": "string"},
					{"name": "children", "type": ["null", {"type": "array", "items": "treenode"}]}
				]
			}
		]
	]]

	local sample_data = {
		a = 1234,
		b = 'Hello',
		dayofweek = 'Wed',
		temperatures = {21, 21.3, 21.6, 22, 22.5, 23, 22.7, 21},
		tree = {name = 'T0', children = {
			{name = 'T1-1', children = {
				{name = 'T1-1-1'},
				{name = 'T1-1-2'},
			}},
			{name = 'T1-2'},
			{name = 'T1-3'},
		}}
	}

	local schema_raw = json_decode(schema_json)
	Libre_DataInitSchema(schema_raw)
    -- Open a table named 'table1', create if table doesn't exist
	local table = Libre_DataOpenTimeSeries("table1", true);

    local count, min, max = Libre_DataStatTimeSeries(table)
    if count > 0 then   -- Read latest record
        local latest = Libre_DataReadTimeSeries(table, max, max)[1]
    end

    -- Write a new one, format is the first in schema list "TestRecord"
	Libre_DataWriteTimeSeries(table, sample_data, 1)
    -- Libre_DataWriteTimeSeries(table, sample_data, "TestRecord") does the same thing with explicit schema type name.
end

function qa_test_avro()
	test_1()
end