User Tools

Site Tools


metertimeseriesdataformat

This is an old revision of the document!


Table of Contents

Goal

The aim is to specifiy and create a generic software library for timeseries with measured data. The library must be able to store timeseries with associated metadata. The library should consist of two parts:

  • An interface for other components that would like to store or access timeseries data. The library must enforce certain restrictions on incoming data, such as a valid timestamp. For accessing the data an database-cursor like interface seems appropriate.
  • The library must be able to characterize the data based on an empirical cumulative distribution function (ecdf). This means that min/max, average, median etc. should be represented based on the full data, not only the data that has been reduced in resolution. NOTE: This is currently an open issue, to be solved using complex event processing techniques. See e.g. http://subs.emis.de/LNI/Proceedings/Proceedings65/GI-Proceedings.65-24.pdf.
  • The library must store metadata for a given sensor, including free-text comments.
  • The library must perform sanity checks.

Requirements

The library is to be used in a wide range of applications, ranging from recording data locally (on a small embedded device like the Chumby) to the webservers of the project. It must therefore be written in C++, as portable as possible. It is safe to assume a POSIX architecture, the boost libraries can also be used.

Metainformation

  • Description
  • meterID
  • Unit
  • Timezone
  • time-reference: left-justified or right-justified
  • Scaling Factor (e.g. SCALING_FACTOR = 10.0)
  • Offset/Bias
  • Missing Value (e.g. MISSING_VALUE = -1) or (better) don't store missing values. Whats not stored is not available.

Data

  • Timestamp (unixtimestamp)
  • Value (floatingpoints)

Version 0.1

The development of this library will be an ongoing process. In the first version, a simple usecase should be satisfied: the data measurements of individual devices (measured by the plugwise system) must be stored. Basically, this means that 10 individual sensors must be stored with synchronized timesteps. The measurement interval will be in the 1-2 second range. Since the data will be used to develop algorithms, a portable and compact (single-file) storage is adequate. Performance of the storage mechanism is not a critical requirement.

Basic operations for this usecase:

  • Create an empty storage container file
  • add/remove a sensor from the container (including dataset and metadata)
  • add a reading to a sensor's storage
  • dump all readings of a sensor
metertimeseriesdataformat.1295947308.txt.gz · Last modified: 2012/10/30 10:37 (external edit)