1.2. Example 2: Grab data based on metadata¶

In this example, we will learn how to grab the actual data based on metadata.

1.2.1. Filter metadata¶

Just like how we have done in the previous example, let’s search for files containing camera/front-center.

[1]:

import os
from pydtk.db import DBHandler

db_handler = DBHandler(
    db_class='meta',
    db_host='./example_db',
    base_dir_path='../test'
)
db_handler.read(pql='"contents.camera/front-center" == exists(True)')
db_handler.content_df

[1]:

	Record ID	File path	content	tag
0	csv_model_test	/opt/pydtk/test/records/csv_model_test/data/te...	{'camera/front-center': {'tags': ['camera', 'f...	NaN

Note that metadata here is associated to each file containing the actual data.

Thus, if a file has more than one contents (e.g., A rosbag file can store multiple signals), then those which are other than camera/front-center are also retrieved.

1.2.2. Iterate metadata¶

You can get metadata one-by-one as DBHandler works as an iterator.
To get a sample, just use next() method.
Metadata will be returned as a dict.

[2]:

sample = next(db_handler)
sample

[2]:

{'description': 'Description',
 'record_id': 'csv_model_test',
 'data_type': 'raw_data',
 'path': '/opt/pydtk/test/records/csv_model_test/data/test.csv',
 'start_timestamp': 1489728491.0,
 'end_timestamp': 1489728570.957,
 'content_type': 'text/csv',
 'contents': {'camera/front-center': {'tags': ['camera',
    'front',
    'center',
    'timestamps']}},
 '_id': '6ee6a5ceb79111eba234acde48001122'}

1.2.3. Grab data¶

Based on the metadata, we can grab the actual data as a numpy array from the corresponding file.
BaseFileReader automatically chooses an appropriate model to load the file based on the given metadata.
Thus, you can simple call read function to grab data as follows.

[3]:

from pydtk.io import BaseFileReader, NoModelMatchedError

reader = BaseFileReader()

try:
    timestamps, data, columns = reader.read(sample)
    print('# of frames: {}'.format(len(timestamps)))
except NoModelMatchedError as e:
    print(str(e))

WARNING:root:Failed to load models in autoware
WARNING:root:Failed to load models in movie.py
WARNING:root:Failed to load models in rosbag.py

# of frames: 2400

Let’s check the ndarray.

[4]:

timestamps?

[5]:

data?

[6]:

columns?