Removing the Need to Preprocess Telemetry Streams for Data Analysis

Abstract

Current post-test telemetry data analysis systems require that the record structures of the data files be in a well defined format. This has traditionally been accomplished by either developing custom analysis tools for each data stream in question, or, more commonly, by pre-processing the data into a prescribed format. Pre-processing data causes several problems: 1) new generation data files must be created, often consuming gigabytes of disk storage, 2) pre-processors must be written for each data stream, involving programmer as well as computer time, and 3) time must be taken to do the pre-processing, increasing the delay between test and analysis of data. A Flexible File Server (FFS) has been developed to mitigate these problems. This software eliminates the pre-processing phase traditionally occurring before data analysis can take place thus permitting users to directly access and analyze raw telemetry data streams written in arbitrarily complex structures. The FFS works in conjunction with a table-driven Data Dictionary which allows analysts direct access to individual variables by symbolic name. While the FFS decodes the data stream structure, the dictionary describes the telemetry frame structure and how variables are recorded inside these frames. This paper will discuss how the structure of recorded data, both data stream structure (e.g., time tags and frame id's) and frame structure (e.g., sub- and super-commutation), is described using the Flexible File Server and data dictionary, and how these concepts are incorporated into an advanced data analysis software package.