EXTENSION OF A COMMON DATA FORMAT FOR REAL-TIME APPLICATIONS

Abstract

The HDF5 (Hierarchical Data Format) data storage family is an industry standard format that allows data to be stored in a common format and retrieved by a wide range of common tools. HDF5 is a widely accepted industry standard container for data storage developed by the National Center for Supercomputing Applications (NCSA) at the University of Illinois at Urbana-Champaign. The HDF5 data storage family includes HDF-Time History, intended for data processing, and HDF-Packet, intended for real-time data collection; each of these is an extension to the basic HDF5 format, which defines data structures and associated interrelationships, optimized for that particular purpose. HDF-Time History, developed jointly by Boeing and NCSA, is in the process of being adopted throughout the Boeing test community and by its external partners. The Boeing/NCSA team is currently developing HDF-Packet to support real-time streaming applications, such as airborne data collection and recording of received telemetry. The advantages are significant cost reduction resulting from storing the data in its final format, thus avoiding conversion between a myriad of recording and intermediate formats. In addition, by eliminating intermediate file translations and conversions, data integrity is maintained from recording through processing and archival storage. As well, HDF5 is a general-purpose wrapper, into which can be stored processed data and other data documentation information (such as calibrations), thus making the final data file self-documenting. This paper describes the basics of the HDF-Time History, the extensions required to support real-time acquisition with HDF-Packet, and implementation issues unique to real-time acquisition. It also describes potential future implementations for data acquisition systems in different segments of the test data industry.