Browsing International Telemetering Conference Proceedings, Volume 53 (2017) by Authors
EFFICIENT DATA STORAGE, SAMPLING, AND RETRIEVAL BY LEVERAGING OPEN SOURCE TECHNOLOGIESFerrill, Micha; Avionics Test & Analysis Corporation (International Foundation for Telemetering, 2017-10)This paper demonstrates the use of open-source software tools to manage large data sets. Advances in technology have greatly reduced the cost of data storage and processing systems. The ability to handle large amounts of data efficiently while retaining fine-grain control of the data retrieval process becomes a challenge. In particular, traditional data processing applications are inadequate to handle the large data sets typically encountered in IRIG-106 Chapter 10 data files. We answer this challenge by using readily available, open-source tools that efficiently store and retrieve IRIG-106 Chapter 10 data to/from a file-based database. We will demonstrate a method that facilitates a separation between the parsing of raw input data and the display of desired information at a user-defined sample rate. This open-source based solution provides a low-cost, reliable, and efficient means for handling large amounts of data at a high rate of speed.