OPAL: Leveraging Open Source

Abstract

The current explosion of test and evaluation data being collected from various systems has exposed a strong need for low-cost digital infrastructure to facilitate scalable analytics across all available data. Private industry and academic research have built such systems utilizing Open-Source Software (OSS) with tremendous success. The 309th Software Egineering Group (SWEG) developed OPAL (Open Platform for Advanced Learning) platform is a government owned and developed solution to address this gap and provide data discovery, analytics, and warehousing all license free. OPAL leverages best-in-breed Open-Source Software including JupyterLab (Python analysis environment), MinIO (S3-compliant, redundant and object-versioning data backend), Postgres (Data cataloging), and Dask (scalable compute), among others. In addition to Open-Source tooling, custom integration and software piping are used to further lower the analysts’ barrier to available data: custom Chapter 10 parsing and translating at high speed (10GB/min) into Apache Parquet format, a web-based data catalog for discovery, and lightweight arbitrary object storage organization. This paper will delineate design choices, our DevOps paradigm, benchmarking numbers, and results against a publicly available commercial flight dataset.