Implementing Procedure Tracking Systems with Computer Vision and Sensors

Abstract

When performing complex step-by-step procedures, users may require assistance and extensive instruction. In this project, we implement a proof-of-concept automated procedure tracking system using computer vision and sensors. Our system operates on a single-camera system—an overhead stationary camera on the Jetson Orin Nano single-board computer (SBC)—to ensure good camera coverage as the user completes the procedure. In addition, the tools the user interacts with in the procedure are augmented with sensors such as accelerometers and gyroscopes. To simulate the expected complexity and use case of a space station maintenance task, we focus on tracking the procedure of installing the bottom bracket of a bike. During the procedure, the program will identify the progress and accuracy of each step and have visual cues to notify users of deviations from pre-defined instructions.