A Hardware-in-the-Loop Dynamic Data Driven Adaptive Multi-Scale Simulation (DDDAMS) System for Crowd Surveillance via Unmanned Vehicles

Abstract

Planning and control of unmanned vehicles play a major role in multi-vehicle systems since accomplishing challenging missions requires not only an extensive decision-making process but it also demands execution of those decisions based on the received information from multiple sensors. In this dissertation, a simulation-based planning and control system is designed, developed and demonstrated for effective and efficient crowd surveillance via collaborative operation of unmanned aerial vehicles (UAVs) and unmanned ground vehicles (UGVs). The dissertation research works involve three phases. At phase one, a dynamic data driven adaptive multi-scale simulation (DDDAMS)-based planning and control framework is designed and developed, where the major components include 1) integrated controller, 2) integrated planner, 3) decision module for DDDAMS, and 4) real system. Moreover, crowd detection, tracking, and motion planning modules are implemented in this framework to perform the crowd surveillance mission. This framework adopts dynamic data driven application system (DDDAS) paradigm, where the integrated planner is invoked on a temporal or event basis to incorporate dynamic data from onboard sensors of unmanned vehicles into the simulation and select the best control strategy. At phase two, a testbed is designed and constructed using agent-based hardware-in-the-loop simulation, which involves various hardware components (i.e. real UAVs and UGVs containing onboard sensors and computers) and software components (agent-based simulation and hardware interface). The agent-based simulation, a major component of this testbed, is developed by modeling the behavior of the unmanned vehicles while utilizing the terrain elevation data obtained from GIS. Moreover, a social force model is used to mimic the crowd dynamics in the simulated environment. The constructed testbed is used to evaluate the effectiveness and computational efficiency of the proposed planning and control framework. At phase three, a team formation approach is proposed for allocating unmanned vehicles to different crowd clusters using their geometry and available number of resources based on two different criteria (i.e. overall coverage of all clusters and uniform assignment of resources among clusters). This approach is used in crowd splitting scenarios when the crowd starts to divide into clusters, and the existing team of unmanned vehicles is not able to continue following all the clusters. Moreover, control strategies for team formation, information aggregation, and motion planning of unmanned vehicles are introduced, and a method for determining the value of the control strategy parameter for the information aggregation of UAVs and UGVs is proposed. In conclusion, we believe this work has a profound impact on both the research community and practitioners using unmanned vehicles. Also, the developed hardware-in-the-loop DDDAMS system has the potential to be deployed in real-world applications such as border patrol.