Project No: 16213526

Title: Edge Computing for the Mission Planning in Energy-constrained UAVs for VLM-based Applications

Principal Investigator: Prof. Dan WANG

Abstract:

Low-altitude Unmanned Aerial Vehicles (UAVs) are increasingly being deployed for complex tasks, e.g., payload delivery that requires onboard video understanding. For example, in a precision environmental remediation application (e.g., by Zenatech), a UAV will fly in an area to disperse remediation payloads (e.g., Algaecides). This mission requests for searching for the disperse locations in this area that maximize the remediation effectiveness. UAVs need to hover in a location to capture a video sequence, perform reasoning with vision-language models (VLMs) to identify the growth of algal blooms, and make disperse decisions. Similar applications include oil spill relief, invasive species management, and precision agriculture. UAVs are critically limited in their battery resources. The energy consumption of UAVs is influenced by the flight trajectory. In particular, UAVs need to hover to capture a video of certain length; so that video-language analysis can explore the long-term temporal dependencies within the video sequence. A longer observation period increases the accuracy; yet this also requires a longer hovering period and more energy; thus restricting the coverage of future areas. The energy consumption of UAVs is also affected by weather; in particular, wind can have an impact of 40%. The UAV mission planning requires onboard computing and potential interactions with the cloud, a typical edge-computing scenario. Current UAVs are equipped with edge computing hardware and software platforms. However, existing edge-computing frameworks, UAV mission planning algorithms, and wind forecasting models; are insufficient to support the new UAV applications. In this project, we plan to develop EdgeVLM-UAV (Edge Computing for UAVs with VLM applications), a new edge-computing framework for UAVs with missions that involve VLM-based applications. We will develop new (1) edge-cloud vision-language analysis algorithms that is computing and communication efficient. We plan to develop algorithms based on scene-graph to efficiently explore long-term temporal dependencies, (2) wind forecasting models. We plan to explore the recent weather foundation models (e.g., GraphCast from Google, Pangu from Huawei, etc.) and develop new models through physics-informed knowledge distillation techniques, and (3) mission planning algorithms. We plan to develop algorithms on the UAV trajectory given energy constraints. Given the uncertainty from the wind forecasting, we further develop robust mission planning algorithms. Our project proposes new edge computing problems and develops new techniques on weather foundation model distillation, scene-graph-based algorithms, and robust optimization algorithms. We will implement an open source EdgeVLM-UAV prototype, evaluate EdgeVLM-UAV through simulators, e.g., AirSim from Microsoft, and enhance existing software platforms.