Refining perception contracts: Case studies in vision-based safe auto-landing

Perception contracts provide a method for evaluating safety of control systems that use machine learning for perception. A perception contract is a specification for testing the ML components, and it gives a method for proving end-to-end system-level safety requirements. The feasibility of contract-based testing and assurance was established earlier in the context of straight lane keeping—a 3-dimensional system with relatively simple dynamics. The paper advances the methodology by introducing an algorithm for constructing data and requirement guided refinement of perception contracts (DaRePC). The method is applied to analyze two 6 and 12-dimensional flight control systems that use multi-stage, heterogeneous, ML-enabled perception. The resulting analysis provides testable contracts which establish the state and environment conditions under which an aircraft can safely touchdown on a runway and a drone can safely pass through a sequence of gates. It can also discover conditions (e.g. low-horizon sun) that can possibly violate the safety of the vision-based control system.