Post by Nishantha Ruwan

This paper introduces a mixed-reality hardware-in-the-loop (HIL) testbed for autonomous vehicles (AVs) and connected autonomous vehicles (CAVs) that tightly integrates physical robots with a high-fidelity CARLA simulation environment. The testbed is designed to bridge the critical "sim-to-real" gap—the discrepancy between algorithmic performance in simulation and real-world deployment. By combining the scenario diversity of simulation with the realism of physical hardware, the platform enables rigorous validation of perception, planning, and control algorithms under safety-critical conditions. The system uses physical AgileX Limo robots with Ackermann steering, equipped with LiDAR, RGB-D cameras, and IMU sensors, alongside virtual agents in CARLA. A key innovation is the use of "digital twins"—virtual projections of the physical robots in CARLA, with their internal physics disabled and states continuously synchronized with the real robots. This setup allows physical and simulated agents to coexist in a shared mixed-reality environment, supporting multi-agent research at scale. A Road-Side Coordinator (RSU) manages vehicle-to-infrastructure (V2I) communication via ROS, coordinating maneuvers like platooning and intersection crossing. The testbed also features a web-based remote access interface for configuring experiments and downloading logged multimodal sensor data and trajectories. The authors propose a novel decentralized, online self-supervised learning control framework based on Control Barrier Functions (CBFs). The CBF-based controller enforces safety constraints (e.g., rear-end and merging safety) during both training and deployment, ensuring inherent safety. Each agent solves a parametric Quadratic Program (QP) at each time step to compute a safe control input, and the controller parameters are learned online without requiring labeled data. A sensor fusion pipeline—combining RGB and LiDAR data—provides state estimates for the controller. Experiments were conducted across multiple traffic densities using a curriculum learning approach. The online learning algorithm converged quickly within each curriculum. The end-to-end system was validated on a fleet of 12 vehicles (including 5 physical robots). Fine-tuning the controller with real-world robot data successfully bridged the sim-to-real gap, achieving a loss comparable to simulation within two policy updates. The testbed’s ability to simulate photorealistic conditions for perception development was also demonstrated. Overall, the work presents a powerful, open, and remote-accessible platform to accelerate safe AV/CAV research. https://lnkd.in/g5HgYzaT