FleetAgent: Teleoperation Assistant for Autonomous Fleets via Vectorized V2N Messages

FleetAgent is a cloud-hosted multimodal large language model (MLLM) designed to assist in the teleoperation of autonomous vehicle fleets by processing compact vectorized vehicle-to-network (V2N) messages. It employs a novel vector-to-embedding interface called VecFormer, which enhances batch processing efficiency by reducing context length and GPU KV-cache growth, while achieving up to 625 times reduction in uplink payload compared to raw images and 16.54 times reduction in KV-cache memory compared to original text descriptions. The system's performance is validated on the VecEval dataset, showing a 16.8% improvement in Lingo-Judge scores and a 19.9% reduction in intervention failure rates compared to the Qwen2.5-VL-7B model, highlighting its potential for efficient and explainable teleoperation in large-scale fleet management.