Proactive Conversational Assistant for a Procedural Manual Task based on Audio and IMU

The article presents a novel real-time conversational assistant designed for procedural manual tasks, utilizing only audio and IMU data from wearable devices to ensure user privacy and reduce computational demands. The system demonstrates significant improvements through model fine-tuning, achieving a 50% increase in precision and a 150% increase in recall for answering user questions, while being implemented on edge devices without cloud reliance. This approach is particularly relevant for practitioners seeking efficient, privacy-preserving AI solutions in task-oriented applications.