Trustworthy Self-Composable Big-Data-as-a-Service: An LLM-Orchestrated Multi-Agent Framework for Automated Data Engineering, AutoML, MLOps Deployment, and Drift-Aware Lifecycle Optimization

This paper presents a novel framework for Big-Data-as-a-Service (BDaaS) that utilizes LLM-orchestrated multi-agent collaboration to automate the entire data engineering lifecycle, including data ingestion, cleaning, feature engineering, AutoML training, deployment, and monitoring. The architecture features a central LLM orchestration layer that manages agent execution and workflow context, while ensuring artifact governance and drift-aware adaptation. Evaluation on benchmark datasets shows that this multi-agent approach outperforms traditional methods in predictive performance and lifecycle reliability, indicating its potential for enhancing automation in production environments.