Lightweight Transformer Models for On-Device Fault Detection: A Benchmark Study on Resource-Constrained Deployment

This study benchmarks lightweight transformer models (DistilBERT, TinyBERT-6L, TinyBERT-4L, MobileBERT) against traditional ML methods (Random Forest, XGBoost, SVM, Logistic Regression) for on-device fault detection across three datasets. Key findings indicate that TinyBERT-4L offers a favorable trade-off with a model size of 55 MB and a CPU inference latency of 18 ms, while INT8 quantization can reduce model size by 25% with minimal impact on classification performance (86.9% F1). The results underscore the challenges of deploying accurate models in resource-constrained environments, particularly in scenarios with extreme class imbalance.