Less is More: Lightweight Prompt Compression for Question Answering Applications on Edge Devices

The paper introduces CORE, a two-stage sentence-level prompt compression method designed for question answering applications on edge devices, eliminating the need for auxiliary small language models. CORE utilizes named entity recognition and semantic matching to construct and refine answer and clue sets, resulting in a 30.19% accuracy improvement, 50.47% memory reduction, and 1.94 times speedup on an NVIDIA Jetson AGX Orin, along with a 95.74% energy reduction compared to the LLMLingua2 method on smartphones. This advancement is significant for practitioners developing efficient AI applications on resource-constrained devices, enabling enhanced performance with lower computational overhead.