Scaling Performance and Low-Resource Annotation with Many-Shot In-Context Learning for Named Entity Recognition

This study investigates the effectiveness of many-shot in-context learning (ICL) for Named Entity Recognition (NER), revealing that scaling to hundreds of demonstrations allows large language models (LLMs) to match or exceed the performance of fine-tuned BERT models. The authors demonstrate that using around one hundred human-labeled examples as ICL demonstrations can produce high-quality labeled data, resulting in a 10% absolute F1 improvement when fine-tuning BERT for low-resource NER tasks. This approach is significant for practitioners as it reduces the need for extensive labeled datasets while enhancing model performance in structured tasks.