Evaluating Large Language Models for Hausa and Fongbe Machine Translation: Benchmarks, Failures, and Metric Reliability

The study evaluates the translation capabilities of four large language models (GPT-4o Mini, Claude Sonnet 4, Gemini 2.5 Flash, and Qwen2.5-7B) for English-to-Hausa and English-to-Fongbe translations, revealing significant disparities in translation quality and metric reliability. Hausa achieved acceptable translation quality with human scores of 4.0-4.5, while Fongbe scored poorly (1.0-2.2), highlighting a consistent 3x BLEU gap. The research emphasizes the necessity of multi-metric evaluation for low-resource languages and establishes that a minimum of 2,500 sentences is required for reliable model ranking, indicating that smaller samples can lead to misleading conclusions.