Where Does the Signal Live? A Web Data Recipe for Medical Encoder Pretraining

The article presents the FineMed corpus and DoctoBERT, a state-of-the-art French medical encoder family, developed through a novel web data curation approach for pretraining encoders in dense-terminology domains. The method employs medical-term density filtering and signal-amplifying rephrasing, demonstrating that filtered web data significantly enhances performance on downstream tasks compared to traditional educational quality filters. This advancement is crucial for practitioners as it allows for the effective utilization of web-scale data in training medical NLP models, improving scalability and diversity in language representation.