One Polluted Page Is Enough: Evaluating Web Content Pollution in Generative Recommenders

The article introduces FORGE (Fake Online Recommendations in Generative Environments), a benchmark designed to evaluate the susceptibility of search-augmented LLMs to promote fake products due to polluted web content. The study reveals that all tested models, including 12 commercial and open-weight LLMs, exhibit significant vulnerability, with fooled rates reaching up to 73.8% when exposed to manipulated search results. This research highlights critical implications for practitioners, emphasizing the need for robust defenses against content pollution in generative recommenders, as traditional reasoning and skepticism methods may inadvertently exacerbate the issue.