Data pipelines for training large language models typically filter raw web crawl text using five specific heuristics: minimum document length, minimum token entropy, language identification, perplexity filtering against a small reference model, and blacklist-based removal of known low-quality domains.

Authors

• Person: M. Brenndoerfer Organization: mbrenndoerfer.com
  Hallucination Causes: Why Language Models Fabricate Facts

Sources

• Hallucination Causes: Why Language Models Fabricate Facts mbrenndoerfer.com M. Brenndoerfer · mbrenndoerfer.com via serper

Referenced by nodes (1)

• Large Language Models concept