According to Shannon’s source coding theorem (Shannon, 1948), the minimum expected number of bits required to encode data is the entropy, which represents the theoretical limit of a model’s compression performance.

Authors

• Person: Not available Organization: arXiv
  A Survey on the Theory and Mechanism of Large Language Models

Sources

• A Survey on the Theory and Mechanism of Large Language Models arxiv.org arXiv via serper

Referenced by nodes (1)

• entropy concept