Mahankali et al. (2023) proved that when covariates are sampled from a Gaussian distribution, the pretraining loss with a single-layer linear attention is minimized via one-step gradient descent.

Authors

• Person: Not available Organization: arXiv
  A Survey on the Theory and Mechanism of Large Language Models

Sources

• A Survey on the Theory and Mechanism of Large Language Models arxiv.org arXiv via serper

Referenced by nodes (1)

• gradient descent concept