concept

memorization

Facts (11)

Sources

A Survey on the Theory and Mechanism of Large Language Models arxiv.org arXiv Mar 12, 2026 10 facts

referenceThe paper 'Memorization in deep learning: a survey' was published in ACM Computing Surveys.

referenceKandpal et al. (2022) argue that data repetition is the primary driver of memorization that leads to privacy risks, and they demonstrated that re-training models on sequence-level deduplicated data significantly reduces these privacy risks.

claimThe memorization of contaminated data, particularly sensitive information, creates significant privacy vulnerabilities in large language models.

referenceThe paper 'Generalization v.s. memorization: tracing language models’ capabilities back to pretraining data' investigates the relationship between memorization and generalization in language models.

claimMemorization in Large Language Models is deeply intertwined with the model's learning and generalization capabilities, rather than being solely a privacy risk (Wei et al., 2024).

claimHuang et al. (2024b) observed a 'cliff-like decline' in GPT-4's performance on medium-to-hard problems when tested on novel competition problems released after its training data cut-off, suggesting reliance on memorization rather than algorithmic reasoning.

referenceThe paper 'Memorization sinks: isolating memorization during LLM training' was published in the Proceedings of the 42nd International Conference on Machine Learning, Vol. 267, pp. 19307–19326, edited by A. Singh, M. Fazel, D. Hsu, S. Lacoste-Julien, F. Berkenkamp, T. Maharaj, K. Wagstaff, and J. Zhu.

referenceThe paper 'Memorization or interpolation? detecting llm memorization through input perturbation analysis' is an arXiv preprint, identified as arXiv:2505.03019.

referenceThe paper 'Entropy-memorization law: evaluating memorization difficulty of data in llms' is an arXiv preprint, identified as arXiv:2507.06056.

claimBiderman et al. (2023) demonstrated that using a partially trained model to predict memorization is more effective than using a small model.

Investigating the impact of sleep quality on cognitive functions ... frontiersin.org Frontiers 1 fact

claimThe Japanese education system's emphasis on rote learning and memorization may increase reliance on cognitive processes sensitive to sleep deprivation, such as working memory and attention, among students in Tokyo.