concept

AI alignment

Also known as: alignment

Facts (18)

Sources

A Survey on the Theory and Mechanism of Large Language Models arxiv.org arXiv Mar 12, 2026 8 facts

referenceThe paper 'Vision superalignment: weak-to-strong generalization for vision foundation models' (arXiv:2402.03749) is cited in the survey 'A Survey on the Theory and Mechanism of Large Language Models' regarding alignment.

referenceThe paper 'Fundamental limitations of alignment in large language models' exists as an arXiv preprint (arXiv:2304.11082) and was also published in the Proceedings of the 41st International Conference on Machine Learning, pages 53079–53112.

referenceThe research paper 'Trustworthy llms: a survey and guideline for evaluating large language models’ alignment' was published as an arXiv preprint (arXiv:2505.21598) and cited in section 2.2.1 of the survey.

referenceThe paper 'Ai alignment: a comprehensive survey' is an arXiv preprint, identified as arXiv:2310.19852.

referenceThe paper 'Murphys laws of ai alignment: why the gap always wins' explores the challenges and inevitable difficulties in AI alignment.

claimLin et al. (2024) proposed Heterogeneous Model Averaging (HMA) to balance the trade-off between alignment and the retention of pre-training knowledge.

claimThe survey titled 'A Survey on the Theory and Mechanism of Large Language Models' organizes the theoretical landscape of Large Language Models into a lifecycle-based taxonomy consisting of six stages: Data Preparation, Model Preparation, Training, Alignment, Inference, and Evaluation.

claimThe academic community has established two primary theoretical pillars for AI alignment: the pursuit of mathematical safety guarantees and the mechanistic analysis of Reinforcement Learning (RL) dynamics.

Building Trustworthy NeuroSymbolic AI Systems - arXiv arxiv.org arXiv 3 facts

claimThe National Science Foundation (NSF) identifies grounding, instructability, and alignment as the three fundamental attributes of ensuring AI safety.

referenceNgo, Chan, and Mindermann (2022) analyzed the AI alignment problem from the perspective of deep learning.

claimAlignment in language models refers to ensuring that a model designed to follow instructions does not produce unsafe results, a concept discussed by MacDonald in 1991.

A Survey of Incorporating Psychological Theories in LLMs - arXiv arxiv.org arXiv 2 facts

referenceYang Liu, Yuanshun Yao, Jean-Francois Ton, Xiaoying Zhang, Ruocheng Guo, Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, and Hang Li authored 'Trustworthy llms: a survey and guideline for evaluating large language models’ alignment', published as an arXiv preprint in 2023.

claimAdding reward variability in reinforcement learning may reduce premature convergence and improve alignment with human intent.

Cybersecurity Trends and Predictions 2025 From Industry Insiders itprotoday.com ITPro Today 1 fact

claimAvani Desai identifies AI alignment—the tailoring of AI models to serve specific geopolitical motives—as a critical emerging concern, noting that these tools could be engineered to exploit vulnerabilities in a rival's infrastructure.

A Comprehensive Review of Neuro-symbolic AI for Robustness ... link.springer.com Springer Dec 9, 2025 1 fact

referenceWagner, B.J. and Garcez, A. proposed a neuro-symbolic approach to AI alignment in a 2024 preprint titled 'A neurosymbolic approach to AI alignment'.

LLM-empowered knowledge graph construction: A survey - arXiv arxiv.org arXiv Oct 23, 2025 1 fact

referenceThe Graphusion framework (Yang et al., 2024) uses a unified, prompt-based paradigm to perform fusion subtasks—including alignment, consolidation, and inference—within a single generative cycle.

Practices, opportunities and challenges in the fusion of knowledge ... frontiersin.org Frontiers 1 fact

claimThe mismatch in tokenization between Large Language Model (LLM) and Knowledge Graph (KG) embeddings can lead to information loss during alignment.

The Evidence for AI Consciousness, Today - AI Frontiers ai-frontiers.org AI Frontiers Dec 8, 2025 1 fact

claimAI alignment work has historically focused on preventing artificial intelligence from becoming dangerous through methods of control, containment, and corrigibility.