Knowledge Tree
pre-trained language models
Also known as: pretrained language models, pre-trained models, pre-trained language representation, pre-trained language model, Pre-trained Language Model, PLM, PLMs, enhanced pre-trained language models
Facts (44)
Sources
Practices, opportunities and challenges in the fusion of knowledge ... frontiersin.org 21 facts
referenceCao and Liu (2023) proposed RELMKG, a method for reasoning with pre-trained language models and knowledge graphs for complex question answering, published in Applied Intelligence.
referenceXu et al. (2023) developed a pre-trained language model with prompts specifically for temporal knowledge graph completion.
referenceAUTOPROMPT (Shin et al., 2020) automatically generates prompts through gradient-guided search to assist pre-trained models in performing tasks.
referenceWang et al. (2022) developed 'SIMKGC', a simple contrastive knowledge graph completion method utilizing pre-trained language models.
referenceTAGREAL (Jiang P. et al., 2023) automatically generates high-quality query hints and retrieves supporting information from large text corpora to detect knowledge within pre-trained language models (PLMs).
referenceWang et al. (2022) explored knowledge prompting in pre-trained language models for natural language understanding.
referenceMedLAMA (Meng et al., 2021) creates a benchmark based on UML and introduces the Contrastive-Probe, a self-supervised contrastive probing method that can adjust the representation space of the underlying pre-trained models without any task-specific data.
referenceThe paper 'Rewire-then-probe: a contrastive recipe for probing biomedical knowledge of pre-trained language models' by Meng, Z., Liu, F., Shareghi, E., Su, Y., Collins, C., Collier, N. introduces a contrastive method for probing biomedical knowledge in pre-trained language models.
referenceWang et al. (2021) created 'KEPLER', a unified model designed for both knowledge embedding and pre-trained language representation.
referenceB. Kim, T. Hong, Y. Ko, and J. Seo published 'Multi-task learning for knowledge graph completion with pre-trained language models' in the Proceedings of the 28th International Conference on Computational Linguistics in 2020.
referenceHe et al. (2019) developed a method for integrating graph contextualized knowledge into pre-trained language models.
referenceThe GenKGC model (Xie et al., 2022) leverages pre-trained language models to convert the knowledge graph completion task into a sequence-to-sequence generation task.
referenceJ. Li, T. Tang, W. X. Zhao, Z. Wei, N. J. Yuan, and J.-R. Wen published 'Few-shot knowledge graph-to-text generation with pretrained language models' as an arXiv preprint in 2021.
referenceJiang et al. (2023) proposed a method for text-augmented open knowledge graph completion using pre-trained language models.
referencekNN-KGE (Wang P. et al., 2023) uses a pre-trained language model and k-nearest neighbors to perform linear interpolation of entity distributions, which are calculated based on the distance between entity embeddings and knowledge storage.
referenceThe paper 'Knowledge graph extension with a pre-trained language model via unified learning method' (Knowl.-Based Syst. 262:110245) proposes a unified learning method for extending knowledge graphs using pre-trained language models.
referenceHao et al. (2022) introduced 'Bertnet', a system for harvesting knowledge graphs with arbitrary relations from pre-trained language models.
referenceLAMA (Petroni et al., 2019) converts knowledge into cloze-style questions to evaluate the relational knowledge and recall ability of pre-trained models.
referenceThe paper 'Mem-kgc: masked entity model for knowledge graph completion with pre-trained language model' (IEEE Access 9, 132025–132032) introduces a masked entity model approach for knowledge graph completion using pre-trained language models.
referenceThe paper 'Pretrain-kge: learning knowledge representation from pretrained language models' was published in the Findings of the Association for Computational Linguistics: EMNLP 2020.
referenceBERTRL, proposed by Zha et al. in 2022, leverages pre-trained language models and fine-tunes them using relation instances and reasoning paths as training samples.
Combining large language models with enterprise knowledge graphs frontiersin.org Aug 26, 2024 10 facts
claimFully fine-tuning Pre-trained Language Models (PLMs) is often costly and inefficient, requiring substantial computational resources and time, and because these models are tailored for narrow applications, updates are challenging, according to Razuvayevskaya et al. (2023).
claimInaccurate Named Entity Recognition and Relation Extraction prompting results can be corrected through active learning techniques (Wu et al., 2022) or by distilling large Pre-trained Language Models into smaller models for specific tasks (Agrawal et al., 2022).
perspectiveA hybrid approach that combines Pre-trained Language Models (PLMs), Knowledge Graph (KG) structure understanding, and domain expertise is recommended to ensure privacy compliance in industrial settings.
perspectiveThe authors advocate for Pre-trained Language Model (PLM)-based Knowledge Graph Embedding (KGE) approaches that treat the Large Language Model (LLM) as a modular component, allowing for easy replacement to integrate context-specific models trained on domain-specific knowledge to enhance system relevance and accuracy.
claimPrompting with large Large Language Models (like GPTs) can underperform in Named Entity Recognition compared to fine-tuned smaller Pre-trained Language Models (like BERT derivations), especially when more training data is available (Gutierrez et al., 2022; Keloth et al., 2024; Pecher et al., 2024; Törnberg, 2024).
claimMethodological frameworks for Pre-trained Language Model (PLM)-based Knowledge Graph Embedding (KGE) techniques generally fall into two categories: model finetuning and prompting.
perspectiveTo adapt to evolving Large Language Models (LLMs), Pre-trained Language Models (PLMs) should be treated as plug-and-play components to ensure versatility and longevity.
claimThe primary challenges of implementing corporate Knowledge Graph Embedding (KGE) solutions are categorized into four areas: (i) the quality and quantity of public or automatically annotated data, (ii) developing sustainable solutions regarding computational resources and longevity, (iii) adaptability of PLM-based KGE systems to evolving language and knowledge, and (iv) creating models capable of efficiently learning the Knowledge Graph (KG) structure.
claimThe main challenges for enterprise Large Language Model (LLM)-based solutions for Knowledge Graph Embedding (KGE) include the high cost and resource intensity of creating tailored Pre-trained Language Model (PLM)-based KGE solutions, the mismatch between public benchmark datasets and enterprise use cases due to structural differences, the need for robust methods to combine automated novelty detection with human-curated interventions, and the requirement for a shift from classification to representation learning to accommodate novelty and encode Knowledge Graph (KG) features.
claimFinetuning PLM-based KGE models is generally costly and requires large amounts of annotated data, whereas prompting is more cost-effective but introduces privacy-related risks.
Unlocking the Potential of Generative AI through Neuro-Symbolic ... arxiv.org Feb 16, 2025 2 facts
claimTransfer learning enables pre-trained models to adapt their knowledge to new domains using minimal data, making it particularly advantageous in resource-constrained scenarios.
referenceYukun Huang, Yanda Chen, Zhou Yu, and Kathleen McKeown published 'In-context learning distillation: Transferring few-shot learning ability of pre-trained language models' as an arXiv preprint (arXiv:2212.10670) in 2022.
A Survey on the Theory and Mechanism of Large Language Models arxiv.org Mar 12, 2026 2 facts
claimOpenAI introduced the "weak-to-strong generalization" (W2SG) paradigm (Burns et al., 2024), which demonstrates that strong pre-trained language models fine-tuned using supervision signals from weaker models consistently surpass the performance of their weak supervisors.
referenceThe paper 'Why do pretrained language models help in downstream tasks? an analysis of head and prompt tuning' analyzes the efficacy of head and prompt tuning.
A survey on augmenting knowledge graphs (KGs) with large ... link.springer.com Nov 4, 2024 2 facts
claimLAMA (LAnguage Model Analysis) is a benchmark for evaluating the factual knowledge contained in pre-trained language models by testing their ability to recall factual information without additional context.
referenceThe paper 'A unified model for knowledge embedding and pre-trained language representation' was published in the Transactions of the Association for Computational Linguistics in 2021 (Volume 9, pages 176–94).
The Synergy of Symbolic and Connectionist AI in LLM-Empowered ... arxiv.org Jul 11, 2024 1 fact
claimFine-tuning adapts pre-trained models to specific tasks or domains by utilizing a smaller, task-specific dataset to optimize performance for particular applications.
On Hallucinations in Artificial Intelligence–Generated Content ... jnm.snmjournals.org 1 fact
claimTransfer learning, which involves leveraging publicly pretrained models and fine-tuning them on local data, is an effective strategy for balancing generalization and specialization to mitigate hallucinations.
Construction of Knowledge Graphs: State and Challenges - arXiv arxiv.org 1 fact
claimThe use of pre-trained language models has advanced the state-of-the-art in neural relation extraction.
[2509.04664] Why Language Models Hallucinate - arXiv arxiv.org Sep 4, 2025 1 fact
claimHallucinations in pretrained language models originate as errors in binary classification, arising through natural statistical pressures when incorrect statements cannot be distinguished from facts.
Large Language Models Meet Knowledge Graphs for Question ... arxiv.org Sep 22, 2025 1 fact
claimQi Zhao, Qi Song, Tian Xie, Haiyue Zhang, Hongyu Yang, and Xiangyang Li published the paper 'Improving pre-trained language models with knowledge enhancement and filtering framework' in 2025.
Applying Large Language Models in Knowledge Graph-based ... arxiv.org Jan 7, 2025 1 fact
referenceHu, N., Wu, Y., Qi, G., Min, D., Chen, J., Pan, J.Z., and Ali, Z. conducted an empirical study of pre-trained language models in simple knowledge graph question answering, published in the journal World Wide Web in 2023.
Combining Knowledge Graphs and Large Language Models - arXiv arxiv.org Jul 9, 2024 1 fact
referenceYusheng Su, Xu Han, Zhengyan Zhang, Yankai Lin, Peng Li, Zhiyuan Liu, Jie Zhou, and Maosong Sun developed 'CoKEBERT', a model for contextual knowledge selection and embedding towards enhanced pre-trained language models, as described in their 2021 paper (AI Open, 2:127–134).