Knowledge Tree
human-in-the-loop
Also known as: human-in-the-loop refinement, human-in-the-loop workflows, HITL
Facts (21)
Sources
Construction of Knowledge Graphs: State and Challenges - arXiv arxiv.org 8 facts
referenceThe ScienceWise project uses the XI Pipeline with a human-in-the-loop approach to annotate papers and build a research knowledge graph.
claimUsing an iterative human-in-the-loop process allows for continuous improvement and refinement of knowledge graphs, enhancing the overall reliability and trustworthiness of the data.
claimQuality evaluation frameworks for knowledge graphs can support mechanisms for quality improvement, such as human-in-the-loop approaches or automatic error correction.
claimThe XI Pipeline performs entity linking by combining probabilistic models and microtask crowdsourcing, which outperforms models that do not use a human-in-the-loop paradigm.
claimThe Health Knowledge Graph Builder (HKGB) is a platform designed to semi-automatically construct clinical knowledge graphs with heavy human-in-the-loop involvement, consuming Electronic Medical Records (EMR) as input and producing graph data in OWL and RDF formats.
claimHuman-in-the-loop approaches for knowledge graph evaluation may require KG sampling to evaluate only sub-graphs of the entire knowledge graph due to the degree of automation involved.
referenceThe NELL project proposed a human-in-the-loop approach where learned extraction patterns were validated by a user after a certain number of iterations.
claimThe SAGA system supports live graph curation through a human-in-the-loop approach and powers question answering, entity summarization, and text annotation (NER) services.
Combining large language models with enterprise knowledge graphs frontiersin.org Aug 26, 2024 7 facts
claimThe authors of 'Combining large language models with enterprise knowledge graphs' identify LLMs, knowledge graph, relation extraction, knowledge graph enrichment, AI, enterprise AI, carbon footprint, and human in the loop as the primary keywords for their research.
referenceThe paper 'How to invest my time: lessons from human-in-the-loop entity extraction' by Zhang et al. (2019) discusses strategies for human-in-the-loop entity extraction.
claimThe human-in-the-loop (HITL) paradigm has been successfully employed to dynamically curate and expand databases based on subject matter expert feedback (Gentile et al., 2019).
procedureThe procedure for Human-in-the-loop (HITL) methods in Natural Language Understanding (NLU) involves: (1) starting with a small set of annotated data, (2) selecting challenging samples for the model, (3) having humans annotate these samples, (4) updating the model with the new annotations, and (5) repeating the process.
referenceThe paper 'A survey of human-in-the-loop for machine learning' by Wu et al. (2022) provides a comprehensive survey of human-in-the-loop methodologies within the field of machine learning.
claimThe human-in-the-loop (HITL) paradigm integrates human expertise into the modeling process to manage machine learning model uncertainty, as noted by Wu et al. (2022).
claimHuman-in-the-loop (HITL) methods effectively handle scarce or sparse data for Named Entity Recognition (NER) (Shen et al., 2017), address mislabeling (Muthuraman et al., 2021), and enhance data processing, model training, and inference stages of the machine learning pipeline (Zhang et al., 2019; Klie et al., 2020; Wu et al., 2022).
A Comprehensive Review of Neuro-symbolic AI for Robustness ... link.springer.com Dec 9, 2025 3 facts
claimNeuro-symbolic AI supports iterative human-in-the-loop refinement during training and debugging.
referenceNatarajan et al. (2025) explored the distinction between human-in-the-loop and AI-in-the-loop systems, specifically addressing automation versus collaboration.
referenceEthical deployment of neuro-symbolic systems requires supporting procedural fairness and democratic accountability through mechanisms such as human-in-the-loop intervention, the ability to audit and revise symbolic logic, and participatory rule curation that reflects diverse values, as suggested in reference [186].
LLM Hallucination Detection and Mitigation: State of the Art in 2026 zylos.ai Jan 27, 2026 1 fact
procedureHigh-stakes LLM decisions should incorporate human-in-the-loop processes, including flagging low-confidence responses for human review, implementing approval workflows, and building feedback loops to improve detection.
Reducing hallucinations in large language models with custom ... aws.amazon.com Nov 26, 2024 1 fact
claimThe combination of Amazon Bedrock Agents, Amazon Knowledge Bases, and RAGAS evaluation metrics allows for the construction of a custom hallucination detector that remediates hallucinations using human-in-the-loop processes.
Medical Hallucination in Foundation Models and Their ... medrxiv.org Mar 3, 2025 1 fact
perspectiveThe authors argue that the deployment of any current large language model in clinical settings necessitates rigorous, task-specific validation protocols, continuous performance monitoring, and careful integration within human-in-the-loop workflows, regardless of overall performance metrics.