concept

Creole

Also known as: CREOLA, Clinical Review of LLMs and AI

Facts (17)

Sources

A framework to assess clinical safety and hallucination rates of LLMs ... nature.com Nature May 13, 2025 16 facts

measurementThe authors used the CREOLA platform to analyze the impact of prompting techniques on the safety of large language model outputs, achieving low hallucination and omission rates that outperform previously reported model and human error rates.

claimThe researchers built CREOLA, an in-house platform designed to enable clinicians to identify and label relevant hallucinations and omissions in clinical text to inform future experiments and implement the researchers' framework at scale.

claimThe name CREOLA (Clinical Review of LLMs and AI) pays tribute to Creola Katherine Johnson, a pioneering human computer at NASA, drawing a parallel between the role of human computers in Apollo moon missions and the role of clinicians in safely integrating AI into clinical practice.

referenceCREOLA (Clinical Review of LLMs and AI) is a platform that combines experiment design, hallucination and omission taxonomy, and clinical safety evaluation to identify changes in generated clinical documentation resulting from changes in LLM architecture.

claimThe framework utilizes a 'clinician in the loop' approach, where clinicians identify clinical errors in LLM-generated documentation, supported by a specialized annotation platform called CREOLA.

procedureThe authors of the study 'A framework to assess clinical safety and hallucination rates of LLMs' introduced a framework for clinical note generation consisting of four components: (1) a clinically and technically-informed error taxonomy to classify LLM outputs, (2) an experiment structure to comprehensively and iteratively compare outputs within an LLM document generation pipeline, (3) a clinical safety framework to assess potential harms of errors in LLM outputs, and (4) an encompassing graphical user interface (GUI) named CREOLA to perform and assess all previous steps.

claimThe CREOLA framework provides a sandbox environment to validate or discredit LLM architectures and prompt approaches prior to clinical deployment.

procedureThe CREOLA framework requires clinicians to review and annotate model outputs to assess the veracity of clinical facts, as clinicians possess the unique skills necessary for this task.

claimThe CREOLA platform provides a sandbox environment to buffer users and patients from harm if LLM iterations result in increased clinical error rates.

claimThe CREOLA platform was built by M.D. and S.K. to facilitate clinical safety and hallucination rate assessments in LLMs.

claimThe CREOLA platform, used for annotating model outputs, was hosted as a Streamlit web application at https://creola.tortus.ai/.

claimThe CREOLA framework is designed for the clinical safety assessment of large language models in clinical documentation scenarios.

procedureTo ensure annotators understood the annotation process, the CREOLA study team initially provided one-to-one tuition, which was later replaced by a short online course followed by a mandatory questionnaire that annotators had to complete correctly to participate.

claimThe authors propose a framework for assessing clinical safety and hallucination rates in large language models (LLMs) that includes an error taxonomy for classifying outputs, an experimental structure for iterative comparisons in document generation pipelines, a clinical safety framework to evaluate error harms, and a graphical user interface named CREOLA.

claimThe CREOLA platform is an in-house tool designed to allow clinicians to identify and label hallucinations and omissions in LLM-generated clinical text.

claimThe CREOLA platform provided a communication channel for annotators to contact the study team to resolve issues promptly.

Associations between dietary diversity and self-rated health in a ... link.springer.com Springer Feb 28, 2025 1 fact

procedureThe study participants were selected using a combination of non-probability sampling methods: (a) convenience sampling through haphazard and chance meetings, (b) quota sampling to ensure equitable sex ratios and the inclusion of specific subpopulations such as Creole, Indigenous, and Brazilian populations on the Oyapock River, and (c) snowball sampling through participant networks.