Knowledge Tree
Home Research Docs About
concept

Prompt Sensitivity

Also known as: Prompt Sensitivity (PS)

Related (2)

Facts (24)

Sources
Survey and analysis of hallucinations in large language models frontiersin.org Frontiers Sep 29, 2025 24 facts
claimMistral shows balanced behavior across the dimensions of Factuality, Coherence, Prompt Sensitivity, Model Variability, and Usability, indicating a mixed attribution of hallucination sources.
claimMixed-origin models, such as Mistral 7B and OpenChat-3.5, display moderate Prompt Sensitivity (PS) and Model Variability (MV) scores, indicating that both prompt and model factors contribute equally to hallucinations.
claimThe authors of the survey introduce 'Prompt Sensitivity (PS)' as a concrete metric designed to systematically measure the effect of prompt changes on model hallucinations.
referenceResearch directions for hallucination evaluation include the development of integrated, multi-task, multilingual benchmarks with unified annotation schemas (Liu et al., 2023) and the use of attribution-aware metrics incorporating Prompt Sensitivity (PS) and Model Variability (MV).
claimThe radar plot in Figure 4 of the study 'Survey and analysis of hallucinations in large language models' visualizes the comparative performance of DeepSeek, Mistral, and LLaMA 2 across five behavioral dimensions: Factuality, Coherence, Prompt Sensitivity, Model Variability, and Usability.
claimThe authors of the 'Survey and analysis of hallucinations in large language models' define Prompt Sensitivity (PS) and Model Variability (MV) as metrics to quantify the contribution of prompts versus model-internal factors to hallucinations.
referenceHallucinations can be categorized into four attribution types based on Prompt Sensitivity (PS) and Model Variation (MV) scores: Prompt-dominant (high PS, low MV), Model-dominant (low PS, high MV), Mixed-origin (high PS, high MV), and Unclassified/noise (low PS, low MV).
claimPrompt-dominant models, such as LLaMA 2, exhibit high Prompt Sensitivity (PS), meaning their hallucination rates fluctuate based on prompt structure and can be effectively steered using structured prompting techniques like Chain-of-Thought.
claimLLaMA 2 (13B) exhibits high prompt sensitivity, allowing for fine control via prompts but also making it susceptible to poorly worded questions.
claimThe hallucination attribution framework provides interpretable quantitative scores, specifically Prompt Sensitivity (PS), Model Variability (MV), and Joint Attribution Score (JAS), which are used for benchmarking and tracking improvements in Large Language Models.
procedureThe evaluation framework presented in 'Survey and analysis of hallucinations in large language models' utilizes QAFactEval and hallucination rate metrics to compute Prompt Sensitivity (PS) and Model Variability (MV), allowing for the differentiation between prompt-induced and model-intrinsic hallucinations.
claimModel-dominant models, such as DeepSeek 67B, show low Prompt Sensitivity (PS) but high Model Variability (MV), meaning hallucinations persist regardless of prompt variation due to internal knowledge limitations or inference biases.
claimModels with higher PS (Prompt Sensitivity) and MV (Model Variance) metrics generally performed worse on factuality benchmarks like TruthfulQA (Lin et al., 2022) and HallucinationEval (Wu et al., 2023), while models with low MV, such as GPT-4, achieved better TruthfulQA scores.
measurementLLaMA 2 exhibits high Prompt Sensitivity (PS), while DeepSeek shows high Model Variability (MV).
claimGPT-4's lower hallucination rate is more stable across prompts (lower Prompt Sensitivity) compared to GPT-3.5, as observed in the analysis and supported by prior findings from Liu et al. (2023).
claimPrompt Sensitivity (PS) is a metric that measures the variation in output hallucination rates under different prompt styles for a fixed model, where high PS indicates that hallucinations are primarily prompt-induced.
measurementThe 'Survey and analysis of hallucinations in large language models' reports Prompt Sensitivity (PS) and Model Variability (MV) scores for LLMs as follows: LLaMA 2 (13B) (PS: 0.091, MV: 0.045), Mistral 7B (PS: 0.078, MV: 0.053), DeepSeek 67B (PS: 0.060, MV: 0.080), OpenChat-3.5 (PS: 0.083, MV: 0.062), and Gwen (PS: 0.079, MV: 0.057).
perspectiveFor developers deploying Large Language Models, selecting models based on attribution patterns (Prompt Sensitivity vs. Model Vulnerability) can inform fine-tuning strategies.
claimLLaMA-2's hallucinations are primarily prompt-driven, characterized by high Prompt Sensitivity (PS) and low Model Variance (MV), meaning the model fails when prompts are suboptimal.
claimPrompt Sensitivity, as a metric for language model evaluation, indicates the extent to which a model's output is influenced by different prompt formulations, where higher sensitivity implies a greater risk of prompt-induced hallucination.
procedureTo establish objective thresholds for 'low' versus 'high' Prompt Sensitivity and Model Variability, the authors collect the values for all evaluated models, plot the distributions, and use the median value of each distribution as the cutoff.
procedureThe Prompt Sensitivity (PS) measurement protocol involves evaluating each model on multiple variants of prompts systematically varied along three axes: Format (e.g., declarative vs. interrogative vs. instruction-style), Structure (e.g., straight forward query vs. Chain-of-Thought, zero-shot vs. few-shot, inclusion of context), and Specificity (vague vs. explicitly detailed).
formulaPrompt sensitivity (PS) is defined as the degree of variation in hallucination frequency observed across different prompt types.
measurementCPS (Prompt Sensitivity) analysis showed LLaMA 2 had a value of 0.15 for vaguely specified prompts, indicating susceptibility to prompt-induced hallucinations.