Large Language Models ↔ Vectara

Relations (1)

cross_type 0.70 — strongly supporting 7 facts

Vectara is related to Large Language Models because it develops tools like the HHEM [1] and the LLM Hallucination Leaderboard [2] to evaluate, measure, and improve the truthfulness and accuracy of Large Language Models {fact:4, fact:5}.

Facts (7)

Sources

vectara/hallucination-leaderboard - GitHub github.com Vectara 5 facts

claimThe Vectara hallucination leaderboard serves as an indicator for the accuracy of Large Language Models when deployed in Retrieval Augmented Generation (RAG) and agentic pipelines, where the model acts as a summarizer of search results.

procedureVectara used a temperature setting of 0 when querying Large Language Models for the hallucination leaderboard, except in cases where that setting was impossible or unavailable.

claimThe Vectara hallucination leaderboard authors chose to evaluate hallucination rates in summarization tasks rather than attempting to determine if a response was hallucinated without a reference source, because the latter would require training a model as large or larger than the LLMs being evaluated.

claimThe dataset used for the Vectara hallucination leaderboard is curated to be not publicly available to prevent overfitting by Large Language Models, contains over 7700 articles from diverse sources including news, technology, science, medicine, legal, sports, business, and education, and includes articles ranging from 50 words to 24,000 words in length.

claimThe Vectara hallucination leaderboard evaluates hallucination rates within summarization tasks as an analogue for determining the overall truthfulness of Large Language Models (LLMs).

Real-Time Evaluation Models for RAG: Who Detects Hallucinations ... cleanlab.ai Cleanlab 1 fact

referenceThe Hughes Hallucination Evaluation Model (HHEM) is a Transformer model trained by Vectara to distinguish between hallucinated and correct responses from various Large Language Models across different context and response data.

EdinburghNLP/awesome-hallucination-detection - GitHub github.com GitHub 1 fact

referenceThe Vectara LLM Hallucination Leaderboard is a resource for evaluating hallucinations in large language models.