The AWS approach for detecting hallucinations in RAG-based systems involves the following steps: 1) Create a dataset with questions, context, and the response to classify. 2) Send a call to an LLM providing the statement and the context. 3) Instruct the LLM to tag sentences in the statement that are directly based on the context. 4) Parse the outputs to obtain sentence-level numeric scores between 0 and 1. 5) Ensure the LLM, memory, and parameters used for evaluation are independent of those used for Q&A. 6) Tune the decision threshold for the hallucination scores based on the specific dataset or domain. 7) Use the threshold to classify the statement as a hallucination or fact.

Authors

• Person: Not available Organization: Amazon Web Services
  Detect hallucinations for RAG-based systems - AWS

Sources

• Detect hallucinations for RAG-based systems - AWS aws.amazon.com Amazon Web Services via serper

Referenced by nodes (2)

• RAG concept
• Amazon Web Services entity