The token similarity detection approach for hallucination detection involves extracting unique sets of tokens from the answer and the context, then calculating similarity using metrics such as BLEU score over different n-grams, ROUGE score, or the proportion of shared tokens between the two texts.

Authors

• Person: Not available Organization: Amazon Web Services
  Detect hallucinations for RAG-based systems - AWS

Sources

• Detect hallucinations for RAG-based systems - AWS aws.amazon.com Amazon Web Services via serper

Referenced by nodes (1)

• hallucination detection concept