# [2401.00396] RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models
Abstract page for arXiv paper 2401.00396: RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models

# Computer Science > Computation and Language

[Submitted on 31 Dec 2023 (v1), last revised 17 May 2024 (this version, v2)]

# Title:RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models

Authors: Cheng Niu, Yuanhao Wu, Juno Zhu, Siliang Xu, Kashun Shum, Randy Zhong, Juntong Song, Tong Zhang
View a PDF of the paper titled RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models, by Cheng Niu and 7 other authors
View PDF HTML (experimental)

> Abstract:Retrieval-augmented generation (RAG) has become a main technique for alleviating hallucinations in large language models (LLMs). Despite the integration of RAG, LLMs may still present unsupported or contradictory claims to the retrieved contents. In order to develop effective hallucination prevention strategies under RAG, it is important to create benchmark datasets that can measure the extent of hallucination. This paper presents RAGTruth, a corpus tailored for analyzing word-level hallucinations in various domains and tasks within the standard RAG frameworks for LLM applications. RAGTruth comprises nearly 18,000 naturally generated responses from diverse LLMs using RAG. These responses have undergone meticulous manual annotations at both the individual cases and word levels, incorporating evaluations of hallucination intensity. We not only benchmark hallucination frequencies across different LLMs, but also critically assess the effectiveness of several existing hallucination detection methodologies. Furthermore, we show that using a high-quality dataset such as RAGTruth, it is possible to finetune a relatively small LLM and achieve a competitive level of performance in hallucination detection when compared to the existing prompt-based approaches using state-of-the-art large language models such as GPT-4.

Subjects: | Computation and Language (cs.CL)
Cite as: | arXiv:2401.00396 [cs.CL]
| (or arXiv:2401.00396v2 [cs.CL] for this version)
| https://doi.org/10.48550/arXiv.2401.00396 arXiv-issued DOI via DataCite

## Submission history

From: Yuanhao Wu [view email]
[v1] Sun, 31 Dec 2023 04:43:45 UTC (7,284 KB)
[v2] Fri, 17 May 2024 06:29:31 UTC (8,095 KB)
Full-text links:

## Access Paper:

View a PDF of the paper titled RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models, by Cheng Niu and 7 other authors
- View PDF
- HTML (experimental)
- TeX Source

view license
Current browse context:
cs.CL
< prev | next >
new | recent | 2024-01
Change to browse by:
cs

### References & Citations

- NASA ADS
- Google Scholar
- Semantic Scholar

export BibTeX citation Loading...

### Bookmark

Bibliographic Tools

# Bibliographic and Citation Tools

Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Demos
Related Papers
About arXivLabs
Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)