RAG Evaluator is a comprehensive Python library for evaluating Retrieval-Augmented Generation (RAG) systems and generated responses. It provides multiple metrics to assess the quality, accuracy, and reliability of AI-generated text against reference contexts.
This tool is designed for researchers, developers, and data scientists who need to systematically evaluate and improve their RAG pipelines, ensuring high-quality outputs for production applications.
- BLEU Score: Measures n-gram overlap between generated output and reference text
- ROUGE-1: Evaluates unigram overlap for content coverage assessment
- BERTScore: Leverages BERT embeddings to measure semantic similarity
- Perplexity: Assesses language model prediction quality and fluency
- Diversity: Quantifies the uniqueness and variety of generated content
- Racial Bias Detection: Identifies potential biased language in outputs
Future versions will include advanced evaluation metrics:
- Faithfulness: Measure how well responses adhere to provided context
- Context Precision: Evaluate the relevance of retrieved documents
- Context Recall: Assess completeness of information retrieval
- Hallucination Detection: Identify unsupported or fabricated claims
- LLM-as-a-Judge: Automated evaluation using large language models
- Answer Relevance: Measure how directly responses address queries
- Response Completeness: Assess thoroughness of generated answers
Install the library using pip:
pip install rag-evaluatorFor local development:
git clone https://github.com/itsananytripathi/rag-evaluator.git
cd rag-evaluator
pip install -r requirements.txtfrom rag_evaluator import RAGEvaluator
# Initialize the evaluator
evaluator = RAGEvaluator()
# Input data
question = "What are the causes of climate change?"
response = "Climate change is caused by human activities."
reference = "Human activities such as burning fossil fuels cause climate change."
# Evaluate the response
metrics = evaluator.evaluate_all(question, response, reference)
# Print the results
print(metrics)Launch the interactive evaluation dashboard:
cd "streamlit app"
pip install -r requirements.txt
streamlit run app.pyThe web app provides a user-friendly interface for:
- Interactive RAG system evaluation
- Real-time metric visualization
- Comparative analysis across multiple responses
- Export capabilities for reporting
| Metric | Description | Use Case |
|---|---|---|
| BLEU | N-gram overlap measurement | Translation quality, text similarity |
| ROUGE-1 | Unigram overlap scoring | Summarization evaluation |
| BERTScore | Semantic similarity via embeddings | Context-aware quality assessment |
| Perplexity | Language model prediction confidence | Fluency and coherence evaluation |
| Diversity | Bigram uniqueness measurement | Output variety assessment |
| Racial Bias | Hate speech detection | Ethical AI compliance |
Run the test suite:
python -m unittest discover -s rag_evaluator -p "test_*.py"- Add Faithfulness metric
- Add Context Precision and Recall
- Implement Hallucination Detection
- Add comprehensive documentation
- LLM-as-a-Judge evaluation framework
- Answer Relevance scoring
- Response Completeness metric
- Batch evaluation capabilities
- Full RAG pipeline evaluation suite
- Custom metric plugins
- Enterprise features
- Performance optimizations
Anany Tripathi
- GitHub: itsananytripathi
- Email: anany.tripathivns@gmail.com
This project is maintained and enhanced by Anany Tripathi, building upon the original open-source RAG evaluation framework. We acknowledge the contributions of the original authors and the open-source community.
Original Attribution:
- Copyright (c) 2024 Gurpreet Kaur
- Original Author: AI Anytime
This project is licensed under the MIT License - see the LICENSE file for details.
Please read and follow our Code of Conduct to ensure a welcoming and inclusive community.
This project is licensed under the MIT License - see the LICENSE file for details.
See CHANGELOG.md for a history of changes and updates.