Open-source LLM evaluation framework by Confident AI (Apache 2.0). Pytest-native, 50+ research-backed metrics, multi-modal, model-agnostic. 100M+ daily evals, 150K+ developers, 50%+ Fortune 500. G-Eval, DAG, QAG techniques + Golden Synthesizer + Conversation Simulator.
DeepEval is an open-source framework for evaluating LLMs (Large Language Models), LLM applications, AI agents, RAG systems, and prompts. Released under the Apache 2.0 licence, available at github.com/confident-ai/deepeval (250+ contributors). Built by Confident AI with the mission to make LLM evaluation as native to developers as pytest is for unit tests. It has reached the scale of 100M+ daily evaluations, 150K+ developers, and adoption at more than 50% of Fortune 500 companies.
Key characteristics: pytest-native — evaluations run as pytest tests in CI/CD or as Python scripts. Iterate locally, in your own environment, on your own criteria. Command line: `deepeval test run tests/test_agent.py`. More than 50 research-backed metrics ready out-of-the-box: Hallucination, Faithfulness, Answer Relevancy, Summarization, Toxicity, Bias, plus native conversational metrics (Role adherence, Knowledge retention, Conversation completeness) for multi-turn dialogues. Multi-modal by default — text, images, audio with one runner and the same metrics.
Flexible SOTA evaluation techniques: G-Eval (criteria-based chain-of-thought scoring via form-filling for reliable subjective evals), DAG (Directed Acyclic Graph metrics for objective multi-step conditional scoring), QAG (Question-Answer Generation for close-ended reference-grounded scoring). All in the same runner — giving developers a rich toolkit for diverse evaluations. Trace, grade, and iterate directly in your editor — no dashboards to open, no context switch. Span-level scores with reasons.
No dataset? No problem: (1) Golden Synthesizer — generating synthetic goldens from a knowledge base (chunking → extracting context → generating → evolving → filtering → applying styles), (2) Conversation Simulator — simulating full conversations for multiple user personas (pondering scenario → analyzing user profile → simulating user response). Widely used by "vibe coding agents" — Cursor, Claude Code, Codex shell out to a single CLI, read scored traces with reasons, patch the failing span, and re-run to confirm. Closes the build → eval → patch loop.
Integrations: model providers (OpenAI, Anthropic Claude, Google Gemini, Azure OpenAI, AWS Bedrock, Vertex AI, Mistral, LiteLLM, Portkey), frameworks (LangChain, LlamaIndex, CrewAI, OpenAI Agents, LangGraph, PydanticAI, Anthropic, Google ADK, AgentCore, Strands, Vercel AI SDK, OpenTelemetry), CI/CD (GitHub Actions, GitLab CI, Jenkins, CircleCI, Buildkite, Azure Pipelines). Enterprise-scale via integration with the Confident AI SaaS platform ("Vercel for DeepEval") for observability, dataset management, prompt versioning, human annotation, and production monitoring. Sister product: DeepTeam (adversarial testing / red-teaming) for detecting vulnerabilities.
