Evals: AI tools compared (October 2026)

Giskard

Paris-based platform for continuous red teaming and testing of LLM agents (Giskard Hub) with an open-source library, hosted in the EU or US.

Langfuse

Open-source platform from Berlin for prompt management, tracing and evals of LLM apps, as a cloud with an EU region or self-hosted.

Braintrust

Evaluation and observability platform for AI agents with experiments, scorers, trace analysis and the Loop assistant.

Prolific

Human data platform with verified participants for AI training, preference data, evaluations and research.

Scale AI

Training data, human feedback and evaluation for AI models, plus AI applications for enterprises and government.

Arize Phoenix

Open-source platform from Arize AI for tracing and evaluating LLM applications based on OpenTelemetry, self-hosted.

Galileo

Evaluation and observability platform for GenAI applications and agents, with Luna evaluation models and Protect guardrails.

DeepEval

Open-source framework from Confident AI for testing LLM applications like pytest, with metrics for RAG, agents and safety.

GSM8K

OpenAI's set of 8,500 grade school math word problems with worked solutions, used to test multi-step reasoning of language models.

Label Studio

Open-source labeling tool for text, images, audio, video and agent traces, with commercial cloud and enterprise editions from HumanSignal.

MMLU-Pro

Evaluation set of about 12,000 expert-reviewed multiple-choice questions with ten options each, used to compare language models.

OpenCompass

Open-source LLM evaluation platform from Shanghai with CompassRank leaderboards, the CompassHub benchmark catalog and VLMEvalKit.

OSWorld

Benchmark of 369 real computer tasks on Ubuntu, Windows and macOS that tests multimodal agents operating desktop and web apps.

Prompt flow

Open-source development tools from Microsoft to build LLM flows from prompts and code, test them on datasets and check them in CI/CD.

promptfoo

Open-source tool for testing prompts, models and RAG pipelines and for red teaming, usable in CI/CD, part of OpenAI since 2026.

Scale Labs Leaderboards

Scale AI's leaderboards with expert-built and partly private evaluations such as Humanity's Last Exam, SWE-Bench Pro and safety tests.

Vectara Hallucination Leaderboard

Public leaderboard of how often language models add hallucinations when summarizing documents, scored with Vectara's open HHEM model.

LiveBench

LLM benchmark that adds new questions regularly and scores answers against objective ground truth, without an LLM judge.

τ-bench (tau-bench)

Sierra's benchmark for AI agents that talk to users, call tools and follow policy in retail, airline, telecom and banking.

SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

ICLR 2024 paper introducing SWE-bench: 2,294 real GitHub issues from 12 Python projects as a test bed for language models.