Skip to main content

Tools

TruLens

3,665

Evaluation and Tracking for LLM Experiments and AI Agents

Detection Post-generation Graduated HITL AI-based Experimental Activity band E Environment
Activity band details

Activity band A to F: a bin of a score that adds GitHub stars (log scale), a push to the upstream repository in the 12 months before `as_of`, the number of declared integration surfaces and, where the catalogue records them, citing papers and benchmark evaluations. It measures popularity and activity. It is not a level of evidence: no band means a paper cites the tool or that it was measured. Recency is judged as of 2026-11-22.

Computed band: E (raw score 3.93). Shown band: E .

Facets: documented_in=0evaluated_on=0integration_surfaces=0.5paper_citations=0recent_push=2stars=1.43

Source of truth: pipeline/maturity/score.py

Description

TruLens provides open-source tools for evaluating and tracking LLM experiments with programmable feedback functions. It enables developers to instrument, evaluate, and iterate on LLM applications using feedback functions for quality assessment, including coverage of context relevance, groundedness, and answer relevance.

Why it's included

Occupies detection × post_generation × graduated_hitl with the RAG Triad — context relevance, groundedness, and answer relevance — as programmable feedback functions over instrumented traces. Distinguished from general LLM-observability platforms (arize-phoenix) by centering RAG-specific groundedness metrics and from deepeval by exposing those metrics as live feedback functions rather than Pytest assertions. Useful as the canonical RAG-evaluation exemplar in the post-generation detection cell.

Addresses failure modes

These tags are the catalog's claim about what TruLens is built to catch or prevent. A tag is not a test result. SAICA-Bench has not measured TruLens, so none of these tags is tested here.

  • Incomplete execution — Agent claims completion while subtasks are skipped, stubbed, or silently dropped.

Records evidence of

TruLens can record evidence of these failures (traces, or checks you configure yourself), but it does not detect or prevent them on its own. They are not counted as failure modes it addresses.

View raw YAML on GitHub →