Home › Directory › Observability & Evals
Observability & Evals
Tracing, monitoring and evaluation tools for LLM apps and agents.
JSON: /api/v1/directory.json (filter on category=observability-evals)
Langfuse
Open-source observability for LLM apps: traces, evals, prompt management and cost tracking.
Free tierAPI keyLangSmith
Tracing, evaluation and monitoring platform for LLM applications and agents.
Free tierAPI keyOpik
Open-source LLM evaluation and observability from Comet.
Free tierAPI keyOpenLLMetry
OpenTelemetry-based instrumentation for LLM applications.
FreeNo authOpenInference
Open conventions and instrumentation for AI observability based on OpenTelemetry.
FreeNo authLunary
Open-source observability and prompt management for LLM apps.
Free tierAPI keyDatadog LLM Observability
Tracing and evaluation of LLM apps inside Datadog.
PaidAPI keypromptfoo
Open-source CLI for testing, evaluating and red-teaming prompts and models.
FreeNo authDeepEval
Open-source LLM evaluation framework in the style of pytest.
FreeNo authRagas
Evaluation framework for RAG pipelines.
FreeNo authTruLens
Evaluation and tracking for LLM experiments and agents.
FreeNo authOpenAI Evals
Framework and registry for evaluating LLMs.
FreeNo authEleutherAI LM Evaluation Harness
Standard framework for few-shot evaluation of language models.
FreeNo authInspect AI
UK AI Security Institute's framework for LLM evaluations.
FreeNo authSentry
Error tracking and performance monitoring with AI SDK integrations.
Free tierAPI keyOpenTelemetry
Vendor-neutral standard for traces, metrics and logs.
FreeNo authGrafana
Dashboards and observability stack, including AI observability integrations.
Free tierAPI keyPostHog LLM Analytics
Product analytics with LLM cost and latency tracking.
Free tierAPI keyLaminar
Open-source observability and evals for AI agents.
Free tierAPI keyPatronus AI
Automated evaluation and guardrails for LLM failures.
PaidAPI key