Telemetry

Telemetry (telemetry.dev) is an observability platform built specifically for AI and LLM applications. It captures every model call, tool step, and data retrieval as OpenTelemetry traces, then attaches token counts, cost, latency, and error information to each one — so a team can debug a failure or track spend across models, providers, and environments from a single place instead of piecing it together from provider dashboards and application logs.

Cost is computed per call using live pricing across more than 4,700 models spanning OpenAI, Anthropic, Google Gemini, Mistral AI, Cohere, Meta Llama, Hugging Face, Groq, Ollama, xAI, DeepSeek, and Perplexity, among others. Integration doesn't require Telemetry's own SDK: any OpenTelemetry exporter can send traces to its OTLP/HTTP endpoint and they arrive normalized, though first-party SDKs exist for TypeScript and Python, plus drop-in integrations for LangChain and the Vercel AI SDK. The platform emphasizes vendor neutrality — "no lock-in" across whichever AI infrastructure a team uses — and the company states a first trace typically shows up within about five minutes of setup.

1. Sign up for a free account at telemetry.dev — the Free plan requires no credit card and includes 10,000 ingestion units (spans, log records, or metric data points) per month.
2. Point your application at Telemetry's OTLP/HTTP endpoint using any standard OpenTelemetry exporter, or install the first-party TypeScript or Python SDK, or use the drop-in integration for LangChain or the Vercel AI SDK, for the fastest setup.
3. Run your AI application as usual — model calls, tool steps, and retrievals are automatically captured as traces, with tokens (input, output, cached, reasoning), cost, and latency attached to each one.
4. Use the dashboard to view volume, cost, p95 latency, and error rates broken down by model, provider, environment, or user, and search or filter traces by name, status, environment, framework, or session.
5. Set per-environment input/output capture settings for privacy, and rely on automatic secret redaction at ingest (with support for custom redaction patterns) to keep sensitive data like API keys out of stored traces.
6. Upgrade to Pro or Team as ingestion volume and retention needs grow, for more projects, longer trace retention, and more team seats.

• Debugging a failed or slow AI request by inspecting its full trace — every model call, tool step, and retrieval involved — instead of piecing together logs from multiple providers.
• Tracking AI spend across providers and models, broken down by environment or user, using cost computed per call from live pricing across 4,700+ models.
• Monitoring latency and error rates (including p95 latency) for an AI application in production, to catch regressions or provider issues quickly.
• Adding observability to an existing AI stack without changing providers, using standard OpenTelemetry rather than a proprietary, lock-in SDK.
• Keeping sensitive prompt/completion content out of stored traces via per-environment capture toggles and automatic secret redaction, for teams with privacy or compliance requirements.
• Comparing cost and performance across models or providers (for example OpenAI vs. Anthropic vs. an open-weight model on Groq) to inform routing or pricing decisions.