Capture detailed traces of every step in your LLM application, then systematically measure and improve quality through datasets, evaluators, and experiments. LangSmith combines production monitoring with human feedback to close the loop between debugging failures and deploying better versions.
LangSmith is an observability and evaluation platform built by the LangChain team, purpose-built for developers who create applications powered by large language models (LLMs). While general-purpose monitoring tools exist for traditional software, LLM applications present a unique set of challenges — non-deterministic outputs, complex multi-step reasoning chains, latency variability, and difficulty measuring correctness — that LangSmith is specifically designed to address.
This document gives you a map of the LangSmith domain: what problem it solves, what its core ideas are, and how its components fit together.
Building an LLM application is deceptively straightforward at first. You write a prompt, call an API, and get a response. But the moment you move toward production, critical questions arise:
LangSmith provides infrastructure to answer all of these questions systematically.
LangSmith is organised around three interconnected ideas:
Every time your LLM application runs, LangSmith records a trace: a structured, hierarchical log of every operation that took place. A trace captures the inputs and outputs of each step, the model used, the latency, the token counts, and any metadata you attach. Complex pipelines — chains, agents, tool calls, retrievers — are represented as nested runs inside the trace, so you can drill down from the top-level response all the way to an individual LLM call.
This is the foundation of everything else in LangSmith. You cannot debug or evaluate what you cannot see.
Once you can see what happened, the next step is deciding whether it was good. LangSmith provides an evaluation framework that lets you:
Evaluation in LangSmith is not a one-time step; it is a continuous practice woven into the development workflow.
Automated evaluation is powerful but imperfect. LangSmith supports human-in-the-loop annotation, where team members (or end users) can attach structured feedback to individual runs — thumbs up/down, ratings, corrected outputs, or free-text notes. This feedback becomes part of your data, usable for fine-tuning evaluation criteria or curating new datasets.
The main components of LangSmith and how data flows between them. Traces are the starting point; they feed datasets, playgrounds, evaluations, and the monitoring dashboard.
Every execution of your application is captured as a tree of runs. The root run represents the top-level invocation; child runs represent sub-steps (an LLM call, a tool invocation, a retrieval step). Each run stores its inputs, outputs, start/end timestamps, model name, and token counts.
A dataset is a versioned collection of examples — typically {input, output} pairs — that define what correct or representative behaviour looks like. Datasets can be created manually, imported from files, or assembled by selecting interesting runs directly from traces.
An evaluator is a function that takes a run's output (and optionally a reference output from a dataset) and returns a numeric score or categorical label. Evaluators can be:
When you run your application against a dataset under a specific configuration (prompt version, model, parameters), that constitutes an experiment. LangSmith tracks experiments over time and lets you compare metrics side by side — making it straightforward to answer "did this prompt change help?"
Runs can be routed to an annotation queue, where human reviewers apply structured labels. This is especially valuable for high-stakes domains where automated scoring is insufficient, or for bootstrapping a new evaluation rubric.
The Playground is an interactive environment inside LangSmith where you can take any traced prompt, edit it, re-run it against different models or parameters, and immediately see the new outputs — without leaving the platform.
The Hub is a registry for sharing and versioning prompts. Teams can publish, pull, and manage prompt templates from a central location, with full version history.
For production deployments, the monitoring dashboard aggregates trace data into time-series metrics: p50/p95 latency, token consumption, error rates, and feedback scores. This gives you continuous visibility into how your application is behaving at scale.
A typical LangSmith workflow looks like this:
This loop — trace → evaluate → improve → monitor — is the central rhythm of working with LangSmith.
LangSmith is primarily aimed at engineers and teams building LLM-powered products who need to move beyond ad-hoc prompt testing toward systematic quality assurance. It integrates natively with the LangChain and LangGraph libraries, but also works with any LLM framework or direct API calls through its REST API and Python/TypeScript SDKs.
LangSmith addresses the hardest operational challenges of LLM development: visibility, measurement, and continuous improvement. Its components — tracing, datasets, evaluators, experiments, annotation queues, playground, and monitoring — form a cohesive system that supports the full lifecycle of an LLM application, from early debugging through production monitoring.