What It Is

LangSmith is LangChain's observability, evaluation, and prompt management platform. It provides tracing (every LLM call, tool use, and retrieval), evaluation suites (run test cases against your prompts), prompt versioning, and monitoring for production LLM apps. Integrates natively with LangChain but also works with any framework via the LangSmith SDK.

How It Works

Instrument your app with the LangSmith SDK (one line of code if using LangChain). Every LLM call is captured as a trace with inputs, outputs, latency, cost, and any tool calls. You can create datasets from production traces to build regression tests. Evals can be LLM-as-judge (use another LLM to grade outputs), heuristic (regex, exact match), or human-in-the-loop.