How-to guides

Step-by-step guides that cover key tasks and operations in LangSmith.

Setup

See the following guides to set up your LangSmith account.

• Create an account and API key
• Set up an organization
  • Create an organization
  • Manage and navigate workspaces
  • Manage users
• Set up a workspace
  • Create a workspace
  • Manage users
  • Configure workspace settings
• Set up billing
• Update invoice email, tax id and, business information
• Set up access control (enterprise only)
  • Create a role
  • Assign a role to a user

Tracing

Get started with LangSmith's tracing features to start adding observability to your LLM applications.

• Annotate code for tracing
  • Use @traceable/traceable
  • Wrap the OpenAI API client
  • Use the RunTree API
  • Use the trace context manager (Python only)
• Toggle tracing on and off
• Log traces to specific project
  • Set the destination project statically
  • Set the destination project dynamically
• Set a sampling rate for traces
• Add metadata and tags to traces
• Implement distributed tracing
  • Distributed tracing in Python
  • Distributed tracing in TypeScript
• Access the current span within a traced function
• Log multimodal traces
• Log retriever traces
• Log custom LLM traces
  • Chat-style models
  • Specify model name
  • Stream outputs
  • Manually provide token counts
  • Instruct-style models
• Prevent logging of sensitive data in traces
  • Rule-based masking of inputs and outputs
• Export traces
  • Use filter arguments
  • Use filter query language
• Share or unshare a trace publicly
• Compare traces
• Trace generator functions
• Trace with LangChain
  • Installation
  • Quick start
  • Trace selectively
  • Log to specific project
  • Add metadata and tags to traces
  • Customize run name
  • Access run (span) ID for LangChain invocations
  • Ensure all traces are submitted before exiting
  • Trace without setting environment variables
  • Distributed tracing with LangChain (Python)
  • Interoperability between LangChain (Python) and LangSmith SDK
  • Interoperability between LangChain.JS and LangSmith SDK
• Trace with LangGraph
  • Interoperability between LangChain and LangGraph
  • Interoperability between @traceable/traceable and LangGraph
• Trace with Instructor (Python only)
• Trace with the Vercel AI SDK (JS only)
• Trace without setting environment variables
• Trace using the LangSmith REST API
• Calculate token-based costs for traces

Datasets

Manage datasets in LangSmith to evaluate and improve your LLM applications.

• Manage datasets in the application
  • Create a new dataset and add examples manually
  • Dataset schema validation
  • Add inputs and outputs from traces to datasets
  • Upload a CSV file to create a dataset
  • Generate synthetic examples
  • Export a dataset
  • Create and manage dataset splits
• Manage datasets programmatically
  • Create a dataset from list of values
  • Create a dataset from traces
  • Create a dataset from a CSV file
  • Create a dataset from a pandas DataFrame
  • Fetch datasets
  • Fetch examples
  • Update examples
  • Bulk update examples
• Version datasets
  • Create a new version of a dataset
  • Tag a version
• Share or unshare a dataset publicly

Evaluation

Evaluate your LLM applications to measure their performance over time.

• Evaluate an LLM application
  • Run an evaluation
  • Use custom evaluators
  • Evaluate on a particular version of a dataset
  • Evaluate on a subset of a dataset
  • Evaluate on a dataset split
  • Evaluate on a dataset with repetitions
  • Use a summary evaluator
  • Evaluate a LangChain runnable
  • Return multiple scores
• Bind an evaluator to a dataset in the UI
• Run an evaluation from the prompt playground
• Evaluate on intermediate steps
• Use LangChain off-the-shelf evaluators (Python only)
  • Use question and answer (correctness) evaluators
  • Use criteria evaluators
  • Use labeled criteria evaluators
  • Use string or embedding distance metrics
  • Use a custom LLM in off-the-shelf evaluators
  • Handle multiple input or output fields
• Compare experiment results
  • Open the comparison view
  • View regressions and improvements
  • Filter on regressions or improvements
  • Update baseline experiment
  • Select feedback key
  • Open a trace
  • Expand detailed view
  • Update display settings
• Evaluate an existing experiment
• Unit test LLM applications (Python only)
• Run pairwise evaluations
  • Use the evaluate_comparative function
  • Configure inputs and outputs for pairwise evaluators
  • Compare two experiments with LLM-based pairwise evaluators
  • View pairwise experiments
• Audit evaluator scores
  • In the comparison view
  • In the runs table
  • In the SDK
• Create few-shot evaluators
  • Create your evaluator
  • Make corrections
  • View your corrections dataset
• Fetch performance metrics for an experiment
• Run evals using the API only
  • Create a dataset
  • Run a single experiment
  • Run a pairwise experiment
• Upload experiments run outside of LangSmith with the REST API
  • Request body schema
  • Considerations
  • Example request
  • View the experiment in the UI

Human feedback

Collect human feedback to improve your LLM applications.

• Capture user feedback from your application to traces
• Set up a new feedback criteria
• Annotate traces inline
• Use annotation queues
  • Create an annotation queue
  • Assign runs to an annotation queue
  • Review runs in an annotation queue

Monitoring and automations

Leverage LangSmith's powerful monitoring and automations features to make sense of your production data.

• Filter traces in the application
  • Create a filter
  • Filter for intermediate runs (spans)
  • Advanced: filter for intermediate runs (spans) on properties of the root
  • Advanced: filter for runs (spans) whose child runs have some attribute
  • Filter based on inputs and outputs
  • Filter based on input / output key-value pairs
  • Copy the filter
  • Manually specify a raw query in LangSmith query language
  • Use an AI Query to auto-generate a query
• Use monitoring charts
  • Change the time period
  • Slice data by metadata or tag
  • Drill down into specific subsets
• Set up automation rules
  • Create a rule
  • View logs for your automations
• Set up online evaluations
  • Configure online evaluations
  • Set API keys
• Set up webhook notifications for rules
  • Webhook payload
  • Example with Modal
• Set up threads
  • Group traces into threads
  • View threads

Prompts

Organize and manage prompts in LangSmith to streamline your LLM development workflow.

• Create a prompt
  • Compose your prompt
  • Save your prompt
  • View your prompts
  • Add metadata
• Update a prompt
  • Update metadata
  • Update the prompt content
  • Version a prompt
• Manage prompts programmatically
  • Configure environment variables
  • Push a prompt
  • Pull a prompt
  • Use a prompt without LangChain
  • List, delete, and like prompts
• LangChain Hub

Playground

Quickly iterate on prompts and models in the LangSmith Playground.

• Use custom TLS certificates
• Use a custom model
• Save settings configuration