signafAIde
Documentation menu

Audit evidence from LangSmith traces

LangSmith shows your team what a LangChain agent did. With its OpenTelemetry integration switched on, one span processor on the tracer provider also writes a signed decision and outcome record for each tool call, on your machine.

LangSmith keeps receiving its traces: the processor reads the spans inside your process and exports nothing. How it works, the verify output and sampling: Audit log from OpenTelemetry traces.

Python #

Add the package to the project that has langsmith[otel] and LangChain. Set a tracer provider before the agent is built: LangSmith finds it and writes its spans there.

Terminal
pip install "signafaide[otel]"
Python
import os

from langchain.agents import create_agent
from langchain_core.tools import tool
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from signafaide.otel import SignafaideSpanProcessor

os.environ["LANGSMITH_OTEL_ENABLED"] = "true"  # LangSmith's own line
os.environ["LANGSMITH_TRACING"] = "true"  # LangSmith's own line

provider = TracerProvider()
provider.add_span_processor(
    SignafaideSpanProcessor(agent="support-bot")
)  # the one line
trace.set_tracer_provider(provider)

@tool
def refund(order: str, amount: float) -> str:
    """Refund an order."""
    return f"refunded {amount} on {order}"

agent = create_agent(model, tools=[refund])  # model: the model you already use
agent.invoke({"messages": [("user", "Refund order A1, 5 pounds")]})

The sample runs with langsmith 0.14 and LangChain 1.5. LangSmith's API key and endpoint stay in the environment as they are.

TypeScript #

LangSmith documents this integration for Python. In a TypeScript LangChain agent, wrap the tools with the SDK: Audit log for LangChain agents.

Before the tool ran, or after #

After. LangSmith builds its spans from finished runs, in the background, so the decision and the outcome are both written after the tool ran, when LangSmith sends its batch. signafaide verify says so:

telemetry: _pending/nokey/support-bot: 1 tool is recorded from telemetry spans: refund (decision written after the tool ran)
telemetry: _pending/nokey/support-bot: telemetry sampling follows the caller: a tool call inside a trace that arrives marked as not sampled is not recorded

For a decision written before the tool runs, add the langchain-signafaide middleware or wrap the tools: Audit log for LangChain agents. The call is then recorded once, by the wrapper.

What is recorded and what is not #

  • Recorded. Each tool call: the tool's name, its arguments and its result, the model's call id, the span's start time and its trace and span ids, signed and chained.
  • One record per call. A tool that is also wrapped with the SDK, or runs through the middleware, is recorded once, by the wrapper.
  • Sampling. A span the sampler drops never reaches the processor, so its tool call has no record: Sampling.
  • Protecting a tool needs the SDK. The processor never stops a tool. To run a tool only once its record is confirmed, use the middleware or the wrapper to protect the tool.
  • Checks stay Calibrating. With a checks file in place, the processor records the tool-call checks of each tool call it reads, always Calibrating: it notes what they find and never blocks, and it never asks the witness for a ruling.
  • Evidence, not certification. Records prove integrity and timing, not that a decision was right. The co-signature and the public timestamp come after signafaide init, which creates a free sandbox and sends only salted hashes.

Next: start free, with the SDK alone and no account. Quickstart