signafAIde
Documentation menu

Audit evidence from Datadog LLM Observability traces

Datadog LLM Observability, which Datadog's documentation now also calls Agent Observability, takes OpenTelemetry spans in the GenAI and OpenInference conventions. Add one span processor to the tracer provider that sends them and each tool call also becomes a signed decision and outcome record on your machine.

Datadog keeps receiving the same spans: the processor reads them inside your process and exports nothing. How it works, the verify output and sampling: Audit log from OpenTelemetry traces.

This page is for the OpenTelemetry route. For an agent traced with Datadog's own SDK instead, wrap its tools with the signafAIde SDK, starting from the Quickstart.

Python #

A LangChain agent traced by the OpenLLMetry LangChain instrumentation. DATADOG_OTLP_ENDPOINT and DD_API_KEY stand for the endpoint of your Datadog site and your key.

Terminal
pip install "signafaide[otel]"
Python
from langchain.agents import create_agent
from langchain_core.tools import tool
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.langchain import LangchainInstrumentor
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from signafaide.otel import SignafaideSpanProcessor

provider = TracerProvider(
    resource=Resource.create({"service.name": "support-bot"})
)
trace.set_tracer_provider(provider)
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(  # Datadog's exporter
    endpoint=DATADOG_OTLP_ENDPOINT,
    headers={
        "dd-api-key": DD_API_KEY,
        "dd-otlp-source": "llmobs",
        "DD-LLMObs-Enabled": "true",
    },
)))
provider.add_span_processor(SignafaideSpanProcessor())  # the one line
LangchainInstrumentor().instrument()

@tool
def refund(order: str, amount: float) -> str:
    """Refund an order."""
    return f"refunded {amount} on {order}"

agent = create_agent(model, tools=[refund])  # model: the model you already use
agent.invoke({"messages": [("user", "Refund order A1, 5 pounds")]})

TypeScript #

A Vercel AI SDK agent, whose spans follow the GenAI convention.

Terminal
npm install signafaide
TypeScript
import { OpenTelemetry } from "@ai-sdk/otel";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-proto";
import { resourceFromAttributes } from "@opentelemetry/resources";
import { BatchSpanProcessor } from "@opentelemetry/sdk-trace-base";
import { NodeTracerProvider } from "@opentelemetry/sdk-trace-node";
import { generateText, registerTelemetry, stepCountIs, tool } from "ai";
import { SignafaideSpanProcessor } from "signafaide/otel";
import { z } from "zod";

const provider = new NodeTracerProvider({
  resource: resourceFromAttributes({ "service.name": "support-bot" }),
  spanProcessors: [
    new BatchSpanProcessor(new OTLPTraceExporter({ // Datadog's exporter
      url: DATADOG_OTLP_ENDPOINT,
      headers: {
        "dd-api-key": DD_API_KEY,
        "dd-otlp-source": "llmobs",
        "DD-LLMObs-Enabled": "true",
      },
    })),
    new SignafaideSpanProcessor(), // the one line
  ],
});
provider.register();
registerTelemetry(new OpenTelemetry());

await generateText({
  model, // the model you already use
  prompt: "Refund order A1, 5 pounds",
  stopWhen: stepCountIs(3),
  tools: {
    refund: tool({
      description: "Refund an order.",
      inputSchema: z.object({ order: z.string(), amount: z.number() }),
      execute: async ({ order, amount }) => `refunded ${amount} on ${order}`,
    }),
  },
});
await provider.shutdown(); // sends the buffered spans before the script exits

The samples run with the OpenTelemetry Python SDK 1.45 and opentelemetry-instrumentation-langchain 0.63, and with the OpenTelemetry JavaScript SDK 2 and AI SDK 7.

Before the tool ran, or after #

It follows the instrumentation, not Datadog.

  • Python sample: after. The OpenLLMetry LangChain instrumentation sets a span's attributes after the span has started, so the decision and the outcome are both written when the span ends, after the tool ran.
  • TypeScript sample: before. The Vercel AI SDK starts each tool span with its attributes set, so the decision is signed and chained before the tool runs.

Every record says which it was, and so does signafaide verify: Before the tool ran, or after. For a LangChain agent, the langchain-signafaide middleware writes the decision before the tool runs: Audit log for LangChain agents.

What is recorded and what is not #

  • Recorded. Each tool call whose span follows the OpenTelemetry GenAI, OpenInference or OpenLLMetry convention: the tool name and arguments, then the result or the error.
  • Not recorded. A tool call with no span, and a span the sampler dropped: Sampling.
  • Protecting a tool needs the SDK. The processor never stops a tool. To run a tool only once its record is confirmed, wrap it with the SDK.
  • Checks stay Calibrating. With a checks file in place, the processor records the tool-call checks of each tool call it reads, always Calibrating: it notes what they find and never blocks, and it never asks the witness for a ruling.
  • Evidence, not certification. Records prove integrity and timing, not that a decision was right. The co-signature and the public timestamp come after signafaide init, which creates a free sandbox and sends only salted hashes.

Next: start free, with the SDK alone and no account. Quickstart