signafAIde
Documentation menu

Audit log from OpenTelemetry traces

An agent that is already traced with OpenTelemetry gets signed records with no tool wrapping. Add one span processor where tracing is set up, and every tool-call span becomes a decision and an outcome record. Python and TypeScript.

The processor runs inside your process and exports nothing. Its records are signed, chained and kept in .signafaide/ exactly as a wrapped tool's are. What recording covers and where it stops: Tool coverage.

Python #

Terminal
pip install "signafaide[otel]"
Python
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from pydantic_ai import Agent
from pydantic_ai.models.instrumented import InstrumentationSettings
from signafaide.otel import SignafaideSpanProcessor

provider = TracerProvider(
    resource=Resource.create({"service.name": "support-bot"})
)
provider.add_span_processor(SignafaideSpanProcessor())  # the one line

Agent.instrument_all(InstrumentationSettings(tracer_provider=provider))
agent = Agent(model)  # model: the model you already use

@agent.tool_plain
def refund(order: str, amount: float) -> str:
    """Refund an order."""
    return f"refunded {amount} on {order}"

agent.run_sync("Refund order A1, 5 pounds")

The tracer provider is the one you already have: the only new line is add_span_processor. To record every call, give the provider an always-on sampler, TracerProvider(sampler=ALWAYS_ON) or sampler: new AlwaysOnSampler(): no span is dropped, and no sampling note is written. The example traces a Pydantic AI agent; any instrumentation that follows one of the three conventions below is recorded the same way.

TypeScript #

TypeScript
import { OpenTelemetry } from "@ai-sdk/otel";
import { NodeTracerProvider } from "@opentelemetry/sdk-trace-node";
import { generateText, registerTelemetry, stepCountIs, tool } from "ai";
import { SignafaideSpanProcessor } from "signafaide/otel";
import { z } from "zod";

const provider = new NodeTracerProvider({
  spanProcessors: [new SignafaideSpanProcessor({ agent: "support-bot" })], // the one line
});
provider.register();
registerTelemetry(new OpenTelemetry());

await generateText({
  model, // the model you already use
  prompt: "Refund order A1, 5 pounds",
  stopWhen: stepCountIs(3),
  tools: {
    refund: tool({
      description: "Refund an order.",
      inputSchema: z.object({ order: z.string(), amount: z.number() }),
      execute: async ({ order, amount }) => `refunded ${amount} on ${order}`,
    }),
  },
});
await provider.shutdown(); // sends the buffered spans before the script exits

signafaide/otel is part of the signafaide package and uses @opentelemetry/api, which every OpenTelemetry set-up already has. The example traces the Vercel AI SDK through @ai-sdk/otel.

  • agent names the record chain. Without it the name is the tracer resource's service.name, and without that the project folder's name.
  • store_content=True (storeContent: true) also sends inputs and outputs, encrypted. declare_tools=False (declareTools: false) stops the processor declaring the tools it sees.

A verify run #

Run the agent once, then signafaide verify (npx signafaide verify in a TypeScript project). This is a local run of the Python example, in which the agent called refund once: two records for the tool list, one sampling note, one telemetry note, one decision and one outcome. The tracer provider has OpenTelemetry's default sampler.

$ signafaide verify
local state: /home/dev/support-agent/.signafaide
 1. Envelope schema              PASS         6 records
 2. Record hashes                NOT CHECKED  0 records; 6 not yet shipped (no receipt); nothing co-signed yet
 3. Client signatures            PASS         6 records
 4. Agent chains                 PASS         6 records
 5. Merkle proofs                NOT CHECKED  0 records; nothing co-signed yet
 6. Batch hashes and signatures  NOT CHECKED  0 batches; nothing co-signed yet
 7. Batch chain                  NOT CHECKED  0 batches; nothing co-signed yet
 8. Signed heads                 NOT CHECKED  0 heads; nothing co-signed yet
 9. drand rounds                 NOT CHECKED  0 rounds
10. Anchors                      NOT CHECKED  0 anchors; local state holds no anchors
11. OpenTimestamps               NOT CHECKED  0 proofs
12. Content hashes               PASS         6 records
coverage: _pending/nokey/support-bot: the 1 declared tool is recorded
telemetry: _pending/nokey/support-bot: 1 tool is recorded from telemetry spans: refund (decision written before the tool ran)
telemetry: _pending/nokey/support-bot: telemetry sampling follows the caller: a tool call inside a trace that arrives marked as not sampled is not recorded
checked here: signatures, chain links and content hashes of all 6 records, using this machine's keys
co-signature, public timestamp: not yet. Run `signafaide init` to send hashes, or self-host (SIGNAFAIDE_URL)
PASS (local checks only): 4 of 12 checks passed, 8 not checked (nothing co-signed yet)

The two telemetry: lines are notes, never failures. signafaide status shows the same two lines.

Which spans are recorded #

A span is a tool call when it follows one of three conventions:

  • OpenTelemetry GenAI: gen_ai.operation.name is execute_tool. Pydantic AI, the Vercel AI SDK (@ai-sdk/otel) and the OpenLLMetry instrumentations write these.
  • OpenInference: openinference.span.kind is TOOL. The OpenInference instrumentors write these.
  • OpenLLMetry: traceloop.span.kind is tool.

The decision holds the tool's name and the arguments the span carries; the outcome holds the result the span carries, or the error when the span's status is an error. What the instrumentation leaves out of the span is not in the record.

Before the tool ran, or after #

A wrapped tool's decision is always written before the tool runs. A span can only be recorded when it says it is a tool call, and instrumentations differ in when they say so.

  • At span start: before. Pydantic AI and the Vercel AI SDK start the span with its attributes set. The decision is signed and chained before the tool runs.
  • At span end: after. The LangChain instrumentations of OpenInference and OpenLLMetry set the attributes after the span has started. The decision and the outcome are both written when the span ends, after the tool ran.

Every record says which it was, in signed data: meta.telemetry.decision_written in its content is before or after, and a signed note in the chain says the same for a verifier that holds only hashes. That is the first telemetry: line above.

Sampling #

A span the sampler drops never reaches the processor, so its tool call has no record. The processor writes one signed note per process that says what the sampler can leave out, and verify and status repeat it. There are three cases:

  • Always-on: no span is dropped and there is no note.
  • Parent-based with an always-on root, OpenTelemetry's default: every trace your program starts is kept, and a trace that arrives from another service follows that caller's decision. The note is the second telemetry: line in the run above.
  • A sampler that can drop your own traces (ratio-based, always-off, parent-based with another root, a custom one): records come from sampled telemetry: some tool calls may not be recorded.

In TypeScript a span does not name its sampler. The processor reads it from the registered tracer provider (provider.register(), or the Node SDK); for a provider that is not registered, pass the provider's sampler as new SignafaideSpanProcessor({ sampler }). With neither, the processor cannot tell and writes no note.

Protect a tool #

The processor cannot protect a tool: by the time a span reaches it the tool is already running. With signafaide.checks.json in place it runs the tool-call checks on the arguments and results its spans carry and writes what they found into the records, as notes only, and no call asks signafAIde for a ruling. To have signafAIde allow or block a tool before it runs, protect that tool with the SDK as well (protect(fn), or "protect": true for a wrapped framework tool); the wrapper's records then stand and the span adds none.

What is recorded and what is not #

  • Recorded. Every tool call that produces a span in one of the three conventions and passes the sampler: the tool name and the arguments, then the result or the error. Inputs and outputs stay in .signafaide/ on your machine. Nothing is sent until you run signafaide init, and then only salted hashes. The processor forwards no span data anywhere.
  • Not recorded. A tool call with no span, a span the sampler dropped, and a span in another convention leave no record. With the default sampler that is a call inside a trace that arrives marked as not sampled. The tools declared are the ones seen in spans so far, not the agent's full list: give that to declare_tools or declareTools and signafaide verify names the tools that are not recorded.
  • Protecting a tool needs the SDK. "Do not run the tool unless it is recorded" is a feature of wrapping the tool with the SDK, not of the telemetry processor. The processor never raises into your application: when a record cannot be written the tool still runs, and one warning is logged.
  • One record per call. A tool that is also wrapped with the SDK, or runs through the LangChain middleware, is recorded once, by the wrapper. A tool that an MCP gateway on the same machine records for the same agent is left to the gateway.
  • Evidence, not certification. Records prove integrity and timing, not that a decision was right. The eight checks not run above need a witness: the co-signature and the public timestamp come after signafaide init.

Next: what a full tool list adds, and where recording stops. Tool coverage