← Vetro Attest · Free Art. 12/14 checker · Free EU AI Act tools

Audit log for LlamaIndex agents

LlamaIndex agents mix retrieval, LLM calls and tool calls. For an audit you need all three in order, per run, in a form nobody can quietly edit.

LlamaIndex.TS: the callback manager

LlamaIndex.TS routes LLM and tool calls through a CallbackManager (Settings.callbackManager). The events that matter:

• llm-start and llm-end: the messages sent and the response.
• llm-tool-call and llm-tool-result: the tool, its arguments and its output.

Vetro ships a listener for these events with no dependency on the llamaindex package, verified against the framework's own source rather than its docs. Each event is appended to a hash-chained ledger and the run is signed with ed25519; policies are checked before the tool call runs.

LlamaIndex in Python: OpenTelemetry

In Python, the practical route is an OpenTelemetry instrumentation for LlamaIndex (the OpenInference and OpenLLMetry projects both provide one). Export the traces as OTLP JSON and you have a first reading of the run without changing code.

OTLP traces are not signed at the source, so they prove less than a ledger written at the call site: they show what happened according to your trace pipeline. That is still far better than nothing, and it is where most teams start (see OpenTelemetry GenAI traces as evidence).

What retrieval adds to the audit

For RAG agents, record which documents were retrieved for each answer. When an answer is wrong or harmful, the first question is whether the model or the corpus was at fault, and only the retrieval record can say. It also matters for Article 10 data governance.

Check your own logs

Paste a Vetro ledger or an OpenTelemetry GenAI trace export into the free checker: it verifies integrity and shows which record-keeping obligations your evidence covers, in your browser. When you need something to hand to an auditor or a customer, the countersigned EU AI Act report costs 49 USD, one-off: an ed25519 counter-signature over your evidence plus a mapping to the articles, including an explicit list of what the log cannot show.

See a sample ledger verified in one click, check your own logs free or get the countersigned EU AI Act report.