Skip to content
Docs/CITADEL Governance/Evidence & Audit
Edit on GitHub

Evidence & Auditor Walkthrough

The WORM chain is evidence. Making it admissible evidence — for a regulator, a court, or an internal audit — requires a chain of custody that documents who had access to it, when, how integrity was verified at each handover, and what proof accompanies the data when it leaves CITADEL. This page explains the chain-of-custody model, the export bundle format, and provides a step-by-step auditor walkthrough for verifying a chain, validating anchors, and exporting a custody manifest.

Overview

Every WORM entry is cryptographically linked to its predecessor and content-addressed via TripleHash (SHA-256 + SHA-512 + BLAKE3 in parallel). Every 100 entries, CITADEL writes a chain anchor — an Ed25519 signature over the current chain state — that elevates an internally consistent chain into an externally attestable one. When evidence leaves CITADEL, it travels with those anchors, the corresponding public keys, and a custody manifest that records who authorised the export, who received it, and a bundle-level hash the receiver can recompute to detect transit tampering.

What CITADEL records automatically

Every WORM entry carries the following fields. Each field serves a specific custody purpose that an auditor can verify independently:

FieldTypeCustody purpose
idUUIDUnique reference that cannot be re-used; identifies the exact evidence item
sequence_numint64Position in the chain; gaps are detectable — a missing sequence number is a finding
ts_utctimestampCITADEL's server-authoritative time — not the caller's — establishing when the entry was committed
source + event_typestringProvenance — which subsystem produced this evidence and what class of event it records
project_idUUIDScope — which investigation this evidence belongs to, enabling targeted export
payloadBYTEAThe evidence itself, byte-for-byte, as submitted by the originating platform
triple_hashhex stringContent-addressable digest — 128-byte composite of SHA-256 + SHA-512 + BLAKE3; proof the bytes have not changed
chain_hashhex stringSHA-256 of the previous chain_hash concatenated with this entry's payload; proves the entry exists at this sequence position
prev_hashhex stringThe chain_hash of the immediately preceding entry; proves this entry came after the prior one
created_attimestampDB-level insert timestamp; must match ts_utc — divergence is an integrity flag

Chain anchors

Every 100 entries, CITADEL writes a chain anchor: an Ed25519 signature over (sequence_num, ts_utc, chain_hash) using the master CITADEL signing key. Anchors are what elevate an internally consistent chain into an externally attestable one — they bind the chain state to a private key only CITADEL holds, enabling external notarisation.

TripleHash and chain structure
Bash
"cm"># Per-entry integrity (128-byte composite digest)
triple_hash = SHA-256(payload) || SHA-512(payload) || BLAKE3(payload)

"cm"># Chain linkage (forward hash)
chain_hash[n] = SHA-256(chain_hash[n-1] || payload_bytes[n])

"cm"># Ed25519 anchor — written every 100 entries
anchor_payload = sequence_num + "|" + ts_utc + "|" + chain_hash
anchor_digest  = SHA-512(anchor_payload)
anchor_sig     = Ed25519_Sign(master_key, anchor_digest)

All three hash algorithms in TripleHash must be broken simultaneously to forge an entry. The Ed25519 anchor every 100 entries binds the chain to the CITADEL signing key. After any disaster recovery — even from a cold backup — the restored log can be verified against the original anchors; tampered recovery is detectable.

Evidence export bundle

When evidence leaves CITADEL — to an auditor, a court, or a regulator — the export bundle must contain all five components for the evidence to be attestable:

  1. WORM entries (worm_entries.jsonl) — the evidence itself, one JSON object per line, sorted by sequence_num ASC.
  2. Chain anchors (chain_anchors.jsonl) — every anchor covering the exported range, one JSON object per line.
  3. Public keys (pubkeys.yaml) — all Ed25519 public keys that signed anchors covering this range, with id, pubkey_hex, issued, revoked, and replaced_by.
  4. Bundle hash (bundle.sha256) — SHA-256 over the concatenated bytes of worm_entries.jsonl, chain_anchors.jsonl, and pubkeys.yaml. The receiver recomputes this to detect transit tampering.
  5. Custody manifest (manifest.yaml) — who authorised the export, who received it, when, and a reference to the bundle hash. This is the operator's responsibility; CITADEL does not generate it automatically.
Note: Do not hand out raw database dumps in lieu of a proper bundle. The manifest and anchor public keys are what make the evidence attestable. A raw dump contains the bytes but not the cryptographic proof.

Custody manifest format

The custody manifest is a YAML document produced by the operator and included in every export bundle as manifest.yaml. It creates the meta-chain-of-custody: the manifest references the bundle hash, and the bundle contains the WORM entry that authorised the export. Any future dispute about whether an export was authorised can be resolved against the chain itself.

manifest.yaml — recommended format
YAML
bundle_id: "citadel-export-2026-04-19-0001"
produced_at: "2026-04-19T14:32:00Z"
produced_by: user_id: 42
  role: admin
  jwt_sub: "alice@example.com"
authorised_by: user_id: 99
  role: admin
  jwt_sub: "bob@example.com"
received_by: name:               "EU DG-CONNECT audit team"
  contact: "audit-lead@dg-connect.europa.eu"
  received_at: "2026-04-19T14:35:00Z"

evidence: time_range:
    from: "2026-01-01T00:00:00Z"
    to: "2026-03-31T23:59:59Z"
  entries_count: 12847
  first_sequence_num: 10201
  last_sequence_num: 23047
  anchors_count: 129

integrity: bundle_sha256:      "<hex>"
  anchor_pubkeys: - id:             "citadel-anchor-2026Q1"
      pubkey_hex: "<32-byte Ed25519 pubkey hex>"
      issued: "2026-01-01"
      revoked: null

Export authorisation

Exporting evidence is itself a governance-relevant action. It passes through the full MARSHAL pipeline:

  • action.type is DATA_EXPORT — picked up by AUGUR rule_03, which requires a non-empty incident_id (the reason for the export). Attempting a DATA_EXPORT without an associated incident results in an immediate HARD_STOP.
  • Gate 3 (NDS) enforces separation of duties — two distinct identities must sign off: the operator requesting the export and a verifier from a different role group.
  • Gate 5 (WORM) records the export action itself, including operator, verifier, reason, and the range exported, in the immutable audit chain.

The WORM entry for the export is included in the bundle itself, so the bundle is self-referencing: the custody manifest references the bundle hash, and the bundle contains cryptographic proof that the export was properly authorised.

Retention policy

Per NIS2 Directive Article 21(2)(b), incident-related evidence must be retained long enough to support authority review. CITADEL's default policy is indefinite — WORM entries are never deleted.

Retention windowWhen to apply
7 yearsDefault for regulated entities (financial services, healthcare, critical infrastructure)
10 yearsWhere national law requires it (defence contractors, classified handling)
30 daysDevelopment and staging environments only — must be a completely separate CITADEL instance

There is no TTL on WORM entries in v1.0.0. Archival tiering (moving entries older than one year to cold storage) is planned for v2.0; today, storage cost scales linearly with time.

Key rotation and revocation

Never delete an anchor public key, even after an Ed25519 key is rotated. Every anchor signed with the old key remains valid for the duration of the evidence retention period. The key's custody record shows:

  • issued — date the key came into use.
  • revoked — date a replacement was issued. The old key no longer signs new anchors, but all existing signatures still verify against it.
  • replaced_by — the new pubkey_id.

Export bundles for a given time range automatically include all public keys that signed anchors touching that range, including rotated keys. An auditor receiving a bundle for a range that spans a key rotation will see both the old and new key in pubkeys.yaml.

Auditor walkthrough

The following steps are written for external or internal auditors receiving a CITADEL evidence bundle. Complete all six steps in order — any failure at any step means the bundle should be rejected, either for re-export (if transit corruption is suspected) or for escalation (if tampering is suspected). A legitimate CITADEL deployment never produces a failing bundle.

Step 1 — Bundle integrity

Recompute the bundle hash and compare it against bundle.sha256:

Step 1 — recompute bundle_sha256
Bash
"cm"># Recompute the bundle hash (concatenated bytes of the three data files)
cat worm_entries.jsonl chain_anchors.jsonl pubkeys.yaml | sha256sum

"cm"># Compare the output against bundle.sha256
diff <(cat worm_entries.jsonl chain_anchors.jsonl pubkeys.yaml | sha256sum | awk '{print $1}')      <(cat bundle.sha256 | tr -d '[:space:]')

"cm"># Clean result: empty diff — no output, exit 0
"cm"># Mismatch: the bundle was modified in transit — reject and request re-export

Step 2 — Pubkey registry

For each public key in pubkeys.yaml, verify:

  • pubkey_hex is a valid Ed25519 public key — exactly 32 bytes (64 hexadecimal characters).
  • issued and revoked dates are plausible: not in the future, no unexplained gaps between consecutive keys.
  • If revoked is set, replaced_by must point at another key in the bundle whose issued date is on or after the revocation date.
  • No anchor in chain_anchors.jsonl references a key outside that key's [issued, revoked) validity window.

Step 3 — Anchor signatures

For each anchor in chain_anchors.jsonl, verify the Ed25519 signature:

Step 3 — verify anchor signatures (verification script)
Bash
"cm"># The verification script (provided with the bundle) runs this for all anchors:
"cm">#
"cm">#   payload = sequence_num + "|" + ts_utc + "|" + chain_hash
"cm">#   digest  = SHA-512(payload)
"cm">#   ed25519.Verify(pubkey_for(anchor.pubkey_id), digest, anchor.signature) == true
"cm">#
"cm"># Run the bundled script:
./citadel-verify anchors --bundle ./export-bundle/

"cm"># Expected output on success:
"cm"># [OK] 129/129 anchors verified

Every anchor must verify. A single failing anchor voids the integrity claim for the range that anchor covers. Reject the bundle on any anchor failure.

Step 4 — Linear chain walk

Iterate entries in worm_entries.jsonl sorted by sequence_num ascending. For each entry:

  1. Recompute TripleHash from the raw payload bytes and confirm it matches the stored triple_hash:
    TripleHash recomputation
    Bash
    "cm"># For each entry:
    expected_triple = sha256(payload) || sha512(payload) || blake3(payload)
    assert expected_triple == entry.triple_hash
  2. Recompute chain_hash from the previous entry's chain_hash and this entry's payload:
    chain_hash recomputation
    Bash
    "cm"># For each entry:
    expected_chain = sha256(hex_decode(entry.prev_hash) || entry.payload_bytes)
    assert expected_chain == entry.chain_hash
  3. Check continuity — each entry's prev_hash must equal the preceding entry's chain_hash:
    Continuity check
    Bash
    "cm"># For each entry[i] where i > 0:
    assert entry[i].prev_hash == entry[i-1].chain_hash
    
    "cm"># Any mismatch indicates insertion, deletion, or re-ordering — reject the bundle
Step 4 — full chain walk via verification script
Bash
./citadel-verify chain-walk --bundle ./export-bundle/ --verbose

"cm"># Expected output on success:
"cm"># [OK] 12847 entries walked
"cm"># [OK] TripleHash verified: 12847/12847
"cm"># [OK] chain_hash verified: 12847/12847
"cm"># [OK] Continuity: no gaps or sequence breaks

Step 5 — Anchor-over-chain-walk

Confirm that the first and last entries in the bundle are covered by anchors that verified in Step 3. An anchor covers entries [anchor.prev_sequence_num + 1, anchor.sequence_num]. The chain_hash at anchor.sequence_num must match the value derived from the chain walk in Step 4.

Any chain_hash from the walk that does not appear in the anchor set means anchor coverage is incomplete. The bundle is valid as far as the walk proves, but the uncovered range has weaker evidence than anchor-sealed range — flag this as a finding and ask the deployer to re-export with complete anchor coverage.

Step 5 — anchor coverage check
Bash
./citadel-verify anchor-coverage --bundle ./export-bundle/

"cm"># Expected output on success:
"cm"># [OK] First entry sequence 10201 covered by anchor at sequence 10300
"cm"># [OK] Last entry sequence 23047 covered by anchor at sequence 23100
"cm"># [OK] No gaps in anchor coverage

Step 6 — Time-range sanity

Confirm the bundle's claimed range (from manifest.yaml's evidence.time_range) is consistent with the actual entries:

  • evidence.first_sequence_num ≤ the first entry's sequence_num in worm_entries.jsonl.
  • evidence.last_sequence_num ≥ the last entry's sequence_num.
  • evidence.time_range.from ≤ the first entry's ts_utc.
  • evidence.time_range.to ≥ the last entry's ts_utc.
Step 6 — time range sanity check
Bash
./citadel-verify time-range --bundle ./export-bundle/

"cm"># Expected output on success:
"cm"># [OK] Manifest range 2026-01-01T00:00:00Z → 2026-03-31T23:59:59Z
"cm"># [OK] First entry: seq=10201, ts=2026-01-01T00:01:14Z  (within range)
"cm"># [OK] Last  entry: seq=23047, ts=2026-03-31T23:58:02Z  (within range)
"cm"># [OK] Entry count matches manifest: 12847

Claims a green bundle supports

A bundle that passes all six steps supports the following claims in an audit or legal context:

  1. These specific payloads existed at their timestamps. The chain_hash at each sequence_num encodes the full history up to that point; re-deriving it from the payloads and the genesis hash yields the same value.
  2. The timestamps are CITADEL-authoritative. ts_utc is set by CITADEL's server clock at append time — not by the calling platform.
  3. The payloads have not been altered since committal. TripleHash is content-addressable; any mutation would fail the chain walk at Step 4.
  4. The chain was not re-ordered or spliced. Continuity checks (Step 4.3) detect insertion, deletion, and re-ordering at any position in the range.
  5. The range is cryptographically sealed. The Ed25519 anchor signatures (Step 3) bind the chain_hashes to a private key only CITADEL holds; an attacker would need that key to forge anchors over a modified chain.

Claims a bundle does not support

Be explicit with stakeholders about what the bundle cannot prove:

  • The payloads are truthful. CITADEL proves a payload was submitted and committed, not that its business-semantic content is accurate. If a platform submitted incorrect data, the WORM entry faithfully records the inaccuracy.
  • The actor was who they claimed to be. CITADEL uses whatever identity the calling platform's IdP asserted. A compromised credential produces genuine-looking WORM entries.
  • Actions were carried out. The WORM records the MARSHAL decision, not the execution outcome. Evidence for the effect of a decision lives in the downstream system's own records — you need both layers for a complete picture.

Common findings

FindingLikely causeAction
Gaps in sequence_num (e.g. 1234 → 1236, 1235 missing)Gaps should not occur — the WORM table serialises appends and assigns sequence numbers monotonically with LOCK TABLERaise as a finding; the export may have missed entries, or the chain has been tampered with
Two anchors with identical sequence_numCannot happen legitimately — each anchor covers a unique rangeDuplicate entries in the anchor table; this is itself an incident — escalate immediately
Payload is not UTF-8 / not JSONpayload is BYTEA — legally any bytes; in practice always JSON, but the format does not enforce thisChain integrity is still valid; the payload is opaque to this analysis but the custody proof holds
Bundle SHA-256 mismatch on receiptTransit corruption or tamperingReject and request re-export; if the deployer cannot produce a matching bundle, treat as a security incident

Do's and don'ts

Do not re-sign anchors. If a deployer or counterparty requests a "fresh signature" over historical entries — decline. The whole point of anchors is that the signature is fixed at the time the entry range was sealed. A request to re-sign historical anchors is a red flag and should be treated as a potential incident.
  • Do verify the bundle hash before beginning any other step — transit corruption is the most common non-adversarial failure mode.
  • Do include the WORM entry for the export action itself in the bundle so the custody chain is self-referencing.
  • Do retain rotated Ed25519 public keys indefinitely alongside the anchors they signed.
  • Do not hand out raw database dumps. A dump has the bytes but not the cryptographic proof chain.
  • Do not modify or redact a payload in place. Redaction is done via a compensating WORM entry that references the old entry and marks it superseded — the original entry remains in the chain unchanged.
  • Do not attempt a DATA_EXPORT without an associated incident ID. AUGUR rule_03 will force a HARD_STOP and create a P1 incident automatically.

SIN — Security Intelligence Network — Open-source cybersecurity & compliance for the EU Digital Decade.

Apache-2.0 & AGPL-3.0 · GitHub · Contribute