ThreatFlow

ThreatFlow is the threat intelligence hub for the opensecstack ecosystem. It ingests indicators of compromise (IOCs) from multiple feed types, normalises them to STIX 2.1 format, maps them to the MITRE ATT&CK framework, and publishes enriched intelligence bundles to APIGuard, IRFlow, and NIS2 Compass. Every ingestion, correlation, and enrichment operation is governed by CITADEL MARSHAL and WORM-logged.

Overview

ThreatFlow is a Go service licensed under Apache-2.0. It listens on port 8091, backed by PostgreSQL 16 as the IOC store and Redis 7 for hot IOC lookup caching (10-minute TTL, invalidated on upsert or revoke). All intelligence objects are stored and exchanged as STIX 2.1 bundles. Deduplication is performed by hashing the indicator pattern with SHA-256 before insertion.

Key features

• Multi-source IOC ingestion — pulls from TAXII 2.1 servers, CSV feeds (abuse.ch, OTX), MISP instances (/events/restSearch with to_ids=true), STIX 2.1 bundles, and manual API uploads.
• STIX 2.1 native — all indicators are normalised to STIX 2.1 Indicator objects on ingest; bundles are available via GET /api/v1/stix/bundles/{id}.
• MITRE ATT&CK mapping — 19 embedded techniques, 16 auto-rules, and feed-provided extraction for automatic TTP classification of ingested indicators.
• Correlation engine — 5 built-in rules (duplicate, resolves-to, subdomain-of, same-network, shares-cve) cross-reference new IOCs with APIGuard scan findings, IRFlow incidents, and existing IOCs from other feeds. High-confidence matches produce STIX Relationship objects and trigger webhook notifications.
• Feed confidence scoring — each feed source has a configurable confidence_base (0–100) propagated to all IOCs ingested from that source, based on historical accuracy.
• CITADEL MARSHAL governance — every mutation (IOC_INGEST, STIX_BUNDLE_IMPORT, etc.) is gated through the MARSHAL 5-gate engine before being persisted.
• CITADEL WORM logging — all ingestion and export events are emitted to the CITADEL WORM audit chain via a bounded async queue with graceful drain on shutdown.
• JWT + API key + RBAC — HS256 bearer tokens with 4 roles; API keys are stored as SHA-256 hashes.
• Redis match cache — GET /match?type=X&value=Y lookups are cached in Redis with a 10-minute TTL; cache is invalidated on upsert or revoke.
• Rate limiting — per-IP token bucket with configurable requests-per-second and burst.

Architecture

ThreatFlow is built on Go 1.22+ with chi v5 for routing, zerolog for structured JSON logging, and Cobra + Viper for the CLI and configuration. The data pipeline has three stages:

1. Ingestion — the Feed Poller pulls from configured sources; the STIX Parser normalises indicators; the Deduplicator checks SHA-256 of each pattern; the MARSHAL gate evaluates whether ingestion should proceed; PostgreSQL stores the IOC; a WORM event (threatflow.ioc.ingested) is emitted to CITADEL.
2. Correlation — the Correlation Engine cross-references new IOCs with APIGuard scan findings, IRFlow open incidents, and IOCs from other feeds. Matches produce STIX Relationship objects; high-confidence matches trigger outbound webhooks.
3. Export — the Export Engine assembles STIX 2.1 bundles on demand or on schedule, filtered by consumer (IRFlow receives incident-relevant IOCs, APIGuard receives URL/domain IOCs). Each export is WORM-logged with the bundle hash.

Ports & endpoints

"cm"># Run locally
cd threatflow
go run ./cmd/threatflow serve

"cm"># Or with Docker
docker build -t threatflow .
docker run -p 8091:8091 threatflow

"cm"># Health check
curl http://localhost:8091/api/v1/health
"cm"># {"service":"threatflow","status":"ok"}

"cm"># Ingest an IOC
curl -X POST http://localhost:8091/api/v1/iocs \
  -H "Content-Type: application/json" \
  -d '{"type":"ipv4-addr","value":"198.51.100.42","source":"manual"}'

Integration

sinauth (identity)

ThreatFlow authenticates operators via sinauth SSO (OAuth 2.0 / OIDC, authorization code + PKCE). Access tokens are RS256-signed JWTs issued by https://auth.sin.to and validated against the sinauth JWKS endpoint at https://auth.sin.to/.well-known/jwks.json. See the sinauth integration guide for RBAC mapping and MFA configuration.

CITADEL (governance)

ThreatFlow evaluates every mutation (IOC_INGEST, STIX_BUNDLE_IMPORT, and others) through CITADEL MARSHAL before persisting anything. WORM events are emitted via a bounded async queue — the queue drains gracefully on shutdown. Configure with THREATFLOW_CITADEL_API_URL, THREATFLOW_CITADEL_KEY_ID, and THREATFLOW_CITADEL_KEY_SECRET. When THREATFLOW_CITADEL_API_URL is empty, CITADEL governance is disabled (development mode).

IRFlow (incident response)

ThreatFlow pushes STIX 2.1 IOC bundles to IRFlow for enrichment of open incidents, and receives incident artefacts from IRFlow for retroactive IOC matching. High-confidence correlation matches trigger HMAC-signed outbound webhook notifications to IRFlow. IRFlow also forwards sighting events to POST /api/v1/sightings when it observes indicator values in incident data.

APIGuard (API security)

ThreatFlow receives scan target URLs and discovered API endpoints from APIGuard for IOC extraction. It returns enrichment data — known-malicious URLs and IP indicators — as STIX 2.1 bundles for the APIGuard Unsafe Consumption of APIs (A10) and SSRF (A7) modules.

NIS2 Compass (compliance)

ThreatFlow forwards supply chain IOCs to NIS2 Compass as STIX 2.1 artefacts for the Article 21(2)(d) supply chain security control. This provides verifiable, machine-readable evidence of threat intelligence coverage.

Full documentation

The complete ThreatFlow documentation lives in the threatflow/docs folder on GitHub. Key references:

• architecture.md — system design, data flow, component interactions
• api-reference.md — complete HTTP API documentation
• stix-integration.md — STIX 2.1 object types, bundle format, and TAXII polling
• ioc-feeds.md — feed sources, ingestion pipeline, deduplication
• mitre-attack.md — TTP classification and tagging
• citadel-integration.md — MARSHAL governance and WORM logging details
• configuration.md — environment variables and config file format