Issue Research Pipeline

The issue research pipeline discovers modern-slavery and human-rights allegations for supplier organisations and persists them as structured, provenance-tagged records. This page is the landing page for the subsystem — it covers what the pipeline produces, the two engines behind it, the three-layer architecture, the job lifecycle, and how the pipeline was built.

What it produces

Each discovered allegation is persisted as an Issue node attached to an Organisation via a HAS_ISSUE relationship, tagged with a MODERN_SLAVERY satellite mode (schemas/issue.graphql). See Neo4j Infrastructure for the graph database these nodes live in. Every issue carries:

• involvement — the OECD/UNGP classification of the organisation's relationship to the impact (see Involvement Definitions)
• sentiment — a severity score
• Provenance — source (which pipeline/list found it), tier (genuine modern slavery vs broader worker/human-rights issues), and slaveryPractice (the specific practice type, where known)

Issues surface on the organisation's scorecard as a live issue count, in the Analyst and Spotlight reported-issue lists, and in generated reports. The count is never a stored scalar — it is always computed live from issuesConnection.totalCount (IssueRepository.countByOrganisationAndModule).

Two engines, one flag

The pipeline has two independent engines, selected per run by a single kill-switch flag:

The flag defaults OFF in every environment, and OFF is byte-for-byte identical to the pre-harness legacy behaviour — no new steps, no realtime publish, no schema changes to the write path. The legacy engine remains the production default while the flag is OFF; it is a kill-switch fallback, not a deprecated code path.

Three-layer architecture

The native pipeline is organised into three layers, composed at a single root.

Engine kernel (application/ai/issues-core/) is a domain-agnostic plan → map → reduce orchestrator (run-research.ts). It knows nothing about modern slavery specifically: it plans coverage-cell search queries, fans out a map worker per cell (search + classify), optionally runs an adaptive-depth analyst round, and reduces the results into consolidated issues. Both a batch (runIssuesResearch) and a streaming (streamIssuesResearch) entry point are exposed, plus planIssueCells so the plan can be computed ahead of the phase that consumes it.

Modern-slavery pack (application/ai/issues-research/) plus fast-issues sources (application/ai/pipelines/fast-issues/) supply the domain knowledge. The MS pack binds the kernel's four pluggable stages to modern-slavery-specific prompts, a skill file, and a coverage skeleton. The fast-issues sources are the Phase-1 screening pipelines: one profile per sanctions list (OFAC, UN, UK, UFLPA, BIS, EU, DFAT) plus a dedicated BHRRC (Business & Human Rights Resource Centre) map-reduce pipeline over RSS/corpus articles.

Orchestrator + additive persistence (application/use-cases/issue/) ties the two together for production. run-issues-research.ts is the composition root: it binds the MS pack's stages to the kernel and exports the batch/streaming entry points the two-phase use case and evals call. generate-organisation-issues-two-phase.ts drives Phase 1 (BHRRC + sanctions, in parallel) and Phase 2 (deep research over the planned cells), persisting each source's results through an additive-merge reconciliation rather than overwriting prior issues.

The architecture at a glance

Job lifecycle

The Inngest function organisation-generate-issues (application/jobs/organisation/generate-issues.ts) runs the whole pipeline for one organisation:

1. Freshness self-gate — reads the organisation and skips the run entirely (a cheap no-op) when issues are already fresh for the requested regime, per enrichmentNeeds.
2. markIssuesPending — sets issuesStatus to PENDING before any research starts (idempotent — the enrich orchestrator may already have set it).
3. Route — resolves feature/issues-research-harness once and memoises the decision in an Inngest step, so a mid-run flag flip can't route a partially-complete run onto the other engine.
4. Phases — the legacy engine runs as a single step; the native engine runs Phase 1 and the Phase-2 planner in parallel, then Phase 2.
5. markIssuesCompleted (sets issuesCheckedAt) on success, or markIssuesFailed on failure. Partial-degradation errors (a bounded, transient failure inside an otherwise-healthy run) are excluded from the FAILED decision — only a total wipeout or a genuine crash marks the run FAILED.

The function has retries: 1 and runs at interactive priority (INTERACTIVE_PRIORITY_RUN), so both background enrichment fan-out and an interactive spotlight/analyst trigger share one function and pool without background runs starving interactive ones.

How it was built

The native pipeline shipped as an arc of six PRs, each independently mergeable. #1535, #1536, and #1572 landed the engine kernel inert behind the harness flag; #1576 activated the pipeline (schema, two-phase orchestrator, and Inngest routing, still default OFF). #1585 and #1609 then built on top of that already-activated pipeline.

Rollout flags

Three independent, reversible kill switches gate the pipeline. All default OFF:

Each flag can be flipped independently without redeploying, and flipping it OFF fully reverts that piece of behaviour. See Configuration & Operations for env vars, models, and deploy prerequisites.

Read next