Data Sources & Screening

Phase 1 of the native brain screens a company against verified reference sources before Phase 2's open-web research ever starts. This page covers the sanctions/BHRRC screening pipelines, the corpus-vs-live-RSS path for BHRRC article bodies, the dedicated Blob store that hosts all of it, and its caching. See Issue Research Pipeline for how Phase 1 fits into the two-phase pipeline as a whole.

Fast-issues Phase 1: verified-source screening

Phase 1 (application/ai/pipelines/fast-issues/) is deterministic: it name-matches the company against flat-file reference lists loaded from Blob storage. There is no web search anywhere in Phase 1 — that is what distinguishes it from Phase 2, which runs an agentic web_search_batch → save_issues loop over live Tavily results (see Two-Phase Native Brain).

Every source is a self-contained profile: it owns its own Blob index key, its own name-matching configuration, and the logic that maps one matched index entry to a ConsolidatedIssue. All eight profiles — the seven SANCTIONS_PROFILES plus BHRRC — are fanned out in parallel from executePhase1 (generate-organisation-issues-two-phase.ts) and run behind two shared generic runners: runScreeningResearch/runProfileScreening for sanctions, runBhrrcResearch for BHRRC.

Failure is isolated per source, not per phase. Each profile's promise is settled with its own .then(onSuccess, onError); a rejection is categorised (sanctions-research-failed[<source>] / bhrrc-research-failed) and recorded against the run's error list rather than rejecting the whole Promise.all — one source's outage never blocks the other seven from persisting. Whether that recorded error ultimately marks the organisation FAILED depends on the partial-degradation rules covered in Two-Phase Native Brain.

Sanctions & forced-labour lists

SANCTIONS_PROFILES (shared/sanctions-profiles-registry.ts) holds the seven sanctions profiles in a fixed order, preserved verbatim from the pre-refactor implementation (with DFAT appended last):

Each source tag is the exact value persisted issues are tagged with, and each blob key is one of the SOURCE_INDEX_BLOB_KEYS (see The source-documents Blob store).

runProfileScreening (issues-core/run-source-profile-screening.ts) drives every profile identically:

1. Load — SourceIndexRepository.load(profile.blobKey, profile.entrySchema) fetches and validates the flat-file index from Blob.
2. Match — tieredNameMatch (issues-core/tiered-name-match.ts) scores the company's name and aliases against every index entry's name and aliases, across four built-in tiers: exact, prefix, reverse-prefix, and fuzzy (a stemmed-suffix match, e.g. "Industries" → "Industry"). Names are canonicalised and stripped of a trailing corporate-suffix vocabulary (group, holdings, corporation, ltd, llc, inc, …) before comparison.
3. Map — every matched entry is passed to the profile's entryToIssue to build a ConsolidatedIssue, tagged with a forced-labour/broader-issue severity split (sanctionsTierAttrs) computed per-source (e.g. OFAC's UHRPA/GLOMAG program codes, UFLPA's unconditional nexus, BIS's Xinjiang/China keyword heuristics). See Data Model & Persistence for how a ConsolidatedIssue is reconciled into the Neo4j schema.

Two false-positive guards keep the tiers precise: a reverse-prefix match only counts when the shorter (candidate) side is multi-token or at least five characters, so a short acronym alias can't prefix-match an unrelated query; a fuzzy match only counts when the stemmed name is at least four characters. ofac, eu-sanctions, and dfat-sanctions also apply an isOrganisation filter to skip individuals/persons in the index before matching.

BHRRC allegations

runBhrrcResearch (sources/bhhrc/run-bhrrc-research.ts) follows a longer pipeline than the sanctions profiles, because BHRRC allegations require both a company lookup and a full map-reduce over article text:

1. Load candidates — SourceIndexRepository.load(SOURCE_INDEX_BLOB_KEYS.bhrrc, bhrrcIndexEntrySchema) fetches bhrrc-index.json (each entry: slug, contentId, name, country, sectors).
2. lookupBhrrcCompany — tiered name-matching as above, but with BHRRC-specific tuning: a narrower trailing-suffix vocabulary (BHRRC's index omits "corporation"/"group" that sanctions lists keep), an extra parenthetical tier that matches a token inside the index entry's name (e.g. "H&M" against "Hennes & Mauritz (H&M)"), and a country/sector affinity score boost. Matching ranks by tier first (exact > prefix/reverse-prefix/fuzzy > parenthetical) so the affinity boost can never let a weaker tier outrank a stronger one — it only breaks ties within the winning tier. The result is a list of matched contentIds.
3. fetchAllegations(contentIds) — resolves to either the live RSS path or the Blob corpus path, depending on feature/bhrrc-corpus (see Corpus vs live RSS).
4. mapBhrrcArticle — one flash-lite tool-agent call per allegation article, calling save_issues once. The prompt caps article content at 12,000 characters (MAX_ARTICLE_CHARS, covering the ~9k-character p90 of real articles while bounding runaway token usage on outliers) and explicitly instructs the model not to search the web — only the supplied article text is available. Article mapping runs via Promise.allSettled, so one malformed article never drops the rest; failures are counted in mapFailures.
5. reduceBhrrcIssues — clusters the mapped issues into ConsolidatedIssue[] using the same reduce-issues.ts kernel module Phase 2 uses, bound to the flash-lite model for speed (comparable quality, faster than the full-flash Phase-2 reducer).

If the index has no candidates, the lookup has no matches, or the allegation fetch returns nothing, runBhrrcResearch short-circuits to an empty result — no LLM calls are made for a company with no BHRRC signal.

Corpus vs live RSS

fetchAllegations is a flag-routed function, built by makeDefaultFetchAllegations(resolveBhrrcCorpus, fetchBhrrcAllegationsFromCorpus, fetchBhrrcAllegations): it resolves feature/bhrrc-corpus once per call and dispatches to the corpus path when true, the live path (default) when false.

Live (fetch-bhrrc-allegations.ts) queries BHRRC's unfiltered RSS feed per matched contentId — deliberately not the contains_allegations=YES-filtered feed, since that misses articles relevant to modern slavery but not formally BHRRC-tagged as allegations. Feed and article-body fetches are serial, with a 1-second gap between requests and exponential 429 backoff (2s, 4s, 8s, 16s, 32s), to stay under BHRRC's rate limits.

Corpus (fetch-bhrrc-allegations-from-corpus.ts) reads from the Blob-hosted article corpus via BhrrcArticleRepository. It first loads the scraped-ids manifest (_manifest.json) and splits the matched contentIds:

• In the manifest → read bhrrc-articles/{contentId}.json. An absent blob for an in-manifest id means "scraped, zero articles" (a genuinely clean company), not a miss.
• Not in the manifest → never scraped — a true corpus miss. These ids fall back to the live fetch, with a telemetry warning.
• Corrupt or unreadable blob for an in-manifest id → also falls back to live, for just that id (failedIds), so one bad blob never zeroes out the whole batch.
• Empty/degraded manifest (absent, unreadable, or schema-invalid) → every matched id falls back to live — the safe direction, since it never manufactures a false "no allegations" result.

The corpus is a point-in-time scrape, so it has a recency boundary the never-scraped fallback does not close: a company already in the manifest, with a new article published against it since the last scrape, is silently missed by the corpus path — there is no fallback for "already scraped, but stale" (only for "never scraped"). The article-cache TTL only bounds how fast a re-upload of the corpus propagates; it does not trigger a re-scrape. Running feature/bhrrc-corpus ON therefore requires a recurring re-scrape-and-upload cadence to hold recall at live parity.

The source-documents Blob store

Every third-party source document — the sanctions/BHRRC flat-file indexes and the BHRRC article corpus — lives in one dedicated, private Vercel Blob store, deliberately separate from BLOB_READ_WRITE_TOKEN (which now serves only user-generated content: reports and uploads, via BlobStorage). Every read against it authenticates with its own token, resolved by getSourceDocumentsToken() from the SOURCE_DOCS_BLOB_READ_WRITE_TOKEN environment variable (Vercel also injects an unused SOURCE_DOCS_BLOB_STORE_ID, since the SDK addresses the store by token, not store id).

Key layout:

Two repositories front these reads:

• SourceIndexRepository (infrastructure/repositories/source-index-repository.ts) — a thin wrapper over loadSourceIndexFromBlob, exposing SOURCE_INDEX_BLOB_KEYS (ofac, bis, eu, uk, un, uflpa, bhrrc → their respective *-index.json pathnames) as the single source of truth, so a mistyped key is a compile error instead of a runtime "failed to load" error.
• BhrrcArticleRepository (infrastructure/repositories/bhrrc-article-repository.ts) — a thin wrapper over load-bhrrc-articles.ts, exposing load(contentId) (point lookup), loadMany(contentIds) (bounded-parallel, returns { records, failedIds }), and scrapedContentIds() (the manifest set).

SourceDocumentsTokenError, thrown by getSourceDocumentsToken() when the token is unset or blank, distinguishes a configuration failure from a data failure. Corpus-path callers check instanceof SourceDocumentsTokenError specifically, so a missing/rotated token is logged loudly (console.error, alertable) as "corpus unavailable for all matched ids", distinct from the benign per-id warning logged for a single corrupt article blob.

Always-on vs opt-in

Caching

Both caches de-duplicate concurrent in-flight loads for the same key — N simultaneous requests for the same uncached pathname or contentId trigger exactly one Blob fetch, not N. A load that fails (not-found, unparseable, schema-invalid) is never cached, so the next call retries rather than being stuck with a permanent miss.

The article cache's byte accounting counts 2 bytes per character for body/title/url, since V8 stores a string at 2 bytes/char as soon as any character is non-Latin1 — routine in international BHRRC text (accented names, smart quotes). This deliberately over-counts pure-ASCII bodies, so the cache evicts a little early rather than risking an out-of-memory worker; least-recently-used entries are evicted first, and at least one entry is always retained even at a 0-byte budget. The scraped-ids manifest is only cached when non-empty, so a transient or degraded (empty) read retries on the very next call instead of freezing the corpus off for a full TTL window.