Ingestion Vision: From Governed Text to Governed Meaning

Status: vision, 2026-09-25. Not a decision and not a plan. Each part that becomes work needs its own ADR, following the Design First rule. Nothing here changes an accepted ADR.

Scope: how the text of a governing document (a contract, a trial protocol, a lending covenant, an entitlement scheme, a regulation) becomes LATTICE content that the compilers can use, with a person deciding what is accepted. Examples are domain-neutral, following the repository’s rule for public material.

Inputs: two working notes, MORK for document ingestion and scaling domain ontology mapping. Where this paper adopts, refines or departs from them, §17 says so.

1. The problem

LATTICE can hold what a governing document means: who owes what to whom (Party, Instrument), under which conditions (Eligibility), with which quantities and bounds (Quantification), and how it changes over time (Behaviour). The compilers turn that meaning into SPARQL, SHACL, SWRL and design-time OWL classes (ADR-A19, ADR-A24, ADR-A90). What LATTICE does not yet have is a route from the document’s text to that meaning.

Two facts shape the route:

• Machine extraction works. Frontier LLMs, given a rich applied ontology, have turned whole contracts into OWL of high quality. The MERIDIAN contract ontologies are the reference case.
• Whole-document extraction does not scale. One such extraction cost about USD 300 per contract on a frontier model. Most of that cost is structural: the document is resent with every call, the model re-derives terms it has already resolved, and it spends tokens on Turtle’s verbosity. At the scale of a book of documents, that cost and the review burden it creates are both prohibitive.

Comparable open-source systems (Knwler, TrustGraph, Semantica) turn each document’s text into graph facts, every time, for every document. MORK was built for a different cost curve: turn a source’s shape into a reviewed, reusable mapping once, then compile it and run it deterministically for every instance. The vision uses both curves, each where it fits (§2.1).

The goal is a pipeline in which a small chunk of text, given to a model with the right context, comes back as a proposal already in LATTICE’s own terms, and a person accepts or rejects it. Once proposals are compatible with Quantification and Eligibility, they enter the compilation pipeline unchanged, and that is the ingestion mechanism.

2. Principles

Each principle is stated once here and cited by number below.

2.1 Three kinds of output

The mapping layer (intent nodes, mapping DAGs, compiled executables) earns its cost when what it produces is reused. It is a detour when the output is used once. So the pipeline separates three kinds of output by how often each is reused:

Bespoke text is a rule with a reuse count of one. It compiles into executable artefacts like any rule, but the mapping layer’s cost is not amortised, so it goes the direct route with the facts.

For governing documents the middle row carries most of the value. Standard wording is common, so reviewing a wording’s meaning once and reusing it across a market or programme is where the per-document cost collapses.

3. Two complementary routes to meaning

The routes converge on the same targets (§5.3) and pass through the same gate (§7). Where both exist for one wording, each checks the other: agreement raises confidence, and disagreement is a signal for review. A controlled language’s compiled logic is itself a source schema from MORK’s point of view, so its mapping to LATTICE is a structure mapping, reviewed once and reused.

4. The pipeline

flowchart TD
    A["S0 Acquire and parse layout"] --> B["S1 Segment, fingerprint, tag spans"]
    B --> T{"S2 Known document type?"}
    T -->|yes| TS["Reuse the validated structure mapping"]
    T -->|no| TP["Propose a structure mapping (rule route)"]
    TP --> H
    TS --> W{"Known wording, per element?"}
    W -->|exact or near match| R["Reuse the meaning template, extract slot values"]
    W -->|novel standard wording| D["S3 Classify and route"]
    W -->|bespoke text| F2["S4b Extract facts directly"]
    D --> E["S4a Propose a meaning template (rule route)"]
    R --> F2
    E --> V["S5 Decode, resolve concepts, validate"]
    F2 --> V
    V --> G["S6 Resolve references and entities"]
    G --> H["S7 Review"]
    H -->|accepted| I["S8 Promote and compile"]
    H -->|declined, deferred, taught| J["S9 Learn"]
    I --> J
    J -.-> T
    J -.-> W

S2 is where cost falls fastest. A document of a known type, whose elements are mostly known wording, costs little beyond S0 and S1: its structure mapping runs deterministically, and only slot values and bespoke text are extracted.

5. The intermediate representation for rules

This section applies to the two rule routes of §2.1. Instance facts skip it and are extracted in LATTICE’s terms directly (§7).

5.1 MORK’s intent algebra

A meaning template is extracted as MORK intent nodes (IV7):

Every node records its source text (mork:hasNaturalLanguageSource) and a confidence (mork:hasIntentConfidence). Nodes form a join-semilattice under mork:refinesIntent: one root subject, with constraints and scopes refining it, and compound intents expressed by multi-typing (an exclusion that is also a spatial scope). Intent nodes reference only SKOS concepts, datatypes and source text, never a target class, so the same template can be projected into more than one domain ontology.

A structure mapping needs no new vocabulary, as a working hypothesis. A document type’s element tree is a source representation in MORK’s existing terms (mork:RepresentationScheme, mork:Representation, mork:compositeNarrower, a mork:SerializationFormat), and its mapping to Instrument elements is ordinary MORK mapping, compiled to a deterministic parser the way a schema mapping compiles to RML. Whether document structure needs a sibling vocabulary instead is §15 Q9.

5.2 Projection to the domain

mork:intentMapping links each intent node to the mork:DataMapping that realises it in the target ontology. The mapping scheme, not the intent graph, holds the domain binding. Where the target cannot receive what the text says, the mapping is a mork:UncertainMapping marked incomplete, with a recommendation naming exactly what is missing, and nothing is asserted (IV10). A minimum rate with no stated basis, or a reference to an external standard with no identifier, stays uncertain until a person supplies the rest.

5.3 Targets in LATTICE’s layers

A domain ontology built on LATTICE sorts its content into four kinds. Every template and every fact lands in one of them (IV8):

Quantities and bounds land in Quantification throughout: qnt:Quantity, qnt:Bound, qnt:Range and qnt:RangeSet, alternative bounds per unit (ADR-A95), derived rate spaces for percentages of a base (ADR-A93), calendar units for business days (ADR-A94).

5.4 Worked example

Text of a standard wording: “Term facilities between EUR 1m and EUR 25m, to borrowers incorporated in the EU, excluding Cyprus, at a margin of at least 150 basis points.”

As a meaning template, with the amounts, the region and the margin as slots:

The admission profile is elg:AllRequired over the three conditions. The jurisdiction condition reads its candidate through an evidence binding from the facility to its borrower’s jurisdiction of incorporation (ADR-A91). Every downstream artefact, including the design-time OWL class that tells a reviewer whether a revision widens the criteria (ADR-A90), is then generated, not written.

When a later document uses the same wording with EUR 2m and EUR 40m, S2 recognises the template, S4b extracts the two amounts as facts, and nothing else is proposed or reviewed.

6. Making it affordable

6.1 Structure-first chunking

S0 and S1 split the document before any model reads it (IV5). Each call carries one element, its variables, and the context §6.3 and §6.4 supply. The document is never resent. Cost then scales with the number of novel elements, not with document length times calls.

6.2 Packaging a target ontology

The MORK Teaching Pack (MTP, ADR-A44) packages MORK for a model: a deterministic, hash-pinned build of doctrine, a decision ladder, lenses, minimal pairs and validated cassettes. It compresses well because MORK is small by design and its authors are the experts on its semantics, so its doctrine was written once by the people best placed to write it.

A target domain ontology does not compress that way. It is meant to carry its domain’s real distinctions, and the pipeline’s builders are not experts in every ontology a deployment brings. Generating minimal pairs for an arbitrary ontology is knowledge acquisition, not compression, and a wrong generated pair produces confident, wrong mappings, which is the failure the whole design exists to avoid. So a target ontology is packaged in tiers, by risk:

Readiness depends on annotation density, not size. A large ontology with a definition and a usage note on every term and worked examples is a good retrieval target. A small one with bare names is a poor one, and no compilation fixes that. LATTICE’s literate-spec discipline (a definition and fnd:utility on every term, non-domain examples before mechanism prose) is therefore a prerequisite for onboarding a target ontology, and its absence is a finding to report to the ontology’s owners.

6.3 Grounding, and reproducible retrieval

Before proposing a new term, the model asks whether the target ontology already has it. A grounding interface answers exact, broad or none, with a target IRI, a confidence and a rationale that becomes the proposal’s evidence note. The default is exact-label lookup over the tier-1 index. Better recall comes from tier-2 retrieval, embeddings and entity resolution. Grounding retrieves, and never generates ontology content.

Live retrieval is not stable from run to run unless its index is fixed. So each index is a derived artefact (ADR-A92): built deterministically from declared sources, hashed, versioned, and recorded with the pack version in every run’s provenance (IV11). Reproducing a run means replaying it against the same index hash.

6.4 Document memory

A long document is processed element by element, so the model needs to know what earlier elements settled: defined terms, the parties’ role occupancies, open deferrals, prior hypotheses and their confidence, cross-references. A memory interface records and recalls these, and answers “what bears on this element” by querying the partially built graph rather than replaying the text. The same graph answers questions about the document’s state during ingestion. A reviewer’s correction to a remembered term flows back to every element that used it. The memory’s state at each call is snapshotted by hash, under IV11.

6.5 The wording library and structure mappings

Many governing documents are assembled from standard wording on recurring document types. Once a wording’s meaning or a document type’s structure is reviewed, it is reused: S2 recognises it, and only slot values and bespoke text are extracted. Extraction then runs once per template for a whole market or programme, and per document only for what is new. Near-match thresholds are set conservatively and calibrated by sampling the band just below them for review. Whether “have we seen this shape before” reuses MORK’s community-detection machinery or needs its own similarity model is §15 Q11.

6.6 Extraction tiers and model tiering

Stages differ in what they need. The extraction tiers are:

• Tier 1, lexical and symbolic, never an LLM: grammars, codebook lookup, exact and near matching, DL constraint filtering
• Tier 2, statistical, mostly not an LLM: trained taggers and classifiers, embeddings, community detection
• Tier 3, an LLM, escalatable to a person: novel or complex text, ambiguous relations

Frontier models are kept for the tier-3 residual, which shrinks as the library and the trained models grow:

A pipeline analysis estimated that, for a 100-page document against a mature library, this design moves the cost from over USD 70 (an agentic whole-document approach) to about USD 5 with the staged pipeline, and about USD 3 with trained models in the cheap stages. These are estimates, not measurements. Measuring them is part of the roadmap (§14).

6.7 Caching and local-first defaults

Every model call is keyed on the element’s hash, the pack hash, the index hash and the model identity, so an identical request is answered from cache and a changed input is never answered from a stale one. Batches are resumable. Local models are the default wherever they meet a stage’s calibrated accuracy, and hosted frontier models are reserved for the tier-3 residual.

7. Wire formats

A model’s output format is a cost lever, separate from what the output means. There are two, one per kind of output, both decoded deterministically before anything else sees them:

• MCN for rules. MORK’s compact notation is the proposal wire format for MORK content (ADR-A25). It measured about three times fewer structural tokens than Turtle on the repository’s examples, decodes losslessly, and binds annotations to the triple they annotate. Structure mappings and meaning templates are MORK content, so MCN serves both.
• Generated compact JSON for facts. MCN compresses MORK’s vocabulary, not a target ontology’s instances, so facts need their own format. Each target layer gets a JSON Schema generated from its structural shapes and vocabulary as an execution/ artefact, with types inferred from structural position and keys from the tier-1 codebook (§6.2). JSON Schema validation catches malformed output before RDF compilation, and SHACL catches the rest after it.

Both need the same compiler steps after decoding: concept-label resolution through Vocabulary’s cascade (exact prefLabel, altLabel, embeddings, then disambiguation or review), scoped to the scheme contract bound to the property, then SHACL and OWL validation. JSON-LD is kept for publishing compiled graphs to external tools, not for model output.

8. Gaps, uncertainty and Undetermined

Text is often silent or ambiguous. The pipeline records that instead of resolving it (IV10):

Every consumer of a decision declares what it does with Undetermined: surface it, queue it, block on it. Coercing it to either outcome is a defect.

9. Symbolic constraints on neural output

The ontology is not only the target. It constrains every stage (IV4):

• Disjointness rejects a proposal whose target is disjoint from a class the same section is already aligned to.
• Cardinality rejects a second value where the target allows one.
• Universal restrictions reject a filler outside the declared range.
• MORK’s co-occurrence axioms reject structurally incomplete mappings before they reach the graph.
• Scheme contracts restrict concept resolution to the bound scheme, so a place name cannot resolve to a currency.
• Shapes mask a classifier’s output space, so an element cannot be given a qualifier its kind does not allow.

Rejected proposals are returned to the proposer as negative evidence, so trained models learn the constraints over time while the constraints stay exact.

10. Review, assurance and provenance

10.1 Review

The MORK review workbench provides the review act. A review snapshot pins the tenant, project, mapping graph revision, section and evidence projection. Six typed decisions apply: Confirm, Retarget, Reshape, Decline, Teach and Defer. For ingestion, the reviewer sees the proposal beside its source span (IV1), and a domain steward sees evidence, not notation or lint diagnostics. A newer snapshot makes older decisions stale, so a decision never lands on changed evidence.

Review effort follows reuse (§2.1). Rules are reviewed item by item, because an error in a rule repeats in every document that uses it. Facts are numerous and each matters once, so they are confirmed in batches, which queue calibration governance allows only when precision meets a threshold for the exact pack, profile and model. Below it, or for a low-confidence item, review is item by item. A batch confirmation is still a person’s act, recorded as one. Confirmed items are sampled for audit.

An ontology change that affects approved mappings reopens them. Retrospective challenge records later doubt without mutating the original decision. Corrections, to rules and to facts alike, become graph data that later runs learn from. Active learning concentrates review on proposals whose top candidates are closest, which is where a person’s time changes the outcome.

10.2 Provenance

Every accepted statement can answer where it came from, in PROV-O, through Foundation’s evidence mixin and derived-artefact contract (ADR-A26, ADR-A92):

document version
  → element (S0, S1)                 prov:Entity, Source role
  → span                             prov:Entity
  → extraction activity (S4)         prov:Activity: model, pack hash, index hash, memory snapshot, profile
  → proposal                         prov:Entity, Mapping role, with confidence
  → review activity (S7)             prov:Activity, associated with the reviewer, item or batch
  → accepted statement (S8)          Declaration or Assertion role
  → compiled artefact                exe:ExecutablePlan, exe:GeneratedArtefact, fnd:DerivedArtefact
  → runtime decision                 Execution role

A fact extracted from a known template also records the template and structure mapping it was shaped by, so a later correction to either reaches every fact that depended on it. ADR-A13’s statement provenance properties (source artefact, extraction activity, asserting agent, confidence, mapping status) cover each link. The chain is complete only when every link is present, which governance can check.

11. Components and boundaries

The pipeline needs capabilities LATTICE does not build itself: document parsing, embeddings, models, graph stores, agent memory. Each sits behind an interface LATTICE owns (IV12):

Semantica, a graph-native framework for context and accountable AI, is one candidate behind several of these at once: storage-agnostic graph and vector stores, pluggable reasoners with explanation, agent context and a decision lifecycle with PROV-O export, and entity resolution and conflict detection that S6 needs across documents. It fits as grounding, memory and backend, not as an extraction stage in front of MORK. Its ontology generation is not used: the target ontology is authored and governed.

The isolation rules follow ADR-A83’s pattern:

• no adapter is a dependency of any core package, and core never imports one
• backends are selected by configuration through a registry, never by conditional import
• core CI runs with no optional package installed, and each adapter passes LATTICE-owned contract tests in a separate, non-blocking job
• adapters version independently, so a change in a third-party API never forces a core release

12. What stays deterministic

• Compilation. The compilers are pure functions of validated graphs (ADR-A19). Determinism is what makes artefacts auditable and regenerable (ADR-A27). Structure mappings compile to deterministic parsers under the same rule.
• Validation. SHACL and OWL checks are exact. Trained anomaly detectors complement them and never replace them.
• Provenance. Recording where a statement came from is bookkeeping, and must be complete.
• Packaging and indexing. Tier-1 codebooks and indexes are generated deterministically.
• The production gate. No model-originated node compiles in production mode without governance sign-off (ADR-A25, ADR-A28).

13. Relationship to what exists

ontology-architecture.md §8.5 and the MORK README place Semantica in front of MORK as data acquisition. This vision refines that: Semantica-like services supply grounding, memory and cross-document entity resolution behind LATTICE’s interfaces.

13.1 Comparable systems

None of the three separates reusable rules from one-off facts. That separation is where LATTICE’s cost curve differs.

14. Roadmap sketch

Indicative only. Each phase needs its own ADRs and plan.

MTP’s deferred Phase 7 (output scoring, held-out evaluation sets, metrics) is the natural home for the measurement work each phase needs.

15. Open questions

16. Risks

17. How this paper uses its inputs