> ## Documentation Index > Fetch the complete documentation index at: https://docs.getcollate.io/llms.txt > Use this file to discover all available pages before exploring further. # Reasoning & Validation | Collate Knowledge Graph > Inference levels, materialized SPARQL CONSTRUCT rules, custom ontology extensions, and SHACL validation over the Collate knowledge graph. # Reasoning & Validation A knowledge graph earns its keep when it can tell you things nobody explicitly wrote down, and refuse to accept things that contradict the model. Those are the two halves of this page: **inference** (derive) and **SHACL validation** (check). ## Inference Levels Every SPARQL query can request a reasoning level: | Level | What it derives | | --------------------- | ----------------------------------------------------------------------------------------------------- | | `NONE` | Nothing. Only asserted triples. The default. | | `RDFS` | RDFS entailment, subclass and subproperty closure, domain/range typing. | | `OWL_LITE` / `OWL_DL` | OWL entailment at the corresponding profile: inverse properties, transitivity, symmetry, equivalence. | | `CUSTOM` | Collate's own rule set, transitive lineage traversal, tag propagation, domain inheritance. | Configure the default with `RDF_DEFAULT_INFERENCE_LEVEL`, and enable inference at all with `RDF_INFERENCE_ENABLED`. Callers can override per query (`inference=` on the REST endpoint, `inferenceLevel` on the MCP tool). `GET /api/v1/rdf/status` reports which levels are available and what the default is. ### Two Execution Strategies **In-process inference** builds a Jena inference model over the dataset in memory at query time. Accurate for any level, but bounded: if the store exceeds `RDF_MAX_IN_MEMORY_INFERENCE_TRIPLES` (default 100,000), the query silently falls back to direct execution **without** inference and returns a `warning` field saying so. Enable `cacheInferredTriples` to reuse bounded in-memory models for 60 seconds. **Materialized inference** (`RDF_MATERIALIZED_INFERENCE_ENABLED=true`) runs each rule as a SPARQL `CONSTRUCT` inside the triple store and writes the results to a durable named graph, one per rule. Queries then read materialized triples with no in-memory model at all. **Use materialized inference for anything past a small catalog.** In-process inference does not scale past the triple limit, and a silent fallback that returns un-inferred results is worse than no inference at all, you get an answer that looks right and is incomplete. Materialization has the opposite failure mode: staleness, which is visible as a `dirty` flag. ## Materialized Inference Rules A rule is a SPARQL `CONSTRUCT` query with a name, priority, and enabled flag. Its output lands in `https://open-metadata.org/graph/inferred/{rule}`. ### The Starter Pack Four rules ship and are always present: Materializes indirect lineage by walking `prov:wasDerivedFrom` transitively, so SPARQL can answer "all upstream tables of dashboard X" without users writing property paths. ```sparql theme={null} PREFIX prov: PREFIX om: CONSTRUCT { ?x om:transitivelyDerivedFrom ?y } WHERE { ?x prov:wasDerivedFrom+ ?y . FILTER(?x != ?y) } ``` If a column carries a `PII.*` tag and another column receives data from it via column-level lineage, propagate the tag downstream. The derived tag is marked with `om:inferredTagSource` pointing at the upstream column, so a propagated tag is always distinguishable from a curated one. ```sparql theme={null} PREFIX om: CONSTRUCT { ?downstream om:hasTag ?piiTag . ?downstream om:inferredTagSource ?upstream } WHERE { ?upstream om:hasTag ?piiTag . ?piiTag om:tagFQN ?fqn . FILTER(STRSTARTS(?fqn, "PII.")) ?colLineage om:fromColumn ?upstream ; om:toColumn ?downstream . } ``` This is the rule that turns "we tagged the source" into "we know every downstream column that is now also sensitive", the question most privacy reviews actually ask. Propagates tags down the containment hierarchy: a tag on a `DatabaseSchema` is inherited by every `Table` in it, and a tag on a `Table` by every `Column`. Inferred tags carry `om:inferredTagSource`. If a `Table` belongs to a `Domain`, every `Column` of that table inherits the membership, so `om:belongsToDomain` queries return both table- and column-level results without a separate lookup. Rules run in **priority order** (lower first; ties broken by name), which matters: schema→table→column tag inheritance at priority 300 runs after PII propagation at 200, so inherited tags do not feed back into the propagation pass in the same run. ### Managing Rules | Endpoint | Purpose | | ------------------------------------ | --------------------------------------------------------------------------------------------------------------------------- | | `GET /api/v1/rdf/rules` | List rules with materialization state: `dirty`, `graphUri`, `tripleCount`, `lastMaterializedAt`, `lastError`, `systemRule`. | | `GET /api/v1/rdf/rules/{name}` | One rule. | | `PUT /api/v1/rdf/rules/{name}` | Create or update a custom rule. Saving marks it dirty. | | `DELETE /api/v1/rdf/rules/{name}` | Delete a custom rule and drop its materialized graph. System rules cannot be deleted. | | `POST /api/v1/rdf/rules/validate` | Validate a rule body without saving. | | `POST /api/v1/rdf/rules/materialize` | Materialize now. | All admin-only. **Dirty tracking**: when source RDF changes after a successful materialization, the rule is flagged `dirty`, and `GET /api/v1/rdf/rules` reports that flag alongside `tripleCount` and `lastMaterializedAt`. Staleness is therefore always visible rather than silent. ### Running the Materializer Materialization is done by an application, not by the query path. **RDF Inference Materialization** (`RdfInferenceApp`) appears under **Settings → Applications** once RDF is enabled. * **Schedule**: every **5 minutes** by default. It picks up rules whose dirty flag is set and rebuilds only those. * **Execution**: each rule runs as a SPARQL `CONSTRUCT` *inside* Fuseki, writing to that rule's named graph. No triples pass through the Collate JVM, so materialization cost does not scale with server heap. If you have just changed a rule and do not want to wait for the next run, select **Run Now** on the application. ### Writing Your Own Rule ```json theme={null} { "name": "contract-coverage", "displayName": "Tables governed by a data contract", "description": "Marks any table that has an attached data contract, so coverage can be queried directly.", "ruleType": "CONSTRUCT", "priority": 500, "enabled": true, "tags": ["governance"], "ruleBody": "PREFIX om: \nCONSTRUCT { ?t om:hasGovernedContract true }\nWHERE { ?c a om:DataContract ; om:appliedTo ?t }" } ``` ```bash theme={null} curl -X PUT "$COLLATE_HOST/api/v1/rdf/rules/contract-coverage" \ -H "Authorization: Bearer $ADMIN_TOKEN" \ -H "Content-Type: application/json" \ -d @rule.json ``` A `CONSTRUCT` whose `WHERE` clause is expensive is expensive **every time it materializes**, over the whole graph. Validate it, then run it against a non-production dataset before enabling it in production. Unbounded property paths combined with unbound subjects are the usual culprit. ### Explaining an Inference When materialized inference is on, the Ontology Studio relation panel shows an **inference explanation** for a derived relation, which rules contributed, how many triples each produced, and when they last ran. `POST /api/v1/ontology/reasoning/explanations` exposes the same thing over the API. This matters more than it sounds. An inferred PII tag that nobody can explain is a compliance problem, not a feature. ## Custom Ontology Extensions Extend the canonical ontology with your own classes and properties without forking it. Extensions live in a reserved namespace, `https://open-metadata.org/ontology-extension/`, so a custom class can never collide with a core `om:` term or be mistaken for one. | Endpoint | Purpose | | ----------------------------------------------- | ------------------------ | | `GET /api/v1/rdf/ontology/extensions` | List extensions. | | `GET /api/v1/rdf/ontology/extensions/{name}` | Fetch one. | | `PUT /api/v1/rdf/ontology/extensions/{name}` | Create or update. | | `DELETE /api/v1/rdf/ontology/extensions/{name}` | Remove. | | `POST /api/v1/rdf/ontology/extensions/validate` | Validate without saving. | Each extension declares custom OWL classes and properties (object or datatype) with a description explaining why they are needed. Validation rejects URIs outside the extension namespace. Prefer modeling in the **ontology** (concepts and relationship types) over extending the **catalog vocabulary**. Extensions are for describing kinds of *metadata* Collate does not have, not for describing your business, which is what the ontology is for. ## SHACL Validation SHACL (Shapes Constraint Language) is the graph equivalent of a schema check. Collate ships canonical shapes at `rdf/shapes/openmetadata-shapes.ttl`, loaded into `https://open-metadata.org/graph/shapes`, covering base entity constraints (every entity has exactly one id, one name matching `^[a-zA-Z0-9_-]+$`, one FQN, at most one description, a positive version) plus per-class shapes for tables, columns, and the rest. ### Running It ```bash theme={null} # Whole dataset (expensive — admin only) curl "$COLLATE_HOST/api/v1/rdf/validate" -H "Authorization: Bearer $ADMIN_TOKEN" # Scoped to one entity curl "$COLLATE_HOST/api/v1/rdf/validate?entityUri=https://open-metadata.org/entity/table/{uuid}&format=jsonld" \ -H "Authorization: Bearer $ADMIN_TOKEN" ``` The response is a standard `sh:ValidationReport` in Turtle (default) or JSON-LD. An `OM-SHACL-Conforms` response header carries the boolean verdict, so CI can gate on it without parsing the body. Agents call the same thing through the `shacl_validate` MCP tool, which additionally returns `conforms` and `violationCount` in full even when the report body is truncated. ### Validation Modes `RDF_SHACL_VALIDATION_MODE` controls the policy: | Mode | Behavior | | ----------------- | -------------------------------------------------------------- | | `OFF` | No validation. | | `REPORT` | Validate and report. Never blocks. **The default.** | | `ENFORCE_IMPORTS` | Additionally reject ontology **imports** that fail validation. | **Validation never blocks the write path.** SHACL here is a diagnostic, not a gate on entity creation, a platform that refuses to ingest a table because of a shape violation is worse than one that ingests it and tells you. `ENFORCE_IMPORTS` is the one place strictness is opt-in, because a malformed imported ontology corrupts a model rather than one row. ## OWL Profile Guardrails `RDF_STRICT_OWL_PROFILE=true` (the default) makes Collate reject authored axioms outside the supported OWL 2 DL profile. This is not pedantry: OWL Full is undecidable, and a reasoner over an undecidable ontology can run forever. The guardrail is what keeps `OWL_DL` inference a bounded operation. ## Next Centrality, communities, and paths over the materialized graph. Every endpoint, with parameters.