runex Architecture

What this is

runex is a graph-shaped ontology engine: a typed graph store with declarative semantics for object lifecycles and reactive behavior. Object types, state machines, actions, and their triggers all live as data in the graph itself. New behavior is added by writing data — not Python — and takes effect immediately, without restart.

The N+M ingest pipeline (one adapter per source, one upsert routine for all sinks) is one consumer of this engine, not its purpose. Everything downstream of a write — link extraction, state transitions, derived structure — happens through reactive cascades on store events, not through call-site coupling.

Layers

L4   Interface         CLI · Ontology facade               ← contract lives here
L3   Ontology engine   DSL eval · Loader · Engine · Kernels · ReactiveBus
L2   Store             nodes / fields / links / identities · tx_log · on_commit
L1   SQLite

Dependency is strictly one-way L4 → L3 → L2 → L1. L2 doesn’t know what a machine or action is — it just emits mutation events through on_commit. Removing src/runex/ontology/ and src/runex/dsl/ leaves L2 + the pipeline functional.

The contract is an L4 concern — Result envelope, closed Event taxonomy, manifest(), cursor-resumable event stream. L3 stays a pure bool/raise executor; L4 wraps it. JSON is truth; Rich output is a view.

Core abstractions

Node — the only object type

nodes ── node_tags ── supertags
   │
   ├── fields            (typed EAV: text/longtext/number/date/bool/blob)
   ├── links             (typed directed edges with props)
   └── node_identities   (natural_key → node_id projection)

A node is the universal record. A Bookmark, a Person, a state-machine definition, and a kernel reference are all nodes — differentiated by which supertag they carry.

Supertag — type declaration

A supertag declares:

• Field schema (names, types, required, multi-value)
• Optional inheritance via extends
• Optional natural_key field that uniquely identifies real-world objects

Three supertags are reserved for the engine’s own metadata: MachineDefinition, ActionDefinition, KernelRef.

Supertag definitions are (supertag …) forms in .scm bundles, loaded via o.load(). The bundled system, domain, and pack ontologies ship under src/runex/skill/ontology/. The engine applies Policy-A migration gating: additive changes (new field, new supertag) always apply at runtime; destructive changes (drop a field, change a field’s type, change the natural key) are refused if the supertag already has tagged nodes — unless allow_destructive=True. Re-applying a byte-identical definition is a true no-op (no event emitted, no churn on repeated load).

Machine — lifecycle template (data)

(machine "Name"
  (supertag "Name")
  (initial  "s1")
  (states ("s1" "act-a")
          ("s2" "act-b")
          ("s3")))                  ; terminal

Current state is stored on the node as the reserved field __state__<MachineName>. Only the engine’s (transition …) primitive may write this field; manual set-field on it bypasses discipline.

Action — guarded transition with effect (data)

(action "act-a"
  (machine     "Name")
  (from-states "s1")
  (trigger     TRIGGER-EXPR)
  (guard       GUARD-EXPR)            ; pure boolean
  (effect      EFFECT-EXPR)           ; mutations + (transition "s2")
  (priority    100)                   ; optional, lower runs first
  (conflict-policy "last-write-wins")) ; optional

from-states is a hard precondition checked before guard. Guards read but never write. Effects compose store primitives + kernel calls.

Reactive actions are ordered by priority ASC, name ASC — enforced at dispatch time, not at load time. If multiple matching actions write the same literal field, the default last-write-wins policy makes the last writer’s value final and the bus records the conflict in reactive_dispatch_plan. An action marked error-on-conflict that participates in a same-field conflict blocks the whole conflict group and emits reactive_conflict_blocked. It also fails closed on opaque field targets: if the field name is not a literal (e.g., it comes from a kernel result or variable), the bus refuses rather than risk a silent accidental winner.

Trigger — when an action fires

(manual)                        only via explicit Ontology.dispatch
(on EVENT-TYPE PRED …)          fires when event + predicates match
(any-of TRIGGER …)              disjunction

Event types synthesized from store mutations:

Predicates: (field N) (supertag T) (rel R) (machine M) (to S).

Design principle: trigger on the event that signals the data the action reads is in place, not the event that signals the object exists. An action consuming a body field should trigger on (on field-set (field "body") …), not on (on tagged …) — the tag event fires before the body is written.

Read primitive — guard-safe Python helper

Read primitives are Python functions injected into the DSL env by the engine. They are available in guards and effects, must not mutate the store, and must not perform external I/O. Use this boundary for deterministic computations that guards need, such as timestamp arithmetic or business-day counting.

Current engine-owned read primitives beyond store reads include date-diff and business-days-between.

Kernel — effect-only Python escape hatch

Kernels are developer-registered Python functions:

engine.register_kernel("yt-dlp-download",
                       lambda url: {"blob_key": "…", "metadata": "…"})

(call-kernel "name" arg …) is the only way for an effect to reach outside the store: HTTP, LLM, filesystem, subprocess. Agents can compose kernels via DSL but cannot define new ones. This is the safety boundary. Guards cannot call kernels.

Built-in kernels (pure, auto-installed): extract-wiki-links, extract-wiki-link-pairs, extract-hashtags, regex-find, regex-find-all, sum-field-on.

(writeback "sink-key") is the outbound sibling of call-kernel: same effect-only escape-hatch category, same registration model (engine.register_sink(key, sink), the Python/bootstrap author wires a configured SinkAdapter; the DSL author only names it), same guard-disallowed boundary. It projects $node to a typed CanonicalItem and hands it to the sink — the node’s data does not cross the DSL.

Reactive bus — events → dispatch

Every successful transaction commit fires on_commit(events). The bus classifies each event into a high-level signal and matches every loaded action’s trigger_spec against it. Matching actions are sorted by priority ASC, name ASC, then dispatched synchronously.

Before dispatching a matched group, the bus writes a reactive_dispatch_plan audit event containing the source signal, ordered actions, literal field writes, detected same-field conflicts, and the ordering rule. If error-on-conflict blocks a conflict group, the bus also writes reactive_conflict_blocked. No reactive outcome is silent — guard rejections (reactive_guard_rejected), illegal transitions (reactive_illegal_transition), and dispatch errors (reactive_dispatch_error) all emit typed events under actor system:reactive. Imperative dispatch returns ok:false with a structured error; it never raises at the caller.

Cascades nest; depth is bounded by max_depth=10.

Ontology — high-level facade (L4)

o = Ontology.open("data/data.db")    # store + engine + bus + default kernels
o.load("path/to/ontology.scm")       # → Result{ok, data:{machines,actions}, events}
o.dispatch("act-a", node_id)         # → Result{ok, data:{ran}, events, cursor}
o.manifest()                         # → Result{ok, data:{supertags,machines,...}}
o.analyze_conflicts()                # → Result{ok, data:{field_conflicts,...}}
o.events(since=cursor, kinds=[…])    # → Result{ok, data:{count}, events, cursor}
trace = o.trace("act-a", node_id)    # debug only — OntologyTrace, not Result

Every operation returns a Result envelope (ok, data, error, events, cursor). trace() is the single deliberate exception — a debug introspection tool that returns the richer OntologyTrace. See agent-contract.md for the full contract specification.

CLI mirror: runex ontology {list|load|describe|run|trace|manifest|events|check}. Every subcommand accepts --json and emits the Result envelope verbatim; exit code mirrors ok.

Datasource registry

Adapters self-describe via a module-level SPEC: DataSourceSpec in src/runex/adapters/<name>.py. The registry (adapters/registry.py) lazily imports them and exposes registry(), get(key), and all_specs(). manifest().data.datasources surfaces the registry as structured data — agents and products consume it without reading source.

DataSourceSpec.build(ctx) validates required params and returns an adapter instance. A missing required param raises a structured error naming exactly what is missing.

A spec also declares requires: tuple[str, ...] — the capability keys the adapter needs. build() constructs those via the capability broker (adapters/capabilities.py) and injects them into the adapter; the adapter performs zero direct I/O and imports no filesystem/db/http library. A capability (e.g. FileCapability) is the only code that touches the OS on a channel’s behalf — Invariant 4’s sanctioned external-I/O surface made structural rather than conventional, the same broker-injection pattern Engine._make_env uses for the DSL env, one layer out. manifest() exposes requires so the contract stays fully introspectable. All six adapters are split; the shipped capabilities are file, sqlite, command, http (mcp deferred until a consumer exists).

The seam is enforced by an executable criterion, not convention. tests/test_capability_seam.py greps each channel module for I/O verbs — open(, .read_text, .read_bytes, subprocess, sqlite3, httpx, requests., .glob(, .iterdir( — and any hit fails the suite (a leaked seam), the way tests/test_lint_gate.py makes ruff real. The line is the I/O boundary, not the typing boundary: path algebra (Path(...), .name, .parts) is data-shaping and stays in the channel; path I/O verbs (.read_text, .glob, open() move to a capability. Behaviour is preserved iff the channel emits a byte-identical CanonicalItem stream before and after a split. The capability set is derived, never designed — only capabilities an existing adapter provably exercises ship; http shipped with its first consumer (NocoDB v3), not before.

The registry is bidirectional. SinkSpec is the outbound mirror of DataSourceSpec (same lazy registry, same param/capability injection, same zero-direct-I/O seam; manifest().data.sinks exposes it). The inbound boundary is external → CanonicalItem → upsert_node_from_item; the outbound boundary is its exact mirror — project_node_to_item reads a node into a typed CanonicalItem, a SinkAdapter writes it out. The same typed boundary carries data both directions; it never degrades to JSON through the DSL.

Hot-discovery and the trust boundary

At Ontology.open() the registry also scans ~/.runex/extensions/{kernels,sources,sinks}/*.py and registers any module exposing a KERNELS dict (capabilities) or a module-level SPEC (channels). Dropping a file in and restarting the one process adds a channel or kernel — no core edit, no rebuild. This makes runtime extensibility literal for I/O: a channel break (Notion renames a field, WeChat bumps its schema) is a contained, hot-swappable failure in an extension, never a core regression. The core is the stable plug socket; the plugs are extensions — the way an Emacs package is to the C core, or a mod is to a game engine.

Hot-discovery means arbitrary Python executes at open(). This is sound only under runex’s single-user-local model: the extension directory shares the .db file’s trust principal (the Emacs init.el model). The moment a shared or product deployment loads an extension directory it does not own, this is remote code execution. A product on top must not widen this without an isolation story (Invariant 12).

Ingest boundary — Connector, Recipe, Job, Preset

The repository currently exposes two parallel catalogs:

• runnable business DataSourceSpecs such as obsidian-thought and wechat-session
• transport-level CoreDataSourceSpec metadata such as file-markdown, sqlite-query, and http-json

This is transitional. CoreDataSourceSpec is currently descriptive only; it cannot enumerate records. DataSourceSpec is runnable, but combines transport, semantic mapping, watch behavior, and target Supertag in one object.

The target ingest contract, following ADR-0008, is:

Capability → Connector → RawRecord → Recipe → CanonicalItem → Store
                              ╲          ╱
                               IngestConnection

• Capability owns the sanctioned I/O verb (file, http, sqlite, command).
• Connector owns external record enumeration and observation. It emits a declared RawRecord shape and knows no target Supertag.
• Recipe owns semantic interpretation. It accepts one or more RawRecord shapes, validates against manifest(), and emits typed CanonicalItems. It performs no I/O.
• IngestConnection binds a configured Connector, Recipe, run mode, cursor, and deletion policy.
• Preset is a named preconfigured Connection template. Existing DataSourceSpecs remain compatibility presets during migration.

These are separate manifest collections. Transport keys and business preset keys are not peers and must not share one flat namespace.

A RawRecord is a change envelope, not an untyped payload:

source_uri · record_shape · op(upsert/delete) · observed_at · version
payload · metadata · checkpoint

This lets file watches, incremental SQLite reads, command pulls, and future webhooks enter the same Job runner. Recipes consume record_shape + payload; they do not receive filesystem Paths, HTTP requests, webhook acknowledgments, or connector cursors as I/O handles.

A saved IngestConnection is itself an ordinary ontology node. Connector and Recipe definitions are discoverable runtime specs; a user’s configured vault path, mode, cursor, and deletion policy are durable graph state, not a new configuration table.

For v0.2 the first complete implementation is intentionally Obsidian-centered:

obsidian-markdown Connector
  → RawRecord {
      path, vault, frontmatter, body, wikilinks, embeds,
      attachments, content_hash, mtime
    }
  → ontology-frontmatter Recipe
      type/supertag resolved from manifest
      schema fields and ref declarations mapped generically
  → CanonicalItem

The Connector understands Obsidian-flavored Markdown syntax and file lifecycle, but not Thought, Task, or Customer. The Recipe understands ontology schema, identity, fields, and links, but not filesystem watching.

The existing obsidian-thought adapter becomes a legacy Preset:

obsidian-thought ~= obsidian-markdown + legacy-thought recipe + watch defaults

Future Feishu support follows the same seam:

feishu-webhook Connector → RawRecord → business/ontology Recipe

Webhook acknowledgment, signature verification, cursoring, and retry belong to the Connector/Job; ontology mapping remains in the Recipe. Feishu work is not part of Phase 1, but Phase 1 must not hard-code file polling into Recipe or ingest orchestration.

Phase 1 does not implement every speculative core datasource. Only the Obsidian connector is promoted to executable status. file-json, file-csv, sqlite-query, http-json, and command-json remain descriptive candidates until a real Job dogfoods each interface.

Target extension: proactive behavior runtime

The next architecture increment turns the existing synchronous ontology runtime into a host for long-running proactive behavior without making the reactive bus asynchronous.

It does not add a fifth persistence model or a queue subsystem. The core layering remains unchanged:

Outside core    Human shells · Worker host · Agent runtimes · Channels
                              │
L4 Interface    CLI · Ontology facade · Result / manifest / events
                              │
L3 Semantics    DSL eval · Loader · Engine · Kernels · ReactiveBus
                              │
L2 Graph        nodes · supertags · fields · links · identities · tx_log
                              │
L1 Storage      SQLite

The worker host is an L4 consumer, like the CLI or a product shell. It is not inserted between L3 and L2 and does not receive direct ownership of SQLite.

The proactive flow is:

 External facts / human edits
            │
            ▼
 Connector → Recipe → CanonicalItem → Store
                                      │
                                      ▼
                         synchronous ReactiveBus
                         guard → signal → transition
                                      │
                            creates/updates WorkItem
                                      │
                                      ▼
                         durable cursor + worker host
                         claim → query fresh state
                         → run Agent / tool / channel
                                      │
                                      ▼
                         validated command/result writeback
                                      │
                                      ▼
                         synchronous ReactiveBus reacts
                                      │
                         Approval / Conflict / Task / Sink

Everything durable remains a node

The ontology rule is literal: every durable object is a node.

The proactive framework introduces no tasks, jobs, approvals, assertions, or conflicts tables. It models each concept through the existing graph substrate:

This exactly follows the existing implementation: MachineDefinition, ActionDefinition, and KernelRef are already regular nodes with internal supertags, read back into runtime dataclasses by L3.

Task and WorkItem are intentionally different ontology types. A Task is work a human or business process cares about. A WorkItem is a durable request for a worker to perform an external effect. Either can link to the other; neither gets a dedicated L1 table.

WorkItem — an ontology node for durable intent

A slow effect is represented as a typed WorkItem node, not an in-memory callback and not a blocking kernel call. Its lifecycle is:

pending → claimed → running → succeeded
                    ├──────→ retry_wait → pending
                    └──────→ failed

The signal action writes only the intent and stable references needed to reconstruct the work. The worker claims the item with a lease, then queries the referenced nodes again before acting. Event payloads carry references (node_id, work_item_id, action_name, source_event_cursor), not copied subgraphs.

This preserves the answer to “what is true now?”: the Store is queried at execution time rather than trusting a stale event snapshot.

The fields that make a WorkItem claimable (work_key, handler, subject, attempts, lease owner, lease expiry, timestamps) are ordinary typed fields. Its references to source event, subject, result, proposal, and artifacts are ordinary links.

The only possible L2 addition: generic conditional mutation

The current Store writes each field in its own transaction. That is sufficient for deterministic single-threaded cascades, but two worker processes cannot safely implement “claim this pending node if it is still pending” as a read-then-write sequence.

If multi-worker claiming is included, L2 may gain one generic primitive:

conditional_mutate_node(node_id, expected_fields, field_writes)

It atomically checks current field values and applies field writes plus tx_log events in the existing tables. It has no knowledge of WorkItem, lease, Agent, Task, or approval. No L1 table or business-specific column is added. L3/L4 may use this primitive for WorkItem claims, compare-and-set business rules, or other ontology objects.

For a single-worker MVP this primitive can be deferred; the ontology model is the same either way.

Worker host — asynchronous work at the L4 edge

The worker host is a long-lived process outside the core that consumes L4. It reads the persisted event stream or queries claimable WorkItem nodes, but all graph mutations go through the same Ontology/facade contract.

Responsibilities:

• persist and resume its event cursor
• atomically claim work with an owner and lease expiry
• dispatch a named handler (Agent, channel, or deterministic tool)
• validate structured results against the current manifest
• write success, failure, retry, and provenance back to the graph
• recover abandoned work after process restart

The worker host does not own business branching. Machines, guards, approval requirements, and follow-up reactions remain runtime-loaded ontology data.

Approval and conflict are states, not blocked threads

Human-in-the-loop behavior uses typed objects and state transitions:

Proposal → pending_review → approved / rejected / expired
Conflict → unresolved     → accepted / superseded / dismissed

No process waits for a person. A product shell, Obsidian file, or external channel edits the review object; that write re-enters the ordinary reactive path. The human remains sovereign over destructive, externally visible, or ambiguous actions.

Obsidian is the human-sovereign mirror

The proactive loop is not human-usable if state only changes inside SQLite. For v0.2, Obsidian is a required bidirectional shell, not an optional future projection.

The responsibility split:

• SCM owns type definitions (fields, machines, actions) — no I/O concepts
• Guard actions (e.g. correct_task_status) protect machine-state fields from being overwritten by stale file values during re-ingest
• Ingest writes all fields from the file into the graph without filtering
• Sink writes all non-internal fields back to the file without filtering
• StoreWatcher dispatches project_ by naming convention

The mirror loop is:

Human/Agent edits Markdown
  → FileWatcher + Frontmatter ingest
  → CanonicalItem → Store → ReactiveBus
  → guard action detects stale fields (e.g. state disagreement)
  → corrects to machine-authoritative value
  → writeback to file
  → FileWatcher suppresses the matching machine echo

Before writeback, project_* actions compare the current file hash against _sync_file_hash baseline via (call-kernel "file-hash" …). If the human changed the file since the last baseline, writeback fails closed and creates a Conflict node.

The primary approval interface follows the same loop: an Approval/Proposal node is projected as an Obsidian note; the human changes a Frontmatter value; ingest advances the machine. CLI approval is only an operator fallback.

Agent role

An Agent is an effect handler, not the top-level orchestrator. It receives a bounded task, fresh referenced context, an allowlisted tool set, and a machine-readable output contract. Its output is a proposal or validated command batch; ontology rules decide whether the result is applied directly, sent for approval, or rejected.

Invariants

These hold across all changes; violating any of them is a regression.

1. L3 → L2 is one-way. Removing src/runex/{ontology,dsl}/ leaves L2 + adapters + pipeline functional (and __state__* fields become inert text).
2. No business names in framework code, extended to no channel names in core. Bookmark, Thought, etc. appear only in config/ and tests/; WeChat, Notion, etc. live only in extension specs, never in the engine. Verifiable by grep.
3. __state__* fields are write-protected by convention. Only (transition …) writes them. Direct writes bypass the discipline that makes state machines meaningful.
4. Read primitives and kernels are separate boundaries. Guards may use read primitives only. Effects can mutate L2 through store primitives or invoke a registered kernel. No eval, no Python interop, no untrusted import.
5. Reactive order is deterministic. priority ASC, name ASC is enforced at the point of dispatch. Same input event → same cascade. Reloading actions in a different order does not change outcome.
6. error-on-conflict fails closed. If an action asserts a hard guarantee and the bus cannot prove it (same-field co-writer present, or field target is opaque), the whole conflict group is blocked. Ambiguity is refused loudly, never resolved by accident.
7. Slow work never runs on the reactive bus. Agent calls, network I/O, subprocesses, downloads, transcription, and other unbounded work execute only in a worker host after durable intent has been recorded.
8. Work is reference-first and recoverable. A worker re-queries current state from stable IDs, claims work with a lease, and can resume after a crash without relying on an in-memory callback or copied subgraph.
9. Human review is modeled as state. Approval never blocks a thread or transaction; it suspends progress in the ontology until a later write advances the review object.
10. No business-specific persistence tables. New durable concepts are supertagged nodes with fields and links. L2 additions, if any, must be domain-neutral graph/store primitives reusable by arbitrary ontology types.
11. Projected human edits fail closed. A machine write must not overwrite a file changed since its last ingested baseline. Checksum suppresses echoes; ownership/version checks detect conflicts.
12. The extension directory shares the .db’s trust principal. Hot-loaded capability kernels and channel specs are the sole sanctioned external-I/O surface, demarcated from the pure trusted core. This holds only under the single-user-local model; a shared or product deployment must not load an extension directory it does not own without an isolation story.

Extending the engine

The DSL covers everything except external I/O. A .scm file is sufficient for new lifecycles; no Python edit, no restart.

Pipeline ingest

Adapters parse external sources (Obsidian vaults, chat JSONLs, …) into CanonicalItem:

@dataclass
class CanonicalItem:
    source_uri:  str
    source_type: str
    supertag:    str
    name:        str
    fields:      dict[str, tuple[type, value]]
    links:       list[CanonicalLink]
    extra_tags:  list[str]
    description: str | None
    attachments: list[Attachment]

upsert_node_from_item(store, item) upserts the item via L2 primitives: get-or-create by natural_key → tag → set fields → resolve RefByName targets → set links → store attachments. Each step emits events; the reactive bus handles everything downstream.

The adapter doesn’t know about ontology. The pipeline doesn’t know about ontology. The cascade is the connection.

Where to look

Expected scale & constraints

User profile

Single-person heavy knowledge worker + 1–3 agent collaborators. Not SaaS, not multi-tenant, not team workspace. This assumption drives every bound below.

Data volume (per user)

Active agent extraction (WorkItem → proposal → approval) adds roughly 1.5–2× to daily write volume, but even the peak stays under 500 tx/day.

Ontology size

Actions have simple trigger patterns (field-set, node-created, manual, scheduled) — no cross-entity join conditions in trigger spec. Guard/effect expressions are evaluated per-node, per-action, ~1ms each.

Bottleneck ranking (highest concern first)

1. writeback file I/O — each reactive cascade tail writes back to .md files. Disk writes are 100–1000× slower than SQLite. This is the real latency wall, not rule matching.
2. tx_log unbounded growth — ~200 appends/day, no retention policy. After one year ~70k rows; full scans degrade without an index on ts or periodic pruning.
3. reactive cascade depth — a single action triggering 5+ nested cascades, each writing files and re-triggering ingest, risks oscillation. Protected by max_depth=10 and echo suppression, but cascades beyond ~3 should be reviewed.
4. FTS5 beyond 100k nodes — trigram tokeniser is resilient, but at this scale consider search_longtext query planning.
5. trigger matching — even at 200 actions, naive prefix matching on trigger specs stays well under 1ms. No optimisation needed.

Why not a Rete-based rule engine

Rete forward-chaining optimises cross-fact join matching (the classic “1000+ rules, many share subconditions” case). runex has ~100 total actions, all ECA (event-condition-action) with per-node guard evaluation — no multi-fact join patterns exist in the trigger or guard layer. Bottlenecks are on I/O and cascade depth, not rule matching. Rete would add complexity (compiled network, cache invalidation, no hot-replace of rules stored as data) for zero gain at this scale.

If the bottleneck profile ever changes (10× actions, cross-entity trigger patterns emerge), the right response is not Rete — it’s a read cache layer and/or CQRS decomposition of the busy write path.

When this engine is the wrong fit

• Sub-millisecond latency budgets (bus is synchronous, single-threaded)
• High-volume OLTP (single SQLite process)
• Strongly-typed RPC schemas (CanonicalItem is loose by design)
• General-purpose high-throughput workflow orchestration; the durable work protocol is deliberately scoped to Agent/tool effects around ontology state

When these constraints bite, reach for a proper workflow engine or stream processor instead.

Future direction

L1 + L2 + L3 are a generic graph-ontology substrate that doesn’t know about runex’s specific domain. The plan is to extract them as a separate pip package once a second consumer appears, or a multi-graph use case emerges. The invariants above are what keep that extraction cheap.

Distribution shape (2026-07-06). runex today ships as a Nuitka onefile binary inside runex-macos-arm64.zip. src/runex/ is the build-time source-of-truth; it is no longer a pip install runex artifact. See the 2026-07-06 entry in changelog.md for the release-channel shape and migration story.