Troubleshooting

CLI failures include a human-readable message. Add --json where supported to receive structured fields suitable for automation. Every HTTP error the server answers, an unknown route, a method the route does not serve and a malformed body, query string or path parameter included, uses a JSON body with error, code, disposition and optional details for conflicts, limits, Blob ranges, external sources, or recovery. A malformed request is 400, a body without a JSON content type 415. MCP tool calls report the same body beside the HTTP status (MCP).

Do not parse human-readable error text when a structured field is present.

Server discovery

Remote graph commands first make an anonymous HEAD /healthz request to the configured service root. Discovery sends neither the data bearer token nor URL Basic-auth credentials; reverse proxies must allow that public probe. A timeout or connection failure is reported with a credential-safe cause category and means the data request was not sent. HTTP discovery refusals retain their status.

HTTP errors

Each code is sent under the statuses ErrorCode in openapi.json names beside its meaning: one, except branch_start_refusal, which is 409, or 410 for not_retained.

A GET or HEAD /blob 500 logs error_kind="blob_pre_header_internal" with a stage. target is a failure resolving a snapshot target for a policy-gated request. cell is any failure of the engine's Blob read, including the branch or snapshot resolution the engine does itself. transport is the server's own refusal before headers, logged with error_variant="unclassified". A managed Blob body that fails after the headers logs the byte range under one of blob_payload_read (with the error class), blob_payload_short_read (with the returned and expected byte counts) or blob_payload_permit_closed.

A change route (GET /changes, POST /changes/baseline, GET /commits/{commit_id}/changes) answers an internal failure with a fixed 500 and logs error_kind="change_route_internal" with error_variant, storage_kind and manifest_kind. A change route that requires recovery logs the operation_id the 503 reports. An export or change-baseline response that fails after its 200 headers ends with an incomplete body and logs error_kind="served_stream_failed" with stream (export or change_baseline) and the same error class. None of these logs carries the error's message, which can hold object URIs or credentials.

A graph-head 412 includes precondition_failure with expected and, when available, actual. A change-feed 410 includes change_feed_gap; retrying the same cursor cannot recover its missing history.

A 400 that refuses a query includes diagnostic: the stable code (Q… parse, T… typecheck), position (line, column, byte) for a parse refusal or stage (and expression when known) for a later one, expected, and fix when one exists. Act on the fix; a retry of the same source fails the same way. See Diagnostics.

Failed data-write commands

Every server error body, and every CLI data-write and read failure, carries a disposition: effects is none, partial or unknown, and action is fix, refresh, retry, recover or reconcile. The server decides it for the request it answered by the first row that matches:

A refusal's own action follows its code: recover for a recovery intent or a full-text index to build, retry for too_many_requests, service_unavailable, graph_unavailable and request_timeout, fix for unauthorized, bad_request, api_contract_mismatch, actor_mismatch, not_found, method_not_allowed, unattached_cannot_publish, branch_start_refusal, payload_too_large, unsupported_media_type and range_not_satisfiable, and refresh for the rest: forbidden, precondition_failed, conflict, external_blob_source, change_feed_gap and internal. A write the server could not settle answers 500.

The CLI reports the server's disposition when that request was the command's only effectful one, and otherwise its own account of the whole command, so a lost response is unknown with reconcile. A read command changes nothing, so its failure always has effects: none. JSON mode writes the failure to stdout, or to stderr when part of the result, such as a streamed page, already reached stdout. Stdout then keeps the bytes already written, which can end inside an unfinished object, as for diff, branch diff, commit changes and changes poll with --json when they follow every page, and the failure document is on stderr. Human mode prints the disposition line on stderr. A target refusal on these commands exits 1 with action: fix. HTTP failures also retain http_status and, when present, the original retry_after string.

This includes commit publish and commit drop, as well as their mutate statement forms. If a response or result output is lost, inspect the commit's current state before another attempt; an output failure does not undo the write. The same applies to load, blob put, blob clear and schema apply: a closed output pipe or failed result flush can follow a successful write. The CLI exits with a failure and does not repeat the operation. If stdout cannot carry a JSON failure, stderr reports the undelivered diagnostic and its disposition, and the exit still follows it.

For the data commands the CLI classifies (queries, reads, load, mutate, branch, commit, schema and Blob commands, and aliases), the exit follows from the disposition:

Maintenance commands (init, upgrade, optimize, rebuild-full-text-indexes, repair, cleanup, schema upgrade-system-columns), cluster, policy, target, credentials, mcp and the managed commands keep their own exits for now: their exit 1 does not by itself say that nothing changed, so read their report; mcp answers each relayed request with its own disposition.

A served write refused with 401 unauthorized, 403 forbidden or 400 api_contract_mismatch exits 1 with effects: none, and action: refresh for the policy denial and fix for the others. The server answers these before it admits the write: bearer authentication, then the API contract header, then policy. Fix the credential, the policy grant or the client before running the command again. If a server target with auth: {provider: anonymous} receives a 401, the CLI adds missing_credential with target, auth: "anonymous" and a setup hint. A missing credential for a token or exec provider refuses locally before any HTTP request. The server's error text is unchanged. The status counts only with its matching code in a response that carries the server's API contract header: a bare 403 from a proxy stays unknown with reconcile, and a refusal that also reports recovery work stays recover.

fix means correct the credential, input, path or client; sent unchanged, the command fails the same way. refresh means read the current state and decide again; it does not grant automatic replay. partial effects mean an earlier step stands although the failing one wrote nothing; read the target before deciding. recover means finish the reported recovery obligation. reconcile means establish the original operation's outcome before another attempt. Only retry with exit 75 grants the bounded retry described above; the CLI performs no automatic retry.

The disposition covers the entire command. A later refused subrequest cannot undo an earlier branch creation, load or publication. An embedded write's own refusal says what the engine did, projected as a server would project it: a policy refusal precedes every publication, a load --from fork included, so it has effects: none on both planes. The engine marks an error that follows a fork which did publish: a typed refusal then has effects: partial on both planes, and an error the engine cannot prove wrote nothing has unknown. A load --from onto a branch that already exists forks nothing. Any later failure, such as rendering a committed receipt, stays unknown. Disconnecting an admitted HTTP write does not cancel it: the server keeps running the original operation, and a lost response can leave the caller's outcome unknown. There is no durable request-result lookup in this release.

Conflicts

A 409 is not one universal retry signal:

• A read-set or version conflict means another writer changed an input. Inspect the whole-command disposition and re-read before deciding on a new operation.
• A key conflict means strict insertion found an existing ID. Change the ID or use merge/upsert semantics; repeating the strict insert is not useful.
• A merge conflict requires an explicit resolution on one branch before merging again.
• A full-text incompatibility includes full_text_index_rebuild_required: { "index": "…", "reason": "…" } with code: "conflict". This condition persists until an operator rebuilds the affected branch's indexes; do not automatically retry the same query. Follow the full-text upgrade procedure.
• A full-text index with no part at the query's snapshot includes full_text_index_required: { "index": "Type.property" } with code: "conflict". It persists until the branch's index is built (omnigraph optimize on main, omnigraph rebuild-full-text-indexes --branch <branch> elsewhere); do not retry the same query. See Search indexes.
• “Already initialized” means the target already contains a graph. Choose a new root or deliberately use the command's destructive option when appropriate.

Writes are atomic at the graph-commit boundary. A normal validation, conflict, or limit error does not mean that a subset became visible. When recovery_required identifies an already published schema change, its table pins and accepted schema contract are durable together; only the process view may still need refresh. Reopen the graph read-write or restart its server rather than replaying the change. Legacy recovery sidecars require the original release, as described under maintenance failures.

Storage-format mismatch

If a graph was written by a different storage-format generation, the binary refuses to open it and names the required release line. Follow the qualified explicit routes in Upgrading to preserve graph identity and history; normal open does not migrate or reset the graph.

Do not edit internal metadata or copy files from individual backing datasets between graph roots.

Cluster failures

• Run cluster validate before plan or apply.
• Apply refuses existing unmanaged roots and missing managed graphs that remain declared. Restore authoritative data to keep a missing graph, or remove its declaration to delete it; see deployment boundaries.
• A server restarted after a crash takes over its predecessor's leased S3 lock once the predecessor's heartbeats have been silent for a lease and a half (45 seconds by default). A lock with no lease (on a local or Azure root, or an older binary's) requires prior-owner and accepted-I/O quiescence, exclusion of other admissions/unlocks, and its exact ID; follow ownership transfer.
• Directory boot reads cluster.yaml to resolve storage, but served graph, query, and policy resources come from applied state. Submit configuration changes with cluster apply --target for live activation.
• By default one graph that cannot open is quarantined while healthy graphs serve. Use --require-all-graphs when partial startup is unacceptable.
• external_blob_base_overlaps_storage_root: an external_blobs base lies inside, or contains, the cluster storage root that holds every graph and the applied state. For a new declaration, move the base to a sibling prefix before applying. Correct existing external-Blob bindings in the desired configuration and apply them. An invalid applied binding keeps its graph blocked; if every graph is blocked, startup fails with cluster_no_healthy_graphs. See External Blob references.
• external_blob_storage_root_uncomparable: the cluster storage root is spelled with a path component an external Blob base cannot express (an empty component, or a percent sign in a local path), so a base of the same storage kind cannot be proven to lie outside it. Moving the base does not help: correct the declaration or remove the allow-list before applying. Graph roots remain fixed. Validation refuses the base, and an invalid applied binding keeps the graph blocked, as for an overlap.

See Operating a cluster.

Maintenance failures

If optimize, repair, cleanup, rebuild-full-text-indexes or upgrade cannot write or flush its output, the CLI returns a failure without repeating the operation. An output failure does not undo completed work. Inspect the graph's state before another attempt; for an interrupted upgrade, follow the upgrade recovery procedure.

• Recovery required: reopen the graph read-write or restart its server. A graph carrying a sidecar from a release before 0.12 must first be opened read-write with that release (omnigraph snapshot <graph> is such an open).
• Foreign drift: repair reports Lance commits above a table's last linear version as foreign_drift; no read or write uses them and no command adopts them (below).
• Cleanup row with error: the collector's trace of that table did not finish, so nothing of that table was deleted; fix the named cause and rerun. Other tables are collected and the command exits 0.
• Azure admission failure: inspect the lease owner before using the admission tool's break-glass flow.

Foreign drift

omnigraph repair classifies a graph table as foreign_drift when its Lance linear history carries commits above the table's recorded last linear version (omnigraph.last_linear_version on the registration). Every OmniGraph write is a detached commit that a graph commit pins, so such a commit came from something else writing the table directory directly. The condition is per table.

• Queries, mutations, loads, merges, index builds, schema apply, optimize and cleanup are unaffected: none of them resolves the linear HEAD.
• repair prints the last linear version, the HEAD and the count of foreign versions, takes no action and exits 0. The removed confirmation flags are rejected; repair never adopts the foreign commit.
• cleanup never deletes a foreign version or its files; the table's result row lists them under foreign_versions.

There is no command to run before writing: the write path has no precondition on a table's linear history. To discard the foreign commits, export the graph and load it into a new one.

See Maintenance and Deployment.

Useful diagnostics

omnigraph version
omnigraph snapshot --target ./graph.omni --json
omnigraph commit list --target ./graph.omni --json
omnigraph cluster status --target ./company-brain --json
omnigraph repair --target ./graph.omni --json

For server requests, retain the HTTP status, response body, request path, timestamp, and server logs. Redact bearer tokens and storage credentials before sharing diagnostics.