# Maintenance

Source: https://www.omnigraph.dev/docs/operations/maintenance

OmniGraph provides four direct-storage maintenance commands:

* `optimize` compacts data and reconciles declared indexes on `main`.
* `index build` brings one branch's declared indexes up to its rows.
* `index rebuild --kind full-text` replaces full-text indexes on one branch.
* `cleanup` permanently removes unretained table versions and unused table
  forks left by branches created before storage format v11.

They do not run through the HTTP server; a cluster server runs optimize as a
[recorded operation](https://www.omnigraph.dev/docs/operations/recorded-operations.md). Address a graph directly, or from a cluster root:

```bash
omnigraph optimize --target ./graph.omni
omnigraph optimize --target ./cluster-root --graph knowledge
```

Follow the command-specific concurrency requirements below. Registered direct
targets accept local files only; an inline [S3 cluster root](https://www.omnigraph.dev/docs/operations/s3-cluster-roots.md)
is the one cloud-direct exception. See
[operator configuration](https://www.omnigraph.dev/docs/cli/targets.md#transport-and-capability-boundaries).

Each command acts as its `--as` actor, and the commit it publishes records
that actor. On a cluster graph the CLI installs the policy the cluster applied
to it, and an embedded host may install its own; either checks the
[policy actions](https://www.omnigraph.dev/docs/operations/policy.md#actions) before any effect: `maintain` for optimize,
index builds and the full-text rebuild, and `reclaim` for cleanup. A standalone
graph has no policy. The actor is self-declared, so storage permissions remain
the trust boundary.

## Optimize

```bash
omnigraph optimize --target ./graph.omni
omnigraph optimize --target ./graph.omni --json
```

Optimize rewrites small fragments into fewer larger fragments, rewrites every
fragment that still stores a dropped property's values, rebuilds each scalar or
vector index whose coverage lags behind appended rows, and builds missing
declared indexes that are ready to build. It works on `main`. `index build`
does only the index part, on any branch including `main`; compaction, the
dropped-value rewrite and the traversal artifact stay with optimize. Optimize
does not delete old versions or collect unused table forks. Use `cleanup` for
storage reclamation.

Dropping a property removes it from the schema without rewriting its table (see
[schema changes](https://www.omnigraph.dev/docs/schema.md#schema-changes)), so its values stay in the
table's data files. Every optimize rewrites each fragment that still holds them,
whatever its size, copying only the remaining properties; the dropped values are
not read. A `cleanup` then deletes the old files, Blob values included, once
nothing it retains references them: no commit its retention keeps, no live
branch created from a commit before that optimize, and no tag naming such a
commit. Merging a branch does not release its files; deleting it does. A table
holding no dropped values and nothing else to compact is left alone.

Each table's work is staged as detached Lance versions of the table's current
pin and published in one graph commit, like any other write. A run that fails
before that commit leaves the graph unchanged and no recovery state; after it
the published pins are the tables' versions, with nothing left to finish.
Optimize runs beside live writers: a write that lands on a table while its
compaction is staged fails the run with a read-set conflict, and the next run
re-plans from the new state. A write prepared before an optimize published goes
ahead, or prepares again once, holding writes off, if it changes a row the run
moved; none reports a conflict for it. Lance compacts neighbouring fragments only when the
same indexes cover them, so a table whose index coverage was uneven before a run
may coalesce fully only on the next run, after the rebuilt coverage is in place.

Compacting a table with Blob properties reads each managed Blob value into
memory. Optimize sizes each compaction batch from its largest row, whatever
`LANCE_DEFAULT_BATCH_SIZE` says, so one batch holds at most 32 MiB of managed
Blob payload; a larger row is read whole, external references are not read,
and the writer's own copies come on top. A Blob descriptor the decoder rejects
refuses the compaction before that table is rewritten. Optimize plans up to
`OMNIGRAPH_MAINTENANCE_CONCURRENCY` tables ahead (default 8) and rewrites up to
that many at once, with that many compaction tasks, index folds and builds in
all (a Blob table's tasks one at a time): budget for that many Blob tables.

### Bounding a run

Writes to `main` wait while optimize holds its writer gates. A run given
`--max-gate-hold 30s` starts no table after that long except the first: the
rest are `deferred`, finished tables publish, and a writer waits at most the
bound plus one rewrite. Tables run in type-key order; `resume_at` names the
first deferred one, and `--tables` runs only the listed tables, in order. Each
table's `outcome` is `optimized`, `unchanged`, `deferred` or
`failed` (with its `error`; the other tables still publish). The report adds
`input_commit`, `graph_commit_id`, `gate_hold_ms` and the `configuration_clear`
and `artifact` outcomes, which never fail a published run.

Optimize also persists the traversal-adjacency artifact
(`__graph_index/csr-current.bin`), which cold traversal builds load instead of
scanning every edge dataset. It is derived: every load is verified and falls
back to the in-memory build, so a stale copy or a failed write (the `artifact`
outcome) costs only its speedup. Optimize builds it after releasing the writer
gates, and only when the run advanced an edge dataset or no artifact exists; a
build skipped for its size is counted again, up to the limit, by each such run.

Existing full-text indexes are preserved rather than incrementally merged.
Unindexed rows remain searchable, but a growing uncovered tail can cost more to
scan. Use `index rebuild --kind full-text` to refresh that coverage or migrate
an old analyzer generation. Optimize reports uncovered full-text coverage under
`pending_indexes` with this remedy. Deferred coverage alone creates no graph
commit or maintenance work. A vector index whose property has no usable vectors
also stays pending; run optimize again after loading or generating vectors.

## Build indexes

```bash
omnigraph index build --target ./graph.omni --branch review
omnigraph index build --target ./cluster-root --graph knowledge \
  --branch review --json
```

On any branch, `index build` builds every declared index a table lacks, such as
those of a type whose first rows were loaded there, and rebuilds whole each
BTREE or vector index that leaves rows uncovered. Every table publishes in one
graph commit on that branch; other branches and snapshots keep their indexes,
and a branch with nothing to do publishes nothing. The default branch is `main`.
As with optimize, a write that lands on a table while its batch is staged fails
the run with a read-set conflict; run the command again. A vector property with
no vectors yet and full-text coverage stay pending: only `index rebuild --kind
full-text` refreshes a full-text index's coverage, and queries match uncovered
rows with the index's analyzer meanwhile.

JSON reports `branch`, `graph_commit_id` (null for a no-op), `indexes` with each
`type_key`, `property` (the indexed column, `__id` for the identity index),
`kind` (`btree`, `full_text` or `vector`) and `action` (`built` or `refreshed`),
and `pending_indexes` with each `type_key`, `property` and `reason`.

## Rebuild full-text indexes

```bash
omnigraph index rebuild --kind full-text --target ./graph.omni --branch main
omnigraph index rebuild --kind full-text --target ./cluster-root \
  --graph knowledge --branch review --json
```

`--kind` is required. The deprecated spelling `rebuild-full-text-indexes` still
works for one release. Under a policy, the rebuild needs `maintain` on the
selected branch.

The command fully replaces text-search indexes from current entities using the
engine's default English analyzer, including already indexed entities. It builds
declared node full-text indexes and replaces existing physical full-text indexes
on nodes or edges; it does not create edge-property indexes from declarations.
External custom tokenizer settings are not preserved. Completed rebuilds warn
on stderr, or in the JSON `warnings` array, that those settings were replaced.

All rebuilt datasets become visible in one graph commit. JSON reports `branch`,
`graph_commit_id`, and `rebuilt_indexes` with each `type_key` and `property`.
Success means the selected branch's planned rebuild was published, or an explicit
no-op with an empty index list, null commit, and empty `warnings` array. The default
branch is `main`.

Other branches and historical snapshots are not rewritten. Rebuild every live
branch that needs full-text search; restoring an older snapshot may require
rebuilding again. The operation does not regenerate embeddings or alter entity
values. Stop overlapping writers and preserve a backup before an upgrade; see
[full-text upgrades](https://www.omnigraph.dev/docs/operations/upgrade.md#full-text-index-upgrade).
Unknown legacy or external index kinds require a controlled migration, not a
guessed replacement; see [unsupported inventory](https://www.omnigraph.dev/docs/operations/upgrade.md#unsupported-index-inventory).

## Cleanup

Cleanup is a tracing collector over the graph's table storage. Every table write
is a detached Lance commit that a graph commit pins, so what a table keeps is
decided by which graph commits the run retains. Per live branch the run computes
the graph commits its policy keeps, takes the table versions those commits pin
as roots, marks every file the root manifests reference (base and overlay data
files, deletion files, index directories, transaction files), and deletes
unretained table versions and the files only they reference. Selected merge
bases and native tagged snapshots are retained in addition to the versions
selected by the policy. Historical table lifetimes participate even after a type
is dropped or its name is re-added under a new identity. Blob sidecars follow
their parent data file's reachability.

`omnigraph cleanup` is the only cleanup to run against a graph root. Lance's
own cleanup (`cleanup_old_versions`, and every vacuum or retention tool built
on it) does not trace OmniGraph's detached table versions. On Lance 13 it can
delete files used only by detached versions once those files are older than
seven days, leaving the manifests present but affected snapshots unreadable.
This can affect the current graph or a recent commit that inherits an old file;
files shared with retained attached versions remain protected. The same risk
applies to the graph's catalog dataset, whose unpublished proposals and
unattached snapshots are detached versions. If an external cleanup has run,
stop writes and check for missing files; restore the whole root from a verified
backup if data was deleted.

A table version no graph commit ever named is unpublished staging: a write
in flight, or one that failed before its publication. Unless a native tag
protects it, it is deleted when the branch incarnation and graph head its
commit recorded can no longer be published against: its valid incarnation is absent from a complete inventory taken
after the staged manifests were listed, or the captured head moved past its
publication without naming it. Missing or malformed ownership is kept.
Cleanup validates its complete branch and tag capture after all table
inventories; an overlapping change refuses the plan before deletion. Cleanup also removes table forks
left by branches created before storage format v11 once nothing references
them and the graph branch incarnation in their name is gone. Branch deletion
and later writes leave reclamation to this command. Without `--confirm`, the
CLI only echoes the requested retention policy and exits before opening the
graph; it does not enumerate candidates:

```bash
omnigraph cleanup --keep 10 --older-than 7d --target ./graph.omni
```

Run the reviewed policy with `--confirm`:

```bash
omnigraph cleanup --keep 10 --older-than 7d --confirm --target ./graph.omni
```

At least one retention option is required:

| Option                  | Meaning                                                                                                                                                                                            |
| ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--keep N`              | Retain the newest `N` graph commits of every live branch, and every table version they pin                                                                                                         |
| `--older-than DURATION` | Also retain every graph commit newer than the cutoff, on every live branch; and collect a pre-v11 table fork only once every data and branch-reference object of the fork is older than the cutoff |

A graph commit survives when either option retains it. The current HEAD of
every live branch is always retained, and a named branch also keeps its
oldest commit, the copy of the parent's commit it was created from, while
the branch lives; for a branch started at a past commit, that is the commit
it names. A commit cleanup no longer retains starts no branch
([start a branch at a commit](https://www.omnigraph.dev/docs/branching.md#start-a-branch-at-a-commit)).
Cleanup also retains the selected merge bases between
current branch heads and accepted merge inputs. Native tags protect the exact
snapshots they name. A tagged graph snapshot retains all its table versions; a
tagged table snapshot retains that version, including a detached version or a
version on main.
`--keep` counts graph commits on that branch, never Lance versions of a dataset: `--keep 10` keeps the
last ten graph commits of each live branch readable, with every table version one of them pins, whatever
their age. A pre-v11 fork that no registration references is collected once the graph branch incarnation
in its name is gone: immediately with `--keep` alone, and with `--older-than` only after every object of
the fork is older than the cutoff. Native tags, readable graph snapshots and required native ancestors
also protect forks. Choose a policy that matches your rollback and audit needs. Planning lists each live
branch's graph commits. `--keep` alone reads only the commits it retains and the merge bases, so an
unreadable commit it drops no longer stops it; `--older-than` also reads the time of every commit not
otherwise retained, up to 64 at once on object storage, and stops on one it cannot read.

Before cleanup:

1. stop long-lived Blob readers and readers of snapshots the policy removes;
2. verify important branches and snapshots;
3. make or verify a backup/export;
4. review the retention policy against the snapshots and branches you still
   need;
5. review the exact retention command and confirmation target.

Cleanup fails closed per table: a trace that does not finish (a pinned
version missing from the table's version listing, or a read that failed)
deletes nothing for that table and reports why in its result row; fix the
cause and rerun cleanup to converge. Cleanup deletes a table version's unused
files before the version itself. If a run stops between the two steps, the
next run finds that version again, counts the files already gone as removed,
and deletes the version. The run derives each branch's history
and pins from one captured version, then validates that the live inventory,
versions, incarnations and tags still match after all table inventories.
A changed observation refuses the whole plan before deletion; rerun with the
same policy. This includes a branch deleted after cleanup listed it: cleanup
returns a retryable conflict, and a new run captures the current branches.
Retirement archives exact branch identity and ancestry inside
its native tree, then removes the active ref. Creation does not scan retired
histories. Cleanup removes unneeded retired trees and their archives, while
preserving trees needed by live descendants, merge bases or tags.

Ordinary graph writes and branch merges may overlap cleanup. A merge protects
its accepted source, target and base snapshots with durable native tags;
source advancement does not revoke those inputs. A cancelled merge or one
whose publication returns an in-doubt error keeps its tags until the target
witness proves it can no longer publish. An unchanged or unresolvable target
keeps them. Graph-branch creation/deletion and cleanup retain the existing
single-writer-process control boundary.

Legacy native snapshots can borrow another location's files. Cleanup retains
those physical origins, including overlay and index files. An incomplete origin
trace prevents physical-file deletion across the run. Unregistered native trees
can retain extra origin files until a later cleanup after those trees vanish.

Ordinary writes retain unpublished staging, because the branch head its commit recorded is still the
branch's head. A write that publishes during the run either landed before
the snapshot, in which case its version is a root, or after it, in which
case its staging is kept for the same reason. Staging left by a write that
failed before publication is kept until that branch publishes again or is
deleted; on a branch that never publishes again it stays, and its table's
result row counts it under `unpublished_manifests`. Age alone never decides
anything: an old staging is not proof that its writer stopped, and an old
pinned version is never removed while a retained graph commit names it.
Files no listed manifest references follow Lance's seven-day unverified-file
rule, including temporary manifests. Each orphan Blob sidecar must satisfy
that age rule itself, even when its parent data file is older.

Each table's result row (under `--json`) carries `bytes_removed`,
`manifests_removed` (table versions deleted: published ones no retained
graph commit names, and dead staging), `unpublished_manifests` and
`unpublished_bytes` (the staging the run found, dead ones included),
`foreign_versions` (Lance versions above the table's last linear version,
which the run never deletes) and `error`, set only when the table's trace
did not finish, in which case its counts are zero. `old_versions_removed`
stays for one release beside `manifests_removed` and carries the same
number; nothing defers a table, so the row has no `deferred` field.

Exit 0 means every table was visited and that nothing a retained graph
commit pins, and no file such a commit references, was removed. It does not mean every table was collected: read the result rows
for tables with an `error`.

### Foreign versions

Every Omnigraph table write is a detached Lance commit that a graph commit
pins, so a table's linear Lance history ends where the table was created. The
registration records that point as its last linear version
(`omnigraph.last_linear_version`). A linear commit above it came from
something other than Omnigraph writing the table directory, such as a Lance
client pointed at it. No read or write of the graph resolves the linear
HEAD, so a foreign version changes no query result and blocks no writer, and
no command adopts it. A confirmed cleanup run (`cleanup --keep N --confirm`,
which also removes what its policy does not retain) lists these versions per
table, under `foreign_versions` with `--json` and by name in the human report,
and leaves them and their files in place. The preview without `--confirm`
does not open the graph and lists nothing. Embedded hosts can read the same
list from the read-only `Omnigraph::cleanup_plan`. To discard them,
export the graph and load it into a new one; see
[Troubleshooting](https://www.omnigraph.dev/docs/operations/troubleshooting.md#foreign-versions).

## Suggested cadence

* Run `optimize` after large loads or on a regular cadence for write-heavy
  graphs.
* Run `cleanup` from an explicit retention policy after backups and rollback
  requirements have been reviewed.

Moving a graph to another storage format is not maintenance. See
[Upgrading](https://www.omnigraph.dev/docs/operations/upgrade.md).