Drew McCormackandClaude Opus 5 f168aa90f9 Queryable projections, and local-first SQLite over LLVS (#11)
* Add design for the projection layer

LLVS is a log, which is good at history and merging and bad at
answering "show me today's notes". The three shortcomings raised
(no querying except by ID, cloud data too fine-grained, no
coalescing) are one problem: a log needs pairing with something
that is not a log.

The design keeps LLVS as truth and projects version diffs into a
SQLite current-values view, which is what gets queried. Map
diffing already supports this: differences() is a set difference
between two arbitrary versions, so a merge commit or a move
backwards onto a branch needs no replay and no reverse diff.

Records the two decisions with consequences downstream: the
projected columns are an index rather than a copy, so adding a
queryable field is a rebuild and never a migration; and a blob
that will not decode is skipped and reported, so one unreadable
value neither freezes sync nor disappears unannounced.

Map bucketing must be fixed first. LLVSModel IDs put every note
in one bucket, which is O(N) per write and fatal at the target
size.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013kU8zpV5HdgtcqyTbDBkQg

* Add implementation plan for the projection layer

Seven tasks, each with its own test cycle: flip model value IDs off
the crowded Map bucket, add transactions to SQLiteDatabase, expose a
correct diff between two arbitrary versions, then build LLVSProjection
on top of those three.

Also corrects the spec. Its claim that the DAG case was already served
by existing API was wrong: Map and Map.Diff are internal, and the one
public route passes the first version as its own common ancestor and
calls fatalError on the two-branch forks, so it traps on exactly the
case the projector needs. Resolving the ancestor and folding those
forks is now a task rather than an assumption.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013kU8zpV5HdgtcqyTbDBkQg

* Put the instance identifier first in model value IDs

The Map buckets values by the first two characters of their ID, and
LLVSModel built IDs as "Type/instance", so every instance of a type
landed in one bucket and each save rewrote a node listing all of
them. That is O(N) per write. Audit item 7.

The instance identifier now leads. The helpers split on the last
slash rather than the first, so an app-supplied instance identifier
may contain slashes and the type name still survives.

fetchAllModels matched an ID prefix, which the flip breaks. It now
compares the type identifier through the same helper, so the ID
layout has one owner. The cost is that it scans every reference at
the version, there being no prefix left to narrow with. Nothing
else relied on the layout.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013kU8zpV5HdgtcqyTbDBkQg

* Add transactions to SQLiteDatabase

A projection must apply its rows and record the version they came
from together, or a crash between the two leaves no way to tell what
had been applied. SQLiteDatabase had no transaction support, so
there was no way to express that.

The rollback is deliberately try?: the caller's error is the one
worth reporting, and a rollback failure would mask it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013kU8zpV5HdgtcqyTbDBkQg

* Add a public diff between two arbitrary versions

Keeping a derived view in step with a store means asking what
differs between the version the view holds and the version it must
reach. After a sync and merge those two are sideways from each
other, not in a line.

Map.differences answers that correctly given a true common
ancestor, but Map is internal. The one public route,
valueChanges(madeBetween:and:), passes the first version as its own
common ancestor, which holds only when it is an ancestor of the
second, and calls fatalError on the forks that arise when it is
not. It traps on exactly the case a derived view needs.

The new call resolves the real ancestor and then decides each
change from whether the value exists at each end rather than from
the fork label, so the two-branch forks need no special handling.
The existing call is left alone; it is public API with callers.

Resolving the ancestor walks the graph, so cost grows with history
length rather than with the size of the difference. Every merge
already pays this. Worth measuring before it matters.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013kU8zpV5HdgtcqyTbDBkQg

* Add the LLVSProjection target and its schema description type

ProjectedType is the one thing an app must write: which table a
stored model type becomes, which columns it carries, and how to get
those column values out of a value's data.

The columns are an index rather than a copy. Declaring only what is
queried or sorted on keeps writes cheap, and leaves the store as the
only place the whole object lives, which is what makes adding a
field later a rebuild rather than a migration.

extract throws to mark a value unreadable, for when another device
wrote a model this build cannot decode.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013kU8zpV5HdgtcqyTbDBkQg

* Project version diffs into SQLite in one transaction

Each pass asks the store what differs between the version the
database holds and the version it must reach, then writes those
rows and the new version marker together. A failure part-way rolls
both back, so the database is either current or behind, never
half-updated, and behind is repaired by running again.

Because the diff is a set difference rather than a replay, a merge
commit or a move backwards onto a branch needs no special handling.

An unreadable value is skipped and its ID returned. Failing the
pass would let one value another device wrote freeze the whole
projection; dropping it silently would lose it from the UI with
nothing to show for it.

schemaVersion is the app's own number for the shape of its tables.
Raising it rebuilds rather than diffing onto tables built for
different columns, which is why there are no migrations here.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013kU8zpV5HdgtcqyTbDBkQg

* Test that a failed projection pass rolls back whole

The single-transaction claim was asserted but never checked. These
tests fail a pass part-way and require that nothing it wrote
survives and that the version marker did not move.

Getting this to test anything took two attempts. Failing the pass by
dropping the table failed its very first write, so there was nothing
written to roll back and the tests passed with transactions removed.
They now fail on a CHECK constraint that rejects one title, after
earlier rows have already been written.

Verified by removing BEGIN and COMMIT and watching two of them fail,
then restoring. Caching hid that on the first attempt: swift test
--filter reported a pass against a build that did not contain the
mutation, which is worth knowing the next time a mutation looks
survivable.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013kU8zpV5HdgtcqyTbDBkQg

* Follow the coordinator's version, and document LLVSProjection

ProjectionFollower keeps a projection in step with a coordinator,
either by being awaited after each save or by draining the version
stream.

It is an actor owning both the projector and the SQLite connection,
which the first attempt was not. Following from a background task
while the test saved on another crashed the suite: Store is
@unchecked Sendable but only its history is behind a mutex, so the
value map is read and written concurrently. That is audit item 9,
and it is the library reporting a real race rather than a test
problem. Neither object is thread-safe, so neither is now reachable
from outside the actor: the database is opened from a URL rather
than handed in, and queries go through query().

The README example is also compiled, as ReadmeExampleCompileCheck.
The samples went stale once before and had to be rewritten wholesale
(audit item 22); one that is built cannot drift unnoticed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013kU8zpV5HdgtcqyTbDBkQg

* Record the concurrency constraint in the spec, plan and README

The design said nothing about threads, which turned out to be the
hardest constraint in the work: Store's value map is unguarded, so
a pass reading it while the app saves crashed the suite with a
signal. The spec now carries that, and the plan's Task 7 is marked
superseded rather than left describing an API that cannot compile.

The README says the part an adopter needs: await
projectCurrentVersion() after a save, because the actor cannot stop
an app saving on another thread while a pass runs. Closing that
properly is audit item 9.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013kU8zpV5HdgtcqyTbDBkQg

* Make the projection rollback tests catch a lost transaction

Code review found the rollback tests were not reliable guards:
with BEGIN and COMMIT removed they still passed 2 of 12 and 4 of 12
runs respectively.

The cause is ordering. Map.differences builds its result from Sets
of keys and IDs, and Swift seeds its hasher per process, so the
order of changes in a pass varies from run to run. Failing the pass
by rejecting one row's content therefore hit first on some runs and
last on others, and on the runs where it went first nothing had
been written and there was nothing to roll back.

They now fail on a trigger counting inserts, which holds whatever
the order, and catch a removed transaction 12 of 12.

Also recorded which two tests are the atomicity guards. The other
two catch nothing when the transaction goes, correctly: they cover
recovery, and without a transaction the work still completes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013kU8zpV5HdgtcqyTbDBkQg

* Correct why the projection follower is an actor

The crash that drove this design was recorded as a Store data race,
on the grounds that Store is @unchecked Sendable with only its
history behind a Mutex. Code review checked that under Thread
Sanitizer, 300 concurrent saves against 300 concurrent passes, and
found no race: Map holds only a let zone and a Mutex-protected
Cache, and FileZone is the same.

The real cause is SQLiteDatabase, which documents itself as not
thread-safe and serialises nothing. The crashing design had the
test reading the projected database while the follower's task wrote
it. Four threads sharing one SQLiteDatabase, with no LLVS store
involved, reproduces the same signal on its own.

The actor stands, and for a better reason than the one given: it
serialises the database, which genuinely needs it. Store's own
serialisation stays open as audit item 9, and this design does not
depend on it.

The README caveat went too: it warned against saving during a pass
on a premise that turned out to be false.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013kU8zpV5HdgtcqyTbDBkQg

* Skip diff entries both versions already share

Code review noted that a value updated to the same content on both
branches still came back as an update. The diff can also mention a
value neither end touched, and that was reported too.

Both cases are now skipped by comparing stored version ids: two
references to the same stored version are the same bytes. That
costs a map lookup on the from side, which the code already did to
test existence, and saves reading the value and writing it back
unchanged.

Comparing the data itself would also work and catch two separate
copies of identical bytes, but it means reading both values, which
is the expensive half. Identity of the stored version is the part
that is free.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013kU8zpV5HdgtcqyTbDBkQg

* Test that saving during a projection pass stays safe

Every other test here awaits its pass after the save, so nothing
covered the case the follower exists to handle.

These run saves against passes, drive the same thing through the
version stream, and read the projection while it is being written.
They pass, and pass under Thread Sanitizer, which matches what code
review measured independently.

The property is asserted rather than described because it is
accidental: Map's fields are a let zone and a Mutex-backed Cache,
and FileZone is the same, so concurrent readers and writers meet
only in file I/O and a lock. Store promises none of that. A comment
saying so would go stale in silence the day Map gains mutable
state, which is the day this matters; a test starts failing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013kU8zpV5HdgtcqyTbDBkQg

* Guard Store's concurrent-read property, and check it in CI

The concurrency guard was in the projection tests, where it reached
Map's immutability through two intermediary layers: slow, indirect,
and liable to be blamed on the projection when it failed. Code
review pointed out it belongs against Store, which is where audit
item 9 lives and where someone changing Map would look.

It is written to fail without a sanitizer, on a crash or a wrong
answer, because CI runs none by default. A second CI job now runs
the concurrency suites under Thread Sanitizer, so the race is
caught as a race rather than waited on. Both pass today, which
matches what code review measured.

Projector+Coordinator.swift keeps one line about it: the
constraint, that Store promises no serialisation, rather than the
mechanism, which would go stale silently the day Map changes.

Also from review: ProjectedType now rejects a table or column name
that is not a plain ASCII SQL identifier. SQLite has no binding for
identifiers, so they are interpolated; a typo becomes a clear
failure at construction rather than a syntax error at the first
write. And query's doc comment says the compiler prevents the
database escaping, which it does, rather than asking the caller not
to.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013kU8zpV5HdgtcqyTbDBkQg

* Take the history walk off the projection write path

The spec called the common-ancestor walk "not new risk, since every
merge already pays it". Code review pushed back, and was right: a
merge is occasional, a projection pass runs on every save, so the
design had moved an O(history) operation onto the write path.

Measured: one diff cost 0.68 ms at 100 versions and 17.19 ms at
3000. Linear, per save.

Two consecutive versions are almost always on one line, and then
the source version is the common ancestor. isAncestor searches back
from the target alone and stops on a hit, so the full search runs
only when that cannot settle it. The same measurements are now 0.13
ms and 0.36 ms.

A false from isAncestor means "not found within the limit", not
"not an ancestor", so the caller must treat it as inconclusive and
fall back. That is what the limit buys and it is tested.

What remains is not the walk. Over fifteen times the history a diff
still costs about 2.4x; probing the parts shows ancestry flat at
0.002 ms and map lookups flat, so the rest is Map bucket size —
audit item 7, from the read side. With IDs sharing a prefix it is
about 11x.

The new performance test bounds that and fails at 11.7x if the
shortcut is removed, which I checked in a throwaway worktree.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013kU8zpV5HdgtcqyTbDBkQg

* Cap the unreadable IDs a projection pass reports

A version-skew event, where a newer device writes a model shape
this build cannot decode, can make every value in a rebuild
unreadable. Handing the app one ID per value then costs memory
proportional to the store and tells it nothing the count does not.

The list stops at 100 and unreadableCount carries the true number.

Also fixes a README line still describing the old "Type/instance"
value ID layout, which the ID flip changed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013kU8zpV5HdgtcqyTbDBkQg

* Add design for local-first SQLite over LLVS

The projection layer is one-way, so an app still writes through
StoreCoordinator.save and a write to the SQLite is destroyed by the
next rebuild. What an app wants is to use SQLite the way any SQLite
app does, with syncing following from that.

SQLite becomes a second write path into LLVS rather than a second
truth: triggers capture row and column changes, a drain step turns
them into a version, and incoming versions apply back under a
suppression flag. All four mechanisms were checked against SQLite
before writing this down, including that a multi-row UPDATE fires
per row and that a no-op update records nothing.

The merge needs no new code. A row becomes a value, values merge in
LLVS as they always have, and because MergeableArbiter already
merges property by property, a column is a property and
column-level merge falls out. Same-column collisions go to the
MergeArbiter, which is what an app's arbiter is already for.

Scalars get real indexable columns rather than JSON with duplicate
index columns; only nested properties become JSON, and only that
property rather than the whole type. The macro already walks the
stored properties and has the type annotation in reach.

Three questions are named and left open rather than guessed:
owned-table migration, delete and tombstone rules, and whether an
app's own transaction should bound a version.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013kU8zpV5HdgtcqyTbDBkQg

* Specify typed reads for owned SQLite tables

An owned table is generated from a Swift model, so handing its rows
back as columns throws away what the declaration already knows.
Reads now hydrate into the model and carry the llvs_id and the
version they were read at, so a row knows where it came from and a
later typed write has what it needs.

Writes stay SQL on purpose. An UPDATE that sets one column says
only that column changed, and the per-column trigger records that.
A save(model) writing every column would claim they all changed and
destroy the granularity that lets concurrent edits to different
columns merge. A typed write has to diff against the stored row
first, which is worth building deliberately rather than getting for
free by writing whole rows.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013kU8zpV5HdgtcqyTbDBkQg

* Settle the three open questions in the SQLite design

A plan cannot carry a TBD, so each is decided with its reasoning.

Migration is drain-then-rebuild, and the order is forced: rebuilding
first would discard local edits that never reached the truth. A
failed drain cancels the migration, because running on the old
schema beats losing unsynced work.

A delete is a removal, and an edit on another device brings the row
back, which is what the default arbiter already does with
removedAndUpdated. A returning row is visible and fixable; a
silently discarded edit is neither. Stating the rule is the point.

Transaction boundaries need no mechanism. SQLite fires triggers
inside the app's transaction, so a committed transaction yields one
version and a rolled-back one yields nothing. Verified: an update
inside a rolled-back transaction leaves no changelog row.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013kU8zpV5HdgtcqyTbDBkQg

* Add implementation plan for local-first SQLite

Seven tasks. The macro learns each property's column type; an owned
table gets its capture triggers; captured changes drain into a
version; incoming versions apply back under suppression; a round
trip proves two devices editing different columns both keep their
edit; reads come back typed; and a rebuild drains first.

The mechanism the plan rests on was checked rather than assumed:
binding.typeAnnotation?.type.trimmedDescription compiles against
the SwiftSyntax already linked, so the macro can see a declared
type. It cannot see an inferred one, so var x = 0 gets no column
and is reported instead of guessed at.

Typed writes, generating the table from the type alone, wiring an
owned table into ProjectionFollower, and measuring a drain are all
named as deferred with the reason.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013kU8zpV5HdgtcqyTbDBkQg

* Generate a SQLite schema from the model declaration

@MergeableModel already walked every stored property and skipped
let, static and computed ones correctly. It kept only the name,
while binding.typeAnnotation sat unused beside it.

It now also emits sqliteSchema: a real indexable column for each
scalar property, and a JSON TEXT column for a type with no column
shape. Only the awkward property becomes JSON, so an array of tags
does not stop the title being a plain TEXT column.

A property whose type is inferred gets no column. A macro sees only
syntax, so var count = 0 has no visible type, and guessing Int
would be wrong for var count = someExpression. Such properties are
listed in propertiesWithoutColumns instead, which the existing
MultipleBindingModel exercises.

Column names are snake_cased, and one colliding with a SQLite
keyword takes a trailing underscore rather than quoting, so it
stays a plain identifier everywhere it appears. "when" was in the
test set for exactly that reason.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013kU8zpV5HdgtcqyTbDBkQg

* Capture SQL writes to an owned table

Three triggers record what an app's ordinary SQL changes: per row
for an insert or delete, and per column for an update. The per
column part is what the merge needs, so that two devices editing
different columns of one row do not overwrite each other.

Two behaviours come free from SQLite and are tested rather than
assumed. A guard of OLD.col IS NOT NEW.col means a write that
changes nothing records nothing, NULL included. And because
triggers fire inside the app's transaction, a rollback takes the
changelog rows with it, so an app's own BEGIN/COMMIT bounds a
version without any mechanism.

Suppression is a one-row table rather than a Swift property,
because a trigger's WHEN clause is SQL and can only consult the
database. It lifts even when the block throws, or capture would
stay off for good.

Clearing takes a sequence, because an app may write between a drain
reading the changelog and finishing with it; those writes have to
survive for the next drain.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013kU8zpV5HdgtcqyTbDBkQg

* Turn captured SQL writes into LLVS versions

A drain reads the changelog, assembles each touched row, and makes
one version. Several edits to a row collapse into one change: the
changelog says which rows and columns moved, the row says what they
now hold, and the per-column detail is what the merge uses later.

Three cases that would otherwise be wrong. A row inserted and
deleted before the drain reached it produces nothing, rather than a
removal of something never stored. A remove for a value the store
does not hold is dropped for the same reason. And the changelog is
cleared only through the sequence that was read, so a write landing
while the version is being made is kept for next time.

The stored JSON is keyed by property, not column, because that is
what the model decodes from. A null column is omitted rather than
written as JSON null, so an optional property decodes as nil and a
non-optional one fails loudly rather than silently.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013kU8zpV5HdgtcqyTbDBkQg

* Apply incoming versions into an owned table

Closes the loop. A version from another device, or one a merge
produced, lands in the table under suppression, so it is not
recaptured as a local edit and traded back and forth forever.

A value that is not a JSON object is skipped rather than failing
the batch, which is the rule the read-only projection already
follows: one value this build cannot read must not stop the rest.

A property the value does not carry binds as NULL rather than
failing, so a model that has gained a property still applies and
the new column simply reads back empty.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013kU8zpV5HdgtcqyTbDBkQg

* Test the owned table round trip and per-column merge

Two devices, each with its own store and its own SQLite, both edit
one row from a shared base: one changes the title, the other the
body. After the merge both edits are present. That is the claim the
whole design rests on, and until now it was only reasoned about.

Checked that it tests what it claims: with the type unregistered,
so the property-wise merge does not run, the test fails and one
device's edit is lost. It passes only because a column is a
property and MergeableArbiter merges properties independently.

Also covers the stated delete rule, where an edit beats a delete
and the row returns carrying it, and a same-column conflict, which
resolves through the arbiter with the untouched column unharmed
either way.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013kU8zpV5HdgtcqyTbDBkQg

* Read owned table rows as models, and drain before rebuilding

Reads hydrate into the model and carry the value ID and the version
they were read at, so a row knows where it came from and a typed
write would later have what it needs. A row that cannot be decoded
is left out rather than failing the query, so one bad row does not
blank a list.

Writes stay SQL, which is the deliberate half. An UPDATE naming one
column says only that column changed; a whole-row typed write would
claim they all did and cost the per-column merge that the round
trip test proves.

Rebuilding discards a table, so registerOwnedTable and
drainOwnedTables get pending local edits into the store first, and
a failed drain stops before anything is thrown away.

The README's example is compiled as
ReadmeOwnedTableCompileCheck, the same guard the projection example
has, because these samples went stale once before.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013kU8zpV5HdgtcqyTbDBkQg

* Store a Date column as Unix seconds

Codable encodes a Date as seconds since 2001. SQLite's strftime, a
database browser, and every other tool mean seconds since 1970.
Storing Codable's number made the round trip work, because both
ends used the same convention, while making the promise the design
rests on false: an app writing strftime('%s', 'now') and reading it
back got a date 31 years out, with no error.

Found by checking the encoding rather than reading the code. The
round trip test passed throughout, which is why it needed a test
written from the app's side instead: write a Unix timestamp in
plain SQL, read the date it meant.

A Date column is now its own storage case, holding Unix seconds,
with the conversion at the boundary. The macro comment claiming
"seconds since 1970" was wrong and is fixed too.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013kU8zpV5HdgtcqyTbDBkQg

* Key column conversion on the semantic type, not the SQLite one

Code review found that a Bool did not round-trip in either
direction. The drain wrote 1 into the JSON, which Codable refuses,
so the value in the store never decoded; and fetch, which skips a
row it cannot decode, returned an empty list with no error. Any
model with a starred or isDone flag was silently unusable, and the
failure shape was the worst kind.

This is the Date bug again. Int, Bool and Date are all INTEGER, and
the conversion was switching on the declaration string, which
cannot tell them apart. Adding a .bool case would have fixed the
instance and left the third one waiting.

ColumnStorage now names the semantic type for every case, the
declaration derives from it so the two cannot disagree, and both
conversions switch exhaustively. A type that needs new handling is
a compile error rather than a wrong answer.

Also adds the test review asked for: one model carrying every
supported type, driven both ways, asserting equality. Reverting the
Bool conversion fails three of its tests, so it guards what it
claims. That test would have caught both bugs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013kU8zpV5HdgtcqyTbDBkQg

* Apply only a table's own type, and allow a table with no columns

Two findings from code review, both reproduced before fixing.

apply wrote every change it was handed. A change set from
valueChanges(updatingFrom:to:) carries every type in the store, so
with two owned tables each ended up holding every value: rows with
null columns where the properties did not exist, and real but wrong
data where names collided. It now takes only values whose ID names
its own type, which is the check Projector already made.

A table with no columns could not be created at all. SQLite rejects
a trigger with an empty body, and with nothing to capture the
update trigger was exactly that. It is now left out. Reachable from
a marker or tombstone type, or one whose properties all have
inferred types.

Also from review: fetch logs the rows it skipped rather than
silently returning fewer, since silence is what let the Bool bug
look like an empty table; failing to lift capture suppression is
logged rather than swallowed, because leaving it set discards every
later local write; UInt maps to INTEGER instead of falling through
to JSON; and rowid is escaped, as it shadows SQLite's implicit
alias.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013kU8zpV5HdgtcqyTbDBkQg

* Say plainly that fetch's version is a label, not a check

Code review pointed out that atVersion is stamped onto every row
without verification, which is a trap for something whose stated
purpose is future optimistic concurrency.

fetch genuinely cannot check it: an owned table records no version
of its own, and the one that matters lives in
Projector.projectedVersion(). So the documentation now says it is a
label the caller attaches, and to pass the projected version or
nothing, rather than implying a guarantee that is not there.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013kU8zpV5HdgtcqyTbDBkQg

* Close the window where a local write is swallowed by suppression

Code review found data loss with no signal, and reproducing it was
the convincing part: a write landing while apply holds the
suppression flag changes the row but records nothing, so the UI
shows the user's edit and it never syncs.

apply now takes the write lock before suppressing rather than
after, so a writer on another connection waits instead of falling
into that window. Nothing can protect a second thread sharing the
same connection, so the requirement to drive one table from one
place is now stated on the type, on the suppression call, and as
the first of the README's rules — it is the one that loses data.

Also from review: a typealias silently became a JSON column instead
of an indexable one. A macro cannot resolve an alias, since that
needs type checking, so it is reported in propertiesStoredAsJSON
with the type as written rather than left as a silent demotion.

And the drain's store check, which review noted was load-bearing
but not obvious: a row deleted and reinserted between drains has
insert as its last operation while still existing in the store, so
it must go out as an update. Now commented, and covered by tests
for all three sequences review ran by hand.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013kU8zpV5HdgtcqyTbDBkQg

* Measure the drain against table size, and fix a load-sensitive test

The drain reads one row per touched value, which should make its
cost follow what changed rather than what the table holds. That was
the right shape on paper and unmeasured, and the previous review's
scaling predictions were sharper than my own, so it is now
measured: a one-row drain takes 1.85 ms against 100 rows and 2.50
ms against 5000. Flat. The test asserts the shape rather than those
numbers, so it survives a slower machine.

Adding it surfaced a test that was already fragile.
followUpdatesProjectsTheCurrentVersionFirst slept a fixed 200 ms
and hoped, which is long enough on an idle machine and not on a
busy one: it failed once in a full run while passing ten times on
its own. It now waits for the row it expects. Six suite runs clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013kU8zpV5HdgtcqyTbDBkQg

* Stop a scalar in a JSON column killing the process

Code review found an uncatchable crash on a path the previous
commit had just started advertising. JSONSerialization raises an
Objective-C exception, not a Swift error, for a top-level value
that is not a collection: try? cannot catch it and the process
dies. A bare string reaches a JSON column whenever a property's
type could not be resolved — a typealias, or a raw-value enum — so
propertiesStoredAsJSON told a developer their alias was a JSON
column, and the first value through it killed the app.

Worse, the value can arrive from another device, making it a
sync-time crash on a device that did nothing wrong.

Both directions now handle fragments: the write checks
isValidJSONObject and falls back to .fragmentsAllowed, and the read
says explicitly that it allows them rather than relying on the
default. Removing the guard kills the test process with signal 6,
so the test guards what it claims.

Also records the scaling question review raised, which is sharper
than the one I measured: the reads follow what changed, but one
drain is one version, so a bulk UPDATE over 50,000 rows makes a
single version carrying 50,000 values. Batching changes drain's
return type, so the decision is cheaper now than later.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013kU8zpV5HdgtcqyTbDBkQg

* Let an owned table record which version it is a copy of

The design says SQLite is the working copy at a version, and then
nothing recorded which one: the caller passed basedOn: and had to
remember it. A checkout that does not know its own revision is not
a checkout.

Reproducing the consequence is what settled it. A drain based on
nothing, when the store already holds versions, makes a version
with no predecessor — a second root. One device that never synced
with anything ends up with a forked history and two heads, silently.

The table now keeps a state row, written in the same transaction as
the changelog clearing, the way Projector already does for its own
version. drain defaults to it, apply takes an optional atVersion so
a sync can record what it brought, and basedOn: stays for
deliberately building on something else.

Found by code review, which noticed the gap between the framing and
the code.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013kU8zpV5HdgtcqyTbDBkQg

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-21 13:55:15 +02:00

Low-Level Versioned Store (LLVS)

Author: Drew McCormack (@drewmccormack)

Ever wish it was as easy to move your app's data around as it is to push and pull your source code with Git?

LLVS brings the same model to app data. Every save creates a version. Versions branch, merge, and sync between devices, just like commits in a Git repository. Your app gets full version history, conflict resolution, and multi-device sync without writing any networking or diffing code.

The problem

A user edits a note on their phone during a flight. Meanwhile, a share extension updates the same note on their iPad. Later, their Watch app writes a quick addition. When the phone comes back online, three copies of the data have diverged independently.

LLVS handles this the way Git handles divergent branches: it tracks the full ancestry of every change, finds the common ancestor when versions diverge, and merges them back together through a conflict resolver you control.

What you get

  • Version history. Every save is a version. Branch, merge, diff, or read the store as it was at any point in time.
  • Three-way merge. When versions diverge, LLVS finds their common ancestor and diffs both sides. You provide a MergeArbiter to resolve conflicts however you like, or use a built-in one.
  • Sync without networking code. Push and pull versions via CloudKit, WebDAV, Google Drive, OneDrive, Box, pCloud, a shared directory, a peer-to-peer link, or your own exchange.
  • Typed models, if you want them. The LLVSModel library stores Codable structs and merges them property by property with the @MergeableModel macro.
  • Multi-process safe. Share a file-based store between your main app, extensions, and widgets using an app group container.
  • Pluggable storage. File-based storage by default, SQLite via LLVSSQLite, or bring your own backend.
  • Encryption-friendly. LLVS stores opaque Data blobs. Encrypt them however you want; the framework never inspects your data.

Everything asynchronous uses async/await. There are no completion handlers and no Combine.

Quick Start

This walks through a minimal app that syncs a single shared message via CloudKit. The full code is in Samples/TheMessage.

Set up a StoreCoordinator

StoreCoordinator is the simplest entry point. It wraps a Store, tracks the current version for your UI, and orchestrates sync and merging.

import LLVS
import LLVSCloudKit
import CloudKit

let coordinator = try StoreCoordinator()
let container = CKContainer(identifier: "iCloud.com.mycompany.themessage")
coordinator.exchange = CloudKitExchange(
    with: coordinator.store,
    storeIdentifier: "MainStore",
    cloudDatabaseDescription: .publicDatabase(container)
)

StoreCoordinator() puts the store in Application Support. Use StoreCoordinator(withStoreDirectoryAt:cacheDirectoryAt:) if you want to choose the location, for example an app group container.

Save, fetch, sync

let messageId = Value.ID("MESSAGE")

func post(message: String) throws {
    let value = Value(id: messageId, data: message.data(using: .utf8)!)
    try coordinator.save(updating: [value])
    sync()
}

func fetchMessage() -> String? {
    guard let value = try? coordinator.value(id: messageId) else { return nil }
    return String(data: value.data, encoding: .utf8)
}

func sync() {
    Task {
        try? await coordinator.exchange()
        _ = try? coordinator.merge()
    }
}

exchange() sends and receives versions with the cloud. It is a bit like a two-way git fetch: it moves data, but does not touch your current version. merge() then reconciles any concurrent changes and moves the current version forward. That's the entire sync implementation.

React to changes

currentVersionUpdates is an AsyncStream<Version.ID> that yields whenever the current version changes, whether from a local save or a merge.

Task {
    for await _ in coordinator.currentVersionUpdates {
        self.message = fetchMessage() ?? ""
    }
}

This example is deliberately minimal. What happens when data diverges across devices is covered below.

Installation

LLVS is installed with the Swift Package Manager. It requires Swift tools 6.1, and macOS 15, iOS 18, or watchOS 11.

dependencies: [
    .package(url: "https://github.com/mentalfaculty/LLVS.git", from: "0.11.0")
]

In Xcode, choose File > Add Package Dependencies..., enter the repository URL, and select the libraries your target needs.

Library What it is
LLVS The core: Store, StoreCoordinator, merging, file storage, and the in-core exchanges
LLVSSQLite SQLite storage backend
LLVSModel Typed models, the @MergeableModel macro, MergeableArbiter
LLVSCloudKit CloudKit exchange
LLVSWebDAV, LLVSGoogleDrive, LLVSOneDrive Cloud file systems for use with CloudFileSystemExchange
LLVSBox, LLVSPCloud Exchanges that wrap the Box and pCloud SDKs (need a trait, see below)

The core LLVS library depends on ZIPFoundation, which is used for snapshots. LLVSModel depends on swift-syntax for its macro.

Box and pCloud need a package trait

LLVSBox and LLVSPCloud wrap vendor SDKs, and I don't want those SDKs downloaded into apps that never use them. They are gated behind package traits, so you have to opt in:

.package(url: "https://github.com/mentalfaculty/LLVS.git", from: "0.11.0", traits: ["Box"])

The traits are Box and PCloud. Without the trait, the module builds but is empty, and the SDK is not fetched.

Typed Models with LLVSModel

The core framework deals in raw Data. If your model is made of Codable structs, LLVSModel saves you the boilerplate, and gives you a much better merge. This is how the LoCo sample stores its contacts.

import LLVS
import LLVSModel

@MergeableModel
struct Contact: StorableModel, Equatable, Identifiable, Codable {
    static let modelTypeIdentifier = "Contact"
    var id: UUID = .init()
    var firstName: String = ""
    var lastName: String = ""
    var city: String = ""
    var avatarJPEGData: Data?
}

StorableModel is Codable plus a stable modelTypeIdentifier. Each instance is stored as JSON in one value, with the identifier "<instanceIdentifier>/Contact". The instance identifier leads so that instances of one type spread across the value map rather than crowding into a single node; see Upgrading to 0.12 if you have a store written by an earlier version.

@MergeableModel generates a Mergeable conformance that does a three-way merge of each stored property. If one device changes firstName and another changes city, both edits survive. Properties that are themselves Mergeable (including optionals of Mergeable types) are merged recursively; plain Equatable properties are compared against the common ancestor. Nested types only need @MergeableModel; StorableModel is just for the top-level types you save.

To use that merge, register your types with a MergeableArbiter:

let coordinator = try StoreCoordinator()

let arbiter = MergeableArbiter()
arbiter.register(Contact.self)
coordinator.mergeArbiter = arbiter

Values of unregistered types fall through to fallbackArbiter, which defaults to MostRecentChangeFavoringArbiter.

Saving and fetching are typed extensions on StoreCoordinator:

let contact = Contact(firstName: "Ada")
try coordinator.save(contact, instanceIdentifier: contact.id.uuidString)

let one = try coordinator.fetchModel(Contact.self, instanceIdentifier: contact.id.uuidString)
let all = try coordinator.fetchAllModels(Contact.self)

try coordinator.removeModel(Contact.self, instanceIdentifier: contact.id.uuidString)

Querying with LLVSProjection

LLVS answers "what is this value at this version". It does not answer "which contacts were edited this week", because every value is an opaque blob keyed by ID.

LLVSProjection fills that gap without making LLVS query-aware. It keeps a SQLite table of current values, updated from version diffs, and you query that. The store stays the truth; the table is an index over it.

import LLVSProjection

let contacts = ProjectedType(
    typeIdentifier: Contact.modelTypeIdentifier,
    tableName: "contacts",
    columns: [
        ProjectedColumn(name: "name", declaration: "TEXT"),
        ProjectedColumn(name: "age", declaration: "INTEGER"),
    ],
    extract: { value in
        let contact = try JSONDecoder().decode(Contact.self, from: value.data)
        return ["name": .text(contact.name), "age": .integer(Int64(contact.age))]
    }
)

let follower = try ProjectionFollower(
    databaseURL: databaseURL,
    coordinator: coordinator,
    types: [contacts],
    schemaVersion: 1)

Project after a save, then query:

try coordinator.save(inserting: [value])
await follower.projectCurrentVersion()

let names = try await follower.query { database in
    var names: [String] = []
    try database.forEach(matchingQuery: "SELECT name FROM contacts WHERE age > 30 ORDER BY name") { row in
        if let name: String = row.value(inColumnAtIndex: 0) { names.append(name) }
    }
    return names
}

followUpdates(onResult:) drives the same thing from the coordinator's version stream, for an app that would rather not await each save.

Four things are worth knowing before building on it.

The projection is an index, not a copy. Declare only the columns you query or sort on, and read the whole object from the store once a query has named the IDs. That keeps writes cheap and the database small.

Changing your columns is a rebuild, not a migration. Raise schemaVersion, and the next pass throws the tables away and re-projects everything from the store. There is no migration to write, because the truth never lived in SQLite. Losing the database entirely is the same event, and equally survivable.

A value that will not decode is skipped, not fatal. If extract throws, because another device wrote a model this build cannot read, that value is left out and its ID comes back in ProjectionResult.unreadableIds. One unreadable value never blocks the rest, and nothing disappears without your app being told. The ID list is capped, since a version-skew event can make every value unreadable at once; unreadableCount always gives the true number.

A pass is all or nothing. Rows and the version marker are written in one transaction, so a crash part-way leaves the projection on its previous version rather than half-updated. Running again repeats the same work.

ProjectionFollower is an actor, and it owns both the projector and the SQLite connection. Neither is thread-safe, so neither is reachable from outside it — which is why the database is opened from a URL rather than handed in, and why queries go through query.

That matters most for the SQLite connection, which is not thread-safe and would crash if used from two threads at once. Keeping it inside the actor is what makes query and a projection pass safe against each other.

Local-First SQLite

The section above is one-way: LLVS is the truth, the SQLite an index over it, and you write through StoreCoordinator. An owned table turns that around. You write ordinary SQL, and those writes become versions that sync and merge.

Think of it as a checkout. The SQLite holds the state at some version, and writing to it makes a new one.

@MergeableModel
struct Note: StorableModel, Codable, Equatable {
    static let modelTypeIdentifier = "Note"
    var title: String = ""
    var body: String = ""
    var updatedAt: Date = .now
    var tags: [String] = []
}

let table = OwnedTable(
    typeIdentifier: Note.modelTypeIdentifier,
    tableName: "notes",
    schema: Note.sqliteSchema)
for statement in table.createStatements() {
    try database.execute(statement: statement)
}

The table is ordinary. Index it however you like:

try database.execute(statement: "CREATE INDEX notes_updated ON notes(updated_at)")

Write ordinary SQL. Triggers record what changed, and a drain turns it into a version:

try database.execute(statement: "UPDATE notes SET title = ? WHERE llvs_id = ?",
                     withBindingsList: [["New title", noteId]])

let result = try table.drain(in: database, store: store)

The table records which version it is a working copy of, so you do not have to carry that across a launch. table.currentVersion(in: database) reads it back, and basedOn: overrides it when you deliberately want to build on something else.

Read rows back as models:

let rows = try table.fetch(Note.self, in: database, where: "updated_at > ?", bindings: [cutoff])
for row in rows { print(row.model.title, row.id) }

And apply what arrives from elsewhere:

let changes = try store.valueChanges(updatingFrom: oldVersion, to: newVersion)
try table.apply(changes, in: database, atVersion: newVersion)

Passing atVersion records what those changes brought the table to, in the same transaction as the rows, so the next drain continues from what arrived rather than from what this device last wrote.

A column per property

@MergeableModel generates the schema. A property typed String, Int, Double, Bool, Date, UUID or Data — or an optional of one — becomes a real column you can index. Anything else becomes a TEXT column holding JSON, still queryable through json_extract and json_each.

Only the awkward property becomes JSON. A model with an array of tags still gets a plain indexed TEXT column for its title.

A Date column holds Unix seconds, so WHERE updated_at > strftime('%s', 'now', '-7 days') means what it looks like. (Codable encodes dates as seconds since 2001; the conversion happens at the boundary so you never see it.)

A property whose type is inferred rather than written down, such as var count = 0, gets no column, because a macro sees only syntax and guessing would be wrong. Annotate it — var count: Int = 0 — and it gets one. Anything skipped is listed in sqliteSchema.propertiesWithoutColumns.

Column names are snake_cased, and one that would collide with a SQLite keyword takes a trailing underscore, so var when: Date becomes when_.

How conflicts resolve

The merge happens in LLVS. SQLite never sees a conflict.

A row change becomes a value, and values merge the way they always have. Because a column is a property, MergeableArbiter merges them independently: if you edit a note's title on your phone while a share extension edits its body, both survive. Register your types and that is what you get.

let arbiter = MergeableArbiter()
arbiter.register(Note.self)
coordinator.mergeArbiter = arbiter

Two devices changing the same column is a real conflict, and it goes to your MergeArbiter, which is what it is for.

A delete racing an edit brings the row back carrying the edit, because the default arbiter favours the more recent change. A row that reappears is visible and fixable; an edit that silently vanished is neither. Change it in your arbiter if your app wants the opposite.

Four rules worth knowing

Drive one table from one place. This is the one that loses data if you ignore it. Capture is suppressed while a version from elsewhere is applied, and that suppression is global to the table, so a write from your app during an apply is swallowed: the row changes, nothing is captured, and the edit never syncs. The row still shows what the user typed, so nothing looks wrong until it fails to arrive on their other device. apply takes the write lock first, which makes another connection wait, but nothing can protect a second thread sharing yours. Put the table behind an actor or a serial queue — SQLiteDatabase asks the same of you already.

Writes are SQL, reads are typed. That is deliberate. UPDATE notes SET title = ? says only the title changed, and that is what lets your edit and another device's merge. A save(note) writing every column would claim they all changed and throw that away. A typed write is possible, but it has to diff against the stored row first.

A rebuild drains first. Rebuilding discards the table, so anything written but not yet drained must reach LLVS before that happens. Register the table with registerOwnedTable and call drainOwnedTables before rebuilding; a failed drain stops the whole thing rather than losing the work.

Your own transactions need nothing special. Triggers fire inside your transaction, so a committed one yields exactly one version and a rolled-back one yields nothing at all.

Concepts

StoreCoordinator is convenient for common cases, but Store gives you direct access to the version graph: branching, merging, diffing, and time travel.

Creating a Store

let rootDir = FileManager.default
    .containerURL(forSecurityApplicationGroupIdentifier: "group.com.mycompany.myapp")!
    .appendingPathComponent("MyStore")
let store = try Store(rootDirectoryURL: rootDir)

Using an app group container lets your main app, extensions, and widgets share the same store. Each process keeps its history in memory, so call try coordinator.store.reloadHistory() before merging, to pick up versions written by the other processes.

Versions and values

Every write creates a new version.

let value = Value(idString: "ABCDEF", data: "Hello".data(using: .utf8)!)
let firstVersion = try store.makeVersion(basedOnPredecessor: nil, inserting: [value])

Passing nil for the predecessor creates an initial version, like Git's first commit. Subsequent changes build on a predecessor, and inserts, updates, and removes can be combined in a single call:

let secondVersion = try store.makeVersion(
    basedOnPredecessor: firstVersion.id,
    inserting: [newValue],
    updating: [changedValue],
    removing: [obsoleteValueId]
)

Versions are store-wide: once a value is added, it persists in all subsequent versions until explicitly updated or removed. You can retrieve any value at any version.

let value = try store.value(idString: "ABCDEF", at: secondVersion.id)

You can also ask what changed, with store.valueChanges(madeInVersionIdentifiedBy:) and store.valueChanges(madeBetween:and:).

Data is stored once, under the version that wrote it. Later versions just refer to it.

Heads and branches

When concurrent changes happen (edits on two devices between syncs, or writes from both your app and its share extension) the version history naturally diverges. This isn't an error; it's the normal state of decentralized data. The divergence gets reconciled through merging.

Each Version can have up to two predecessors (one for linear history, two for a merge) and any number of successors. A head is a version with no successors, the tip of a line of history. When multiple heads exist, they generally need to be merged.

store.queryHistory { history in
    let heads = history.headIdentifiers
    // ...
}

let latest: Version? = store.mostRecentHead

Always go through queryHistory to touch the History; it serializes access.

Divergence from syncing is anonymous, but you can also make a named Branch for background work, such as a long import. Pass it when saving with the coordinator (save(updating:in:)), and it is recorded in the version's metadata. Named branches are left alone by merge() by default. You bring them in when you are ready, using the headSelection argument.

let importBranch = Branch(rawValue: "import")
try coordinator.save(inserting: importedValues, in: importBranch)
// ...later
try coordinator.merge(headSelection: .allUnbranchedAndSpecificBranches([importBranch]))

Merging and arbiters

When two versions have diverged, LLVS performs a three-way merge: it finds the greatest common ancestor, diffs each side against it, and hands the results to a MergeArbiter. The arbiter decides how to resolve every conflict.

let arbiter = MostRecentChangeFavoringArbiter()
let merged = try store.merge(version: headA, with: headB, resolvingWith: arbiter)

If one version is an ancestor of the other, LLVS fast-forwards without creating a new version, just like Git. If the two versions have no common ancestor at all, it falls back to a two-way merge.

store.mergeHeads(into:resolvingWith:) merges all the other heads into a version in one call, and StoreCoordinator.merge() does the same for the coordinator's current version, using its mergeArbiter. Heads are merged in an order that depends only on the versions themselves, so every device does it the same way.

There are three built-in arbiters:

  • MostRecentChangeFavoringArbiter resolves each conflict individually, keeping whichever change is newer. An update always beats a removal. This is the coordinator's default.
  • MostRecentBranchFavoringArbiter resolves all conflicts in favor of whichever of the two versions has the newer timestamp.
  • MergeableArbiter (in LLVSModel) merges registered model types property by property, as described above.

Conflicts and Value.Fork

For full control, implement the protocol yourself.

public protocol MergeArbiter {
    func changes(toResolve merge: Merge, in store: Store) throws -> [Value.Change]
}

The Merge gives you the two versions, the commonAncestor (if any), and forksByValueIdentifier, a dictionary of Value.Fork describing what happened to each value:

  • .inserted, .updated, .removed (each carrying the branch, .first or .second) and .twiceRemoved are not conflicts. LLVS handles them for you.
  • .twiceInserted, .twiceUpdated, and .removedAndUpdated(removedOn:) are conflicts. fork.isConflicting tells you which is which.

Your arbiter must return a Value.Change for every conflicting fork. Use .preserve(reference) to keep an existing value from one side, .preserveRemoval(id) to keep a removal, or .update(value) to write something new. This is where you encode your app's domain logic. Here is an arbiter where the longer text wins:

final class LongestTextArbiter: MergeArbiter {
    func changes(toResolve merge: Merge, in store: Store) throws -> [Value.Change] {
        let v = merge.versions
        var changes: [Value.Change] = []
        for (valueId, fork) in merge.forksByValueIdentifier {
            switch fork {
            case .twiceInserted, .twiceUpdated:
                let first = try store.value(id: valueId, at: v.first.id)!
                let second = try store.value(id: valueId, at: v.second.id)!
                let winner = first.data.count >= second.data.count ? first : second
                changes.append(.preserve(winner.reference!))
            case let .removedAndUpdated(removedOn):
                let updated = removedOn == .first ? v.second : v.first
                let value = try store.value(id: valueId, at: updated.id)!
                changes.append(.preserve(value.reference!))
            case .inserted, .updated, .removed, .twiceRemoved:
                break
            }
        }
        return changes
    }
}

Structuring your data

If you are not using LLVSModel, how you map your model onto values is up to you, but the granularity matters:

Approach Merging Performance Disk use
One property per Value Best (per-property conflict resolution) Slow (many small reads) Many small files
One entity per Value Good (per-entity conflict resolution) Moderate Moderate
Entire model in one Value Poor (must merge everything manually) Fast (single read) Large per-version files

One entity per Value is a good default, and it is what LLVSModel does. With @MergeableModel you get per-property merging on top, without paying for per-property storage.

Storage Backends

The Storage protocol creates Zone instances, and a Zone is just a raw read/write interface for blobs. Two are included:

  • FileStorage (the default) keeps files on disk under the store's root directory, in 2-character prefix subdirectories to keep the file system happy. It is safe to use from multiple processes.
  • SQLiteStorage (in LLVSSQLite) keeps the same data in SQLite databases.
import LLVSSQLite

let store = try Store(rootDirectoryURL: rootDir, storage: SQLiteStorage())

StoreCoordinator creates its store with file storage. SQLiteStorage is not thread-safe, so use a SQLite-backed store from one queue or actor at a time. To write your own backend, conform to Storage and Zone.

Exchanges

An Exchange sends and receives versions between stores, the equivalent of git push and git pull. Set one on a coordinator and call exchange(), or drive it directly:

let retrievedIds = try await exchange.retrieve()
let sentIds = try await exchange.send()
Backend Library Notes
CloudKitExchange LLVSCloudKit Private (default or custom zone) or public database. Shared databases do not work yet. Supports snapshots.
CloudFileSystemExchange LLVS Works over any CloudFileSystem. Supports snapshots.
WebDAVFileSystem LLVSWebDAV A CloudFileSystem. Base URL plus optional username and password.
GoogleDriveFileSystem LLVSGoogleDrive A CloudFileSystem. Takes an access token or a GoogleDriveAuthenticator.
OneDriveFileSystem LLVSOneDrive A CloudFileSystem. Takes an access token or a OneDriveAuthenticator.
BoxExchange LLVSBox Wraps the Box SDK. Needs the Box trait.
PCloudExchange LLVSPCloud Wraps the pCloud SDK. Needs the PCloud trait.
FileSystemExchange LLVS A shared directory. Good for tests and syncing between processes. Supports snapshots.
MemoryExchange LLVS In memory (an actor). For tests.
MultipeerExchange LLVS Peer to peer, over a PeerTransport that you supply.

A few examples:

// CloudKit, private database
let exchange = CloudKitExchange(
    with: store,
    storeIdentifier: "MyStore",
    cloudDatabaseDescription: .privateDatabaseWithCustomZone(CKContainer.default(), zoneIdentifier: "MyZone")
)

// WebDAV
let webDAV = WebDAVFileSystem(baseURL: serverURL, username: "drew", password: password)
let exchange = CloudFileSystemExchange(cloudFileSystem: webDAV, store: store, basePath: "MyApp")

// A shared directory
let exchange = FileSystemExchange(rootDirectoryURL: sharedDirectoryURL, store: store, usesFileCoordination: false)

CloudFileSystem is a small protocol (exists, list, upload, download, remove), so supporting another file-based service is not much work. MultipeerExchange does not depend on MultipeerConnectivity itself: you implement PeerTransport.send(_:toPeer:) to push bytes to the other peer, and call receiveData(_:) on the exchange when bytes arrive.

Every exchange has a newVersionsAvailable stream (AsyncStream<Void>) that you can use to trigger a sync when the backend is able to tell you something changed. For everything else, conform to Exchange yourself. You only need to implement the primitive operations; retrieve() and send() have default implementations that work out what is missing on each side and transfer it in batches.

Snapshots

When a new device joins, it normally downloads every version from the beginning and rebuilds the store's history. For stores with thousands of versions, this can be slow.

Cloud snapshots solve this by periodically uploading a chunked copy of the entire store. A new device downloads the snapshot, restores it locally, and then uses normal incremental sync to catch up with anything added since. Existing devices are unaffected.

let coordinator = try StoreCoordinator(
    withStoreDirectoryAt: storeURL,
    cacheDirectoryAt: cacheURL,
    snapshotPolicy: .auto
)
coordinator.exchange = myExchange

// On first launch, try to restore from a snapshot before syncing
try? await coordinator.bootstrapFromSnapshot()
try? await coordinator.exchange()
_ = try? coordinator.merge()

bootstrapFromSnapshot() checks whether the exchange and storage support snapshots, whether a compatible snapshot exists, and whether the local store is still empty. If so, it downloads and restores the snapshot. If not, it returns without doing anything, and the app falls back to a full sync with no extra code.

With SnapshotPolicy.auto, the coordinator uploads a new snapshot after an exchange when enough time has passed (minimumInterval, default 7 days) and enough new versions have accumulated (minimumNewVersions, default 20). You can build your own SnapshotPolicy with different numbers. The default is .disabled.

Snapshots need both sides to opt in. The storage must conform to SnapshotCapable (FileStorage and SQLiteStorage do), and the exchange to SnapshotExchange (FileSystemExchange, CloudFileSystemExchange, and CloudKitExchange do). If either side doesn't, snapshot operations are silently skipped.

Samples

The Samples directory has two SwiftUI apps. They are Xcode projects, not part of the package.

  • TheMessage is a minimal app that syncs a single shared message via the public CloudKit database. Good for understanding the basics.
  • LoCo is a contact book that uses LLVSModel, @MergeableModel, and MergeableArbiter, and syncs via a private CloudKit zone.

Upgrading to 0.12

LLVSModel value IDs changed shape. They are now "<instance-id>/<TypeName>" rather than "<TypeName>/<instance-id>".

The Map buckets values by the first two characters of their ID. With the type name leading, every instance of a type landed in one bucket, and each save rewrote a node listing all of them — O(N) per write, which does not hold once a store grows. The instance identifier now leads, so instances spread across buckets.

If you use modelValueID, modelTypeIdentifier(from:) and instanceIdentifier(from:), nothing in your code changes: the signatures are the same, and both accessors now split on the last slash rather than the first, so an instance identifier may itself contain slashes. If you built or parsed these IDs by hand, adjust.

An existing store keeps its old IDs and goes on working, crowded into one bucket as before. There is no migration, and objects written by this version do not match the IDs written by an earlier one, so do not point two builds at one synced store across the upgrade.

fetchAllModels now compares the type identifier rather than matching an ID prefix, so it scans every reference at the version. For a large store, project the type into LLVSProjection and query that instead.

Upgrading to 0.11

The package builds in Swift 6 language mode. Ordinary use is unaffected, and both sample apps needed no changes, but four things break if you extend the framework.

  • Exchange and Zone require Sendable. Your own conformances must be safe to use from more than one thread.
  • Cache requires Sendable keys and values. Its methods take some Hashable & Sendable instead of AnyHashable.
  • @Atomic is now @Guarded. The standard library has its own Atomic, and the two names clashed.
  • SQLiteDatabase.Error.bindingFailed carries a valueDescription: String instead of an Any? value.

Upgrading to 0.10

There are two source breaks.

  • StoreCoordinator.merge() and Store.mergeHeads(into:resolvingWith:) now throw. A failed merge used to crash the app (it was a try! inside). Now you get the error. Add try, or try? if you just want to try again at the next sync. StoreCoordinator.merge() still attempts every head before throwing the first error it met.
  • Box and pCloud need traits. If you use LLVSBox or LLVSPCloud, add traits: ["Box"] or traits: ["PCloud"] to your .package entry, otherwise the module will be empty. Traits need swift-tools-version 6.1 in your own manifest.
S
Description
Low-Level Versioned Store
Readme MIT
2.4 MiB
Languages
Swift 100%