The platform half of chunk 2; the account-add UI is 2d, since it is a design task and the piece that needs an on-device review. A stub ContentProvider turned out to be required and was not in the plan: a sync adapter registers against a content authority, and we publish no provider since :provider was deleted. Without one there is nothing for contentAuthority to name, nothing for requestSync to address, and hasAuthorityAccess() makes every ContentResolver sync call a silent no-op at targetSdk 34+. - CredentialStore: Keystore AES/GCM, blob in its own DataStore file. security-crypto is formally deprecated and terminal. Decryption failure means re-authenticate, never a crash — including ProviderException, which is a RuntimeException and escapes the obvious catches. - CalDavAccounts + SyncAuthenticator: no password reaches AccountManager, which stores them as plain TEXT. The authenticator never returns null — a null is the protocol for "answering asynchronously", and nothing here does, so Settings would wait forever. addAccount refuses with a readable message until 2d ships the screen, rather than opening the home screen and hanging. - SyncAdapterService: enqueue and wait on the unique work *name*, not the request id — enqueueUniqueWork is async so the id is unknown when the wait starts, and under KEEP it may never exist at all. Being deduplicated is not a failure. - Account type and authority are per build variant, so debug and release do not fight over ownership. SyncContractTest guards the Kotlin/resValue pair. - The credential blob is the only thing excluded from backup: Keystore keys are non-exportable, so a restored ciphertext can never be decrypted. Known trade-off recorded in network_security_config.xml and SYNC-PLAN.md: the user CA store is trusted for all traffic, which chunk 5's cert4android should replace rather than sit beside. The instrumented tests here compile but have not been run — device work waits for an explicit go-ahead.
569 lines
33 KiB
Markdown
569 lines
33 KiB
Markdown
# Agendula — CalDAV sync: the build plan
|
||
|
||
**Branch:** `feat/caldav-sync`, cut from `release/v1.0.0`.
|
||
**Design:** [`SYNC.md`](SYNC.md) — the decisions, the traps and the audit trail
|
||
behind them. This document does **not** repeat it. It sequences it.
|
||
**Shape:** five chunks, one branch, one PR at the end.
|
||
|
||
`SYNC.md` is a 1,200-line decision document with roughly forty ⚠️ findings
|
||
scattered through it, most of which are "the obvious implementation is wrong,
|
||
here is why". That is the right shape for a design document and the wrong shape
|
||
for a work queue. This is the work queue.
|
||
|
||
---
|
||
|
||
## Decisions taken before the first line of code
|
||
|
||
`SYNC.md` left three questions open and named two of them as blocking. All three
|
||
are now answered, plus one the phases imply. **These are not reopened by the
|
||
implementation chunks** — if a chunk finds evidence that overturns one, it stops
|
||
and says so rather than quietly picking the other branch.
|
||
|
||
| # | Question | Answer | Consequence |
|
||
|---|---|---|---|
|
||
| 1 | dav4jvm + cert4android distribution, and the Java-21 wall | **Vendor a known-good tree**, recompiled at our Java 17 target | `FAIL_ON_PROJECT_REPOS` stays; no Ktor/guava/xpp3 tail; MPL-2.0 headers retained and the attribution screen becomes mandatory, not optional. Freezes the API churn that shipped two breaking majors nineteen days apart |
|
||
| 2 | Conflict policy on 412 | **Server wins, local edit discarded** | DAVx5's policy. Terminates, always. The discarded edit is *surfaced* — a sync report the user can read — because silence is what makes this policy feel like data loss. The draft's "preserve as a duplicate" stays withdrawn: RFC 4791 §4.1 makes it unuploadable |
|
||
| 3 | External mode's future | **Kept, unchanged** | The mapper and the UI stay dual-capable. Retiring it is a separate decision that does not block sync, and it ships and works today |
|
||
| 4 | iCalendar library | **None — a hand-rolled content-line model.** ⚠️ Overturned during chunk 1; `SYNC.md` § *Libraries* assumed a typed library was the only option | A raw `(name, params, value)` content-line tree is what the unknown-property requirement actually wants: a typed model normalises away exactly what must survive. `ICalendarWriter` already folds in 204 lines, `lib-recur` is already a direct `:app` dependency and does `RRULE`, and `java.time` is native at minSdk 29 — so ical4j's 2.2 MB of duplicated zone data, its production-exhausting `ZoneRulesProvider`, its mandatory properties/cache/registry shims and its release-only R8 failure buy nothing. VTIMEZONE bodies are already on the round-trip allowlist, so they round-trip opaquely. **Cost accepted: we own the lexer.** biweekly was rejected on the same ground as before |
|
||
|
||
---
|
||
|
||
## Scope boundaries
|
||
|
||
**Not in this branch, deliberately:**
|
||
|
||
- **Extraction into floret-kit.** `SYNC.md` § *What lands in floret-kit* is right
|
||
that `core-dav` and `core-sync` belong there and that Calendula needs the same
|
||
primitives. Extracting across a submodule boundary while the API is still
|
||
moving costs more than it buys. **Mitigation, which is a real constraint on
|
||
every chunk:** the DAV and iCalendar layers take no task-domain type and no
|
||
Android UI type, so the later extraction is a file move. Chunk 5 records what
|
||
moves.
|
||
- **Retiring External mode**, per decision 3.
|
||
- **A release.** Nothing here bumps `versionCode` / `versionName`. Sync ships
|
||
when it works against real servers, and that is a decision taken after the
|
||
device review, not inside this plan.
|
||
|
||
**In scope but not verifiable by CI:** everything that needs a real server. The
|
||
per-server trap matrix (`SYNC.md` § *Server reality*) is written as named,
|
||
skipped integration tests in chunks 3–4 and turned on by hand against a live
|
||
Nextcloud/Radicale/Baïkal in chunk 5.
|
||
|
||
---
|
||
|
||
## The five chunks
|
||
|
||
| # | Chunk | Rests on | The thing it must not get wrong |
|
||
|---|---|---|---|
|
||
| 1 | **Mapper** — VTODO ↔ Room | nothing | Unknown properties survive a read-modify-write cycle |
|
||
| 2a | **Vendored DAV foundation** | nothing | Vendor a tree we can actually maintain, and know which of its defects are real |
|
||
| 2b | **Discovery and auth protocol** | 2a | The two collection filters are inversions of the obvious rule |
|
||
| 2c | **The Android account layer** | 2b | Registering the sync adapter, or every `ContentResolver` sync call is a silent no-op |
|
||
| 2d | **Account-add UI** | 2c | Never send a credential into an unvalidated redirect chain |
|
||
| 3 | **Engine core** — read, write, conflicts | 1, 2 | A failed resource must not fail the collection |
|
||
| 4 | **Incremental sync + scheduling** | 3 | Never persist a sync token before the bodies it covers are applied |
|
||
| 5 | **Hardening, compliance, real servers** | 1–4 | Ship no licence violation, and no retry loop on a dead app password |
|
||
|
||
Chunks 1 and 2 are genuinely independent of each other; the ordering between
|
||
them is arbitrary and 1 goes first only because it is the one with a
|
||
specification that can be written down before any network exists.
|
||
|
||
---
|
||
|
||
## Chunk 1 — the mapper
|
||
|
||
**Goal:** a bidirectional VTODO ↔ Room-entity mapper whose *specification is its
|
||
fixture corpus*. Today `ICalendarWriter` is the export half only — write-only,
|
||
hand-rolled, and it silently drops everything it does not model.
|
||
|
||
**The corpus comes first.** Not as a testing courtesy: `SYNC.md` § *Unknown
|
||
properties* is explicit that the fixtures that catch bugs are the adversarial
|
||
ones, and a mapper written before its corpus gets a corpus that ratifies it.
|
||
Minimum set, each one a named failure the design predicts:
|
||
|
||
- a `CLASS:CONFIDENTIAL` task as Nextcloud rewrites it out of a **shared**
|
||
calendar — the one that destroys data on write-back
|
||
- a master plus `RECURRENCE-ID` overrides in one resource
|
||
- unknown properties nested **inside a `VALARM`** — the case a flat
|
||
property-per-row model cannot represent
|
||
- an entirely unknown sub-component
|
||
- a `TZID` the device tzdb does not know
|
||
- a UID containing `/` and `@`
|
||
- one exceeding `CALDAV:max-resource-size`
|
||
- `RRULE` + `DUE` with no `DTSTART`, which has no well-defined `RECURRENCE-ID`
|
||
- `X-MOZ-LASTACK` / `X-MOZ-SNOOZE-TIME` (dropping them causes Thunderbird alarm
|
||
storms) and `X-APPLE-SORT-ORDER`
|
||
- all four recurring-completion models on read (a/b/c/d — we write (a))
|
||
|
||
**Round-trip assertion, stated the way `SYNC.md` corrects it.** Byte-stability is
|
||
unachievable and asserting it produces a corpus that gets normalised until it
|
||
tests nothing. Re-parse both sides into a canonical multiset of
|
||
`(component path, property name, params as a sorted map, unfolded unescaped value)`
|
||
and assert equality **modulo an enumerated allowlist**: `PRODID`, `DTSTAMP`,
|
||
`LAST-MODIFIED`, `SEQUENCE`, VTIMEZONE bodies, fold positions, parameter quoting.
|
||
Byte-equality is asserted where it means something — the unfolded, unescaped
|
||
value octets of every untouched property, and full parameter preservation
|
||
including unknown parameters.
|
||
|
||
**Storage for the unknown half.** A new table, and therefore a **schema
|
||
migration to v2** with an exported schema and a `MigrationTestHelper` test, per
|
||
the repo's existing pattern in `app/schemas/`. It must represent nesting
|
||
(`component path`), ordering, and opaque unknown sub-components. Cap the total
|
||
at ~25 kB per resource, as DAVx5 does, for `CursorWindow` and
|
||
`CALDAV:max-resource-size` both.
|
||
|
||
**Deliverables**
|
||
|
||
- fixture corpus under `app/src/test/resources/vtodo/`, with a README naming what
|
||
each fixture is for
|
||
- `VTodoParser` / `VTodoSerialiser` behind one interface, no Android types
|
||
- unknown-property table, Room migration v1→v2, migration test
|
||
- the `SYNC.md` § *The rest of the minefield* rules as individual tests:
|
||
`SEQUENCE` is preserve-verbatim; `DTSTAMP` regenerates but `LAST-MODIFIED`
|
||
does not; `COMPLETED` is UTC-only; `RELATED-TO;RELTYPE=PARENT` points *up*;
|
||
`PRIORITY` 0 = undefined; a `TRIGGER;RELATED=END` alarm without `DUE` is
|
||
invalid and must be caught **before** PUT, because ordinary UI actions reach it
|
||
- `ICalendarWriter` either becomes the serialiser or is deleted in its favour —
|
||
two VTODO writers in one app is the defect this chunk exists to prevent
|
||
- ical4j's `ical4j.properties`, `MapTimeZoneCache` and registry shim, plus its R8
|
||
keep rules (`-keep class net.fortuna.ical4j.** { *; }` + the `-dontwarn` set),
|
||
because that failure is `release`-only
|
||
|
||
**Also settle here:** the `VALARM` ownership collision. Either the adapter owns
|
||
alarms and `AndroidTasksDataSource.setAlarm` stops bulk-deleting property rows,
|
||
or reminders stay local-only and are never serialised. Picking neither is how one
|
||
local reminder edit destroys every server-side alarm on that task.
|
||
|
||
**Done when:** the corpus round-trips, the migration test passes both directions,
|
||
and no fixture loses an unknown property.
|
||
|
||
### Settled while building it
|
||
|
||
- ⚠️ **No schema migration was needed.** `TaskEntity.unknown_properties` already
|
||
exists at v1, reserved for exactly this and written by nothing. The residue
|
||
fills it. It holds leftover properties followed by whole sub-component blocks —
|
||
`BEGIN`/`END` lines are content lines like any other, so nesting and unknown
|
||
sub-components fall out of the same representation, which is what the plan
|
||
thought needed a second table.
|
||
- **`VALARM` ownership: neither side writes the other's alarms.** `VALARM`s
|
||
round-trip in the residue and are never authored; local reminders live in
|
||
`task_alarms` and are never serialised. The collision `SYNC.md` describes was
|
||
`AndroidTasksDataSource.setAlarm` bulk-deleting *provider property rows* — in
|
||
the own-store world the two stores are disjoint and neither can destroy the
|
||
other. Merging them is a later decision; the point is that it is now a
|
||
decision, not an accident.
|
||
- **The mapper claims a property only when it can reproduce it exactly.**
|
||
Everything else stays in the residue: a value it cannot parse or that is out of
|
||
range (`PRIORITY:11`, `PERCENT-COMPLETE:150`, `SEQUENCE:x`, `STATUS:X-DEFERRED`),
|
||
a time it can read but not reproduce (floating, or a `TZID` this device's tzdb
|
||
lacks), and a `DTSTART`/`DUE` pair that disagrees on value type or timezone —
|
||
one `is_all_day` flag and one `timezone` column cannot author both, and
|
||
flattening the odd one out destroys it. Clamping or defaulting any of these
|
||
would be a silent rewrite of somebody's data. `SUPPRESSED_BY_RESIDUE` makes
|
||
"present in the residue" mean "do not author this".
|
||
- ⚠️ **Suppression is conditional, or editing breaks.** A residue property
|
||
suppresses its column only while the two still agree. If the user edits a field
|
||
whose stamp was unreproducible, the stale residue copy is evicted and the
|
||
column is authored — otherwise changing the due date of an imported task would
|
||
silently do nothing on the server. `RELATED-TO` is the deliberate exception: a
|
||
parent we have not fetched is not the same as no parent, so an unresolvable
|
||
link is kept rather than destroyed.
|
||
- **One normalisation is allowed beyond the stated allowlist**, and it is an
|
||
equivalence rather than a concession: an absent `STATUS` and
|
||
`STATUS:NEEDS-ACTION` mean the same thing for a to-do. Any other `STATUS`
|
||
compares exactly, so losing a completion still fails the corpus.
|
||
|
||
---
|
||
|
||
## Chunk 2a — the vendored DAV foundation
|
||
|
||
⚠️ **Split out of chunk 2 while building it.** Vendoring is a self-contained
|
||
piece with its own verification — upstream's suite has to pass unmodified before
|
||
anything is built on top — and bundling it with discovery and auth would have
|
||
produced one commit nobody could review.
|
||
|
||
**Version:** dav4jvm **2.2.1**, the last OkHttp release. 3.0.0 deleted the OkHttp
|
||
package for Ktor and 4.x requires Java 21, and `SYNC.md`'s entire auth section is
|
||
written in OkHttp terms — preemptive Basic via an `Interceptor`, OkHttp stripping
|
||
`Authorization` on cross-host redirects, `BasicDigestAuthHandler` because OkHttp
|
||
has no Digest. Taking 4.x would have invalidated all of it. A plain JVM module,
|
||
not an Android library: the tree has zero Android imports and must keep it that
|
||
way, or the floret-kit extraction stops being a file move.
|
||
|
||
Full record in [`dav/PROVENANCE.md`](../dav/PROVENANCE.md). Of the two defects
|
||
`SYNC.md` said we would inherit, **one was not real** — 2.2.1 does follow 303 —
|
||
and the other (`dav4jvm#209`, permanent redirects never reaching the caller) is
|
||
fixed. A third, which the audit did not have, was found by writing the tests
|
||
upstream never wrote: **HTTP dates were parsed and formatted in the device's
|
||
local zone**, putting every `getlastmodified` out by the local UTC offset.
|
||
|
||
**Done when:** upstream's 63 tests pass unmodified, `commons-lang3` and the xpp3
|
||
runtime dependency are gone, and `:app` links against the module.
|
||
|
||
---
|
||
|
||
## Chunk 2b — the discovery and auth protocol
|
||
|
||
⚠️ **Split again while building it.** The seam is the platform: everything here
|
||
is protocol, testable on a JVM against MockWebServer, and none of it touches
|
||
Android. Keystore, `AccountManager`, Custom Tabs and the account-add UI are a
|
||
different kind of work with a different kind of verification — including the
|
||
on-device review — so they are chunk 2c.
|
||
|
||
Lands in a new `:caldav` module: MIT, plain JVM, `api`-depending on `:dav`. A
|
||
separate module from the vendored MPL tree keeps the licences unmixed and makes
|
||
"no Android types" a compile-time guarantee rather than a discipline.
|
||
|
||
**One dependency added:** `dnsjava` 3.6.3 (BSD-3) for the SRV/TXT half of
|
||
RFC 6764. Android's own `DnsResolver` is callback-only and cannot do the
|
||
`TXT path=` lookup, and JNDI's DNS provider does not exist on Android at all.
|
||
It is what DAVx5 uses. Behind a `DnsResolver` interface so the pipeline is
|
||
testable without a network.
|
||
|
||
**Done when:** the trap table is executable — Posteo's SRV-only 8443, GMX's
|
||
`TXT path=`, null SRV targets, Google's dead-end record, SOGo's triple
|
||
resourcetype, Nextcloud's trashed calendar, the absent/empty component set, and
|
||
`<D:unauthenticated/>` as a 200 that means failure.
|
||
|
||
---
|
||
|
||
## Chunk 2c — the Android account layer
|
||
|
||
Everything that needs the platform and nothing that needs a screen: Keystore
|
||
credentials, `AccountManager`, the stub sync adapter, the manifest, and the
|
||
network security config. The account-add UI is 2d — it is a design task, it
|
||
needs the `material-3` skill, and it is the piece that needs an on-device review.
|
||
|
||
### ⚠️ A stub `ContentProvider` turned out to be required
|
||
|
||
Not in the original plan, and it is load-bearing. A sync adapter registers
|
||
against a **content authority**, and Agendula publishes no provider — `:provider`
|
||
was deleted when we took our own Room store. With no authority there is nothing
|
||
for `<sync-adapter android:contentAuthority>` to name, nothing for
|
||
`requestSync` to address, and nothing for Settings to render a switch against.
|
||
`SyncStubProvider` stores nothing and exists solely to hold up that end.
|
||
|
||
### Other things settled here
|
||
|
||
- **The account type and authority are per build variant**, generated by
|
||
`resValue`. Two installs cannot own the same account type, so a debug build
|
||
sharing the release build's would fight it. `SyncContractTest` asserts the
|
||
Kotlin constants and the generated strings still agree — drift between them is
|
||
invisible at build time and surfaces as an account the framework will not
|
||
trigger.
|
||
- **The credential blob is the one thing excluded from backup.** It lives in its
|
||
own DataStore file for exactly that reason. Keystore keys are non-exportable,
|
||
so a restored ciphertext is permanently undecryptable; excluding it means the
|
||
user signs in again, which is the honest outcome. Excluding the database or all
|
||
of DataStore would trade a latent bug for a live one.
|
||
- ⚠️ **The network security config trusts the whole user CA store, for all
|
||
traffic.** `SYNC.md` specifies it, and without it a correctly installed private
|
||
CA is not trusted at all — which is the self-hosting case. But `base-config` is
|
||
the widest form: any CA in the user store can intercept the CalDAV connection
|
||
and read the app password from the `Authorization` header. **Chunk 5's
|
||
cert4android should replace this block, not sit beside it** — its per-connection
|
||
approval is the narrow version of the same capability.
|
||
- **`FOREGROUND_SERVICE` appears in the merged manifest** without being declared,
|
||
from `work-runtime`. That is not the FGS route: below API 31 WorkManager
|
||
implements expedited work with a foreground service, and minSdk is 29. Noted in
|
||
the manifest because it shows in F-Droid's permission diff.
|
||
|
||
**Not run:** the instrumented tests here (`CredentialStoreTest`,
|
||
`SyncContractTest`) compile but have not been executed — `CLAUDE.md` reserves
|
||
device interaction for when it is explicitly asked for. They need a device run
|
||
before chunk 2 is done.
|
||
|
||
---
|
||
|
||
## Chunk 2d — the account-add UI
|
||
|
||
**Goal:** the app can add a CalDAV account and list its VTODO collections. No
|
||
syncing yet.
|
||
|
||
**Vendoring, first.** A new Gradle module holding dav4jvm + cert4android sources
|
||
at our Java 17 target, MPL-2.0 headers intact, with a `PROVENANCE.md` recording
|
||
the upstream commit — the repo already has this pattern from `:provider`. Fix the
|
||
two inherited defects while vendoring, since we now can: **303 is not handled**
|
||
(RFC 6764 §5 names it explicitly) and **issue #209** mutates `location` in place
|
||
so permanent redirects never reach the caller.
|
||
|
||
**Discovery** is the nine-step pipeline in `SYNC.md` § *Generic CalDAV
|
||
discovery*, including every step marked ⚠️ MISSING there. The two filter
|
||
corrections are the important part and both are inversions of the obvious:
|
||
|
||
- **`supported-calendar-component-set` absent means "supports everything."**
|
||
Request properties **by name** — RFC 4791 §5.2.3 says the property SHOULD NOT
|
||
come back from an allprop request. Treat empty as all.
|
||
- **Classify on `resourcetype` as an unordered set, with a positive test for
|
||
`CALDAV:calendar`** — not by excluding `schedule-outbox`, because SOGo's main
|
||
personal calendar reports `collection` + `calendar` + `schedule-outbox`
|
||
simultaneously for every non-Apple client, which is exactly what we are.
|
||
|
||
The discovery traps table (Posteo SRV-only on 8443, GMX/Web.de `path=`, null SRV
|
||
target, Google's dead-end SRV, iCloud/Zoho well-known 401, `dav.runbox.com`'s
|
||
HTTPS→HTTP downgrade, `<D:unauthenticated/>` as a 200 that means failure,
|
||
cross-host home sets) becomes a table-driven unit test over recorded responses.
|
||
|
||
**Nextcloud Login Flow v2** with all five corrections: `POST` form-encoded (a
|
||
`GET` is 405); an explicit `User-Agent`, because it becomes the app password's
|
||
name in the user's security settings and `okhttp/4.12.0` defeats the entire point
|
||
of the flow; 404 means *pending only* and anything that is neither 404 nor 200
|
||
stops the poll; validate the `endpoint` origin and refuse an `http` downgrade
|
||
outright; and `loginName` is **not** the uid — it is the Basic auth username and
|
||
nothing else. Custom Tabs needs the `<queries>` entry, the `ACTION_VIEW`
|
||
fallback, `{token, endpoint, deadline}` persisted before launch, and an explicit
|
||
"I finished / Cancel" affordance.
|
||
|
||
**Auth is not just Basic.** Digest (Baïkal defaults to it, OkHttp has none),
|
||
Basic sent preemptively via an `Interceptor` gated to HTTPS and the account's own
|
||
origin, and domain-detection at account-add time for the three servers whose real
|
||
error is not "wrong password": Fastmail (app password; Basic plan has no CalDAV),
|
||
iCloud (app-specific password, 2FA to mint one), Google (OAuth2-only — refuse
|
||
with an explanation, `SYNC.md` drops it as a target).
|
||
|
||
**Credentials:** Keystore `AES/GCM/NoPadding`, blob in DataStore. Do not call
|
||
`setUserAuthenticationRequired`; do **not** set `setUnlockedDeviceRequired`
|
||
(breaks background sync). `AEADBadTagException` and
|
||
`KeyPermanentlyInvalidatedException` mean *re-authenticate*, not crash. Exclude
|
||
the blob — and only the blob — from Auto Backup: a restored ciphertext is
|
||
permanently undecryptable.
|
||
|
||
**The account plumbing, which is the part that silently does nothing if skipped.**
|
||
`AccountManager` **plus a registered `AbstractThreadedSyncAdapter`** whose
|
||
`onPerformSync` enqueues a WorkManager job and waits. Without a registered
|
||
adapter, `hasAuthorityAccess()` makes every `ContentResolver` sync API a silent
|
||
no-op at targetSdk ≥ 34 — no exception, no log, and it passes on a Robolectric
|
||
shadow. Manifest: `INTERNET`, `READ_SYNC_SETTINGS`, `WRITE_SYNC_SETTINGS`,
|
||
`ACCESS_NETWORK_STATE`; the authenticator `<service>` exported and guarded by
|
||
`android.permission.ACCOUNT_MANAGER` (`ACCOUNT_AUTHENTICATOR` does not exist).
|
||
Ship an in-app sync button, since "Sync now" stays greyed out under
|
||
`userVisible="false"`. Add `androidx.work`, `androidx.hilt:hilt-work` and
|
||
`androidx.browser` to the catalogue, with the `HiltWorkerFactory`, the removed
|
||
default `WorkManagerInitializer` and an `@EarlyEntryPoint` for the authenticator.
|
||
|
||
**Also:** ship a `network-security-config` with `<certificates src="user"/>`, or
|
||
a user who correctly installs their private CA is still not trusted.
|
||
|
||
**UI in this chunk needs on-device review before it can be called done** — the
|
||
account-add flow is the first screen a self-hoster meets, and per `CLAUDE.md`
|
||
nothing UI-facing gets tagged before that review.
|
||
|
||
**Done when:** an account can be added against a real Nextcloud and its task
|
||
lists appear, the account shows in system Settings with a working sync switch,
|
||
and the discovery trap table is green.
|
||
|
||
---
|
||
|
||
## Chunk 3 — engine core
|
||
|
||
**Goal:** a full bidirectional sync that is correct but not yet clever — no
|
||
sync-collection, no scheduling beyond a manual trigger.
|
||
|
||
**Read path.** `REPORT calendar-query` with a VTODO comp-filter and **no
|
||
time-range**, matching DAVx5 — which deliberately does not use RFC 6578 for
|
||
tasks, and omits the time-range because some servers return no tasks without a
|
||
time at all. Then `calendar-multiget` in batches, **matching returned hrefs
|
||
against what was asked for**, because real servers reply with responses for
|
||
unrelated URLs. Never request `calendar-data` inside a REPORT that is not
|
||
sanctioned to carry it.
|
||
|
||
**Write path.** `If-None-Match: *` on create, `If-Match` on update and DELETE.
|
||
Send `Prefer: handling=strict` to sabre-based servers — the cheapest single fix
|
||
in the whole audit, because it preserves both our bytes and the ETag. Request
|
||
`Accept-Encoding: identity`. Strip `W/` and keep a weak flag; **if a PUT returns
|
||
no ETag or a weak one, discard it and re-fetch.**
|
||
|
||
**412 means three different things** and they get three code paths: on create it
|
||
means the filename is taken (re-fetch, adopt if the UID matches, else regenerate
|
||
as a UUID); on update it means the server is newer (decision 2: server wins,
|
||
report the discard); on update where the resource is *gone* it is spec-correct
|
||
and not a conflict at all (`HEAD` to disambiguate — 404 is delete-vs-edit).
|
||
|
||
**Never write back a body whose ETag does not match the hash of what we
|
||
downloaded.** This is the Nextcloud shared-calendar landmine and it is the single
|
||
most destructive bug available in this chunk: `CalendarObject::get()` strips
|
||
`VALARM` on read-only shares and reduces `CLASS:CONFIDENTIAL` to a VEVENT-shaped
|
||
whitelist that deletes `DUE`, `STATUS`, `COMPLETED`, `PERCENT-COMPLETE`,
|
||
`PRIORITY` and `RELATED-TO` — while leaving the ETag untouched.
|
||
|
||
**Isolation, in two layers.** A failed collection must not fail the account, and
|
||
— its twin, which the draft omitted — **a failed resource must not fail the
|
||
collection.** Per-resource quarantine with a failure counter, *not* backoff: a
|
||
single HTTP 400 has halted all calendar sync in DAVx5 for weeks. 507 MUST NOT be
|
||
auto-retried (RFC 4918 §11.5), and 5xx is not safely retryable either — a
|
||
contradictory `RRULE`/`EXDATE` pair returns 500 from Nextcloud and will do so
|
||
forever.
|
||
|
||
**Client-side validation before PUT**, because sabre returns 415 for things
|
||
ordinary UI actions produce: `DUE` before `DTSTART`, value-type mismatch between
|
||
them, a present `METHOD`, multiple UIDs, mixed component types in one resource.
|
||
|
||
**href ≠ UID.** Sanitise filenames on vdirsyncer's rule — `a–zA–Z0–9_.-+`,
|
||
**excluding `@`** — cap the basename near 200 bytes, fall back to a UUID.
|
||
DELETE: conditional on `If-Match`, 404/410 count as success, and a locally
|
||
deleted resource that was never uploaded is never DELETEd.
|
||
|
||
**Local change detection** is `is_dirty = 1 OR is_deleted = 1` scoped by list —
|
||
the columns already exist in the v1 schema. Every downstream insert writes
|
||
`is_dirty = 0` explicitly.
|
||
|
||
**Done when:** a task created in Nextcloud web appears locally and vice versa;
|
||
a concurrent edit resolves server-wins with a visible report; a quarantined
|
||
resource does not stop its collection.
|
||
|
||
---
|
||
|
||
## Chunk 4 — incremental sync and scheduling
|
||
|
||
**Goal:** `sync-collection` as the optimisation on top of chunk 3's baseline, and
|
||
sync that runs by itself.
|
||
|
||
Every numbered finding in `SYNC.md` § *RFC 6578* is a test in this chunk. The
|
||
four that are not optional:
|
||
|
||
1. **Invalidation has no status code.** Ignore the status; match
|
||
`<D:valid-sync-token/>` anywhere in the body on **any** 4xx. Observed in the
|
||
wild: 403 (sabre), 400 (Google), 409 (Radicale), 412 (Evolution accepts it).
|
||
Thunderbird accepts only 400 and therefore never recovers from the 403 most of
|
||
the self-hosted world emits.
|
||
2. **Initial sync must not report deletions**, so mark-and-sweep is mandatory and
|
||
`initialIncomplete` must be persisted *atomically with the token* — otherwise a
|
||
resumed partial sync sweeps against an incomplete set and **deletes live
|
||
data**.
|
||
3. **Persist the token only after the bodies are applied**, per page. The RFC's
|
||
own Appendix B has this backwards. Under WorkManager, process death mid-sync is
|
||
routine.
|
||
4. **A token the server accepts over a change log it already pruned** returns 207,
|
||
zero changes, and no error. RFC 6578 gives no signal for it. The only
|
||
mitigation is **periodic full reconciliation** (PROPFIND `Depth: 1` + ETag
|
||
diff) on a slow cadence regardless of the token — so chunk 3's full path is a
|
||
permanent safety net, not a fallback.
|
||
|
||
Plus: no `DAV:limit` (Nextcloud regressed it to a localised HTML error page);
|
||
507 on the SELF href is truncation, 507 as the outer status means retry without
|
||
the limit; an iteration cap **and** a no-progress guard, because the RFC never
|
||
requires the token to advance; tokens are opaque and **never reused across
|
||
collections**, keyed `(accountId, collectionUrl)`; deletion is
|
||
`<D:status>404</D:status>` at *`<D:response>`* level, not inside a `propstat`; a
|
||
207 with no `<D:sync-token>` degrades to PROPFIND rather than throwing; and
|
||
`sync-collection` **never reports collection property changes**, so displayname,
|
||
colour and read-only status refresh only via PROPFIND on the home set.
|
||
|
||
**Three membership edge cases** that each look like a bug: create-then-delete
|
||
between syncs is reported as removed (the delete handler no-ops on an unseen
|
||
href); delete-then-recreate at the same URI is reported as *changed* (re-read the
|
||
UID from the body); and ACL churn may be reported as removal, so **apply a sanity
|
||
threshold before acting on a large delete batch.**
|
||
|
||
**Scheduling has a ceiling.** Make sync chunked and resumable — persist the
|
||
cursor per collection so a killed worker resumes rather than restarts. Periodic
|
||
sync is a plain `PeriodicWorkRequest`, no foreground service. "Sync now" from a
|
||
visible screen uses `setExpedited(RUN_AS_NON_EXPEDITED_WORK_REQUEST)`, and
|
||
`getForegroundInfo` is implemented **unconditionally** — omitting it crashes
|
||
below API 31 and we support 29. Android 15 forbids starting a `dataSync` FGS from
|
||
`BOOT_COMPLETED`, and we have a boot receiver. Sync hard on app open and on
|
||
connectivity-regained, and be honest in the UI about cadence: in the `rare` and
|
||
`restricted` buckets network access is off entirely and the worst case is
|
||
genuinely "once overnight".
|
||
|
||
**Done when:** a token invalidation on any of the four status codes recovers; a
|
||
process kill mid-page loses nothing and duplicates nothing; the periodic worker
|
||
runs and the manual trigger is instant.
|
||
|
||
---
|
||
|
||
## Chunk 5 — hardening, compliance, real servers
|
||
|
||
**Goal:** the things that are not features and are not optional.
|
||
|
||
**The server matrix, turned on.** Nextcloud, Radicale, Baïkal (**on Digest**),
|
||
SOGo, Fastmail, iCloud — each per-server trap from `SYNC.md` § *Server reality*
|
||
as a named test. Two must be explicit because both silently destroy data:
|
||
**round-tripping a task from a shared Nextcloud calendar**, and **an
|
||
ETag-unchanged body change on SOGo** (its ETag is a row-version counter and the
|
||
body is regenerated per principal). Also: Nextcloud's per-calendar UID uniqueness
|
||
(409 `no-uid-conflict`), its trashbin renaming hrefs to `<name>-deleted.ics` so
|
||
delete→recreate→delete returns 403, MKCALENDAR rate-limited 10/hour; SOGo never
|
||
invalidating a token and using second-granularity ones; Radicale never enforcing
|
||
`supported-calendar-component-set`. Feature-detect collection creation via
|
||
OPTIONS and **disable the "new task list" UI where neither MKCALENDAR nor
|
||
extended MKCOL exists** (iCloud is MKCOL-only, Google has neither, Posteo
|
||
disables it).
|
||
|
||
**Revocation, both directions.** On 401: stop that account immediately, mark
|
||
`NEEDS_REAUTH`, notify with a deep link into the login flow, and **do not retry
|
||
on a timer** — Nextcloud's brute-force protection throttles then 429s *per source
|
||
IP*, so a retry loop on a dead app password takes down the user's other Nextcloud
|
||
clients and looks like we broke their server. Distinguish 401 from 403 from
|
||
429/503 (honour `Retry-After`). On account removal: best-effort
|
||
`DELETE /ocs/v2.php/core/apppassword` with `OCS-APIRequest: true`, or
|
||
uninstalling never revokes access.
|
||
|
||
**The backup path, which arms itself the day this ships.** Post-restore
|
||
reconciliation must **offer to re-attach the account**, never prune. Pruning is
|
||
event-driven off `AccountManager`'s account-removed broadcast, not off "absent
|
||
from the visible set". Device-verify by restoring onto a fresh device and
|
||
confirming nothing is deleted — `SYNC.md` is explicit that excluding the database
|
||
from backup is the wrong fix, because Auto Backup is currently Local mode's only
|
||
safety net.
|
||
|
||
**Licence attribution — a plain violation until it exists.** MPL-2.0 §3.2(a)
|
||
(dav4jvm, cert4android) requires telling recipients how to obtain source; §3.4
|
||
requires retaining file headers; BSD-3 (ical4j) requires reproducing the
|
||
copyright notice in binary distributions; Apache-2.0 §4(d) propagates NOTICE.
|
||
Settings exposes only our own MIT `LICENSE` today. Note ical4j's POM declares a
|
||
non-SPDX licence name and a `LICENSE` URL that 404s, so generators produce empty
|
||
output — this one is written by hand.
|
||
|
||
**Play compliance.** A privacy policy, linked in Console **and in the app**.
|
||
File as **Collected, not Shared**, encrypted in transit — "not collected" is not
|
||
defensible, because Play defines collection as transmitting off-device
|
||
irrespective of recipient. Ship a "Remove account and delete local data" action
|
||
even though the Account Deletion policy does not apply. **Do not ship
|
||
`REQUEST_IGNORE_BATTERY_OPTIMIZATIONS` in the Play build** — generic server sync
|
||
is not on the acceptable-use list; use
|
||
`ACTION_IGNORE_BATTERY_OPTIMIZATION_SETTINGS`.
|
||
|
||
**Minified build.** `:app` runs R8 + resource shrinking in `release`, and
|
||
`proguard-rules.pro` already documents two pruning outages. Verify on the
|
||
`releaseTest` build type, on device.
|
||
|
||
**Record the extraction.** A short section listing what moves to floret-kit
|
||
(`core-dav`, `core-sync`, Login Flow v2, credential storage, the licence screen)
|
||
and what stays app-local (the mapper, conflict policy and its UI, storage modes),
|
||
so the follow-up branch is a file move and not a rediscovery.
|
||
|
||
**Done when:** the matrix is green where a server exists to run it against, the
|
||
attribution screen ships, and a `releaseTest` build syncs on a real device.
|
||
|
||
---
|
||
|
||
## Definition of done, every chunk
|
||
|
||
1. It builds: `./gradlew :app:assembleDebug`.
|
||
2. Unit tests pass:
|
||
`./gradlew :app:testDebugUnitTest :dav:test :caldav:test`. ⚠️ Name the plain
|
||
JVM modules explicitly — `testDebugUnitTest` is an Android-variant task and
|
||
does not exist on them, so their suites would otherwise be compiled by nobody
|
||
and run by nobody. Add each new module here **and** to CI.
|
||
3. Lint passes: `./gradlew :app:lintDebug`. Added after chunk 1 shipped a literal
|
||
byte-order mark that builds and tests both accepted and CI's lint step would
|
||
have rejected.
|
||
4. `/code-review high` on the chunk's diff, with the findings either fixed or
|
||
answered in the commit message. For a chunk that vendors code, scope the
|
||
review to the files we wrote — reviewing thousands of lines of verbatim
|
||
upstream is noise, and its defects belong in `PROVENANCE.md`.
|
||
5. One commit per chunk, referencing the chunk number. No release, no tag.
|
||
|
||
Chunk 2 additionally does not count as done until the account-add UI has had an
|
||
**on-device review with an explicit go-ahead** — `CLAUDE.md`'s rule, and this is
|
||
exactly the case it exists for.
|
||
|
||
## What stops the loop
|
||
|
||
- A `SYNC.md` decision that the evidence overturns. Say so; do not pick the other
|
||
branch silently.
|
||
- Anything requiring a real server that is not reachable — write the test, mark
|
||
it skipped, name it in the commit, move on.
|
||
- Anything that would need a device beyond `adb install`.
|