makiolaj/agendula
1
0
Fork 0
Code Issues Pull Requests Actions Releases 7 Activity

docs: decide to build our own store and delete the vendored provider

Browse Source 
The vendored dmfs provider was kept on the grounds that it hands us the
sync bookkeeping for free. The phase-1 sync audit measured that
bookkeeping and found most of it broken, absent, or unusable: _DIRTY not
set on delete, no home for a per-collection sync token, read-only
collections inexpressible, ACCOUNT_TYPE write-once so enabling sync is a
full migration, and cleanUpLists able to delete a user's lists after a
backup restore. Sixteen findings are provider-imposed rather than
platform- or protocol-imposed.

Costing the alternative showed the swap is far smaller than assumed.
TasksDataSource is already a 14-method, domain-shaped interface;
exactly one file above the data layer references TasksContract. The
work is a second implementation behind an interface built for it, not a
rewrite. Against ~5 weeks to build, owning the store removes 2.5-4
weeks from the sync plan, and 8,200 of the vendored 14,555 lines are
things we would never write - 23 migrations from a 2013 schema, 798
lines of full-text search the app has zero call sites for, and 1,581
lines of a type-safe layer over ContentValues that Room deletes.

External mode (OpenTasks, tasks.org) is unaffected and keeps every
file that describes somebody else's schema.

STORAGE-DECISION.md is the reasoning; OWN-STORE.md is the architecture
and the six-phase plan. :provider stays in-tree until phase 5 so
recurrence parity can be tested against it before it goes.

Also corrected here: the provider's JVM test count (51 -> 56, measured
from the test-results XML) and a fourth site of the debunked "switching
sync on is never a migration" claim, in StorageMode.kt.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jean-Luc Makiola
2026-08-13 14:46:07 +02:00
parent 98ed339346
commit 13cb27b2ab
10 changed files with 1984 additions and 39 deletions
Download Patch File Download Diff File Expand all files Collapse all files

439
docs/OWN-STORE.md Normal file
View File

@@ -0,0 +1,439 @@
# Agendula's own task store — architecture and plan
**Branch:** `feat/own-store`
**Decision:** taken. `docs/STORAGE-DECISION.md` costed it; this is the build.
**Supersedes:** the "keep `:provider`" position in `STORAGE-AND-SYNC.md` and the
"Settled" section of `SYNC.md`.
---
## The decision, in one paragraph
Agendula stops vendoring the dmfs OpenTasks provider. The `:provider` module —
14,555 lines of Java, 1.66× the size of the app itself — is **deleted**. In its
place the app gets its own Room database, designed for the two things Agendula
actually does: show tasks, and sync them over CalDAV. Support for *external*
providers (OpenTasks, tasks.org) **stays**, unchanged, as a user choice — so
anyone already syncing through DAVx5 keeps working exactly as they do today.
Agendula becomes a normal Android app with a normal database, plus an optional
compatibility path into somebody else's ContentProvider.
---
## What changes and what does not
```
BEFORE AFTER
UI / ViewModels UI / ViewModels
│ │
TasksRepository TasksRepository ← unchanged
│ │
TasksDataSource (interface) TasksDataSource ← unchanged
│ ╱ ╲
AndroidTasksDataSource RoomTasksDataSource AndroidTasksDataSource
│ │ │
ContentResolver Room / SQLite ContentResolver
│ │ │
┌────┴─────┐ our tables ┌──────┴──────┐
│ │ │ │
:provider OpenTasks OpenTasks tasks.org
(deleted) tasks.org (external, unchanged)
```
**Unchanged above the data layer.** `TasksRepository`, every ViewModel, every
screen. Verified: exactly one file outside `data/tasks` references
`TasksContract` (`domain/Models.kt`, for five constants), and it stops doing so
in phase 0.
**Deleted.** The `:provider` Gradle module, its manifest `<provider>`, its two
custom permissions, its 84 Java files, its 13 translated string resources, and
its three dmfs runtime dependencies from `:provider`'s own build file.
**Kept for External mode.** `TasksContract.kt`, `ColumnReader.kt`,
`TaskMapper.kt`, `TaskWriteMapper.kt`, `AndroidTasksDataSource.kt`,
`ProviderResolver.kt`, `ProviderEnvironment.kt`, `TaskProjections.kt`. These
describe *somebody else's* schema and are exactly right for that job.
---
## Storage modes after the change
```kotlin
enum class StorageMode {
/** Agendula's own Room database. The default; always available. */
OWN,
/** A tasks provider app already installed — OpenTasks, tasks.org. */
EXTERNAL,
}
```
`LOCAL` (meaning "our bundled dmfs provider") is gone. `ProviderResolver.own`
and the `TaskProvider(isOwn = true)` case go with it: in `OWN` mode there is no
authority, no ContentResolver and no permission to grant.
`ProviderResolver` narrows to what it was always really for — **discovering
external providers** — and `ProviderStatus.READY` becomes unconditional in `OWN`
mode.
### Consequences worth stating plainly
- **No runtime permission is needed for the default path.** Today's permission
prompt only ever applied to External mode; now that is visibly true.
- **Third-party apps can no longer read Agendula's tasks.** We publish no
ContentProvider. Users who need interop pick External mode, or wait for a
possible read-only facade (explicitly out of scope — see *Deliberately not
doing*).
- **Auto Backup gets simpler and safer.** One Room file we control, with a
documented restore path, instead of a provider database whose `cleanUpLists`
routine could delete restored lists whose accounts no longer exist.
---
## The schema
Four tables. Designed from Agendula's actual reads and writes plus RFC 5545's
`VTODO`, not inherited from a 2013 schema.
### `task_lists`
| Column | Type | Notes |
|---|---|---|
| `id` | INTEGER PK | |
| `name` | TEXT NOT NULL | |
| `color` | INTEGER NOT NULL | ARGB |
| `account_id` | INTEGER NULL | FK → `accounts`, NULL = device-only |
| `is_visible` | INTEGER NOT NULL | default 1 |
| `is_synced` | INTEGER NOT NULL | default 1 |
| `owner` | TEXT NULL | CalDAV owner display name |
| `is_read_only` | INTEGER NOT NULL | **new** — the provider could not express this at all |
| `sort_order` | INTEGER NOT NULL | user ordering, which the provider also lacked |
| `href` | TEXT NULL | collection URL, relative to the account root |
| `ctag` | TEXT NULL | |
| `sync_token` | TEXT NULL | RFC 6578, **per collection** — not squatted into a shared slot |
| `is_dirty` | INTEGER NOT NULL | a real boolean, not dmfs's monotonic counter |
> `account_id` being nullable is the single most important schema change. In the
> dmfs provider `ACCOUNT_TYPE` is write-once and throws on change, which made
> "turn sync on" a full data migration. Here, attaching a local list to an
> account is `UPDATE task_lists SET account_id = ?`.
### `tasks`
Master rows *and* recurrence overrides live here; an override is a row with
`recurrence_id` set and `parent_task_id` pointing at its master.
| Group | Columns |
|---|---|
| identity | `id`, `list_id`, `uid` (NOT NULL, minted at creation), `href`, `etag` |
| content | `title`, `description`, `location`, `url`, `color` |
| state | `status`, `percent_complete`, `completed_at`, `priority`, `classification` |
| time | `dtstart`, `due`, `duration`, `is_all_day`, `timezone` |
| recurrence | `rrule`, `rdate`, `exdate`, `recurrence_id`, `master_id` |
| hierarchy | `parent_id`, `sort_order` |
| audit | `created_at`, `last_modified`, `sequence` |
| sync | `is_dirty`, `is_deleted`, `unknown_properties` |
Two entries deserve explanation.
**`uid` is NOT NULL and assigned at creation.** Every task gets a real
RFC 4122 UUID the moment it is inserted, in every mode, synced or not. This
closes the gap `ICalendarWriter.uidFor` currently papers over by synthesising
`agendula-<rowid>@…`, and it means any local task can later be pushed to a
server without duplicating. The provider never assigned one.
**`unknown_properties`** holds the raw unfolded iCalendar lines of every
property we do not model — `ATTENDEE`, `CATEGORIES`, `X-*`, `GEO`, and anything
a future RFC adds. On write we re-emit them verbatim after the properties we do
own. This is what makes an honest round-trip possible, and it replaces the
provider's `data0`–`data15` bag with something that cannot silently lose a field
it has no column for.
Indices: `(list_id, is_deleted)`, `(parent_id)`, `(uid)` unique per list,
`(master_id, recurrence_id)`, `(is_dirty)`.
### `task_alarms`
| Column | Notes |
|---|---|
| `id`, `task_id` | FK, `ON DELETE CASCADE` |
| `minutes_before` | positive = before the reference |
| `reference` | `DUE` or `START` |
| `message` | optional |
Replaces `AlarmHandler` (133 lines of Java) and the `dataN` slot convention.
Delete-and-reinsert stops being necessary — the provider's re-validate-everything
behaviour was the only reason `setAlarm` worked that way.
### `accounts`
| Column | Notes |
|---|---|
| `id`, `display_name`, `principal_url`, `home_set_url` |
| `username` | the app password is **not** here — Keystore only, per `SYNC.md` |
| `last_sync_at`, `last_sync_error` |
Not populated until `SYNC.md` phase 2, but the FK exists from v1 so enabling
sync never requires a schema migration.
---
## Recurrence: expand at read, not on write
The provider maintained a materialised `instances` table, recomputed by
`Instantiating.java` on every write — and still only ever materialised **one**
upcoming occurrence.
Agendula expands lazily instead:
```
tasks (masters + overrides) ──► RecurrenceExpander ──► List<Task>
rrule/rdate/exdate (lib-recur, in memory) occurrences
```
This is the right call here because **the repository already filters and sorts
in Kotlin, not SQL**. `TasksRepositoryImpl.loadTasks` reads the whole set,
applies `TaskFiltering.matches`, then `TaskSorting.DEFAULT`. Nothing depends on
the database being able to order by instance time, so nothing is lost — and a
materialised table's entire class of staleness bugs never exists.
- Bounded window: expansion is capped (default: 1 year back, 2 years forward,
hard ceiling of N occurrences per series) so an unbounded `RRULE` cannot hang
the UI.
- `lib-recur` **pinned at 0.12.2** — 0.16.0 removed `RecurrenceSet`. We pin
because we chose to, and the pin is now ours to lift on our own schedule.
- Client-side expansion is required for CalDAV regardless: server-side
`CALDAV:expand` on `VTODO` is broken on every server `SYNC.md` targets.
### Completing one occurrence of a recurring task
The provider's `Detaching.java` implemented **model (d): detach the occurrence
as a brand-new task with its own UID**. We inherited that without ever choosing
it, and it is the model least compatible with CalDAV.
**We implement model (a): a `RECURRENCE-ID` override.** Completing one
occurrence writes a second `tasks` row with the same `uid`, a `recurrence_id`
naming the occurrence, `master_id` pointing at the series, and the completed
state. This is what RFC 5545 specifies and what every other CalDAV client
expects to receive.
This decision is now made explicitly, recorded here, and testable.
---
## Reactivity
`TasksDataSource.registerObserver(onChange: () -> Unit): AutoCloseable` **stays
as-is**. The Room implementation backs it with `InvalidationTracker.Observer`
over the four tables; the External implementation keeps its `ContentObserver`.
One interface, two mechanisms, `TasksRepositoryImpl.observing()` untouched.
Going Flow-native in the DAOs is a later, optional refinement. Doing it now
would change the interface and therefore the External path, for no user-visible
gain.
---
## Migrating existing users
Anyone on v0.3.x has their tasks inside the bundled provider's SQLite file at
`/data/data/de.jeanlucmakiola.agendula/databases/tasks.db` (dmfs schema
version 23). Removing the Gradle module does **not** remove that file — an app
update leaves the data directory intact.
So the migration reads the file directly, with no provider and no
ContentResolver involved:
```
OneShotImport
1. does databases/tasks.db exist? no → nothing to do, mark done
2. open SQLiteDatabase.OPEN_READONLY
3. read tasklists → task_lists (account_type LOCAL → account_id NULL)
4. read tasks → tasks (skip _deleted = 1; mint uid where NULL)
5. read properties → task_alarms (mimetype = …/alarm only)
6. verify counts, inside one Room transaction
7. record completion in DataStore
8. rename tasks.db → tasks.db.imported (kept one release, then deleted)
```
Rules that make this safe:
- **Read-only, single transaction, verified counts.** Either the whole import
lands or none of it does.
- **The source file is renamed, never deleted**, for one release. If the import
is wrong we can still recover from a user's device.
- **Idempotent.** Guarded by a DataStore flag *and* by the rename, so a crash
mid-import cannot double-import.
- **Runs before first UI read**, gated the same way `StorageModeHolder.awaitReady()`
already gates the launch reminder re-sync.
- Tasks that were in an *external* account inside our bundled provider (only
possible if the user had pointed DAVx5 at our authority) are imported as
local lists, with their `uid` preserved. Rare, but preserving the UID is what
lets them be re-attached to an account later.
`tasks.db.imported` is excluded from Auto Backup; the new Room database is
included, which is the whole point of owning it.
---
## Effects on the sync plan
`SYNC.md`'s phase list was written against the provider. Owning the store
deletes work from it outright:
| `SYNC.md` item | Fate |
|---|---|
| "Assign UIDs at creation" (phase 0) | **gone** — `uid` is NOT NULL from v1 |
| Auto Backup / `cleanUpLists` data-loss guard (phase 0) | **gone** — no `cleanUpLists` |
| `lib-recur` pin rationale (phase 0) | reduced to a normal version choice |
| Local → Synced migration (phase 3) | **gone** — `account_id` is a nullable FK |
| ETag / href / CTag squats into `SYNC1`–`SYNC8` | **gone** — real columns |
| Per-collection sync token (phase 3) | **gone** — real column |
| `_DIRTY` set-on-delete workaround (phase 3) | **gone** — tombstones are ours |
| `CALLER_IS_SYNCADAPTER` ignored by instances URI | **gone** — no URIs |
| `Moving` dual-UID collision (phase 4) | **gone** |
| Read-only collections (phase 4) | now *possible* — `is_read_only` exists |
| Recurring-completion model | **decided here** — RECURRENCE-ID override |
| Byte-stable round-trip | improved — `unknown_properties` preserves the rest |
Everything platform-level and protocol-level in `SYNC.md` is untouched: the
`targetSdk 34` sync-framework gate, the stub sync-adapter pattern, credential
storage, Play compliance, discovery, conditional `PUT`, conflict policy, and
every per-server quirk in the server-reality table.
---
## Plan
### Phase 0 — Untangle (0.5 wk)
- `domain/Models.kt` stops importing `TasksContract`; the status and priority
constants move into `domain`. This is the last contract reference above the
data layer.
- `StorageMode`: `LOCAL` → `OWN`; `ProviderResolver` loses `own` / `isOwn`.
- `ProviderChangeReceiver`'s manifest filter drops our own authority.
- Add Room + KSP to the version catalog (KSP is already applied to `:app`).
**Done when:** the app still builds and behaves identically, with the provider
still present and still default.
### Phase 1 — Schema and DAOs (1 wk)
- The four entities above, plus DAOs, plus `schemas/` exported for migration
testing (`room.schemaLocation`, committed).
- `RoomTasksDataSource` implementing all 14 `TasksDataSource` methods except the
recurrence-dependent ones, which throw until phase 2.
- `DataModule` binds by `StorageMode`.
**Done when:** a JVM test creates lists and non-recurring tasks through
`TasksDataSource` against an in-memory Room database and reads them back.
### Phase 2 — Recurrence (1.5–2 wk)
The hard phase. Budget accordingly.
- `RecurrenceExpander` over `lib-recur`: `RRULE`, `RDATE`, `EXDATE`, overrides,
all-day handling, bounded window, `distanceFromCurrent`.
- `RECURRENCE-ID` override creation on single-occurrence edit and completion.
- A test suite that is the deliverable, not an afterthought: daily/weekly/
monthly-by-day/yearly, `COUNT` and `UNTIL`, DST boundaries, all-day series,
a series with an override, a series with an exception, and an unbounded rule
hitting the window ceiling.
**Done when:** `updateInstance` and recurring reads pass parity tests written
against the current provider's observed behaviour, *except* where model (a)
deliberately differs from model (d) — those differences enumerated as tests.
### Phase 3 — Semantics parity (1 wk)
- Completion coherence: `status` ↔ `percent_complete` ↔ `completed_at` ↔ closed,
replacing `AutoCompleting.java` and — importantly — the reopen asymmetry that
`TaskWriteMapper` currently works around in the app.
- Parent/child integrity, orphan handling on delete.
- Validation: `DUE` xor `DURATION`, `due >= dtstart`, all-day pinned to UTC
midnight, list must exist.
- Delete semantics: hard delete when `account_id IS NULL`, tombstone when set.
**Done when:** `TaskWriteMapper`'s provider-quirk workarounds are demonstrably
unnecessary on the Room path (they stay for External).
### Phase 4 — Import and cutover (1 wk)
- `OneShotImport` per the rules above, with tests over a fixture `tasks.db`
captured from a real v0.3.x install.
- `OWN` becomes the default for new installs and for upgraders after import.
- Backup rules updated: include the Room database, exclude `tasks.db.imported`
and the Keystore blob.
**Done when:** an upgrade from a v0.3.2 APK with seeded data lands every task,
list and reminder in Room, verified by count and by content.
### Phase 5 — Delete `:provider` (0.5 wk)
- Remove the module, its `settings.gradle.kts` include, its `:app` dependency,
the three dmfs deps it pulled in, `provider/PROVENANCE.md`.
- Add `lib-recur` (and `rfc5545-datetime`) directly to `:app`.
- Attribution screen: dmfs code is gone, but `lib-recur` stays and is
Apache-2.0. `PROVENANCE.md` is replaced by a short note in
`STORAGE-DECISION.md` recording that the fork existed and why it ended.
**Done when:** `./gradlew build` is green with `:provider` absent, and the APK
declares no ContentProvider and no custom permissions.
### Phase 6 — Harden (1 wk)
- Room migration test infrastructure (`MigrationTestHelper`) wired up, so v1 →
v2 is cheap when sync adds columns.
- Restore-path test: Auto Backup restore into a fresh install.
- Performance check at 5,000 tasks with 20 recurring series.
**Total: 6–6.5 weeks** to a shipping app with its own store, before any CalDAV
work begins. `SYNC.md`'s own estimate drops by 2.5–4 weeks in exchange.
---
## Testing posture
| Layer | How |
|---|---|
| Entities, DAOs, migrations | Room in-memory + `MigrationTestHelper`, JVM |
| `RecurrenceExpander` | pure JVM, no Android — the largest suite |
| Semantics (completion, hierarchy, validation) | JVM through `TasksDataSource` |
| `OneShotImport` | fixture `tasks.db` committed as a test resource |
| External mode | unchanged; existing `TaskMapper` / `TaskWriteMapper` tests stay |
The 93 existing app tests must stay green throughout. The 56 provider tests
leave with the module in phase 5 — replaced, not abandoned: phases 2 and 3 owe
equivalent coverage of the behaviour those tests protected, and phase 2's
parity suite is written against them.
---
## Risks
| Risk | Mitigation |
|---|---|
| **Recurrence is subtler than estimated** | Phase 2 is isolated and pure-JVM; it can overrun without blocking phases 3–4. The provider stays in-tree until phase 5, so we can always compare against it. |
| **Import loses a user's data** | Read-only source, single transaction, count verification, source file renamed not deleted, fixture-based tests. |
| **Regression in a behaviour nobody documented** | Phase 2's parity tests are written *against the provider while it is still present*. That is why deletion is phase 5, not phase 0. |
| **Losing third-party interop** | External mode covers users who need it. A read-only facade stays possible later; nothing in this design forecloses it. |
| **Room + KSP build cost** | KSP is already in the build for Hilt; Room adds one processor. |
---
## Deliberately not doing
- **An exported ContentProvider facade over Room.** Possible later (~1–1.5 wk),
not now. Shipping one would recreate the public-API surface whose validation
and URI plumbing is most of what we are deleting.
- **A domain-native schema.** The table shapes above stay recognisably close to
`TaskContract` where `TaskContract` was right, because it is a proven design
for `VTODO` and because it keeps a future facade cheap.
- **Flow-native DAOs.** Later refinement; changes the interface for no
user-visible gain today.
- **FTS / search.** The provider carried 798 lines of it. The app has never
called it. If search is wanted it is a feature request, designed on its own
terms.
- **Categories and attendees as first-class tables.** They round-trip through
`unknown_properties` until a feature actually needs them.