agendula/docs/OWN-STORE.md

Agendula's own task store — architecture and plan

Branch: feat/own-store Decision: taken. docs/STORAGE-DECISION.md costed it; this is the build. Supersedes: the "keep :provider" position in STORAGE-AND-SYNC.md and the "Settled" section of SYNC.md.

---

The decision, in one paragraph

Agendula stops vendoring the dmfs OpenTasks provider. The :provider module — 14,555 lines of Java, 1.66× the size of the app itself — is deleted. In its place the app gets its own Room database, designed for the two things Agendula actually does: show tasks, and sync them over CalDAV. Support for external providers (OpenTasks, tasks.org) stays, unchanged, as a user choice — so anyone already syncing through DAVx5 keeps working exactly as they do today.

Agendula becomes a normal Android app with a normal database, plus an optional compatibility path into somebody else's ContentProvider.

---

What changes and what does not

BEFORE                                   AFTER

  UI / ViewModels                          UI / ViewModels
        │                                        │
  TasksRepository                          TasksRepository        ← unchanged
        │                                        │
  TasksDataSource  (interface)             TasksDataSource        ← unchanged
        │                                     ╱        ╲
  AndroidTasksDataSource            RoomTasksDataSource   AndroidTasksDataSource
        │                                    │                     │
  ContentResolver                       Room / SQLite        ContentResolver
        │                                    │                     │
   ┌────┴─────┐                         our tables          ┌──────┴──────┐
   │          │                                             │             │
:provider   OpenTasks                                   OpenTasks    tasks.org
(deleted)   tasks.org                                    (external, unchanged)

Unchanged above the data layer. TasksRepository, every ViewModel, every screen. Verified: exactly one file outside data/tasks references TasksContract (domain/Models.kt, for five constants), and it stops doing so in phase 0.

Deleted. The :provider Gradle module, its manifest <provider>, its two custom permissions, its 84 Java files, its 13 translated string resources, and its three dmfs runtime dependencies from :provider's own build file.

Kept for External mode. TasksContract.kt, ColumnReader.kt, TaskMapper.kt, TaskWriteMapper.kt, AndroidTasksDataSource.kt, ProviderResolver.kt, ProviderEnvironment.kt, TaskProjections.kt. These describe somebody else's schema and are exactly right for that job.

---

Storage modes after the change

enum class StorageMode {
    /** Agendula's own Room database. The default; always available. */
    OWN,
    /** A tasks provider app already installed — OpenTasks, tasks.org. */
    EXTERNAL,
}

LOCAL (meaning "our bundled dmfs provider") is gone. ProviderResolver.own and the TaskProvider(isOwn = true) case go with it: in OWN mode there is no authority, no ContentResolver and no permission to grant.

ProviderResolver narrows to what it was always really for — discovering external providers — and ProviderStatus.READY becomes unconditional in OWN mode.

Consequences worth stating plainly

• No runtime permission is needed for the default path. Today's permission prompt only ever applied to External mode; now that is visibly true.
• Third-party apps can no longer read Agendula's tasks. We publish no ContentProvider. Users who need interop pick External mode, or wait for a possible read-only facade (explicitly out of scope — see Deliberately not doing).
• Auto Backup gets simpler and safer. One Room file we control, with a documented restore path, instead of a provider database whose cleanUpLists routine could delete restored lists whose accounts no longer exist.

---

The schema

Four tables. Designed from Agendula's actual reads and writes plus RFC 5545's VTODO, not inherited from a 2013 schema.

task_lists

Column	Type	Notes
id	INTEGER PK
name	TEXT NOT NULL
color	INTEGER NOT NULL	ARGB
account_id	INTEGER NULL	FK → accounts, NULL = device-only
is_visible	INTEGER NOT NULL	default 1
is_synced	INTEGER NOT NULL	default 1
owner	TEXT NULL	CalDAV owner display name
is_read_only	INTEGER NOT NULL	new — the provider could not express this at all
sort_order	INTEGER NOT NULL	user ordering, which the provider also lacked
href	TEXT NULL	collection URL, relative to the account root
ctag	TEXT NULL
sync_token	TEXT NULL	RFC 6578, per collection — not squatted into a shared slot
is_dirty	INTEGER NOT NULL	a real boolean, not dmfs's monotonic counter

account_id being nullable is the single most important schema change. In the dmfs provider ACCOUNT_TYPE is write-once and throws on change, which made "turn sync on" a full data migration. Here, attaching a local list to an account is UPDATE task_lists SET account_id = ?.

tasks

Master rows and recurrence overrides live here; an override is a row with recurrence_id set and parent_task_id pointing at its master.

Group	Columns
identity	id, list_id, uid (NOT NULL, minted at creation), href, etag
content	title, description, location, url, color
state	status, percent_complete, completed_at, priority, classification
time	dtstart, due, duration, is_all_day, timezone
recurrence	rrule, rdate, exdate, recurrence_id, master_id
hierarchy	parent_id, sort_order
audit	created_at, last_modified, sequence
sync	is_dirty, is_deleted, unknown_properties

Two entries deserve explanation.

uid is NOT NULL and assigned at creation. Every task gets a real RFC 4122 UUID the moment it is inserted, in every mode, synced or not. This closes the gap ICalendarWriter.uidFor currently papers over by synthesising agendula-<rowid>@…, and it means any local task can later be pushed to a server without duplicating. The provider never assigned one.

unknown_properties holds the raw unfolded iCalendar lines of every property we do not model — ATTENDEE, CATEGORIES, X-*, GEO, and anything a future RFC adds. On write we re-emit them verbatim after the properties we do own. This is what makes an honest round-trip possible, and it replaces the provider's data0–data15 bag with something that cannot silently lose a field it has no column for.

Indices: (list_id, is_deleted), (parent_id), (uid) unique per list, (master_id, recurrence_id), (is_dirty).

task_alarms

Column	Notes
id, task_id	FK, ON DELETE CASCADE
minutes_before	positive = before the reference
reference	DUE or START
message	optional

Replaces AlarmHandler (133 lines of Java) and the dataN slot convention. Delete-and-reinsert stops being necessary — the provider's re-validate-everything behaviour was the only reason setAlarm worked that way.

accounts

Column	Notes
id, display_name, principal_url, home_set_url
username	the app password is not here — Keystore only, per SYNC.md
last_sync_at, last_sync_error

Not populated until SYNC.md phase 2, but the FK exists from v1 so enabling sync never requires a schema migration.

---

Recurrence: expand at read, not on write

The provider maintained a materialised instances table, recomputed by Instantiating.java on every write — and still only ever materialised one upcoming occurrence.

Agendula expands lazily instead:

tasks (masters + overrides)  ──►  RecurrenceExpander  ──►  List<Task>
        rrule/rdate/exdate         (lib-recur, in memory)    occurrences

This is the right call here because the repository already filters and sorts in Kotlin, not SQL. TasksRepositoryImpl.loadTasks reads the whole set, applies TaskFiltering.matches, then TaskSorting.DEFAULT. Nothing depends on the database being able to order by instance time, so nothing is lost — and a materialised table's entire class of staleness bugs never exists.

• Bounded window: expansion is capped (default: 1 year back, 2 years forward, hard ceiling of N occurrences per series) so an unbounded RRULE cannot hang the UI.
• lib-recur pinned at 0.12.2 — 0.16.0 removed RecurrenceSet. We pin because we chose to, and the pin is now ours to lift on our own schedule.
• Client-side expansion is required for CalDAV regardless: server-side CALDAV:expand on VTODO is broken on every server SYNC.md targets.

Completing one occurrence of a recurring task

The provider's Detaching.java implemented model (d): detach the occurrence as a brand-new task with its own UID. We inherited that without ever choosing it, and it is the model least compatible with CalDAV.

We implement model (a): a RECURRENCE-ID override. Completing one occurrence writes a second tasks row with the same uid, a recurrence_id naming the occurrence, master_id pointing at the series, and the completed state. This is what RFC 5545 specifies and what every other CalDAV client expects to receive.

This decision is now made explicitly, recorded here, and testable.

---

Reactivity

TasksDataSource.registerObserver(onChange: () -> Unit): AutoCloseable stays as-is. The Room implementation backs it with InvalidationTracker.Observer over the four tables; the External implementation keeps its ContentObserver. One interface, two mechanisms, TasksRepositoryImpl.observing() untouched.

Going Flow-native in the DAOs is a later, optional refinement. Doing it now would change the interface and therefore the External path, for no user-visible gain.

---

Migrating existing users

Anyone on v0.3.x has their tasks inside the bundled provider's SQLite file at /data/data/de.jeanlucmakiola.agendula/databases/tasks.db (dmfs schema version 23). Removing the Gradle module does not remove that file — an app update leaves the data directory intact.

So the migration reads the file directly, with no provider and no ContentResolver involved:

OneShotImport
  1. does databases/tasks.db exist?            no  → nothing to do, mark done
  2. open SQLiteDatabase.OPEN_READONLY
  3. read tasklists  → task_lists      (account_type LOCAL → account_id NULL)
  4. read tasks      → tasks           (skip _deleted = 1; mint uid where NULL)
  5. read properties → task_alarms     (mimetype = …/alarm only)
  6. verify counts, inside one Room transaction
  7. record completion in DataStore
  8. rename tasks.db → tasks.db.imported   (kept one release, then deleted)

Rules that make this safe:

• Read-only, single transaction, verified counts. Either the whole import lands or none of it does.
• The source file is renamed, never deleted, for one release. If the import is wrong we can still recover from a user's device.
• Idempotent. Guarded by a DataStore flag and by the rename, so a crash mid-import cannot double-import.
• Runs before first UI read, gated the same way StorageModeHolder.awaitReady() already gates the launch reminder re-sync.
• Tasks that were in an external account inside our bundled provider (only possible if the user had pointed DAVx5 at our authority) are imported as local lists, with their uid preserved. Rare, but preserving the UID is what lets them be re-attached to an account later.

tasks.db.imported is excluded from Auto Backup; the new Room database is included, which is the whole point of owning it.

---

Effects on the sync plan

SYNC.md's phase list was written against the provider. Owning the store deletes work from it outright:

SYNC.md item	Fate
"Assign UIDs at creation" (phase 0)	gone — uid is NOT NULL from v1
Auto Backup / cleanUpLists data-loss guard (phase 0)	gone — no cleanUpLists
lib-recur pin rationale (phase 0)	reduced to a normal version choice
Local → Synced migration (phase 3)	gone — account_id is a nullable FK
ETag / href / CTag squats into SYNC1–SYNC8	gone — real columns
Per-collection sync token (phase 3)	gone — real column
_DIRTY set-on-delete workaround (phase 3)	gone — tombstones are ours
CALLER_IS_SYNCADAPTER ignored by instances URI	gone — no URIs
Moving dual-UID collision (phase 4)	gone
Read-only collections (phase 4)	now possible — is_read_only exists
Recurring-completion model	decided here — RECURRENCE-ID override
Byte-stable round-trip	improved — unknown_properties preserves the rest

Everything platform-level and protocol-level in SYNC.md is untouched: the targetSdk 34 sync-framework gate, the stub sync-adapter pattern, credential storage, Play compliance, discovery, conditional PUT, conflict policy, and every per-server quirk in the server-reality table.

---

Plan

Phase 0 — Untangle (0.5 wk)

• domain/Models.kt stops importing TasksContract; the status and priority constants move into domain. This is the last contract reference above the data layer.
• StorageMode: LOCAL → OWN; ProviderResolver loses own / isOwn.
• ProviderChangeReceiver's manifest filter drops our own authority.
• Add Room + KSP to the version catalog (KSP is already applied to :app).

Done when: the app still builds and behaves identically, with the provider still present and still default.

Phase 1 — Schema and DAOs (1 wk)

• The four entities above, plus DAOs, plus schemas/ exported for migration testing (room.schemaLocation, committed).
• RoomTasksDataSource implementing all 14 TasksDataSource methods except the recurrence-dependent ones, which throw until phase 2.
• DataModule binds by StorageMode.

Done when: a JVM test creates lists and non-recurring tasks through TasksDataSource against an in-memory Room database and reads them back.

Phase 2 — Recurrence (1.5–2 wk)

The hard phase. Budget accordingly.

• RecurrenceExpander over lib-recur: RRULE, RDATE, EXDATE, overrides, all-day handling, bounded window, distanceFromCurrent.
• RECURRENCE-ID override creation on single-occurrence edit and completion.
• A test suite that is the deliverable, not an afterthought: daily/weekly/ monthly-by-day/yearly, COUNT and UNTIL, DST boundaries, all-day series, a series with an override, a series with an exception, and an unbounded rule hitting the window ceiling.

Done when: updateInstance and recurring reads pass parity tests written against the current provider's observed behaviour, except where model (a) deliberately differs from model (d) — those differences enumerated as tests.

Phase 3 — Semantics parity (1 wk)

• Completion coherence: status ↔ percent_complete ↔ completed_at ↔ closed, replacing AutoCompleting.java and — importantly — the reopen asymmetry that TaskWriteMapper currently works around in the app.
• Parent/child integrity, orphan handling on delete.
• Validation: DUE xor DURATION, due >= dtstart, all-day pinned to UTC midnight, list must exist.
• Delete semantics: hard delete when account_id IS NULL, tombstone when set.

Done when: TaskWriteMapper's provider-quirk workarounds are demonstrably unnecessary on the Room path (they stay for External).

Phase 4 — Import and cutover (1 wk)

• OneShotImport per the rules above, with tests over a fixture tasks.db captured from a real v0.3.x install.
• OWN becomes the default for new installs and for upgraders after import.
• Backup rules updated: include the Room database, exclude tasks.db.imported and the Keystore blob.

Done when: an upgrade from a v0.3.2 APK with seeded data lands every task, list and reminder in Room, verified by count and by content.

Phase 5 — Delete :provider (0.5 wk)

• Remove the module, its settings.gradle.kts include, its :app dependency, the three dmfs deps it pulled in, provider/PROVENANCE.md.
• Add lib-recur (and rfc5545-datetime) directly to :app.
• Attribution screen: dmfs code is gone, but lib-recur stays and is Apache-2.0. PROVENANCE.md is replaced by a short note in STORAGE-DECISION.md recording that the fork existed and why it ended.

Done when: ./gradlew build is green with :provider absent, and the APK declares no ContentProvider and no custom permissions.

Phase 6 — Harden (1 wk)

• Room migration test infrastructure (MigrationTestHelper) wired up, so v1 → v2 is cheap when sync adds columns.
• Restore-path test: Auto Backup restore into a fresh install.
• Performance check at 5,000 tasks with 20 recurring series.

Total: 6–6.5 weeks to a shipping app with its own store, before any CalDAV work begins. SYNC.md's own estimate drops by 2.5–4 weeks in exchange.

---

Testing posture

Layer	How
Entities, DAOs, migrations	Room in-memory + MigrationTestHelper, JVM
RecurrenceExpander	pure JVM, no Android — the largest suite
Semantics (completion, hierarchy, validation)	JVM through TasksDataSource
OneShotImport	fixture tasks.db committed as a test resource
External mode	unchanged; existing TaskMapper / TaskWriteMapper tests stay

The 93 existing app tests must stay green throughout. The 56 provider tests leave with the module in phase 5 — replaced, not abandoned: phases 2 and 3 owe equivalent coverage of the behaviour those tests protected, and phase 2's parity suite is written against them.

---

Risks

Risk	Mitigation
Recurrence is subtler than estimated	Phase 2 is isolated and pure-JVM; it can overrun without blocking phases 3–4. The provider stays in-tree until phase 5, so we can always compare against it.
Import loses a user's data	Read-only source, single transaction, count verification, source file renamed not deleted, fixture-based tests.
Regression in a behaviour nobody documented	Phase 2's parity tests are written against the provider while it is still present. That is why deletion is phase 5, not phase 0.
Losing third-party interop	External mode covers users who need it. A read-only facade stays possible later; nothing in this design forecloses it.
Room + KSP build cost	KSP is already in the build for Hilt; Room adds one processor.

---

Deliberately not doing

• An exported ContentProvider facade over Room. Possible later (~1–1.5 wk), not now. Shipping one would recreate the public-API surface whose validation and URI plumbing is most of what we are deleting.
• A domain-native schema. The table shapes above stay recognisably close to TaskContract where TaskContract was right, because it is a proven design for VTODO and because it keeps a future facade cheap.
• Flow-native DAOs. Later refinement; changes the interface for no user-visible gain today.
• FTS / search. The provider carried 798 lines of it. The app has never called it. If search is wanted it is a feature request, designed on its own terms.
• Categories and attendees as first-class tables. They round-trip through unknown_properties until a feature actually needs them.