sync(chunk 4): incremental sync and scheduling
RFC 6578 as an optimisation on top of chunk 3's full path, and sync that runs by itself. - :caldav gains sync-collection, with invalidation matched on DAV:valid-sync-token in the body on any 4xx rather than on a status code — 400, 403, 409 and 412 are all used in the wild, and matching the status is why Thunderbird never recovers from sabre's 403. No DAV:limit (Nextcloud regressed it to an HTML error page); 507 on our own href is truncation. - The engine persists the token per page, after the bodies. An iteration cap and a no-progress guard, since the RFC never requires the token to advance. The full path runs anyway every 24h: a token the server accepts over a pruned change log returns 207, zero changes and no error, and the protocol gives no other way to notice. - The three membership traps: an unknown removed href is a no-op, a delete-then-recreate is re-identified from the UID in the body, and a mass removal is refused in favour of a real listing, because ACL churn looks exactly like one. - Scheduling is a PeriodicWorkRequest with a network constraint, expedited only for the in-app button. getForegroundInfo is implemented unconditionally (setExpedited falls back to a foreground service below API 31 and the default throws) but declares no service type, which would have pulled back the Android 15 dataSync budget and a Play video-demo requirement. initialIncomplete turned out not to be needed: adopting a token only after a full reconciliation completes removes the hazard it guarded, so there is nothing to persist atomically with anything. /code-review high raised 8 findings, all fixed. The two that mattered: the cadence clock sat in the backed-up DataStore, so a restore would have made the engine trust a stale token for a day — both sync-state stores now have their own excluded file; and the incremental download path lacked the write-phase guard, overwriting local edits that had never reached the server. Reasoning in docs/SYNC-PLAN.md. Chunk 2's on-device review is still outstanding; none of this has run on a device or against a real server.
This commit is contained in:
@@ -0,0 +1,52 @@
|
||||
package de.jeanlucmakiola.caldav
|
||||
|
||||
import okhttp3.HttpUrl
|
||||
|
||||
/** What one `sync-collection` REPORT came back with. */
|
||||
sealed interface ChangeSet {
|
||||
|
||||
/**
|
||||
* One page of changes.
|
||||
*
|
||||
* A page, not a result: RFC 6578 lets a server truncate and hand back a token
|
||||
* that resumes where it stopped, so the caller applies this page's bodies,
|
||||
* stores [token], and only then asks for the next.
|
||||
*/
|
||||
data class Page(
|
||||
val changed: List<RemoteRef>,
|
||||
val removed: List<HttpUrl>,
|
||||
/**
|
||||
* The token to store **after** this page is applied.
|
||||
*
|
||||
* ⚠️ Null when the server sent a 207 with no `DAV:sync-token` at all,
|
||||
* which happens. That is not an error and not a reason to throw — the
|
||||
* changes in this page are still real. It means the next run has no
|
||||
* cursor and must reconcile in full.
|
||||
*/
|
||||
val token: String?,
|
||||
/**
|
||||
* The server stopped early and there is more behind [token].
|
||||
*
|
||||
* Signalled by a `507` on the collection's **own** href, which is
|
||||
* truncation rather than failure — a 507 as the *outer* response status
|
||||
* means something else entirely.
|
||||
*/
|
||||
val truncated: Boolean = false,
|
||||
) : ChangeSet
|
||||
|
||||
/**
|
||||
* The token is no longer usable and the collection must be reconciled in full.
|
||||
*
|
||||
* ⚠️ **Invalidation has no status code.** `403` (sabre), `400` (Google),
|
||||
* `409` (Radicale) and `412` (Evolution) are all observed in the wild, so the
|
||||
* only reliable signal is `DAV:valid-sync-token` in the body on **any** 4xx.
|
||||
* Thunderbird matches on 400 alone and therefore never recovers from the 403
|
||||
* that most of the self-hosted world emits.
|
||||
*/
|
||||
data object TokenInvalid : ChangeSet
|
||||
|
||||
/** The server will not do `sync-collection` here. Fall back, do not retry. */
|
||||
data object Unsupported : ChangeSet
|
||||
|
||||
data class Failed(val reason: String) : ChangeSet
|
||||
}
|
||||
@@ -1,6 +1,8 @@
|
||||
package de.jeanlucmakiola.caldav
|
||||
|
||||
import at.bitfire.dav4jvm.DavCalendar
|
||||
import at.bitfire.dav4jvm.DavCollection
|
||||
import at.bitfire.dav4jvm.Error
|
||||
import at.bitfire.dav4jvm.DavResource
|
||||
import at.bitfire.dav4jvm.Response
|
||||
import at.bitfire.dav4jvm.exception.DavException
|
||||
@@ -99,6 +101,72 @@ class CalendarCollection(
|
||||
refs
|
||||
}
|
||||
|
||||
/**
|
||||
* One page of changes since [token] (RFC 6578).
|
||||
*
|
||||
* ⚠️ **No `DAV:limit`.** Nextcloud regressed it to a localised HTML error
|
||||
* page, so asking for a bounded page is how you get an unparseable response
|
||||
* instead of a bounded one. Truncation is the server's decision, signalled
|
||||
* back through [ChangeSet.Page.truncated].
|
||||
*
|
||||
* ⚠️ **A removal is a `404` at `DAV:response` level**, not inside a
|
||||
* `propstat`. A client that only reads statuses out of propstats sees every
|
||||
* deletion as an unremarkable response with no properties and silently keeps
|
||||
* the row.
|
||||
*/
|
||||
override fun changes(token: String?): ChangeSet = try {
|
||||
val changed = mutableListOf<RemoteRef>()
|
||||
val removed = mutableListOf<HttpUrl>()
|
||||
var truncated = false
|
||||
|
||||
val properties = DavCollection(httpClient, url).reportChanges(
|
||||
syncToken = token,
|
||||
infiniteDepth = false,
|
||||
limit = null,
|
||||
GetETag.NAME,
|
||||
) { response, relation ->
|
||||
val code = response.status?.code
|
||||
when {
|
||||
relation == Response.HrefRelation.SELF ->
|
||||
// On our own href this is truncation, not a failure.
|
||||
if (code == INSUFFICIENT_STORAGE) truncated = true
|
||||
|
||||
code == NOT_FOUND || code == GONE -> removed += response.href
|
||||
|
||||
response.isSuccess() ->
|
||||
changed += RemoteRef(response.href, ETag.from(response[GetETag::class.java]))
|
||||
|
||||
else -> Unit
|
||||
}
|
||||
}
|
||||
|
||||
ChangeSet.Page(
|
||||
changed = changed,
|
||||
removed = removed,
|
||||
token = properties.filterIsInstance<SyncToken>().firstOrNull()?.token,
|
||||
truncated = truncated,
|
||||
)
|
||||
} catch (e: HttpException) {
|
||||
when {
|
||||
e.code / 100 == 4 && mentionsInvalidToken(e) -> ChangeSet.TokenInvalid
|
||||
e.code in UNSUPPORTED -> ChangeSet.Unsupported
|
||||
else -> ChangeSet.Failed("HTTP ${e.code}")
|
||||
}
|
||||
} catch (e: DavException) {
|
||||
ChangeSet.Failed(e.toString())
|
||||
} catch (e: IOException) {
|
||||
ChangeSet.Failed(e.toString())
|
||||
}
|
||||
|
||||
/**
|
||||
* ⚠️ The body, not the status. Every observed invalidation status is also a
|
||||
* legitimate answer to something else, and every server picks a different
|
||||
* one — so the element is the signal and the code is noise.
|
||||
*/
|
||||
private fun mentionsInvalidToken(e: HttpException): Boolean =
|
||||
Error.VALID_SYNC_TOKEN in e.errors ||
|
||||
e.responseBody?.contains(VALID_SYNC_TOKEN_ELEMENT) == true
|
||||
|
||||
/**
|
||||
* Downloads [hrefs] with `calendar-multiget`, in batches.
|
||||
*
|
||||
@@ -284,5 +352,17 @@ class CalendarCollection(
|
||||
private const val NOT_FOUND = 404
|
||||
private const val GONE = 410
|
||||
private const val PRECONDITION_FAILED = 412
|
||||
private const val INSUFFICIENT_STORAGE = 507
|
||||
|
||||
/** Matched as a local name, so a server's own namespace prefix is irrelevant. */
|
||||
private const val VALID_SYNC_TOKEN_ELEMENT = "valid-sync-token"
|
||||
|
||||
/**
|
||||
* The server does not implement the report. Deliberately narrow: 400,
|
||||
* 403, 409 and 412 are all *invalidation* codes on some server, so
|
||||
* treating them as "unsupported" would abandon `sync-collection` on a
|
||||
* collection that merely needs a fresh token.
|
||||
*/
|
||||
private val UNSUPPORTED = setOf(405, 415, 501)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -18,6 +18,14 @@ interface RemoteCalendar {
|
||||
|
||||
fun list(): Result<List<RemoteRef>>
|
||||
|
||||
/**
|
||||
* One page of changes since [token], or null for a first run.
|
||||
*
|
||||
* Never throws for anything a server can legitimately answer — a rejected
|
||||
* token is [ChangeSet.TokenInvalid], not an exception.
|
||||
*/
|
||||
fun changes(token: String?): ChangeSet
|
||||
|
||||
fun fetch(hrefs: List<HttpUrl>): Result<FetchResult>
|
||||
|
||||
fun create(name: String, iCalendar: String): PutOutcome
|
||||
|
||||
@@ -0,0 +1,160 @@
|
||||
package de.jeanlucmakiola.caldav
|
||||
|
||||
import com.google.common.truth.Truth.assertThat
|
||||
import okhttp3.OkHttpClient
|
||||
import okhttp3.mockwebserver.MockResponse
|
||||
import okhttp3.mockwebserver.MockWebServer
|
||||
import org.junit.After
|
||||
import org.junit.Before
|
||||
import org.junit.Test
|
||||
|
||||
/** RFC 6578. Every case here is a documented failure of a real client. */
|
||||
class CalendarChangesTest {
|
||||
|
||||
private val server = MockWebServer()
|
||||
private val httpClient = OkHttpClient.Builder().followRedirects(false).build()
|
||||
private lateinit var collection: CalendarCollection
|
||||
|
||||
@Before fun start() {
|
||||
server.start()
|
||||
collection = CalendarCollection(httpClient, server.url("/dav/tasks/"))
|
||||
}
|
||||
|
||||
@After fun stop() = server.shutdown()
|
||||
|
||||
@Test fun `changed and removed are told apart`() {
|
||||
server.enqueue(
|
||||
multistatus(
|
||||
"""
|
||||
<response>
|
||||
<href>/dav/tasks/kept.ics</href>
|
||||
<propstat><prop><getetag>"e1"</getetag></prop>
|
||||
<status>HTTP/1.1 200 OK</status></propstat>
|
||||
</response>
|
||||
<response>
|
||||
<href>/dav/tasks/gone.ics</href>
|
||||
<status>HTTP/1.1 404 Not Found</status>
|
||||
</response>
|
||||
<sync-token>urn:x:2</sync-token>
|
||||
""",
|
||||
),
|
||||
)
|
||||
|
||||
val page = collection.changes("urn:x:1") as ChangeSet.Page
|
||||
|
||||
// ⚠️ The deletion status is at DAV:response level, not inside a propstat.
|
||||
assertThat(page.changed.map { it.href.encodedPath })
|
||||
.containsExactly("/dav/tasks/kept.ics")
|
||||
assertThat(page.removed.map { it.encodedPath })
|
||||
.containsExactly("/dav/tasks/gone.ics")
|
||||
assertThat(page.token).isEqualTo("urn:x:2")
|
||||
assertThat(page.truncated).isFalse()
|
||||
}
|
||||
|
||||
@Test fun `no DAV limit is ever sent`() {
|
||||
server.enqueue(multistatus("<sync-token>urn:x:2</sync-token>"))
|
||||
collection.changes(null)
|
||||
|
||||
val body = server.takeRequest().body.readUtf8()
|
||||
// Nextcloud regressed DAV:limit to a localised HTML error page, so asking
|
||||
// for a bounded page is how you get an unparseable response.
|
||||
assertThat(body).doesNotContain("limit")
|
||||
assertThat(body).contains("sync-level")
|
||||
}
|
||||
|
||||
@Test fun `a 507 on our own href is truncation, not failure`() {
|
||||
server.enqueue(
|
||||
multistatus(
|
||||
"""
|
||||
<response>
|
||||
<href>/dav/tasks/one.ics</href>
|
||||
<propstat><prop><getetag>"e1"</getetag></prop>
|
||||
<status>HTTP/1.1 200 OK</status></propstat>
|
||||
</response>
|
||||
<response>
|
||||
<href>/dav/tasks/</href>
|
||||
<status>HTTP/1.1 507 Insufficient Storage</status>
|
||||
</response>
|
||||
<sync-token>urn:x:2</sync-token>
|
||||
""",
|
||||
),
|
||||
)
|
||||
|
||||
val page = collection.changes("urn:x:1") as ChangeSet.Page
|
||||
|
||||
assertThat(page.truncated).isTrue()
|
||||
assertThat(page.changed).hasSize(1)
|
||||
assertThat(page.token).isEqualTo("urn:x:2")
|
||||
}
|
||||
|
||||
@Test fun `a 207 with no sync-token is not an error`() {
|
||||
server.enqueue(
|
||||
multistatus(
|
||||
"""
|
||||
<response>
|
||||
<href>/dav/tasks/one.ics</href>
|
||||
<propstat><prop><getetag>"e1"</getetag></prop>
|
||||
<status>HTTP/1.1 200 OK</status></propstat>
|
||||
</response>
|
||||
""",
|
||||
),
|
||||
)
|
||||
|
||||
val page = collection.changes("urn:x:1") as ChangeSet.Page
|
||||
|
||||
// The changes are real; only the cursor is missing.
|
||||
assertThat(page.changed).hasSize(1)
|
||||
assertThat(page.token).isNull()
|
||||
}
|
||||
|
||||
// --------------------------------------------------- token invalidation
|
||||
|
||||
@Test fun `invalidation is recognised on every status servers use for it`() {
|
||||
// ⚠️ 403 sabre, 400 Google, 409 Radicale, 412 Evolution. Thunderbird
|
||||
// matches 400 alone and never recovers from the 403 most self-hosted
|
||||
// servers emit.
|
||||
listOf(400, 403, 409, 412).forEach { code ->
|
||||
server.enqueue(
|
||||
MockResponse()
|
||||
.setResponseCode(code)
|
||||
.setHeader("Content-Type", "application/xml; charset=utf-8")
|
||||
.setBody(
|
||||
"<D:error xmlns:D=\"DAV:\"><D:valid-sync-token/></D:error>",
|
||||
),
|
||||
)
|
||||
assertThat(collection.changes("stale")).isEqualTo(ChangeSet.TokenInvalid)
|
||||
}
|
||||
}
|
||||
|
||||
@Test fun `invalidation is recognised whatever prefix the server uses`() {
|
||||
server.enqueue(
|
||||
MockResponse()
|
||||
.setResponseCode(403)
|
||||
.setHeader("Content-Type", "application/xml; charset=utf-8")
|
||||
.setBody("<error xmlns=\"DAV:\"><valid-sync-token/></error>"),
|
||||
)
|
||||
assertThat(collection.changes("stale")).isEqualTo(ChangeSet.TokenInvalid)
|
||||
}
|
||||
|
||||
@Test fun `a 4xx without the marker is not an invalidation`() {
|
||||
server.enqueue(MockResponse().setResponseCode(403).setBody("nope"))
|
||||
// Treating every 403 as invalidation would discard a working token on a
|
||||
// permissions error and re-download the whole collection.
|
||||
assertThat(collection.changes("good")).isInstanceOf(ChangeSet.Failed::class.java)
|
||||
}
|
||||
|
||||
@Test fun `an unimplemented report is reported as unsupported`() {
|
||||
server.enqueue(MockResponse().setResponseCode(501))
|
||||
assertThat(collection.changes(null)).isEqualTo(ChangeSet.Unsupported)
|
||||
}
|
||||
|
||||
@Test fun `a 507 outer status is a failure rather than truncation`() {
|
||||
server.enqueue(MockResponse().setResponseCode(507))
|
||||
assertThat(collection.changes("t")).isInstanceOf(ChangeSet.Failed::class.java)
|
||||
}
|
||||
|
||||
private fun multistatus(body: String) = MockResponse()
|
||||
.setResponseCode(207)
|
||||
.setHeader("Content-Type", "application/xml; charset=utf-8")
|
||||
.setBody("<multistatus xmlns=\"DAV:\">$body</multistatus>")
|
||||
}
|
||||
Reference in New Issue
Block a user