diff --git a/.planning/REQUIREMENTS.md b/.planning/REQUIREMENTS.md index 121fbd8..bf50d1e 100644 --- a/.planning/REQUIREMENTS.md +++ b/.planning/REQUIREMENTS.md @@ -12,7 +12,7 @@ Requirements for this milestone. Each maps to roadmap phases. - [ ] **DB-01**: Application runs on PostgreSQL instead of SQLite - [ ] **DB-02**: All service functions use async database operations - [ ] **DB-03**: Test infrastructure uses PGlite instead of bun:sqlite in-memory databases -- [ ] **DB-04**: Existing SQLite data can be migrated to Postgres via a one-time script +- [x] **DB-04**: Existing SQLite data can be migrated to Postgres via a one-time script - [ ] **DB-05**: Docker Compose provides Postgres for local development ### Authentication @@ -119,7 +119,7 @@ Which phases cover which requirements. Updated during roadmap creation. | DB-01 | Phase 14 | Pending | | DB-02 | Phase 14 | Pending | | DB-03 | Phase 14 | Pending | -| DB-04 | Phase 14 | Pending | +| DB-04 | Phase 14 | Complete | | DB-05 | Phase 14 | Pending | | AUTH-01 | Phase 15 | Pending | | AUTH-02 | Phase 15 | Pending | diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index 2be42f5..d49b174 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -188,7 +188,7 @@ Plans: | 11. Candidate Ranking | v1.3 | 2/2 | Complete | 2026-03-16 | | 12. Comparison View | v1.3 | 1/1 | Complete | 2026-03-17 | | 13. Setup Impact Preview | v1.3 | 0/2 | Not started | - | -| 14. PostgreSQL Migration | v2.0 | 0/? | Not started | - | +| 14. PostgreSQL Migration | v2.0 | 5/6 | In Progress | — | | 15. External Authentication | v2.0 | 0/? | Not started | - | | 16. Multi-User Data Model | v2.0 | 0/? | Not started | - | | 17. Object Storage | v2.0 | 0/? | Not started | - | diff --git a/.planning/STATE.md b/.planning/STATE.md index d9a43f1..526e838 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -2,10 +2,10 @@ gsd_state_version: 1.0 milestone: v2.0 milestone_name: Platform Foundation -status: planning -stopped_at: null -last_updated: "2026-04-03" -last_activity: 2026-04-03 — v2.0 roadmap created (Phases 14-18) +status: executing +stopped_at: Completed 14-05-PLAN.md +last_updated: "2026-04-04T10:28:29Z" +last_activity: 2026-04-04 — Completed 14-05 SQLite-to-Postgres migration script progress: total_phases: 5 completed_phases: 0 @@ -26,9 +26,9 @@ See: .planning/PROJECT.md (updated 2026-04-03) ## Current Position Phase: 14 of 18 (PostgreSQL Migration) -Plan: 0 of ? in current phase -Status: Ready to plan -Last activity: 2026-04-03 — v2.0 roadmap created (Phases 14-18) +Plan: 5 of 6 in current phase +Status: Executing +Last activity: 2026-04-04 — Completed 14-05 SQLite-to-Postgres migration script Progress: [----------] 0% (v2.0 milestone) @@ -45,7 +45,9 @@ Progress: [----------] 0% (v2.0 milestone) ### Decisions -Key decisions made during v2.0 planning: +Key decisions made during v2.0 execution: +- [14-05] Used postgres.js unsafe() for sequence reset DDL instead of drizzle-orm sql template +- [14-05] Row-by-row inserts for better error diagnostics during migration - Platform pivot: single-user to multi-user with discovery-first approach - External auth provider (self-hosted, open-source) — Logto vs Authentik OPEN decision - SQLite to Postgres migration — required by auth provider and multi-user concurrency @@ -64,6 +66,6 @@ None active. ## Session Continuity -Last session: 2026-04-03 -Stopped at: v2.0 roadmap created with 5 phases (14-18) covering 30 requirements +Last session: 2026-04-04T10:28:29Z +Stopped at: Completed 14-05-PLAN.md Resume file: None diff --git a/.planning/phases/14-postgresql-migration/14-05-SUMMARY.md b/.planning/phases/14-postgresql-migration/14-05-SUMMARY.md new file mode 100644 index 0000000..8432915 --- /dev/null +++ b/.planning/phases/14-postgresql-migration/14-05-SUMMARY.md @@ -0,0 +1,93 @@ +--- +phase: 14-postgresql-migration +plan: 05 +subsystem: database +tags: [sqlite, postgres, migration, data-migration, bun-sqlite] + +# Dependency graph +requires: + - phase: 14-01 + provides: PostgreSQL schema definitions (Drizzle pgTable) +provides: + - One-time SQLite-to-PostgreSQL data migration script + - db:migrate-from-sqlite npm script +affects: [14-06, deployment, upgrade-docs] + +# Tech tracking +tech-stack: + added: [] + patterns: [dependency-ordered table migration, unix-to-Date conversion, serial sequence reset] + +key-files: + created: + - scripts/migrate-sqlite-to-postgres.ts + modified: + - package.json + +key-decisions: + - "Used postgres.js unsafe() for sequence reset instead of drizzle-orm sql template (simpler for raw DDL)" + - "Row-by-row insert for error tracing (per-row catch identifies failing record)" + +patterns-established: + - "Migration scripts live in scripts/ directory" + - "Type conversion helpers (unixToDate, intToBool) for SQLite-to-Postgres data transforms" + +requirements-completed: [DB-04] + +# Metrics +duration: 2min +completed: 2026-04-04 +--- + +# Phase 14 Plan 05: SQLite-to-Postgres Migration Script Summary + +**One-time data migration script converting all 13 tables from SQLite to PostgreSQL with timestamp/boolean type conversions and serial sequence reset** + +## Performance + +- **Duration:** 2 min +- **Started:** 2026-04-04T10:26:29Z +- **Completed:** 2026-04-04T10:28:29Z +- **Tasks:** 1 +- **Files modified:** 2 + +## Accomplishments +- Created standalone migration script that reads SQLite and writes to PostgreSQL +- Handles all type conversions: unix epoch integers to Date objects, integer booleans to native booleans +- Migrates tables in FK dependency order (4 waves: no-FK, FK-to-parents, FK-to-intermediates, junction tables) +- Resets all 11 serial sequences after migration to prevent duplicate key errors +- Added `db:migrate-from-sqlite` npm script for easy invocation + +## Task Commits + +Each task was committed atomically: + +1. **Task 1: Create SQLite-to-Postgres migration script** - `b4c3813` (feat) + +## Files Created/Modified +- `scripts/migrate-sqlite-to-postgres.ts` - One-time migration script with type conversions and sequence reset +- `package.json` - Added db:migrate-from-sqlite script + +## Decisions Made +- Used `postgres.js` `unsafe()` for raw `setval` queries instead of drizzle-orm `sql` template -- simpler for dynamic table name interpolation in DDL +- Row-by-row inserts instead of bulk for better error diagnostics (each failed row logs its ID) + +## Deviations from Plan + +None - plan executed exactly as written. + +## Issues Encountered +- Biome lint flagged unused `sql` import from drizzle-orm (used `pg.unsafe()` instead) and unnecessary suppression comments -- removed both + +## User Setup Required + +None - no external service configuration required. + +## Next Phase Readiness +- Migration script ready for use during SQLite-to-Postgres upgrade +- Requires DATABASE_URL env var and existing SQLite file +- Can be tested against a dev Postgres instance with `docker compose up` + +--- +*Phase: 14-postgresql-migration* +*Completed: 2026-04-04*