diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index 2be42f5..f4bb76e 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -121,7 +121,14 @@ Plans: 2. All service-level and route-level tests pass using PGlite in-memory Postgres (no SQLite test infrastructure remains) 3. A one-time migration script converts existing SQLite data into the Postgres database without data loss 4. Docker Compose brings up Postgres alongside the app with a single command for local development -**Plans**: TBD +**Plans:** 6 plans +Plans: +- [ ] 14-01-PLAN.md — Schema rewrite (pg-core), DB config (postgres.js), test helper (PGlite), Drizzle config, initial migration +- [ ] 14-02-PLAN.md — Docker Compose dev/prod + Dockerfile update for PostgreSQL +- [ ] 14-03-PLAN.md — All 9 services async conversion + seed + server index +- [ ] 14-04-PLAN.md — All 9 route handlers async conversion +- [ ] 14-05-PLAN.md — SQLite-to-Postgres data migration script +- [ ] 14-06-PLAN.md — All 18 test files async conversion + full suite green ### Phase 15: External Authentication **Goal**: Users can register and log in via a self-hosted OIDC auth provider, replacing the built-in single-user auth system @@ -188,7 +195,7 @@ Plans: | 11. Candidate Ranking | v1.3 | 2/2 | Complete | 2026-03-16 | | 12. Comparison View | v1.3 | 1/1 | Complete | 2026-03-17 | | 13. Setup Impact Preview | v1.3 | 0/2 | Not started | - | -| 14. PostgreSQL Migration | v2.0 | 0/? | Not started | - | +| 14. PostgreSQL Migration | v2.0 | 0/6 | Not started | - | | 15. External Authentication | v2.0 | 0/? | Not started | - | | 16. Multi-User Data Model | v2.0 | 0/? | Not started | - | | 17. Object Storage | v2.0 | 0/? | Not started | - | diff --git a/.planning/phases/14-postgresql-migration/14-01-PLAN.md b/.planning/phases/14-postgresql-migration/14-01-PLAN.md new file mode 100644 index 0000000..b521d50 --- /dev/null +++ b/.planning/phases/14-postgresql-migration/14-01-PLAN.md @@ -0,0 +1,291 @@ +--- +phase: 14-postgresql-migration +plan: 01 +type: execute +wave: 1 +depends_on: [] +files_modified: + - src/db/schema.ts + - src/db/index.ts + - src/db/migrate.ts + - src/db/seed.ts + - src/shared/types.ts + - tests/helpers/db.ts + - drizzle.config.ts + - package.json +autonomous: true +requirements: [DB-01, DB-03] +must_haves: + truths: + - "Schema defines all 12 tables using drizzle-orm/pg-core (pgTable, serial, text, timestamp, etc.)" + - "Database connection uses postgres.js driver with DATABASE_URL" + - "Test helper creates async PGlite-backed Drizzle instance with migrations applied" + - "Drizzle migrations are generated in drizzle-pg/ directory" + artifacts: + - path: "src/db/schema.ts" + provides: "PostgreSQL table definitions" + contains: "pgTable" + - path: "src/db/index.ts" + provides: "Async Postgres connection" + contains: "drizzle-orm/postgres-js" + - path: "tests/helpers/db.ts" + provides: "PGlite test database factory" + contains: "drizzle-orm/pglite" + - path: "drizzle-pg/" + provides: "PostgreSQL migration files" + - path: "drizzle.config.ts" + provides: "Drizzle Kit config for PostgreSQL" + contains: "postgresql" + key_links: + - from: "tests/helpers/db.ts" + to: "src/db/schema.ts" + via: "import * as schema" + pattern: "import.*schema" + - from: "src/db/index.ts" + to: "src/db/schema.ts" + via: "import * as schema" + pattern: "import.*schema" +--- + + +Rewrite the database foundation from SQLite to PostgreSQL: schema definitions, database connection, test infrastructure, and Drizzle configuration. Install required packages. Generate the initial PostgreSQL migration. + +Purpose: Everything else in this phase depends on these files. Schema and DB config must exist before services, routes, or tests can be converted. +Output: Working schema.ts (pg-core), index.ts (postgres.js), tests/helpers/db.ts (PGlite), drizzle.config.ts (postgresql), generated migration in drizzle-pg/ + + + +@$HOME/.claude/get-shit-done/workflows/execute-plan.md +@$HOME/.claude/get-shit-done/templates/summary.md + + + +@.planning/PROJECT.md +@.planning/ROADMAP.md +@.planning/STATE.md +@.planning/phases/14-postgresql-migration/14-CONTEXT.md +@.planning/phases/14-postgresql-migration/14-RESEARCH.md + +@src/db/schema.ts +@src/db/index.ts +@src/db/migrate.ts +@src/db/seed.ts +@src/shared/types.ts +@tests/helpers/db.ts +@drizzle.config.ts +@package.json + + + + + + Task 1: Install dependencies and rewrite schema + DB config files + package.json, src/db/schema.ts, src/db/index.ts, src/db/migrate.ts, src/db/seed.ts, src/shared/types.ts, drizzle.config.ts + src/db/schema.ts, src/db/index.ts, src/db/migrate.ts, src/db/seed.ts, src/shared/types.ts, drizzle.config.ts, package.json + +**Step 1: Install packages** +```bash +bun add postgres @electric-sql/pglite +bun remove better-sqlite3 @types/better-sqlite3 +``` + +**Step 2: Rewrite `src/db/schema.ts`** — Clean rewrite per D-01. Replace all `sqliteTable` with `pgTable`, all imports from `drizzle-orm/sqlite-core` with `drizzle-orm/pg-core`. + +Column type mapping (apply to ALL 12 tables): +- `integer("id").primaryKey({ autoIncrement: true })` -> `serial("id").primaryKey()` +- `text("col")` -> `text("col")` (unchanged) +- `real("weight_grams")` -> `doublePrecision("weight_grams")` +- `real("sort_order")` -> `doublePrecision("sort_order")` +- `integer("price_cents")` -> `integer("price_cents")` (unchanged) +- `integer("col", { mode: "timestamp" }).$defaultFn(() => new Date())` -> `timestamp("col").notNull().defaultNow()` +- `integer("col", { mode: "timestamp" }).notNull()` (no default, e.g., expiresAt) -> `timestamp("col").notNull()` +- `integer("used").notNull().default(0)` -> `boolean("used").notNull().default(false)` (oauthCodes table) +- `integer("quantity").notNull().default(1)` -> `integer("quantity").notNull().default(1)` (unchanged) + +Tables to rewrite (12 total): categories, items, threads, threadCandidates, setups, setupItems, settings, users, sessions, apiKeys, oauthClients, oauthCodes, oauthTokens. + +Import statement: +```typescript +import { boolean, doublePrecision, integer, pgTable, serial, text, timestamp } from "drizzle-orm/pg-core"; +``` + +Preserve ALL foreign key references and cascade rules exactly as they are. Preserve all `.unique()` constraints. Preserve all `.default()` values. + +For `settings` table: keep `text("key").primaryKey()` (no serial). +For `sessions` table: keep `text("id").primaryKey()` (no serial). + +**Step 3: Rewrite `src/db/index.ts`** per D-03: +```typescript +import { drizzle } from "drizzle-orm/postgres-js"; +import postgres from "postgres"; +import * as schema from "./schema.ts"; + +const connectionString = process.env.DATABASE_URL || "postgresql://gearbox:gearbox@localhost:5432/gearbox"; +const queryClient = postgres(connectionString); +export const db = drizzle(queryClient, { schema }); +``` + +**Step 4: Rewrite `src/db/migrate.ts`**: +```typescript +import { drizzle } from "drizzle-orm/postgres-js"; +import { migrate } from "drizzle-orm/postgres-js/migrator"; +import postgres from "postgres"; + +const connectionString = process.env.DATABASE_URL || "postgresql://gearbox:gearbox@localhost:5432/gearbox"; +const migrationClient = postgres(connectionString, { max: 1 }); +const db = drizzle(migrationClient); + +await migrate(db, { migrationsFolder: "./drizzle-pg" }); +await migrationClient.end(); + +console.log("Migrations applied successfully"); +``` + +**Step 5: Rewrite `src/db/seed.ts`** to async: +```typescript +import { db } from "./index.ts"; +import { categories } from "./schema.ts"; + +export async function seedDefaults() { + const existing = await db.select().from(categories); + if (existing.length === 0) { + await db.insert(categories).values({ + name: "Uncategorized", + icon: "package", + }); + } +} +``` + +**Step 6: Update `src/shared/types.ts`** — No changes needed to the file content itself. The types infer from schema which still exports the same table names. Verify the file still compiles after schema change. + +**Step 7: Update `drizzle.config.ts`** per D-02: +```typescript +import { defineConfig } from "drizzle-kit"; + +export default defineConfig({ + out: "./drizzle-pg", + schema: "./src/db/schema.ts", + dialect: "postgresql", + dbCredentials: { + url: process.env.DATABASE_URL || "postgresql://gearbox:gearbox@localhost:5432/gearbox", + }, +}); +``` + + + grep -q "pgTable" src/db/schema.ts && grep -q "drizzle-orm/pg-core" src/db/schema.ts && grep -q "postgres-js" src/db/index.ts && grep -q "postgresql" drizzle.config.ts && grep -q "async function seedDefaults" src/db/seed.ts && echo "PASS" || echo "FAIL" + + + - src/db/schema.ts contains `import { boolean, doublePrecision, integer, pgTable, serial, text, timestamp } from "drizzle-orm/pg-core"` + - src/db/schema.ts contains `pgTable("categories"` and all 12 table definitions use pgTable + - src/db/schema.ts does NOT contain `sqliteTable` or `drizzle-orm/sqlite-core` or `real(` or `{ mode: "timestamp" }` + - src/db/schema.ts contains `boolean("used")` for oauthCodes table + - src/db/schema.ts contains `doublePrecision("weight_grams")` and `doublePrecision("sort_order")` + - src/db/schema.ts contains `timestamp("created_at").notNull().defaultNow()` pattern + - src/db/index.ts contains `import postgres from "postgres"` and `drizzle-orm/postgres-js` + - src/db/index.ts contains `DATABASE_URL` + - src/db/index.ts does NOT contain `bun:sqlite` + - src/db/migrate.ts contains `drizzle-orm/postgres-js/migrator` and `migrationsFolder: "./drizzle-pg"` + - src/db/seed.ts contains `export async function seedDefaults()` + - src/db/seed.ts contains `await db.select()` and `await db.insert()` + - drizzle.config.ts contains `dialect: "postgresql"` and `out: "./drizzle-pg"` + - package.json contains `"postgres"` in dependencies + - package.json contains `"@electric-sql/pglite"` in devDependencies or dependencies + - package.json does NOT contain `"better-sqlite3"` or `"@types/better-sqlite3"` + + All 12 tables rewritten with pg-core types. DB connection uses postgres.js. Migrate.ts uses postgres-js migrator. Seed is async. Drizzle config targets postgresql dialect with drizzle-pg/ output. + + + + Task 2: Rewrite test helper and generate initial PostgreSQL migration + tests/helpers/db.ts, drizzle-pg/ + tests/helpers/db.ts, src/db/schema.ts + +**Step 1: Rewrite `tests/helpers/db.ts`** per D-07 and D-08: +```typescript +import { drizzle } from "drizzle-orm/pglite"; +import { migrate } from "drizzle-orm/pglite/migrator"; +import * as schema from "../../src/db/schema.ts"; + +export async function createTestDb() { + const db = drizzle({ schema }); + + // Apply migrations from the new PostgreSQL migration directory + await migrate(db, { migrationsFolder: "./drizzle-pg" }); + + // Seed default Uncategorized category + await db.insert(schema.categories).values({ name: "Uncategorized", icon: "package" }); + + return db; +} +``` + +Key changes from current: +- Import from `drizzle-orm/pglite` instead of `drizzle-orm/bun-sqlite` +- `migrate` from `drizzle-orm/pglite/migrator` instead of `drizzle-orm/bun-sqlite/migrator` +- Function is now `async` (returns Promise) +- No `Database` import from `bun:sqlite` +- No `":memory:"` — PGlite creates an in-memory Postgres instance by default +- Migration folder changed to `./drizzle-pg` +- `db.insert(...).values(...).run()` becomes `await db.insert(...).values(...)` + +**Step 2: Generate initial PostgreSQL migration:** +```bash +bunx drizzle-kit generate +``` + +This reads the updated `drizzle.config.ts` (dialect: "postgresql", schema: src/db/schema.ts) and generates SQL migration files in `drizzle-pg/`. + +**Step 3: Verify migration was generated:** +```bash +ls drizzle-pg/ +cat drizzle-pg/*.sql +``` + +Confirm the SQL contains `CREATE TABLE` statements for all 12 tables with correct Postgres types (serial, text, timestamp, boolean, double precision, etc.). + +**Step 4: Quick smoke test — verify PGlite test helper works:** +```bash +bun -e " +import { createTestDb } from './tests/helpers/db.ts'; +const db = await createTestDb(); +const cats = await db.select().from((await import('./src/db/schema.ts')).categories); +console.log('Categories:', cats.length); +if (cats.length !== 1) { console.error('FAIL: expected 1 category'); process.exit(1); } +console.log('PGlite test helper works!'); +" +``` + + + ls drizzle-pg/*.sql && grep -q "drizzle-orm/pglite" tests/helpers/db.ts && grep -q "async function createTestDb" tests/helpers/db.ts && echo "PASS" || echo "FAIL" + + + - tests/helpers/db.ts contains `import { drizzle } from "drizzle-orm/pglite"` + - tests/helpers/db.ts contains `import { migrate } from "drizzle-orm/pglite/migrator"` + - tests/helpers/db.ts contains `export async function createTestDb()` + - tests/helpers/db.ts contains `migrationsFolder: "./drizzle-pg"` + - tests/helpers/db.ts does NOT contain `bun:sqlite` or `drizzle-orm/bun-sqlite` or `.run()` + - drizzle-pg/ directory exists with at least one .sql migration file + - Migration SQL contains CREATE TABLE for categories, items, threads, thread_candidates, setups, setup_items, settings, users, sessions, api_keys, oauth_clients, oauth_codes, oauth_tokens + - PGlite smoke test (bun -e script above) exits 0 + + Test helper returns async PGlite Drizzle instance. Initial PostgreSQL migration generated in drizzle-pg/. Smoke test confirms PGlite can apply migrations and seed data. + + + + + +- `grep -r "sqliteTable\|bun:sqlite\|drizzle-orm/sqlite-core\|drizzle-orm/bun-sqlite" src/db/ drizzle.config.ts tests/helpers/db.ts` returns NO matches +- `grep -c "pgTable" src/db/schema.ts` returns 12+ (one per table, possibly more from import) +- `ls drizzle-pg/*.sql` shows at least one migration file +- PGlite smoke test exits 0 + + + +All database foundation files rewritten for PostgreSQL. Schema uses pg-core types. DB connection uses postgres.js. Test helper uses PGlite. Initial migration generated. No SQLite references remain in these files. + + + +After completion, create `.planning/phases/14-postgresql-migration/14-01-SUMMARY.md` + diff --git a/.planning/phases/14-postgresql-migration/14-02-PLAN.md b/.planning/phases/14-postgresql-migration/14-02-PLAN.md new file mode 100644 index 0000000..599e9dd --- /dev/null +++ b/.planning/phases/14-postgresql-migration/14-02-PLAN.md @@ -0,0 +1,226 @@ +--- +phase: 14-postgresql-migration +plan: 02 +type: execute +wave: 1 +depends_on: [] +files_modified: + - docker-compose.dev.yml + - docker-compose.yml + - Dockerfile + - entrypoint.sh +autonomous: true +requirements: [DB-05] +must_haves: + truths: + - "docker compose -f docker-compose.dev.yml up starts a PostgreSQL 16 instance accessible on localhost:5432" + - "Production docker-compose.yml includes Postgres service with healthcheck and the app depends on it" + - "Dockerfile copies drizzle-pg/ instead of drizzle/ and no longer installs native build tools for better-sqlite3" + artifacts: + - path: "docker-compose.dev.yml" + provides: "Development Postgres service" + contains: "postgres:16-alpine" + - path: "docker-compose.yml" + provides: "Production Postgres + app services" + contains: "postgres:16-alpine" + - path: "Dockerfile" + provides: "Updated container build" + contains: "drizzle-pg" + key_links: + - from: "docker-compose.yml" + to: "Dockerfile" + via: "app service builds from Dockerfile" + pattern: "depends_on" +--- + + +Create Docker Compose configurations for local development and production with PostgreSQL 16, and update the Dockerfile for the Postgres-based app. + +Purpose: Provides the database infrastructure for local dev (DB-05) and production. Must exist before anyone runs the app against real Postgres. +Output: docker-compose.dev.yml (new), docker-compose.yml (rewritten for Postgres), Dockerfile (updated), entrypoint.sh (updated) + + + +@$HOME/.claude/get-shit-done/workflows/execute-plan.md +@$HOME/.claude/get-shit-done/templates/summary.md + + + +@.planning/PROJECT.md +@.planning/ROADMAP.md +@.planning/phases/14-postgresql-migration/14-CONTEXT.md +@.planning/phases/14-postgresql-migration/14-RESEARCH.md + +@Dockerfile +@entrypoint.sh + + + + + + Task 1: Create Docker Compose files for dev and production + docker-compose.dev.yml, docker-compose.yml + docker-compose.yml, Dockerfile, entrypoint.sh + +**Step 1: Create `docker-compose.dev.yml`** per D-10 and D-11: +```yaml +services: + postgres: + image: postgres:16-alpine + environment: + POSTGRES_USER: gearbox + POSTGRES_PASSWORD: gearbox + POSTGRES_DB: gearbox + ports: + - "5432:5432" + volumes: + - pgdata-dev:/var/lib/postgresql/data + healthcheck: + test: ["CMD-SHELL", "pg_isready -U gearbox"] + interval: 5s + timeout: 3s + retries: 5 + +volumes: + pgdata-dev: +``` + +This is a development-only file. The app itself runs locally via `bun run dev` against this Postgres instance using `DATABASE_URL=postgresql://gearbox:gearbox@localhost:5432/gearbox`. + +**Step 2: Rewrite `docker-compose.yml`** for production per D-10: +```yaml +services: + postgres: + image: postgres:16-alpine + environment: + POSTGRES_USER: gearbox + POSTGRES_PASSWORD: ${POSTGRES_PASSWORD} + POSTGRES_DB: gearbox + volumes: + - pgdata:/var/lib/postgresql/data + healthcheck: + test: ["CMD-SHELL", "pg_isready -U gearbox"] + interval: 10s + timeout: 5s + retries: 5 + + app: + image: gearbox:latest + environment: + DATABASE_URL: postgresql://gearbox:${POSTGRES_PASSWORD}@postgres:5432/gearbox + GEARBOX_URL: ${GEARBOX_URL} + ports: + - "3000:3000" + depends_on: + postgres: + condition: service_healthy + volumes: + - uploads:/app/uploads + +volumes: + pgdata: + uploads: +``` + +Key changes from current docker-compose.yml: +- Remove any SQLite volume mounts (data/, gearbox.db references) +- Add postgres service with healthcheck +- App service uses DATABASE_URL env var per D-12 +- App depends_on postgres with service_healthy condition +- POSTGRES_PASSWORD is externalized (not hardcoded in production) + + + grep -q "postgres:16-alpine" docker-compose.dev.yml && grep -q "postgres:16-alpine" docker-compose.yml && grep -q "POSTGRES_PASSWORD" docker-compose.yml && grep -q "DATABASE_URL" docker-compose.yml && echo "PASS" || echo "FAIL" + + + - docker-compose.dev.yml exists and contains `image: postgres:16-alpine` + - docker-compose.dev.yml contains `POSTGRES_USER: gearbox` and `POSTGRES_PASSWORD: gearbox` and `POSTGRES_DB: gearbox` + - docker-compose.dev.yml contains `ports:` with `"5432:5432"` + - docker-compose.dev.yml contains a healthcheck with `pg_isready -U gearbox` + - docker-compose.yml contains `image: postgres:16-alpine` + - docker-compose.yml contains `DATABASE_URL: postgresql://gearbox:${POSTGRES_PASSWORD}@postgres:5432/gearbox` + - docker-compose.yml contains `depends_on:` with `condition: service_healthy` + - docker-compose.yml does NOT contain `gearbox.db` or `DATABASE_PATH` or `sqlite` + + Docker Compose dev file provides local Postgres. Production compose includes Postgres with healthcheck and app service with DATABASE_URL. + + + + Task 2: Update Dockerfile and entrypoint for PostgreSQL + Dockerfile, entrypoint.sh + Dockerfile, entrypoint.sh + +**Step 1: Update `Dockerfile`:** + +The current Dockerfile installs `python3 make g++` for native SQLite bindings (better-sqlite3). These are no longer needed since postgres.js is pure JavaScript. + +```dockerfile +FROM oven/bun:1 AS deps +WORKDIR /app +COPY package.json bun.lock ./ +RUN bun install --frozen-lockfile + +FROM deps AS build +COPY . . +RUN bun run build + +FROM oven/bun:1-slim AS production +WORKDIR /app +ENV NODE_ENV=production +COPY --from=deps /app/node_modules ./node_modules +COPY --from=build /app/dist/client ./dist/client +COPY src/server ./src/server +COPY src/db ./src/db +COPY src/shared ./src/shared +COPY drizzle.config.ts package.json ./ +COPY drizzle-pg ./drizzle-pg +COPY entrypoint.sh ./ +RUN chmod +x entrypoint.sh && mkdir -p uploads +EXPOSE 3000 +HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \ + CMD bun -e "fetch('http://localhost:3000/api/health').then(r=>r.ok?process.exit(0):process.exit(1)).catch(()=>process.exit(1))" +ENTRYPOINT ["./entrypoint.sh"] +``` + +Key changes: +- Remove `RUN apt-get update && apt-get install -y python3 make g++ && rm -rf /var/lib/apt/lists/*` from deps stage (no native bindings needed) +- Change `COPY drizzle ./drizzle` to `COPY drizzle-pg ./drizzle-pg` +- Remove `mkdir -p data` (no SQLite data directory needed) + +**Step 2: Update `entrypoint.sh`** — no changes needed (it already runs `bun run src/db/migrate.ts` which has been rewritten to use postgres-js migrator in Plan 01). Verify it still reads: +```bash +#!/bin/sh +set -e +bun run src/db/migrate.ts +exec bun run src/server/index.ts +``` + + + grep -q "drizzle-pg" Dockerfile && ! grep -q "python3 make g++" Dockerfile && ! grep -q "COPY drizzle ./drizzle" Dockerfile && echo "PASS" || echo "FAIL" + + + - Dockerfile contains `COPY drizzle-pg ./drizzle-pg` + - Dockerfile does NOT contain `COPY drizzle ./drizzle` (the old SQLite migrations line) + - Dockerfile does NOT contain `python3 make g++` or `apt-get install` + - Dockerfile does NOT contain `mkdir -p data` (no SQLite data dir) + - Dockerfile still contains `COPY src/db ./src/db` and `COPY src/server ./src/server` + - entrypoint.sh still contains `bun run src/db/migrate.ts` + + Dockerfile builds without native deps, copies drizzle-pg/ migrations. Entrypoint runs postgres-js based migration on startup. + + + + + +- `docker compose -f docker-compose.dev.yml config` validates successfully +- `docker compose config` validates the production file +- `grep -r "sqlite\|better-sqlite\|bun:sqlite" Dockerfile docker-compose.yml docker-compose.dev.yml` returns NO matches + + + +Docker Compose dev file provides PostgreSQL 16 for local development. Production compose includes Postgres + app with proper dependency chain. Dockerfile is lean (no native build tools) and copies PostgreSQL migrations. + + + +After completion, create `.planning/phases/14-postgresql-migration/14-02-SUMMARY.md` + diff --git a/.planning/phases/14-postgresql-migration/14-03-PLAN.md b/.planning/phases/14-postgresql-migration/14-03-PLAN.md new file mode 100644 index 0000000..b647768 --- /dev/null +++ b/.planning/phases/14-postgresql-migration/14-03-PLAN.md @@ -0,0 +1,202 @@ +--- +phase: 14-postgresql-migration +plan: 03 +type: execute +wave: 2 +depends_on: [14-01] +files_modified: + - src/server/services/item.service.ts + - src/server/services/category.service.ts + - src/server/services/thread.service.ts + - src/server/services/setup.service.ts + - src/server/services/auth.service.ts + - src/server/services/oauth.service.ts + - src/server/services/image.service.ts + - src/server/services/csv.service.ts + - src/server/services/totals.service.ts + - src/server/index.ts +autonomous: true +requirements: [DB-01, DB-02] +must_haves: + truths: + - "Every service function is async and awaits all database calls" + - "No .all(), .get(), or .run() SQLite-only methods remain in any service" + - "Transactions use async callbacks with await on inner operations" + - "Server startup awaits async seed function" + artifacts: + - path: "src/server/services/item.service.ts" + provides: "Async item CRUD operations" + contains: "async function" + - path: "src/server/services/thread.service.ts" + provides: "Async thread operations with async transactions" + contains: "async (tx)" + - path: "src/server/index.ts" + provides: "Async server startup with seed" + contains: "await seedDefaults" + key_links: + - from: "src/server/services/*.ts" + to: "src/db/schema.ts" + via: "import table definitions" + pattern: "from.*db/schema" + - from: "src/server/index.ts" + to: "src/db/seed.ts" + via: "await seedDefaults()" + pattern: "await seedDefaults" +--- + + +Convert all 9 service files from synchronous SQLite operations to async PostgreSQL operations. Update server startup to await async seed. + +Purpose: Services are the data access layer. Every database call must be async for postgres.js. This is the bulk of the mechanical conversion work (~82 call sites per the research). +Output: All service files use async/await. Server index awaits seed. + + + +@$HOME/.claude/get-shit-done/workflows/execute-plan.md +@$HOME/.claude/get-shit-done/templates/summary.md + + + +@.planning/PROJECT.md +@.planning/ROADMAP.md +@.planning/phases/14-postgresql-migration/14-CONTEXT.md +@.planning/phases/14-postgresql-migration/14-RESEARCH.md +@.planning/phases/14-postgresql-migration/14-01-SUMMARY.md + +@src/db/schema.ts +@src/db/index.ts + + + + + + + +Conversion rules (apply to ALL service files): +- `function foo(db)` -> `async function foo(db)` +- `.all()` -> remove (await the query directly, returns array) +- `.get()` -> destructure: `const [row] = await db.select()...` +- `.run()` -> remove (await the query directly) +- `.returning().get()` -> `const [row] = await db.insert()...returning()` +- `db.transaction(() => { ... })` -> `await db.transaction(async (tx) => { await tx... })` + + + + + + Task 1: Convert core data services to async (item, category, thread, setup, totals) + src/server/services/item.service.ts, src/server/services/category.service.ts, src/server/services/thread.service.ts, src/server/services/setup.service.ts, src/server/services/totals.service.ts + src/server/services/item.service.ts, src/server/services/category.service.ts, src/server/services/thread.service.ts, src/server/services/setup.service.ts, src/server/services/totals.service.ts + +Convert each service file following the async conversion rules. Read each file fully before modifying. + +**item.service.ts** — 5 exported functions (getAllItems, getItemById, createItem, updateItem, duplicateItem, deleteItem): +- `getAllItems`: `async`, remove `.all()`, `return await db.select()...` +- `getItemById`: `async`, replace `.get() ?? null` with `const [row] = await db.select()...; return row ?? null` +- `createItem`: `async`, replace `.returning().get()` with `const [row] = await db.insert()...returning(); return row` +- `updateItem`: `async`, existence check uses destructure `const [existing] = await db.select()...`, update uses `const [row] = await db.insert()...returning(); return row` +- `duplicateItem`: `async`, same pattern as createItem +- `deleteItem`: `async`, existence check `const [item] = await db.select()...`, delete `await db.delete()...` + +**category.service.ts** — Has a transaction in `deleteCategory` (moves items to Uncategorized then deletes): +- All functions: `async` +- Transaction: `await db.transaction(async (tx) => { await tx.update()...; await tx.delete()...; })` +- All `.all()` -> remove, `.get()` -> destructure, `.run()` -> remove + +**thread.service.ts** — Has transactions in `resolveThread` and `unresolveThread`: +- All functions: `async` +- `resolveThread` transaction: `await db.transaction(async (tx) => { ... })` with all inner operations awaited +- `unresolveThread` transaction: same pattern +- `.all()` -> remove, `.get()` -> destructure, `.run()` -> remove +- `.returning().get()` -> `const [row] = await ...returning()` + +**setup.service.ts** — Has a transaction in `updateSetupItems` (delete all + re-insert): +- All functions: `async` +- Transaction: `await db.transaction(async (tx) => { await tx.delete()...; for (const item of items) { await tx.insert()...; } })` +- `.all()` -> remove, `.get()` -> destructure, `.run()` -> remove + +**totals.service.ts** — Read-only aggregate queries: +- All functions: `async` +- Remove `.all()`, `.get()` -> destructure + + + ! grep -n "\.all()\|\.get()\|\.run()" src/server/services/item.service.ts src/server/services/category.service.ts src/server/services/thread.service.ts src/server/services/setup.service.ts src/server/services/totals.service.ts && grep -c "async function" src/server/services/item.service.ts | grep -q "[3-9]" && echo "PASS" || echo "FAIL" + + + - item.service.ts: every exported function starts with `export async function` + - item.service.ts: does NOT contain `.all()`, `.get()`, or `.run()` + - category.service.ts: `deleteCategory` contains `await db.transaction(async (tx) =>` + - thread.service.ts: `resolveThread` and `unresolveThread` contain `await db.transaction(async (tx) =>` + - setup.service.ts: `updateSetupItems` contains `await db.transaction(async (tx) =>` + - totals.service.ts: every exported function is async + - No file in this set contains `.all()`, `.get()`, or `.run()` calls on db/tx objects + + Core data services (item, category, thread, setup, totals) fully converted to async with all SQLite-only methods removed. + + + + Task 2: Convert auth/oauth/csv/image services and update server index + src/server/services/auth.service.ts, src/server/services/oauth.service.ts, src/server/services/csv.service.ts, src/server/services/image.service.ts, src/server/index.ts + src/server/services/auth.service.ts, src/server/services/oauth.service.ts, src/server/services/csv.service.ts, src/server/services/image.service.ts, src/server/index.ts + +**auth.service.ts** — User and session management: +- All functions: `async` +- Remove `.all()`, `.get()` -> destructure, `.run()` -> remove +- `.returning().get()` -> `const [row] = await ...returning()` +- Pay attention to boolean checks on `oauthCodes.used` — the column is now native `boolean` (true/false), not integer (0/1). If any code checks `=== 0` or `=== 1` for the `used` field, change to `=== false` or `=== true`. + +**oauth.service.ts** — OAuth client, code, token management: +- All functions: `async` +- Same conversion patterns +- IMPORTANT: The `used` column on `oauthCodes` is now `boolean` type. Any `.set({ used: 1 })` must become `.set({ used: true })`. Any `.where(eq(oauthCodes.used, 0))` must become `.where(eq(oauthCodes.used, false))`. + +**csv.service.ts** — CSV export: +- All functions: `async` +- This is read-only, straightforward `.all()` removal + +**image.service.ts** — Image handling: +- All functions: `async` +- Same conversion patterns. May have fewer DB calls than other services. + +**src/server/index.ts** — Server startup: +- Change `seedDefaults()` to `await seedDefaults()` at the top level +- Since the file is a module (ESM), top-level await is supported. Wrap the seed call: + ```typescript + // Seed default data on startup + await seedDefaults(); + ``` +- If the file structure does not support top-level await cleanly (e.g., exports are synchronous), wrap in an async IIFE or move the await before the export. +- The `seedDefaults` import already points to the async version from Plan 01. + + + ! grep -n "\.all()\|\.get()\|\.run()" src/server/services/auth.service.ts src/server/services/oauth.service.ts src/server/services/csv.service.ts src/server/services/image.service.ts && grep -q "await seedDefaults" src/server/index.ts && echo "PASS" || echo "FAIL" + + + - auth.service.ts: every exported function is `async` + - auth.service.ts: does NOT contain `.all()`, `.get()`, or `.run()` + - oauth.service.ts: every exported function is `async` + - oauth.service.ts: does NOT contain `.set({ used: 1 })` — uses `.set({ used: true })` instead + - oauth.service.ts: does NOT contain `eq(oauthCodes.used, 0)` — uses `eq(oauthCodes.used, false)` instead + - csv.service.ts: every exported function is `async`, no `.all()` calls + - image.service.ts: every exported function is `async` + - src/server/index.ts: contains `await seedDefaults()` + - No file in this set contains `.all()`, `.get()`, or `.run()` calls on db objects + + Auth, OAuth, CSV, and image services fully async. OAuth boolean conversion complete. Server startup awaits async seed. + + + + + +- `grep -rn "\.all()\|\.get()\|\.run()" src/server/services/` returns NO matches (except possibly string literals in error messages) +- `grep -c "async function" src/server/services/*.ts` shows every service has async functions +- `grep "await seedDefaults" src/server/index.ts` returns a match + + + +All 9 service files use async/await for every database operation. No SQLite-only methods (.all, .get, .run) remain. Transactions use async callbacks. OAuth boolean conversion complete. Server index awaits async seed. + + + +After completion, create `.planning/phases/14-postgresql-migration/14-03-SUMMARY.md` + diff --git a/.planning/phases/14-postgresql-migration/14-04-PLAN.md b/.planning/phases/14-postgresql-migration/14-04-PLAN.md new file mode 100644 index 0000000..23b3447 --- /dev/null +++ b/.planning/phases/14-postgresql-migration/14-04-PLAN.md @@ -0,0 +1,159 @@ +--- +phase: 14-postgresql-migration +plan: 04 +type: execute +wave: 2 +depends_on: [14-01] +files_modified: + - src/server/routes/items.ts + - src/server/routes/categories.ts + - src/server/routes/threads.ts + - src/server/routes/setups.ts + - src/server/routes/auth.ts + - src/server/routes/oauth.ts + - src/server/routes/images.ts + - src/server/routes/settings.ts + - src/server/routes/totals.ts +autonomous: true +requirements: [DB-01, DB-02] +must_haves: + truths: + - "Every route handler awaits service function calls" + - "All route handlers that call services are async" + - "No route returns a Promise object instead of resolved data" + artifacts: + - path: "src/server/routes/items.ts" + provides: "Async item route handlers" + contains: "await" + - path: "src/server/routes/settings.ts" + provides: "Async settings handlers with direct DB calls" + contains: "await" + key_links: + - from: "src/server/routes/*.ts" + to: "src/server/services/*.ts" + via: "await service function calls" + pattern: "await .*(get|create|update|delete)" +--- + + +Convert all 9 route handler files to properly await async service calls. Route handlers that call service functions must be async and await the results. + +Purpose: With services now async (Plan 03), route handlers must await them. Missing awaits would return Promise objects as JSON responses instead of actual data. +Output: All route files properly await service calls. + + + +@$HOME/.claude/get-shit-done/workflows/execute-plan.md +@$HOME/.claude/get-shit-done/templates/summary.md + + + +@.planning/PROJECT.md +@.planning/ROADMAP.md +@.planning/phases/14-postgresql-migration/14-CONTEXT.md +@.planning/phases/14-postgresql-migration/14-RESEARCH.md +@.planning/phases/14-postgresql-migration/14-03-SUMMARY.md + +@src/server/routes/items.ts +@src/server/routes/settings.ts + + + + + + + + +Conversion rules for routes: +- Handler callback must be `async (c) => { ... }` +- Every service call: `const result = serviceFunction(db, ...)` -> `const result = await serviceFunction(db, ...)` +- Settings route has direct DB calls: add `await` and remove `.all()/.get()/.run()` +- OAuth routes may have direct DB calls for token validation + + + + + + Task 1: Convert data route handlers to async (items, categories, threads, setups, totals) + src/server/routes/items.ts, src/server/routes/categories.ts, src/server/routes/threads.ts, src/server/routes/setups.ts, src/server/routes/totals.ts + src/server/routes/items.ts, src/server/routes/categories.ts, src/server/routes/threads.ts, src/server/routes/setups.ts, src/server/routes/totals.ts + +For each route file, read the full file first. Then: + +1. Ensure every handler callback is `async (c) => { ... }` (many may already be async for body parsing) +2. Add `await` before every service function call +3. If any handler has direct DB calls (`.select()`, `.insert()`, etc.), apply the same rules as services: remove `.all()/.get()/.run()`, use destructuring for single rows + +**items.ts** — Handlers call: `getAllItems(db)`, `getItemById(db, id)`, `createItem(db, data)`, `updateItem(db, id, data)`, `duplicateItem(db, id)`, `deleteItem(db, id)`. Add `await` before each. + +**categories.ts** — Handlers call: `getAllCategories(db)`, `createCategory(db, data)`, `updateCategory(db, id, data)`, `deleteCategory(db, id)`. Add `await` before each. + +**threads.ts** — Handlers call: `getAllThreads(db)`, `getThreadById(db, id)`, `createThread(db, data)`, `updateThread(db, id, data)`, `deleteThread(db, id)`, `resolveThread(db, id, data)`, `unresolveThread(db, id)`, `addCandidate(db, data)`, `updateCandidate(db, id, data)`, `removeCandidate(db, id)`, `reorderCandidates(db, data)`. Add `await` before each. + +**setups.ts** — Handlers call: `getAllSetups(db)`, `getSetupById(db, id)`, `createSetup(db, data)`, `updateSetup(db, id, data)`, `deleteSetup(db, id)`, `updateSetupItems(db, id, data)`, `updateClassification(...)`. Add `await` before each. + +**totals.ts** — Handlers call totals service functions. Add `await` before each. + + + ! grep -n "= getAllItems\|= getItemById\|= createItem\|= getAllCategories\|= getAllThreads\|= getAllSetups" src/server/routes/items.ts src/server/routes/categories.ts src/server/routes/threads.ts src/server/routes/setups.ts 2>/dev/null | grep -v "await" && echo "PASS" || echo "FAIL" + + + - items.ts: every service call is preceded by `await` + - categories.ts: every service call is preceded by `await` + - threads.ts: every service call is preceded by `await` + - setups.ts: every service call is preceded by `await` + - totals.ts: every service call is preceded by `await` + - No route handler assigns a service call result without `await` + + All data route handlers properly await async service calls. + + + + Task 2: Convert auth, OAuth, settings, and images route handlers to async + src/server/routes/auth.ts, src/server/routes/oauth.ts, src/server/routes/settings.ts, src/server/routes/images.ts + src/server/routes/auth.ts, src/server/routes/oauth.ts, src/server/routes/settings.ts, src/server/routes/images.ts, src/server/middleware/auth.ts + +**auth.ts** — Handlers call auth service functions. Add `await` before each service call. + +**oauth.ts** — Handlers call OAuth service functions. Add `await` before each service call. Also check for any direct DB queries in OAuth routes and apply async conversion. + +**settings.ts** — This route likely accesses the database DIRECTLY (no service layer) using `db.select().from(settings)` etc. Apply full async conversion: +- Remove `.all()` — `const rows = await db.select().from(settings)` +- Remove `.get()` — `const [row] = await db.select().from(settings).where(...)` +- Remove `.run()` — `await db.insert(settings).values(...)` + +**images.ts** — May call image service functions. Add `await` before each service call. + +**Also check `src/server/middleware/auth.ts`** — The auth middleware queries sessions and API keys. If it has direct DB calls, convert them: +- Make the middleware function async (if not already) +- Add `await` before DB queries +- Remove `.get()` -> use destructuring + + + ! grep -n "\.all()\|\.get()\|\.run()" src/server/routes/settings.ts src/server/routes/auth.ts src/server/routes/oauth.ts src/server/routes/images.ts 2>/dev/null && echo "PASS" || echo "FAIL" + + + - auth.ts: every service call is preceded by `await` + - oauth.ts: every service call is preceded by `await` + - settings.ts: does NOT contain `.all()`, `.get()`, or `.run()` + - settings.ts: contains `await db.select()` and `await db.insert()` + - images.ts: every service call is preceded by `await` + - If auth middleware has DB calls, they are all awaited with no `.get()` or `.all()` + + Auth, OAuth, settings, and images routes properly await all DB operations. Auth middleware updated if needed. + + + + + +- `grep -rn "\.all()\|\.get()\|\.run()" src/server/routes/` returns NO matches +- Every route handler that calls a service function uses `await` + + + +All 9 route files await async service calls. Settings route uses async direct DB calls. No route handler will return a Promise object instead of resolved data. + + + +After completion, create `.planning/phases/14-postgresql-migration/14-04-SUMMARY.md` + diff --git a/.planning/phases/14-postgresql-migration/14-05-PLAN.md b/.planning/phases/14-postgresql-migration/14-05-PLAN.md new file mode 100644 index 0000000..e6ca114 --- /dev/null +++ b/.planning/phases/14-postgresql-migration/14-05-PLAN.md @@ -0,0 +1,232 @@ +--- +phase: 14-postgresql-migration +plan: 05 +type: execute +wave: 2 +depends_on: [14-01] +files_modified: + - scripts/migrate-sqlite-to-postgres.ts +autonomous: true +requirements: [DB-04] +must_haves: + truths: + - "Script reads all data from SQLite file and writes it to PostgreSQL" + - "Integer timestamps are converted to Date objects for Postgres timestamp columns" + - "Boolean integers (0/1) are converted to true/false for Postgres boolean columns" + - "All IDs and foreign key relationships are preserved" + - "Serial sequences are reset after data migration to avoid duplicate key errors" + artifacts: + - path: "scripts/migrate-sqlite-to-postgres.ts" + provides: "One-time SQLite to Postgres data migration" + contains: "migrate-sqlite-to-postgres" + key_links: + - from: "scripts/migrate-sqlite-to-postgres.ts" + to: "src/db/schema.ts" + via: "import table definitions for typed inserts" + pattern: "import.*schema" +--- + + +Create the one-time SQLite-to-PostgreSQL data migration script that reads from an existing SQLite database and writes all data into PostgreSQL with proper type conversions. + +Purpose: Existing users need to migrate their data from SQLite to Postgres without data loss (DB-04). This is a standalone script run once during the upgrade. +Output: scripts/migrate-sqlite-to-postgres.ts + + + +@$HOME/.claude/get-shit-done/workflows/execute-plan.md +@$HOME/.claude/get-shit-done/templates/summary.md + + + +@.planning/PROJECT.md +@.planning/ROADMAP.md +@.planning/phases/14-postgresql-migration/14-CONTEXT.md +@.planning/phases/14-postgresql-migration/14-RESEARCH.md +@.planning/phases/14-postgresql-migration/14-01-SUMMARY.md + +@src/db/schema.ts + + + + + + + + + + + + + + + + + +Tables in dependency order: +1. categories, users, settings (no foreign keys to other app tables) +2. items, threads, sessions, apiKeys, oauthClients (FK to categories/users) +3. threadCandidates, setups, oauthCodes, oauthTokens (FK to threads/etc) +4. setupItems (FK to setups + items) + + + + + + Task 1: Create SQLite-to-Postgres migration script + scripts/migrate-sqlite-to-postgres.ts + src/db/schema.ts + +Create `scripts/migrate-sqlite-to-postgres.ts` per D-04, D-05, D-06. + +```typescript +// scripts/migrate-sqlite-to-postgres.ts +import { Database } from "bun:sqlite"; +import { drizzle } from "drizzle-orm/postgres-js"; +import postgres from "postgres"; +import * as schema from "../src/db/schema.ts"; +``` + +**Environment variables:** +- `SQLITE_PATH` — path to SQLite database file (default: `"gearbox.db"`) +- `DATABASE_URL` — PostgreSQL connection string (required) + +**Structure:** +1. Open SQLite database read-only +2. Connect to PostgreSQL via postgres.js + drizzle +3. Migrate tables in dependency order (parents before children) +4. Reset all serial sequences after migration +5. Close both connections +6. Print summary + +**Type conversion functions:** +```typescript +function unixToDate(unix: number | null): Date | null { + if (unix === null || unix === undefined) return null; + return new Date(unix * 1000); // Unix seconds to JS milliseconds +} + +function intToBool(val: number | null): boolean { + return val === 1; +} +``` + +**Migration order and transform functions for each table:** + +1. **categories** — `id` (serial), `name`, `icon`, `createdAt` (unixToDate) +2. **users** — `id` (serial), `username`, `passwordHash`, `createdAt` (unixToDate) +3. **settings** — `key`, `value` (no transforms needed, text PK) +4. **items** — `id` (serial), `name`, `weightGrams`, `priceCents`, `categoryId`, `notes`, `productUrl`, `imageFilename`, `imageSourceUrl`, `quantity`, `createdAt` (unixToDate), `updatedAt` (unixToDate) +5. **threads** — `id` (serial), `name`, `status`, `resolvedCandidateId`, `categoryId`, `createdAt` (unixToDate), `updatedAt` (unixToDate) +6. **sessions** — `id` (text PK), `userId`, `expiresAt` (unixToDate) +7. **apiKeys** — `id` (serial), `name`, `keyHash`, `keyPrefix`, `createdAt` (unixToDate) +8. **oauthClients** — `id` (serial), `clientId`, `clientName`, `redirectUris`, `createdAt` (unixToDate) +9. **threadCandidates** — `id` (serial), all fields, `createdAt`/`updatedAt` (unixToDate), `sortOrder` (keep as number) +10. **setups** — `id` (serial), `name`, `createdAt`/`updatedAt` (unixToDate) +11. **oauthCodes** — `id` (serial), all fields, `expiresAt` (unixToDate), `used` (intToBool) +12. **oauthTokens** — `id` (serial), all fields, `expiresAt`/`refreshExpiresAt`/`createdAt` (unixToDate) +13. **setupItems** — `id` (serial), `setupId`, `itemId`, `classification` + +**For each table, use this pattern:** +```typescript +async function migrateTable(tableName: string, pgTable: any, transform: (row: any) => any) { + const rows = sqlite.query(`SELECT * FROM ${tableName}`).all(); + console.log(` ${tableName}: ${rows.length} rows`); + if (rows.length === 0) return; + + for (const row of rows) { + await db.insert(pgTable).values(transform(row)); + } +} +``` + +**Sequence reset after all data is migrated:** +```typescript +async function resetSequences() { + const tablesWithSerial = [ + "categories", "items", "threads", "thread_candidates", + "setups", "setup_items", "users", "api_keys", + "oauth_clients", "oauth_codes", "oauth_tokens" + ]; + + for (const table of tablesWithSerial) { + await sql`SELECT setval(pg_get_serial_sequence('${sql.raw(table)}', 'id'), COALESCE((SELECT MAX(id) FROM ${sql.raw(table)}), 0))`; + } +} +``` + +Note: Use `db.execute(sql\`...\`)` from drizzle-orm for raw SQL, or use `pg\`...\`` from the postgres.js client directly. The `sql.raw()` helper is needed for dynamic table names. + +**Main function:** +```typescript +async function main() { + const sqlitePath = process.env.SQLITE_PATH || "gearbox.db"; + const databaseUrl = process.env.DATABASE_URL; + + if (!databaseUrl) { + console.error("ERROR: DATABASE_URL environment variable is required"); + process.exit(1); + } + + console.log(`Migrating from SQLite (${sqlitePath}) to PostgreSQL...`); + + const sqlite = new Database(sqlitePath, { readonly: true }); + const pg = postgres(databaseUrl); + const db = drizzle(pg, { schema }); + + // ... migrate all tables in order ... + // ... reset sequences ... + + await pg.end(); + sqlite.close(); + + console.log("Migration complete!"); +} + +main().catch((err) => { + console.error("Migration failed:", err); + process.exit(1); +}); +``` + +**Error handling per table:** Wrap each table migration in try/catch, log which table failed and which row (by ID if available), then re-throw. This aids debugging partial migrations. + +**Add to package.json scripts:** +```json +"db:migrate-from-sqlite": "bun run scripts/migrate-sqlite-to-postgres.ts" +``` + + + test -f scripts/migrate-sqlite-to-postgres.ts && grep -q "bun:sqlite" scripts/migrate-sqlite-to-postgres.ts && grep -q "postgres" scripts/migrate-sqlite-to-postgres.ts && grep -q "setval" scripts/migrate-sqlite-to-postgres.ts && grep -q "unixToDate\|unix.*Date\|\\* 1000" scripts/migrate-sqlite-to-postgres.ts && echo "PASS" || echo "FAIL" + + + - scripts/migrate-sqlite-to-postgres.ts exists + - File imports from `bun:sqlite` (read-only) and `drizzle-orm/postgres-js` and `postgres` + - File imports schema from `../src/db/schema.ts` + - File contains a unix-to-Date conversion function (multiplies by 1000) + - File contains an integer-to-boolean conversion for `used` field + - File migrates all 13 tables in dependency order (categories and users before items and threads, etc.) + - File contains `setval` calls to reset serial sequences after migration + - File reads `DATABASE_URL` from environment and exits with error if missing + - File reads `SQLITE_PATH` from environment with default `"gearbox.db"` + - File opens SQLite in readonly mode + - package.json contains `"db:migrate-from-sqlite"` script + + Migration script reads SQLite, writes to Postgres with type conversions, resets sequences. All IDs and FK relationships preserved per D-06. + + + + + +- `bun run scripts/migrate-sqlite-to-postgres.ts --help` or similar does not crash on syntax errors (will fail on missing DATABASE_URL, which is expected) +- Script contains all 13 table migrations +- Script resets sequences for all tables with serial IDs + + + +One-time migration script exists, handles all type conversions (timestamps, booleans), preserves IDs, resets sequences. Can be run with `DATABASE_URL=... SQLITE_PATH=... bun run scripts/migrate-sqlite-to-postgres.ts`. + + + +After completion, create `.planning/phases/14-postgresql-migration/14-05-SUMMARY.md` + diff --git a/.planning/phases/14-postgresql-migration/14-06-PLAN.md b/.planning/phases/14-postgresql-migration/14-06-PLAN.md new file mode 100644 index 0000000..4a66353 --- /dev/null +++ b/.planning/phases/14-postgresql-migration/14-06-PLAN.md @@ -0,0 +1,261 @@ +--- +phase: 14-postgresql-migration +plan: 06 +type: execute +wave: 3 +depends_on: [14-01, 14-03, 14-04] +files_modified: + - tests/services/item.service.test.ts + - tests/services/category.service.test.ts + - tests/services/thread.service.test.ts + - tests/services/setup.service.test.ts + - tests/services/auth.service.test.ts + - tests/services/oauth.service.test.ts + - tests/services/csv.service.test.ts + - tests/services/image.service.test.ts + - tests/services/totals.test.ts + - tests/routes/items.test.ts + - tests/routes/categories.test.ts + - tests/routes/threads.test.ts + - tests/routes/setups.test.ts + - tests/routes/auth.test.ts + - tests/routes/oauth.test.ts + - tests/routes/images.test.ts + - tests/routes/params.test.ts + - tests/mcp/tools.test.ts +autonomous: true +requirements: [DB-02, DB-03] +must_haves: + truths: + - "All 18 test files use async createTestDb() in beforeEach" + - "All test assertions await async service/route calls" + - "bun test tests/ passes with zero failures" + - "No test file imports from bun:sqlite or drizzle-orm/bun-sqlite" + artifacts: + - path: "tests/services/item.service.test.ts" + provides: "Async item service tests" + contains: "await createTestDb" + - path: "tests/routes/items.test.ts" + provides: "Async item route tests" + contains: "await createTestDb" + - path: "tests/mcp/tools.test.ts" + provides: "Async MCP tools tests" + contains: "await createTestDb" + key_links: + - from: "tests/**/*.test.ts" + to: "tests/helpers/db.ts" + via: "import { createTestDb }" + pattern: "createTestDb" + - from: "tests/services/*.test.ts" + to: "src/server/services/*.ts" + via: "import service functions" + pattern: "from.*services/" +--- + + +Convert all 18 test files to async: await createTestDb(), await all service/route calls, await all assertions involving DB operations. Run the full test suite to confirm everything passes on PGlite. + +Purpose: This is the final verification that the entire stack works on PostgreSQL. Tests must pass on PGlite (DB-03) and confirm async operations work correctly (DB-02). +Output: All tests green. Full `bun test tests/` passes. + + + +@$HOME/.claude/get-shit-done/workflows/execute-plan.md +@$HOME/.claude/get-shit-done/templates/summary.md + + + +@.planning/PROJECT.md +@.planning/ROADMAP.md +@.planning/phases/14-postgresql-migration/14-CONTEXT.md +@.planning/phases/14-postgresql-migration/14-RESEARCH.md +@.planning/phases/14-postgresql-migration/14-01-SUMMARY.md +@.planning/phases/14-postgresql-migration/14-03-SUMMARY.md + +@tests/helpers/db.ts + + + + + + + + + + + + + + +Conversion rules for ALL test files: +1. `beforeEach(() => { db = createTestDb(); })` -> `beforeEach(async () => { db = await createTestDb(); })` +2. Every service call in tests: add `await` (they are now async) +3. Every direct DB call in tests (inserts for setup, selects for assertions): add `await`, remove `.all()/.get()/.run()` +4. Route tests: if using `app.request()`, those are already async. But ensure the test app factory is also async. +5. If `type Db = typeof prodDb` causes type mismatch with PGlite db, use `as any` cast + + + + + + Task 1: Convert all 9 service test files to async + tests/services/item.service.test.ts, tests/services/category.service.test.ts, tests/services/thread.service.test.ts, tests/services/setup.service.test.ts, tests/services/auth.service.test.ts, tests/services/oauth.service.test.ts, tests/services/csv.service.test.ts, tests/services/image.service.test.ts, tests/services/totals.test.ts + tests/services/item.service.test.ts, tests/services/category.service.test.ts, tests/services/thread.service.test.ts, tests/services/setup.service.test.ts, tests/services/auth.service.test.ts, tests/services/oauth.service.test.ts, tests/services/csv.service.test.ts, tests/services/image.service.test.ts, tests/services/totals.test.ts + +For EACH of the 9 service test files, apply these changes: + +**1. Make beforeEach async:** +```typescript +// BEFORE: +let db: any; +beforeEach(() => { + db = createTestDb(); +}); + +// AFTER: +let db: any; +beforeEach(async () => { + db = await createTestDb(); +}); +``` + +**2. Add `await` to every service function call in test bodies:** +```typescript +// BEFORE: +const items = getAllItems(db); +const item = createItem(db, { name: "Test", categoryId: 1 }); + +// AFTER: +const items = await getAllItems(db); +const item = await createItem(db, { name: "Test", categoryId: 1 }); +``` + +**3. Add `await` to direct DB calls used for test setup/assertions:** +```typescript +// BEFORE: +db.insert(schema.items).values({ ... }).run(); +const [cat] = db.select().from(schema.categories).all(); + +// AFTER: +await db.insert(schema.items).values({ ... }); +const [cat] = await db.select().from(schema.categories); +``` + +**4. Make test callbacks async if not already:** +```typescript +// BEFORE: +it("should return all items", () => { + +// AFTER: +it("should return all items", async () => { +``` + +**5. Handle Db type compatibility:** +If TypeScript complains about passing PGlite db to service functions that expect `PostgresJsDatabase`, use `as any` on the db variable: +```typescript +let db: any; // Use any to accommodate PGlite/postgres-js type difference +``` + +**6. OAuth tests — boolean conversion:** +If any OAuth test checks `used === 0` or `used === 1`, change to `used === false` or `used === true`. + +After converting each file, run it individually: +```bash +bun test tests/services/item.service.test.ts +``` +Fix any issues before moving to the next file. + + + bun test tests/services/ 2>&1 | tail -5 + + + - Every service test file has `beforeEach(async () => { db = await createTestDb(); })` + - Every test callback (`it(...)`) that calls service functions or DB is `async` + - No test file contains `.all()`, `.get()`, or `.run()` on db objects + - No test file imports from `bun:sqlite` or `drizzle-orm/bun-sqlite` + - `bun test tests/services/` exits 0 with all tests passing + + All 9 service test files converted to async and passing on PGlite. + + + + Task 2: Convert all route tests + MCP test to async, run full suite + tests/routes/items.test.ts, tests/routes/categories.test.ts, tests/routes/threads.test.ts, tests/routes/setups.test.ts, tests/routes/auth.test.ts, tests/routes/oauth.test.ts, tests/routes/images.test.ts, tests/routes/params.test.ts, tests/mcp/tools.test.ts + tests/routes/items.test.ts, tests/routes/auth.test.ts, tests/mcp/tools.test.ts + +Route tests typically create a test app with a test database injected. The pattern is usually: + +```typescript +// Common route test pattern: +function createTestApp() { + const db = createTestDb(); + // ... create Hono app with db injected + return { app, db }; +} +``` + +This must become: +```typescript +async function createTestApp() { + const db = await createTestDb(); + // ... create Hono app with db injected + return { app, db }; +} +``` + +**For each of the 8 route test files + 1 MCP test file:** + +1. Make the test app factory `async` and `await createTestDb()` +2. Make `beforeEach` async if it calls the factory +3. Route tests use `app.request()` which returns a Promise — these should already be awaited. Verify each test awaits the response. +4. If any test does direct DB calls for setup/assertions, apply same async conversion as service tests +5. Make all test callbacks async + +**MCP test (tests/mcp/tools.test.ts):** +- Same pattern: async createTestDb, await all MCP tool calls +- MCP tools internally call services which are now async + +**After all files converted, run the FULL test suite:** +```bash +bun test tests/ +``` + +This is the gate check. ALL tests must pass. If any test fails: +1. Read the error message carefully +2. Common issues: missing `await`, `.get()` not removed, type mismatch +3. Fix and re-run + +**Also verify no SQLite references remain anywhere in test files:** +```bash +grep -rn "bun:sqlite\|drizzle-orm/bun-sqlite\|\.all()\|\.get()\|\.run()" tests/ +``` +Should return NO matches (except possibly string literals in test descriptions). + + + bun test tests/ 2>&1 | tail -10 + + + - Every route test file has async `createTestApp` or async `beforeEach` with `await createTestDb()` + - Every test callback is `async` + - tests/mcp/tools.test.ts uses `await createTestDb()` + - `grep -rn "bun:sqlite\|drizzle-orm/bun-sqlite" tests/` returns NO matches + - `bun test tests/` exits 0 with ALL tests passing (zero failures) + + All 18 test files pass on PGlite. Full test suite green. No SQLite test infrastructure remains. + + + + + +- `bun test tests/` — ALL tests pass (exit code 0) +- `grep -rn "bun:sqlite\|drizzle-orm/bun-sqlite" tests/` — NO matches +- `grep -rn "\.all()\b" tests/ | grep -v "describe\|it(" ` — NO matches on DB calls (may appear in test descriptions) + + + +All 18 test files converted to async PGlite. Full test suite (`bun test tests/`) passes with zero failures. No SQLite test infrastructure remains anywhere in the tests/ directory. + + + +After completion, create `.planning/phases/14-postgresql-migration/14-06-SUMMARY.md` +