docs: bring phase 14 planning files into worktree
This commit is contained in:
@@ -121,7 +121,14 @@ Plans:
|
||||
2. All service-level and route-level tests pass using PGlite in-memory Postgres (no SQLite test infrastructure remains)
|
||||
3. A one-time migration script converts existing SQLite data into the Postgres database without data loss
|
||||
4. Docker Compose brings up Postgres alongside the app with a single command for local development
|
||||
**Plans**: TBD
|
||||
**Plans:** 6 plans
|
||||
Plans:
|
||||
- [ ] 14-01-PLAN.md — Schema rewrite (pg-core), DB config (postgres.js), test helper (PGlite), Drizzle config, initial migration
|
||||
- [ ] 14-02-PLAN.md — Docker Compose dev/prod + Dockerfile update for PostgreSQL
|
||||
- [ ] 14-03-PLAN.md — All 9 services async conversion + seed + server index
|
||||
- [ ] 14-04-PLAN.md — All 9 route handlers async conversion
|
||||
- [ ] 14-05-PLAN.md — SQLite-to-Postgres data migration script
|
||||
- [ ] 14-06-PLAN.md — All 18 test files async conversion + full suite green
|
||||
|
||||
### Phase 15: External Authentication
|
||||
**Goal**: Users can register and log in via a self-hosted OIDC auth provider, replacing the built-in single-user auth system
|
||||
@@ -188,7 +195,7 @@ Plans:
|
||||
| 11. Candidate Ranking | v1.3 | 2/2 | Complete | 2026-03-16 |
|
||||
| 12. Comparison View | v1.3 | 1/1 | Complete | 2026-03-17 |
|
||||
| 13. Setup Impact Preview | v1.3 | 0/2 | Not started | - |
|
||||
| 14. PostgreSQL Migration | v2.0 | 0/? | Not started | - |
|
||||
| 14. PostgreSQL Migration | v2.0 | 0/6 | Not started | - |
|
||||
| 15. External Authentication | v2.0 | 0/? | Not started | - |
|
||||
| 16. Multi-User Data Model | v2.0 | 0/? | Not started | - |
|
||||
| 17. Object Storage | v2.0 | 0/? | Not started | - |
|
||||
|
||||
@@ -1,16 +1,16 @@
|
||||
---
|
||||
gsd_state_version: 1.0
|
||||
milestone: v2.0
|
||||
milestone_name: Platform Foundation
|
||||
milestone: v1.3
|
||||
milestone_name: Research & Decision Tools
|
||||
status: planning
|
||||
stopped_at: null
|
||||
last_updated: "2026-04-03"
|
||||
stopped_at: Phase 14 plans created (6 plans, 3 waves)
|
||||
last_updated: "2026-04-04T10:12:41.534Z"
|
||||
last_activity: 2026-04-03 — v2.0 roadmap created (Phases 14-18)
|
||||
progress:
|
||||
total_phases: 5
|
||||
completed_phases: 0
|
||||
total_plans: 0
|
||||
completed_plans: 0
|
||||
total_phases: 8
|
||||
completed_phases: 6
|
||||
total_plans: 18
|
||||
completed_plans: 10
|
||||
percent: 0
|
||||
---
|
||||
|
||||
@@ -35,6 +35,7 @@ Progress: [----------] 0% (v2.0 milestone)
|
||||
## Performance Metrics
|
||||
|
||||
**Velocity:**
|
||||
|
||||
- Total plans completed: 0 (v2.0 milestone)
|
||||
- Average duration: --
|
||||
- Total execution time: --
|
||||
@@ -46,6 +47,7 @@ Progress: [----------] 0% (v2.0 milestone)
|
||||
### Decisions
|
||||
|
||||
Key decisions made during v2.0 planning:
|
||||
|
||||
- Platform pivot: single-user to multi-user with discovery-first approach
|
||||
- External auth provider (self-hosted, open-source) — Logto vs Authentik OPEN decision
|
||||
- SQLite to Postgres migration — required by auth provider and multi-user concurrency
|
||||
@@ -64,6 +66,6 @@ None active.
|
||||
|
||||
## Session Continuity
|
||||
|
||||
Last session: 2026-04-03
|
||||
Stopped at: v2.0 roadmap created with 5 phases (14-18) covering 30 requirements
|
||||
Resume file: None
|
||||
Last session: 2026-04-04T10:12:41.532Z
|
||||
Stopped at: Phase 14 plans created (6 plans, 3 waves)
|
||||
Resume file: .planning/phases/14-postgresql-migration/14-01-PLAN.md
|
||||
|
||||
294
.planning/phases/14-postgresql-migration/14-01-PLAN.md
Normal file
294
.planning/phases/14-postgresql-migration/14-01-PLAN.md
Normal file
@@ -0,0 +1,294 @@
|
||||
---
|
||||
phase: 14-postgresql-migration
|
||||
plan: 01
|
||||
type: execute
|
||||
wave: 1
|
||||
depends_on: []
|
||||
files_modified:
|
||||
- src/db/schema.ts
|
||||
- src/db/index.ts
|
||||
- src/db/migrate.ts
|
||||
- src/db/seed.ts
|
||||
- src/shared/types.ts
|
||||
- tests/helpers/db.ts
|
||||
- drizzle.config.ts
|
||||
- package.json
|
||||
autonomous: true
|
||||
requirements: [DB-01, DB-03]
|
||||
must_haves:
|
||||
truths:
|
||||
- "Schema defines all 12 tables using drizzle-orm/pg-core (pgTable, serial, text, timestamp, etc.)"
|
||||
- "Database connection uses postgres.js driver with DATABASE_URL"
|
||||
- "Test helper creates async PGlite-backed Drizzle instance with migrations applied"
|
||||
- "Drizzle migrations are generated in drizzle-pg/ directory"
|
||||
artifacts:
|
||||
- path: "src/db/schema.ts"
|
||||
provides: "PostgreSQL table definitions"
|
||||
contains: "pgTable"
|
||||
- path: "src/db/index.ts"
|
||||
provides: "Async Postgres connection"
|
||||
contains: "drizzle-orm/postgres-js"
|
||||
- path: "tests/helpers/db.ts"
|
||||
provides: "PGlite test database factory"
|
||||
contains: "drizzle-orm/pglite"
|
||||
- path: "drizzle-pg/"
|
||||
provides: "PostgreSQL migration files"
|
||||
- path: "drizzle.config.ts"
|
||||
provides: "Drizzle Kit config for PostgreSQL"
|
||||
contains: "postgresql"
|
||||
key_links:
|
||||
- from: "tests/helpers/db.ts"
|
||||
to: "src/db/schema.ts"
|
||||
via: "import * as schema"
|
||||
pattern: "import.*schema"
|
||||
- from: "src/db/index.ts"
|
||||
to: "src/db/schema.ts"
|
||||
via: "import * as schema"
|
||||
pattern: "import.*schema"
|
||||
---
|
||||
|
||||
<objective>
|
||||
Rewrite the database foundation from SQLite to PostgreSQL: schema definitions, database connection, test infrastructure, and Drizzle configuration. Install required packages. Generate the initial PostgreSQL migration.
|
||||
|
||||
Purpose: Everything else in this phase depends on these files. Schema and DB config must exist before services, routes, or tests can be converted.
|
||||
Output: Working schema.ts (pg-core), index.ts (postgres.js), tests/helpers/db.ts (PGlite), drizzle.config.ts (postgresql), generated migration in drizzle-pg/
|
||||
</objective>
|
||||
|
||||
<execution_context>
|
||||
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
|
||||
@$HOME/.claude/get-shit-done/templates/summary.md
|
||||
</execution_context>
|
||||
|
||||
<context>
|
||||
@.planning/PROJECT.md
|
||||
@.planning/ROADMAP.md
|
||||
@.planning/STATE.md
|
||||
@.planning/phases/14-postgresql-migration/14-CONTEXT.md
|
||||
@.planning/phases/14-postgresql-migration/14-RESEARCH.md
|
||||
|
||||
@src/db/schema.ts
|
||||
@src/db/index.ts
|
||||
@src/db/migrate.ts
|
||||
@src/db/seed.ts
|
||||
@src/shared/types.ts
|
||||
@tests/helpers/db.ts
|
||||
@drizzle.config.ts
|
||||
@package.json
|
||||
</context>
|
||||
|
||||
<tasks>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 1: Install dependencies and rewrite schema + DB config files</name>
|
||||
<files>package.json, src/db/schema.ts, src/db/index.ts, src/db/migrate.ts, src/db/seed.ts, src/shared/types.ts, drizzle.config.ts</files>
|
||||
<read_first>src/db/schema.ts, src/db/index.ts, src/db/migrate.ts, src/db/seed.ts, src/shared/types.ts, drizzle.config.ts, package.json</read_first>
|
||||
<action>
|
||||
**Step 1: Install packages**
|
||||
```bash
|
||||
bun add postgres @electric-sql/pglite
|
||||
bun remove better-sqlite3 @types/better-sqlite3
|
||||
```
|
||||
|
||||
**Step 2: Rewrite `src/db/schema.ts`** -- Clean rewrite per D-01. Replace all `sqliteTable` with `pgTable`, all imports from `drizzle-orm/sqlite-core` with `drizzle-orm/pg-core`.
|
||||
|
||||
Column type mapping (apply to ALL 12 tables):
|
||||
- `integer("id").primaryKey({ autoIncrement: true })` -> `serial("id").primaryKey()`
|
||||
- `text("col")` -> `text("col")` (unchanged)
|
||||
- `real("weight_grams")` -> `doublePrecision("weight_grams")`
|
||||
- `real("sort_order")` -> `doublePrecision("sort_order")`
|
||||
- `integer("price_cents")` -> `integer("price_cents")` (unchanged)
|
||||
- `integer("col", { mode: "timestamp" }).$defaultFn(() => new Date())` -> `timestamp("col").notNull().defaultNow()`
|
||||
- `integer("col", { mode: "timestamp" }).notNull()` (no default, e.g., expiresAt) -> `timestamp("col").notNull()`
|
||||
- `integer("used").notNull().default(0)` -> `boolean("used").notNull().default(false)` (oauthCodes table)
|
||||
- `integer("quantity").notNull().default(1)` -> `integer("quantity").notNull().default(1)` (unchanged)
|
||||
|
||||
Tables to rewrite (12 total): categories, items, threads, threadCandidates, setups, setupItems, settings, users, sessions, apiKeys, oauthClients, oauthCodes, oauthTokens.
|
||||
|
||||
Import statement:
|
||||
```typescript
|
||||
import { boolean, doublePrecision, integer, pgTable, serial, text, timestamp } from "drizzle-orm/pg-core";
|
||||
```
|
||||
|
||||
Preserve ALL foreign key references and cascade rules exactly as they are. Preserve all `.unique()` constraints. Preserve all `.default()` values.
|
||||
|
||||
For `settings` table: keep `text("key").primaryKey()` (no serial).
|
||||
For `sessions` table: keep `text("id").primaryKey()` (no serial).
|
||||
|
||||
**Step 3: Rewrite `src/db/index.ts`** per D-03:
|
||||
```typescript
|
||||
import { drizzle } from "drizzle-orm/postgres-js";
|
||||
import postgres from "postgres";
|
||||
import * as schema from "./schema.ts";
|
||||
|
||||
const connectionString = process.env.DATABASE_URL || "postgresql://gearbox:gearbox@localhost:5432/gearbox";
|
||||
const queryClient = postgres(connectionString);
|
||||
export const db = drizzle(queryClient, { schema });
|
||||
```
|
||||
|
||||
**Step 4: Rewrite `src/db/migrate.ts`**:
|
||||
```typescript
|
||||
import { drizzle } from "drizzle-orm/postgres-js";
|
||||
import { migrate } from "drizzle-orm/postgres-js/migrator";
|
||||
import postgres from "postgres";
|
||||
|
||||
const connectionString = process.env.DATABASE_URL || "postgresql://gearbox:gearbox@localhost:5432/gearbox";
|
||||
const migrationClient = postgres(connectionString, { max: 1 });
|
||||
const db = drizzle(migrationClient);
|
||||
|
||||
await migrate(db, { migrationsFolder: "./drizzle-pg" });
|
||||
await migrationClient.end();
|
||||
|
||||
console.log("Migrations applied successfully");
|
||||
```
|
||||
|
||||
**Step 5: Rewrite `src/db/seed.ts`** to async:
|
||||
```typescript
|
||||
import { db } from "./index.ts";
|
||||
import { categories } from "./schema.ts";
|
||||
|
||||
export async function seedDefaults() {
|
||||
const existing = await db.select().from(categories);
|
||||
if (existing.length === 0) {
|
||||
await db.insert(categories).values({
|
||||
name: "Uncategorized",
|
||||
icon: "package",
|
||||
});
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Step 6: Update `src/shared/types.ts`** -- No changes needed to the file content itself. The types infer from schema which still exports the same table names. Verify the file still compiles after schema change.
|
||||
|
||||
**Step 7: Update `drizzle.config.ts`** per D-02:
|
||||
```typescript
|
||||
import { defineConfig } from "drizzle-kit";
|
||||
|
||||
export default defineConfig({
|
||||
out: "./drizzle-pg",
|
||||
schema: "./src/db/schema.ts",
|
||||
dialect: "postgresql",
|
||||
dbCredentials: {
|
||||
url: process.env.DATABASE_URL || "postgresql://gearbox:gearbox@localhost:5432/gearbox",
|
||||
},
|
||||
});
|
||||
```
|
||||
</action>
|
||||
<verify>
|
||||
<automated>grep -q "pgTable" src/db/schema.ts && grep -q "drizzle-orm/pg-core" src/db/schema.ts && grep -q "postgres-js" src/db/index.ts && grep -q "postgresql" drizzle.config.ts && grep -q "async function seedDefaults" src/db/seed.ts && bun run lint 2>&1 | tail -3 && echo "PASS" || echo "FAIL"</automated>
|
||||
</verify>
|
||||
<acceptance_criteria>
|
||||
- src/db/schema.ts contains `import { boolean, doublePrecision, integer, pgTable, serial, text, timestamp } from "drizzle-orm/pg-core"`
|
||||
- src/db/schema.ts contains `pgTable("categories"` and all 12 table definitions use pgTable
|
||||
- src/db/schema.ts does NOT contain `sqliteTable` or `drizzle-orm/sqlite-core` or `real(` or `{ mode: "timestamp" }`
|
||||
- src/db/schema.ts contains `boolean("used")` for oauthCodes table
|
||||
- src/db/schema.ts contains `doublePrecision("weight_grams")` and `doublePrecision("sort_order")`
|
||||
- src/db/schema.ts contains `timestamp("created_at").notNull().defaultNow()` pattern
|
||||
- src/db/index.ts contains `import postgres from "postgres"` and `drizzle-orm/postgres-js`
|
||||
- src/db/index.ts contains `DATABASE_URL`
|
||||
- src/db/index.ts does NOT contain `bun:sqlite`
|
||||
- src/db/migrate.ts contains `drizzle-orm/postgres-js/migrator` and `migrationsFolder: "./drizzle-pg"`
|
||||
- src/db/seed.ts contains `export async function seedDefaults()`
|
||||
- src/db/seed.ts contains `await db.select()` and `await db.insert()`
|
||||
- drizzle.config.ts contains `dialect: "postgresql"` and `out: "./drizzle-pg"`
|
||||
- package.json contains `"postgres"` in dependencies
|
||||
- package.json contains `"@electric-sql/pglite"` in devDependencies or dependencies
|
||||
- package.json does NOT contain `"better-sqlite3"` or `"@types/better-sqlite3"`
|
||||
</acceptance_criteria>
|
||||
<done>All 12 tables rewritten with pg-core types. DB connection uses postgres.js. Migrate.ts uses postgres-js migrator. Seed is async. Drizzle config targets postgresql dialect with drizzle-pg/ output.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 2: Rewrite test helper and generate initial PostgreSQL migration</name>
|
||||
<files>tests/helpers/db.ts, drizzle-pg/</files>
|
||||
<read_first>tests/helpers/db.ts, src/db/schema.ts</read_first>
|
||||
<action>
|
||||
**Step 1: Rewrite `tests/helpers/db.ts`** per D-07 and D-08:
|
||||
```typescript
|
||||
import { drizzle } from "drizzle-orm/pglite";
|
||||
import { migrate } from "drizzle-orm/pglite/migrator";
|
||||
import * as schema from "../../src/db/schema.ts";
|
||||
|
||||
export async function createTestDb() {
|
||||
const db = drizzle({ schema });
|
||||
|
||||
// Apply migrations from the new PostgreSQL migration directory
|
||||
await migrate(db, { migrationsFolder: "./drizzle-pg" });
|
||||
|
||||
// Seed default Uncategorized category
|
||||
await db.insert(schema.categories).values({ name: "Uncategorized", icon: "package" });
|
||||
|
||||
return db;
|
||||
}
|
||||
```
|
||||
|
||||
Key changes from current:
|
||||
- Import from `drizzle-orm/pglite` instead of `drizzle-orm/bun-sqlite`
|
||||
- `migrate` from `drizzle-orm/pglite/migrator` instead of `drizzle-orm/bun-sqlite/migrator`
|
||||
- Function is now `async` (returns Promise)
|
||||
- No `Database` import from `bun:sqlite`
|
||||
- No `":memory:"` -- PGlite creates an in-memory Postgres instance by default
|
||||
- Migration folder changed to `./drizzle-pg`
|
||||
- `db.insert(...).values(...).run()` becomes `await db.insert(...).values(...)`
|
||||
|
||||
**Step 2: Generate initial PostgreSQL migration:**
|
||||
```bash
|
||||
bunx drizzle-kit generate
|
||||
```
|
||||
|
||||
This reads the updated `drizzle.config.ts` (dialect: "postgresql", schema: src/db/schema.ts) and generates SQL migration files in `drizzle-pg/`.
|
||||
|
||||
**Step 3: Verify migration was generated and is complete:**
|
||||
```bash
|
||||
ls drizzle-pg/
|
||||
cat drizzle-pg/*.sql
|
||||
```
|
||||
|
||||
Confirm the SQL contains `CREATE TABLE` statements for all 12 tables with correct Postgres types (serial, text, timestamp, boolean, double precision, etc.). Count the CREATE TABLE statements -- there must be at least 12 (categories, items, threads, thread_candidates, setups, setup_items, settings, users, sessions, api_keys, oauth_clients, oauth_codes, oauth_tokens).
|
||||
|
||||
**Step 4: Quick smoke test -- verify PGlite test helper works:**
|
||||
```bash
|
||||
bun -e "
|
||||
import { createTestDb } from './tests/helpers/db.ts';
|
||||
const db = await createTestDb();
|
||||
const cats = await db.select().from((await import('./src/db/schema.ts')).categories);
|
||||
console.log('Categories:', cats.length);
|
||||
if (cats.length !== 1) { console.error('FAIL: expected 1 category'); process.exit(1); }
|
||||
console.log('PGlite test helper works!');
|
||||
"
|
||||
```
|
||||
</action>
|
||||
<verify>
|
||||
<automated>ls drizzle-pg/*.sql && grep -c "CREATE TABLE" drizzle-pg/*.sql | tail -1 | grep -qE "^drizzle-pg/.*:1[2-9]$|^drizzle-pg/.*:[2-9][0-9]$" || { echo "WARNING: verify CREATE TABLE count manually"; }; grep -q "drizzle-orm/pglite" tests/helpers/db.ts && grep -q "async function createTestDb" tests/helpers/db.ts && bun run lint 2>&1 | tail -3 && echo "PASS" || echo "FAIL"</automated>
|
||||
</verify>
|
||||
<acceptance_criteria>
|
||||
- tests/helpers/db.ts contains `import { drizzle } from "drizzle-orm/pglite"`
|
||||
- tests/helpers/db.ts contains `import { migrate } from "drizzle-orm/pglite/migrator"`
|
||||
- tests/helpers/db.ts contains `export async function createTestDb()`
|
||||
- tests/helpers/db.ts contains `migrationsFolder: "./drizzle-pg"`
|
||||
- tests/helpers/db.ts does NOT contain `bun:sqlite` or `drizzle-orm/bun-sqlite` or `.run()`
|
||||
- drizzle-pg/ directory exists with at least one .sql migration file
|
||||
- Migration SQL contains CREATE TABLE for all 12+ tables (categories, items, threads, thread_candidates, setups, setup_items, settings, users, sessions, api_keys, oauth_clients, oauth_codes, oauth_tokens)
|
||||
- `grep -c "CREATE TABLE" drizzle-pg/*.sql` shows at least 12 CREATE TABLE statements
|
||||
- PGlite smoke test (bun -e script above) exits 0
|
||||
</acceptance_criteria>
|
||||
<done>Test helper returns async PGlite Drizzle instance. Initial PostgreSQL migration generated in drizzle-pg/ with all 12+ CREATE TABLE statements. Smoke test confirms PGlite can apply migrations and seed data.</done>
|
||||
</task>
|
||||
|
||||
</tasks>
|
||||
|
||||
<verification>
|
||||
- `grep -r "sqliteTable\|bun:sqlite\|drizzle-orm/sqlite-core\|drizzle-orm/bun-sqlite" src/db/ drizzle.config.ts tests/helpers/db.ts` returns NO matches
|
||||
- `grep -c "pgTable" src/db/schema.ts` returns 12+ (one per table, possibly more from import)
|
||||
- `ls drizzle-pg/*.sql` shows at least one migration file
|
||||
- `grep -c "CREATE TABLE" drizzle-pg/*.sql` shows at least 12 tables
|
||||
- PGlite smoke test exits 0
|
||||
- `bun run lint` passes
|
||||
</verification>
|
||||
|
||||
<success_criteria>
|
||||
All database foundation files rewritten for PostgreSQL. Schema uses pg-core types. DB connection uses postgres.js. Test helper uses PGlite. Initial migration generated with all 12+ tables. No SQLite references remain in these files. Lint passes.
|
||||
</success_criteria>
|
||||
|
||||
<output>
|
||||
After completion, create `.planning/phases/14-postgresql-migration/14-01-SUMMARY.md`
|
||||
</output>
|
||||
226
.planning/phases/14-postgresql-migration/14-02-PLAN.md
Normal file
226
.planning/phases/14-postgresql-migration/14-02-PLAN.md
Normal file
@@ -0,0 +1,226 @@
|
||||
---
|
||||
phase: 14-postgresql-migration
|
||||
plan: 02
|
||||
type: execute
|
||||
wave: 1
|
||||
depends_on: []
|
||||
files_modified:
|
||||
- docker-compose.dev.yml
|
||||
- docker-compose.yml
|
||||
- Dockerfile
|
||||
- entrypoint.sh
|
||||
autonomous: true
|
||||
requirements: [DB-05]
|
||||
must_haves:
|
||||
truths:
|
||||
- "docker compose -f docker-compose.dev.yml up starts a PostgreSQL 16 instance accessible on localhost:5432"
|
||||
- "Production docker-compose.yml includes Postgres service with healthcheck and the app depends on it"
|
||||
- "Dockerfile copies drizzle-pg/ instead of drizzle/ and no longer installs native build tools for better-sqlite3"
|
||||
artifacts:
|
||||
- path: "docker-compose.dev.yml"
|
||||
provides: "Development Postgres service"
|
||||
contains: "postgres:16-alpine"
|
||||
- path: "docker-compose.yml"
|
||||
provides: "Production Postgres + app services"
|
||||
contains: "postgres:16-alpine"
|
||||
- path: "Dockerfile"
|
||||
provides: "Updated container build"
|
||||
contains: "drizzle-pg"
|
||||
key_links:
|
||||
- from: "docker-compose.yml"
|
||||
to: "Dockerfile"
|
||||
via: "app service builds from Dockerfile"
|
||||
pattern: "depends_on"
|
||||
---
|
||||
|
||||
<objective>
|
||||
Create Docker Compose configurations for local development and production with PostgreSQL 16, and update the Dockerfile for the Postgres-based app.
|
||||
|
||||
Purpose: Provides the database infrastructure for local dev (DB-05) and production. Must exist before anyone runs the app against real Postgres.
|
||||
Output: docker-compose.dev.yml (new), docker-compose.yml (rewritten for Postgres), Dockerfile (updated), entrypoint.sh (updated)
|
||||
</objective>
|
||||
|
||||
<execution_context>
|
||||
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
|
||||
@$HOME/.claude/get-shit-done/templates/summary.md
|
||||
</execution_context>
|
||||
|
||||
<context>
|
||||
@.planning/PROJECT.md
|
||||
@.planning/ROADMAP.md
|
||||
@.planning/phases/14-postgresql-migration/14-CONTEXT.md
|
||||
@.planning/phases/14-postgresql-migration/14-RESEARCH.md
|
||||
|
||||
@Dockerfile
|
||||
@entrypoint.sh
|
||||
</context>
|
||||
|
||||
<tasks>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 1: Create Docker Compose files for dev and production</name>
|
||||
<files>docker-compose.dev.yml, docker-compose.yml</files>
|
||||
<read_first>docker-compose.yml, Dockerfile, entrypoint.sh</read_first>
|
||||
<action>
|
||||
**Step 1: Create `docker-compose.dev.yml`** per D-10 and D-11:
|
||||
```yaml
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:16-alpine
|
||||
environment:
|
||||
POSTGRES_USER: gearbox
|
||||
POSTGRES_PASSWORD: gearbox
|
||||
POSTGRES_DB: gearbox
|
||||
ports:
|
||||
- "5432:5432"
|
||||
volumes:
|
||||
- pgdata-dev:/var/lib/postgresql/data
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U gearbox"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 5
|
||||
|
||||
volumes:
|
||||
pgdata-dev:
|
||||
```
|
||||
|
||||
This is a development-only file. The app itself runs locally via `bun run dev` against this Postgres instance using `DATABASE_URL=postgresql://gearbox:gearbox@localhost:5432/gearbox`.
|
||||
|
||||
**Step 2: Rewrite `docker-compose.yml`** for production per D-10:
|
||||
```yaml
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:16-alpine
|
||||
environment:
|
||||
POSTGRES_USER: gearbox
|
||||
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
|
||||
POSTGRES_DB: gearbox
|
||||
volumes:
|
||||
- pgdata:/var/lib/postgresql/data
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U gearbox"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
|
||||
app:
|
||||
image: gearbox:latest
|
||||
environment:
|
||||
DATABASE_URL: postgresql://gearbox:${POSTGRES_PASSWORD}@postgres:5432/gearbox
|
||||
GEARBOX_URL: ${GEARBOX_URL}
|
||||
ports:
|
||||
- "3000:3000"
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
volumes:
|
||||
- uploads:/app/uploads
|
||||
|
||||
volumes:
|
||||
pgdata:
|
||||
uploads:
|
||||
```
|
||||
|
||||
Key changes from current docker-compose.yml:
|
||||
- Remove any SQLite volume mounts (data/, gearbox.db references)
|
||||
- Add postgres service with healthcheck
|
||||
- App service uses DATABASE_URL env var per D-12
|
||||
- App depends_on postgres with service_healthy condition
|
||||
- POSTGRES_PASSWORD is externalized (not hardcoded in production)
|
||||
</action>
|
||||
<verify>
|
||||
<automated>grep -q "postgres:16-alpine" docker-compose.dev.yml && grep -q "postgres:16-alpine" docker-compose.yml && grep -q "POSTGRES_PASSWORD" docker-compose.yml && grep -q "DATABASE_URL" docker-compose.yml && echo "PASS" || echo "FAIL"</automated>
|
||||
</verify>
|
||||
<acceptance_criteria>
|
||||
- docker-compose.dev.yml exists and contains `image: postgres:16-alpine`
|
||||
- docker-compose.dev.yml contains `POSTGRES_USER: gearbox` and `POSTGRES_PASSWORD: gearbox` and `POSTGRES_DB: gearbox`
|
||||
- docker-compose.dev.yml contains `ports:` with `"5432:5432"`
|
||||
- docker-compose.dev.yml contains a healthcheck with `pg_isready -U gearbox`
|
||||
- docker-compose.yml contains `image: postgres:16-alpine`
|
||||
- docker-compose.yml contains `DATABASE_URL: postgresql://gearbox:${POSTGRES_PASSWORD}@postgres:5432/gearbox`
|
||||
- docker-compose.yml contains `depends_on:` with `condition: service_healthy`
|
||||
- docker-compose.yml does NOT contain `gearbox.db` or `DATABASE_PATH` or `sqlite`
|
||||
</acceptance_criteria>
|
||||
<done>Docker Compose dev file provides local Postgres. Production compose includes Postgres with healthcheck and app service with DATABASE_URL.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 2: Update Dockerfile and entrypoint for PostgreSQL</name>
|
||||
<files>Dockerfile, entrypoint.sh</files>
|
||||
<read_first>Dockerfile, entrypoint.sh</read_first>
|
||||
<action>
|
||||
**Step 1: Update `Dockerfile`:**
|
||||
|
||||
The current Dockerfile installs `python3 make g++` for native SQLite bindings (better-sqlite3). These are no longer needed since postgres.js is pure JavaScript.
|
||||
|
||||
```dockerfile
|
||||
FROM oven/bun:1 AS deps
|
||||
WORKDIR /app
|
||||
COPY package.json bun.lock ./
|
||||
RUN bun install --frozen-lockfile
|
||||
|
||||
FROM deps AS build
|
||||
COPY . .
|
||||
RUN bun run build
|
||||
|
||||
FROM oven/bun:1-slim AS production
|
||||
WORKDIR /app
|
||||
ENV NODE_ENV=production
|
||||
COPY --from=deps /app/node_modules ./node_modules
|
||||
COPY --from=build /app/dist/client ./dist/client
|
||||
COPY src/server ./src/server
|
||||
COPY src/db ./src/db
|
||||
COPY src/shared ./src/shared
|
||||
COPY drizzle.config.ts package.json ./
|
||||
COPY drizzle-pg ./drizzle-pg
|
||||
COPY entrypoint.sh ./
|
||||
RUN chmod +x entrypoint.sh && mkdir -p uploads
|
||||
EXPOSE 3000
|
||||
HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
|
||||
CMD bun -e "fetch('http://localhost:3000/api/health').then(r=>r.ok?process.exit(0):process.exit(1)).catch(()=>process.exit(1))"
|
||||
ENTRYPOINT ["./entrypoint.sh"]
|
||||
```
|
||||
|
||||
Key changes:
|
||||
- Remove `RUN apt-get update && apt-get install -y python3 make g++ && rm -rf /var/lib/apt/lists/*` from deps stage (no native bindings needed)
|
||||
- Change `COPY drizzle ./drizzle` to `COPY drizzle-pg ./drizzle-pg`
|
||||
- Remove `mkdir -p data` (no SQLite data directory needed)
|
||||
|
||||
**Step 2: Update `entrypoint.sh`** — no changes needed (it already runs `bun run src/db/migrate.ts` which has been rewritten to use postgres-js migrator in Plan 01). Verify it still reads:
|
||||
```bash
|
||||
#!/bin/sh
|
||||
set -e
|
||||
bun run src/db/migrate.ts
|
||||
exec bun run src/server/index.ts
|
||||
```
|
||||
</action>
|
||||
<verify>
|
||||
<automated>grep -q "drizzle-pg" Dockerfile && ! grep -q "python3 make g++" Dockerfile && ! grep -q "COPY drizzle ./drizzle" Dockerfile && echo "PASS" || echo "FAIL"</automated>
|
||||
</verify>
|
||||
<acceptance_criteria>
|
||||
- Dockerfile contains `COPY drizzle-pg ./drizzle-pg`
|
||||
- Dockerfile does NOT contain `COPY drizzle ./drizzle` (the old SQLite migrations line)
|
||||
- Dockerfile does NOT contain `python3 make g++` or `apt-get install`
|
||||
- Dockerfile does NOT contain `mkdir -p data` (no SQLite data dir)
|
||||
- Dockerfile still contains `COPY src/db ./src/db` and `COPY src/server ./src/server`
|
||||
- entrypoint.sh still contains `bun run src/db/migrate.ts`
|
||||
</acceptance_criteria>
|
||||
<done>Dockerfile builds without native deps, copies drizzle-pg/ migrations. Entrypoint runs postgres-js based migration on startup.</done>
|
||||
</task>
|
||||
|
||||
</tasks>
|
||||
|
||||
<verification>
|
||||
- `docker compose -f docker-compose.dev.yml config` validates successfully
|
||||
- `docker compose config` validates the production file
|
||||
- `grep -r "sqlite\|better-sqlite\|bun:sqlite" Dockerfile docker-compose.yml docker-compose.dev.yml` returns NO matches
|
||||
</verification>
|
||||
|
||||
<success_criteria>
|
||||
Docker Compose dev file provides PostgreSQL 16 for local development. Production compose includes Postgres + app with proper dependency chain. Dockerfile is lean (no native build tools) and copies PostgreSQL migrations.
|
||||
</success_criteria>
|
||||
|
||||
<output>
|
||||
After completion, create `.planning/phases/14-postgresql-migration/14-02-SUMMARY.md`
|
||||
</output>
|
||||
221
.planning/phases/14-postgresql-migration/14-03-PLAN.md
Normal file
221
.planning/phases/14-postgresql-migration/14-03-PLAN.md
Normal file
@@ -0,0 +1,221 @@
|
||||
---
|
||||
phase: 14-postgresql-migration
|
||||
plan: 03
|
||||
type: execute
|
||||
wave: 2
|
||||
depends_on: [14-01]
|
||||
files_modified:
|
||||
- src/server/services/item.service.ts
|
||||
- src/server/services/category.service.ts
|
||||
- src/server/services/thread.service.ts
|
||||
- src/server/services/setup.service.ts
|
||||
- src/server/services/auth.service.ts
|
||||
- src/server/services/oauth.service.ts
|
||||
- src/server/services/image.service.ts
|
||||
- src/server/services/csv.service.ts
|
||||
- src/server/services/totals.service.ts
|
||||
- src/server/index.ts
|
||||
autonomous: true
|
||||
requirements: [DB-01, DB-02]
|
||||
must_haves:
|
||||
truths:
|
||||
- "Every service function is async and awaits all database calls"
|
||||
- "No .all(), .get(), or .run() SQLite-only methods remain in any service"
|
||||
- "Transactions use async callbacks with await on inner operations"
|
||||
- "Server startup awaits async seed function"
|
||||
artifacts:
|
||||
- path: "src/server/services/item.service.ts"
|
||||
provides: "Async item CRUD operations"
|
||||
contains: "async function"
|
||||
- path: "src/server/services/thread.service.ts"
|
||||
provides: "Async thread operations with async transactions"
|
||||
contains: "async (tx)"
|
||||
- path: "src/server/index.ts"
|
||||
provides: "Async server startup with seed"
|
||||
contains: "await seedDefaults"
|
||||
key_links:
|
||||
- from: "src/server/services/*.ts"
|
||||
to: "src/db/schema.ts"
|
||||
via: "import table definitions"
|
||||
pattern: "from.*db/schema"
|
||||
- from: "src/server/index.ts"
|
||||
to: "src/db/seed.ts"
|
||||
via: "await seedDefaults()"
|
||||
pattern: "await seedDefaults"
|
||||
---
|
||||
|
||||
<objective>
|
||||
Convert all 9 service files from synchronous SQLite operations to async PostgreSQL operations. Update server startup to await async seed.
|
||||
|
||||
Purpose: Services are the data access layer. Every database call must be async for postgres.js. This is the bulk of the mechanical conversion work (~82 call sites per the research).
|
||||
Output: All service files use async/await. Server index awaits seed.
|
||||
</objective>
|
||||
|
||||
<execution_context>
|
||||
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
|
||||
@$HOME/.claude/get-shit-done/templates/summary.md
|
||||
</execution_context>
|
||||
|
||||
<context>
|
||||
@.planning/PROJECT.md
|
||||
@.planning/ROADMAP.md
|
||||
@.planning/phases/14-postgresql-migration/14-CONTEXT.md
|
||||
@.planning/phases/14-postgresql-migration/14-RESEARCH.md
|
||||
@.planning/phases/14-postgresql-migration/14-01-SUMMARY.md
|
||||
|
||||
@src/db/schema.ts
|
||||
@src/db/index.ts
|
||||
</context>
|
||||
|
||||
<interfaces>
|
||||
<!-- Db type will change after Plan 01. Services use `type Db = typeof prodDb` which will now be a PostgresJsDatabase instance. -->
|
||||
<!-- Key pattern: all services take `db: Db = prodDb` as first parameter -->
|
||||
<!-- After Plan 01, src/db/index.ts exports: `export const db = drizzle(queryClient, { schema })` from postgres-js driver -->
|
||||
|
||||
Conversion rules (apply to ALL service files):
|
||||
- `function foo(db)` -> `async function foo(db)`
|
||||
- `.all()` -> remove (await the query directly, returns array)
|
||||
- `.get()` -> destructure: `const [row] = await db.select()...`
|
||||
- `.run()` -> remove (await the query directly)
|
||||
- `.returning().get()` -> `const [row] = await db.insert()...returning()`
|
||||
- `db.transaction(() => { ... })` -> `await db.transaction(async (tx) => { await tx... })`
|
||||
</interfaces>
|
||||
|
||||
<tasks>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 1: Convert core data services to async (item, category, thread, setup, totals)</name>
|
||||
<files>src/server/services/item.service.ts, src/server/services/category.service.ts, src/server/services/thread.service.ts, src/server/services/setup.service.ts, src/server/services/totals.service.ts</files>
|
||||
<read_first>src/server/services/item.service.ts, src/server/services/category.service.ts, src/server/services/thread.service.ts, src/server/services/setup.service.ts, src/server/services/totals.service.ts</read_first>
|
||||
<action>
|
||||
Convert each service file following the async conversion rules. Read each file fully before modifying.
|
||||
|
||||
**item.service.ts** -- 5 exported functions (getAllItems, getItemById, createItem, updateItem, duplicateItem, deleteItem):
|
||||
- `getAllItems`: `async`, remove `.all()`, `return await db.select()...`
|
||||
- `getItemById`: `async`, replace `.get() ?? null` with `const [row] = await db.select()...; return row ?? null`
|
||||
- `createItem`: `async`, replace `.returning().get()` with `const [row] = await db.insert()...returning(); return row`
|
||||
- `updateItem`: `async`, existence check uses destructure `const [existing] = await db.select()...`, update uses `const [row] = await db.insert()...returning(); return row`
|
||||
- `duplicateItem`: `async`, same pattern as createItem
|
||||
- `deleteItem`: `async`, existence check `const [item] = await db.select()...`, delete `await db.delete()...`
|
||||
|
||||
**category.service.ts** -- Has a transaction in `deleteCategory` (moves items to Uncategorized then deletes):
|
||||
- All functions: `async`
|
||||
- Transaction: `await db.transaction(async (tx) => { await tx.update()...; await tx.delete()...; })`
|
||||
- All `.all()` -> remove, `.get()` -> destructure, `.run()` -> remove
|
||||
|
||||
**thread.service.ts** -- Has transactions in `resolveThread` and `unresolveThread`:
|
||||
- All functions: `async`
|
||||
- `resolveThread` transaction: `await db.transaction(async (tx) => { ... })` with all inner operations awaited
|
||||
- `unresolveThread` transaction: same pattern
|
||||
- `.all()` -> remove, `.get()` -> destructure, `.run()` -> remove
|
||||
- `.returning().get()` -> `const [row] = await ...returning()`
|
||||
|
||||
**setup.service.ts** -- Has a transaction in `updateSetupItems` (delete all + re-insert):
|
||||
- All functions: `async`
|
||||
- Transaction: `await db.transaction(async (tx) => { await tx.delete()...; for (const item of items) { await tx.insert()...; } })`
|
||||
- `.all()` -> remove, `.get()` -> destructure, `.run()` -> remove
|
||||
|
||||
**totals.service.ts** -- Read-only aggregate queries:
|
||||
- All functions: `async`
|
||||
- Remove `.all()`, `.get()` -> destructure
|
||||
</action>
|
||||
<verify>
|
||||
<automated>! grep -n "\.all()\|\.get()\|\.run()" src/server/services/item.service.ts src/server/services/category.service.ts src/server/services/thread.service.ts src/server/services/setup.service.ts src/server/services/totals.service.ts && grep -c "async function" src/server/services/item.service.ts | grep -q "[3-9]" && bun run lint 2>&1 | tail -3 && echo "PASS" || echo "FAIL"</automated>
|
||||
</verify>
|
||||
<acceptance_criteria>
|
||||
- item.service.ts: every exported function starts with `export async function`
|
||||
- item.service.ts: does NOT contain `.all()`, `.get()`, or `.run()`
|
||||
- category.service.ts: `deleteCategory` contains `await db.transaction(async (tx) =>`
|
||||
- thread.service.ts: `resolveThread` and `unresolveThread` contain `await db.transaction(async (tx) =>`
|
||||
- setup.service.ts: `updateSetupItems` contains `await db.transaction(async (tx) =>`
|
||||
- totals.service.ts: every exported function is async
|
||||
- No file in this set contains `.all()`, `.get()`, or `.run()` calls on db/tx objects
|
||||
</acceptance_criteria>
|
||||
<done>Core data services (item, category, thread, setup, totals) fully converted to async with all SQLite-only methods removed.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 2: Convert auth/oauth/csv/image services, update server index, and run PGlite smoke test</name>
|
||||
<files>src/server/services/auth.service.ts, src/server/services/oauth.service.ts, src/server/services/csv.service.ts, src/server/services/image.service.ts, src/server/index.ts</files>
|
||||
<read_first>src/server/services/auth.service.ts, src/server/services/oauth.service.ts, src/server/services/csv.service.ts, src/server/services/image.service.ts, src/server/index.ts</read_first>
|
||||
<action>
|
||||
**auth.service.ts** -- User and session management:
|
||||
- All functions: `async`
|
||||
- Remove `.all()`, `.get()` -> destructure, `.run()` -> remove
|
||||
- `.returning().get()` -> `const [row] = await ...returning()`
|
||||
- Pay attention to boolean checks on `oauthCodes.used` -- the column is now native `boolean` (true/false), not integer (0/1). If any code checks `=== 0` or `=== 1` for the `used` field, change to `=== false` or `=== true`.
|
||||
|
||||
**oauth.service.ts** -- OAuth client, code, token management:
|
||||
- All functions: `async`
|
||||
- Same conversion patterns
|
||||
- IMPORTANT: The `used` column on `oauthCodes` is now `boolean` type. Any `.set({ used: 1 })` must become `.set({ used: true })`. Any `.where(eq(oauthCodes.used, 0))` must become `.where(eq(oauthCodes.used, false))`.
|
||||
|
||||
**csv.service.ts** -- CSV export:
|
||||
- All functions: `async`
|
||||
- This is read-only, straightforward `.all()` removal
|
||||
|
||||
**image.service.ts** -- Image handling:
|
||||
- All functions: `async`
|
||||
- Same conversion patterns. May have fewer DB calls than other services.
|
||||
|
||||
**src/server/index.ts** -- Server startup:
|
||||
- Change `seedDefaults()` to `await seedDefaults()` at the top level
|
||||
- Since the file is a module (ESM), top-level await is supported. Wrap the seed call:
|
||||
```typescript
|
||||
// Seed default data on startup
|
||||
await seedDefaults();
|
||||
```
|
||||
- If the file structure does not support top-level await cleanly (e.g., exports are synchronous), wrap in an async IIFE or move the await before the export.
|
||||
- The `seedDefaults` import already points to the async version from Plan 01.
|
||||
|
||||
**After all conversions, run a PGlite smoke test to verify at least one service works end-to-end:**
|
||||
```bash
|
||||
bun -e "
|
||||
import { createTestDb } from './tests/helpers/db.ts';
|
||||
import * as schema from './src/db/schema.ts';
|
||||
const db = await createTestDb();
|
||||
// Test a basic item service operation
|
||||
const { createItem } = await import('./src/server/services/item.service.ts');
|
||||
const [cat] = await db.select().from(schema.categories);
|
||||
const item = await createItem(db as any, { name: 'Smoke Test', categoryId: cat.id, quantity: 1 });
|
||||
if (!item || !item.id) { console.error('FAIL: createItem returned no result'); process.exit(1); }
|
||||
console.log('Service smoke test PASSED: item created with id', item.id);
|
||||
"
|
||||
```
|
||||
This validates that the async conversion is actually functional, not just structurally correct.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>! grep -n "\.all()\|\.get()\|\.run()" src/server/services/auth.service.ts src/server/services/oauth.service.ts src/server/services/csv.service.ts src/server/services/image.service.ts && grep -q "await seedDefaults" src/server/index.ts && bun run lint 2>&1 | tail -3 && echo "PASS" || echo "FAIL"</automated>
|
||||
</verify>
|
||||
<acceptance_criteria>
|
||||
- auth.service.ts: every exported function is `async`
|
||||
- auth.service.ts: does NOT contain `.all()`, `.get()`, or `.run()`
|
||||
- oauth.service.ts: every exported function is `async`
|
||||
- oauth.service.ts: does NOT contain `.set({ used: 1 })` -- uses `.set({ used: true })` instead
|
||||
- oauth.service.ts: does NOT contain `eq(oauthCodes.used, 0)` -- uses `eq(oauthCodes.used, false)` instead
|
||||
- csv.service.ts: every exported function is `async`, no `.all()` calls
|
||||
- image.service.ts: every exported function is `async`
|
||||
- src/server/index.ts: contains `await seedDefaults()`
|
||||
- No file in this set contains `.all()`, `.get()`, or `.run()` calls on db objects
|
||||
- PGlite smoke test creating an item via service function exits 0
|
||||
</acceptance_criteria>
|
||||
<done>Auth, OAuth, CSV, and image services fully async. OAuth boolean conversion complete. Server startup awaits async seed. PGlite smoke test confirms services work against async DB.</done>
|
||||
</task>
|
||||
|
||||
</tasks>
|
||||
|
||||
<verification>
|
||||
- `grep -rn "\.all()\|\.get()\|\.run()" src/server/services/` returns NO matches (except possibly string literals in error messages)
|
||||
- `grep -c "async function" src/server/services/*.ts` shows every service has async functions
|
||||
- `grep "await seedDefaults" src/server/index.ts` returns a match
|
||||
- `bun run lint` passes
|
||||
- PGlite smoke test exits 0
|
||||
</verification>
|
||||
|
||||
<success_criteria>
|
||||
All 9 service files use async/await for every database operation. No SQLite-only methods (.all, .get, .run) remain. Transactions use async callbacks. OAuth boolean conversion complete. Server index awaits async seed. PGlite smoke test validates at least one service works end-to-end. Lint passes.
|
||||
</success_criteria>
|
||||
|
||||
<output>
|
||||
After completion, create `.planning/phases/14-postgresql-migration/14-03-SUMMARY.md`
|
||||
</output>
|
||||
197
.planning/phases/14-postgresql-migration/14-04-PLAN.md
Normal file
197
.planning/phases/14-postgresql-migration/14-04-PLAN.md
Normal file
@@ -0,0 +1,197 @@
|
||||
---
|
||||
phase: 14-postgresql-migration
|
||||
plan: 04
|
||||
type: execute
|
||||
wave: 2
|
||||
depends_on: [14-01]
|
||||
files_modified:
|
||||
- src/server/routes/items.ts
|
||||
- src/server/routes/categories.ts
|
||||
- src/server/routes/threads.ts
|
||||
- src/server/routes/setups.ts
|
||||
- src/server/routes/auth.ts
|
||||
- src/server/routes/oauth.ts
|
||||
- src/server/routes/images.ts
|
||||
- src/server/routes/settings.ts
|
||||
- src/server/routes/totals.ts
|
||||
- src/server/middleware/auth.ts
|
||||
autonomous: true
|
||||
requirements: [DB-01, DB-02]
|
||||
must_haves:
|
||||
truths:
|
||||
- "Every route handler awaits service function calls"
|
||||
- "All route handlers that call services are async"
|
||||
- "No route returns a Promise object instead of resolved data"
|
||||
- "Auth middleware awaits all DB queries for session and API key validation"
|
||||
artifacts:
|
||||
- path: "src/server/routes/items.ts"
|
||||
provides: "Async item route handlers"
|
||||
contains: "await"
|
||||
- path: "src/server/routes/settings.ts"
|
||||
provides: "Async settings handlers with direct DB calls"
|
||||
contains: "await"
|
||||
- path: "src/server/middleware/auth.ts"
|
||||
provides: "Async auth middleware with awaited DB lookups"
|
||||
contains: "await"
|
||||
key_links:
|
||||
- from: "src/server/routes/*.ts"
|
||||
to: "src/server/services/*.ts"
|
||||
via: "await service function calls"
|
||||
pattern: "await .*(get|create|update|delete)"
|
||||
- from: "src/server/middleware/auth.ts"
|
||||
to: "src/db/schema.ts"
|
||||
via: "session and API key DB queries"
|
||||
pattern: "await.*db\\.select"
|
||||
---
|
||||
|
||||
<objective>
|
||||
Convert all 9 route handler files and the auth middleware to properly await async service calls and DB operations. Route handlers that call service functions must be async and await the results.
|
||||
|
||||
Purpose: With services now async (Plan 03), route handlers must await them. Missing awaits would return Promise objects as JSON responses instead of actual data. The auth middleware queries sessions and API keys on every request -- these direct DB calls must also be async.
|
||||
Output: All route files and auth middleware properly await service/DB calls.
|
||||
</objective>
|
||||
|
||||
<execution_context>
|
||||
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
|
||||
@$HOME/.claude/get-shit-done/templates/summary.md
|
||||
</execution_context>
|
||||
|
||||
<context>
|
||||
@.planning/PROJECT.md
|
||||
@.planning/ROADMAP.md
|
||||
@.planning/phases/14-postgresql-migration/14-CONTEXT.md
|
||||
@.planning/phases/14-postgresql-migration/14-RESEARCH.md
|
||||
@.planning/phases/14-postgresql-migration/14-03-SUMMARY.md
|
||||
|
||||
@src/server/routes/items.ts
|
||||
@src/server/routes/settings.ts
|
||||
@src/server/middleware/auth.ts
|
||||
</context>
|
||||
|
||||
<interfaces>
|
||||
<!-- Route handlers use Hono pattern: app.get("/path", async (c) => { ... }) -->
|
||||
<!-- Services are imported and called: const items = await getAllItems(db) -->
|
||||
<!-- Settings route accesses DB directly (no service layer): await db.select().from(settings) -->
|
||||
<!-- Some handlers may already be async (for body parsing). Add await to service calls. -->
|
||||
<!-- Auth middleware queries sessions table and apiKeys table directly on every authenticated request -->
|
||||
|
||||
Conversion rules for routes:
|
||||
- Handler callback must be `async (c) => { ... }`
|
||||
- Every service call: `const result = serviceFunction(db, ...)` -> `const result = await serviceFunction(db, ...)`
|
||||
- Settings route has direct DB calls: add `await` and remove `.all()/.get()/.run()`
|
||||
- OAuth routes may have direct DB calls for token validation
|
||||
|
||||
Conversion rules for auth middleware:
|
||||
- Middleware function must be async
|
||||
- Session lookup: `db.select()...where(eq(sessions.id, ...))` -> add `await`, remove `.get()`, use destructuring
|
||||
- API key lookup: same pattern
|
||||
</interfaces>
|
||||
|
||||
<tasks>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 1: Convert data route handlers to async (items, categories, threads, setups, totals)</name>
|
||||
<files>src/server/routes/items.ts, src/server/routes/categories.ts, src/server/routes/threads.ts, src/server/routes/setups.ts, src/server/routes/totals.ts</files>
|
||||
<read_first>src/server/routes/items.ts, src/server/routes/categories.ts, src/server/routes/threads.ts, src/server/routes/setups.ts, src/server/routes/totals.ts</read_first>
|
||||
<action>
|
||||
For each route file, read the full file first. Then:
|
||||
|
||||
1. Ensure every handler callback is `async (c) => { ... }` (many may already be async for body parsing)
|
||||
2. Add `await` before every service function call
|
||||
3. If any handler has direct DB calls (`.select()`, `.insert()`, etc.), apply the same rules as services: remove `.all()/.get()/.run()`, use destructuring for single rows
|
||||
|
||||
**items.ts** -- Handlers call: `getAllItems(db)`, `getItemById(db, id)`, `createItem(db, data)`, `updateItem(db, id, data)`, `duplicateItem(db, id)`, `deleteItem(db, id)`. Add `await` before each.
|
||||
|
||||
**categories.ts** -- Handlers call: `getAllCategories(db)`, `createCategory(db, data)`, `updateCategory(db, id, data)`, `deleteCategory(db, id)`. Add `await` before each.
|
||||
|
||||
**threads.ts** -- Handlers call: `getAllThreads(db)`, `getThreadById(db, id)`, `createThread(db, data)`, `updateThread(db, id, data)`, `deleteThread(db, id)`, `resolveThread(db, id, data)`, `unresolveThread(db, id)`, `addCandidate(db, data)`, `updateCandidate(db, id, data)`, `removeCandidate(db, id)`, `reorderCandidates(db, data)`. Add `await` before each.
|
||||
|
||||
**setups.ts** -- Handlers call: `getAllSetups(db)`, `getSetupById(db, id)`, `createSetup(db, data)`, `updateSetup(db, id, data)`, `deleteSetup(db, id)`, `updateSetupItems(db, id, data)`, `updateClassification(...)`. Add `await` before each.
|
||||
|
||||
**totals.ts** -- Handlers call totals service functions. Add `await` before each.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>! grep -n "= getAllItems\|= getItemById\|= createItem\|= getAllCategories\|= getAllThreads\|= getAllSetups" src/server/routes/items.ts src/server/routes/categories.ts src/server/routes/threads.ts src/server/routes/setups.ts 2>/dev/null | grep -v "await" && echo "PASS" || echo "FAIL"</automated>
|
||||
</verify>
|
||||
<acceptance_criteria>
|
||||
- items.ts: every service call is preceded by `await`
|
||||
- categories.ts: every service call is preceded by `await`
|
||||
- threads.ts: every service call is preceded by `await`
|
||||
- setups.ts: every service call is preceded by `await`
|
||||
- totals.ts: every service call is preceded by `await`
|
||||
- No route handler assigns a service call result without `await`
|
||||
</acceptance_criteria>
|
||||
<done>All data route handlers properly await async service calls.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 2: Convert auth, OAuth, settings, images routes and auth middleware to async</name>
|
||||
<files>src/server/routes/auth.ts, src/server/routes/oauth.ts, src/server/routes/settings.ts, src/server/routes/images.ts, src/server/middleware/auth.ts</files>
|
||||
<read_first>src/server/routes/auth.ts, src/server/routes/oauth.ts, src/server/routes/settings.ts, src/server/routes/images.ts, src/server/middleware/auth.ts</read_first>
|
||||
<action>
|
||||
**auth.ts** -- Handlers call auth service functions. Add `await` before each service call.
|
||||
|
||||
**oauth.ts** -- Handlers call OAuth service functions. Add `await` before each service call. Also check for any direct DB queries in OAuth routes and apply async conversion.
|
||||
|
||||
**settings.ts** -- This route likely accesses the database DIRECTLY (no service layer) using `db.select().from(settings)` etc. Apply full async conversion:
|
||||
- Remove `.all()` -- `const rows = await db.select().from(settings)`
|
||||
- Remove `.get()` -- `const [row] = await db.select().from(settings).where(...)`
|
||||
- Remove `.run()` -- `await db.insert(settings).values(...)`
|
||||
|
||||
**images.ts** -- May call image service functions. Add `await` before each service call.
|
||||
|
||||
**src/server/middleware/auth.ts** -- The auth middleware queries sessions and API keys on every authenticated request. These are direct DB calls that must become async:
|
||||
- Make the middleware function async (if not already)
|
||||
- Add `await` before all DB queries (session lookup, API key lookup)
|
||||
- Remove `.get()` -> use destructuring: `const [session] = await db.select()...`
|
||||
- Remove `.all()` if present
|
||||
- This is critical -- the auth middleware runs on every POST/PUT/DELETE request, so missing awaits here would break ALL write operations
|
||||
|
||||
**After all conversions, run a PGlite smoke test to verify routes work end-to-end:**
|
||||
```bash
|
||||
bun -e "
|
||||
import { createTestDb } from './tests/helpers/db.ts';
|
||||
import * as schema from './src/db/schema.ts';
|
||||
const db = await createTestDb();
|
||||
// Verify auth middleware can be imported without errors
|
||||
const authMod = await import('./src/server/middleware/auth.ts');
|
||||
console.log('Auth middleware imports OK');
|
||||
// Verify settings route pattern works
|
||||
const rows = await db.select().from(schema.settings);
|
||||
console.log('Direct DB query works, settings count:', rows.length);
|
||||
console.log('Route smoke test PASSED');
|
||||
"
|
||||
```
|
||||
</action>
|
||||
<verify>
|
||||
<automated>! grep -n "\.all()\|\.get()\|\.run()" src/server/routes/settings.ts src/server/routes/auth.ts src/server/routes/oauth.ts src/server/routes/images.ts src/server/middleware/auth.ts 2>/dev/null && bun run lint 2>&1 | tail -3 && echo "PASS" || echo "FAIL"</automated>
|
||||
</verify>
|
||||
<acceptance_criteria>
|
||||
- auth.ts: every service call is preceded by `await`
|
||||
- oauth.ts: every service call is preceded by `await`
|
||||
- settings.ts: does NOT contain `.all()`, `.get()`, or `.run()`
|
||||
- settings.ts: contains `await db.select()` and `await db.insert()`
|
||||
- images.ts: every service call is preceded by `await`
|
||||
- src/server/middleware/auth.ts: does NOT contain `.get()` or `.all()` on DB calls
|
||||
- src/server/middleware/auth.ts: contains `await` before all DB select queries
|
||||
- All files pass lint
|
||||
</acceptance_criteria>
|
||||
<done>Auth, OAuth, settings, and images routes properly await all DB operations. Auth middleware fully converted to async DB operations. Lint passes.</done>
|
||||
</task>
|
||||
|
||||
</tasks>
|
||||
|
||||
<verification>
|
||||
- `grep -rn "\.all()\|\.get()\|\.run()" src/server/routes/ src/server/middleware/auth.ts` returns NO matches
|
||||
- Every route handler that calls a service function uses `await`
|
||||
- Auth middleware awaits all DB queries
|
||||
- `bun run lint` passes
|
||||
</verification>
|
||||
|
||||
<success_criteria>
|
||||
All 9 route files and auth middleware await async service/DB calls. Settings route uses async direct DB calls. Auth middleware properly awaits session and API key lookups. No route handler will return a Promise object instead of resolved data. Lint passes.
|
||||
</success_criteria>
|
||||
|
||||
<output>
|
||||
After completion, create `.planning/phases/14-postgresql-migration/14-04-SUMMARY.md`
|
||||
</output>
|
||||
233
.planning/phases/14-postgresql-migration/14-05-PLAN.md
Normal file
233
.planning/phases/14-postgresql-migration/14-05-PLAN.md
Normal file
@@ -0,0 +1,233 @@
|
||||
---
|
||||
phase: 14-postgresql-migration
|
||||
plan: 05
|
||||
type: execute
|
||||
wave: 2
|
||||
depends_on: [14-01]
|
||||
files_modified:
|
||||
- scripts/migrate-sqlite-to-postgres.ts
|
||||
autonomous: true
|
||||
requirements: [DB-04]
|
||||
must_haves:
|
||||
truths:
|
||||
- "Script reads all data from SQLite file and writes it to PostgreSQL"
|
||||
- "Integer timestamps are converted to Date objects for Postgres timestamp columns"
|
||||
- "Boolean integers (0/1) are converted to true/false for Postgres boolean columns"
|
||||
- "All IDs and foreign key relationships are preserved"
|
||||
- "Serial sequences are reset after data migration to avoid duplicate key errors"
|
||||
artifacts:
|
||||
- path: "scripts/migrate-sqlite-to-postgres.ts"
|
||||
provides: "One-time SQLite to Postgres data migration"
|
||||
contains: "migrate-sqlite-to-postgres"
|
||||
key_links:
|
||||
- from: "scripts/migrate-sqlite-to-postgres.ts"
|
||||
to: "src/db/schema.ts"
|
||||
via: "import table definitions for typed inserts"
|
||||
pattern: "import.*schema"
|
||||
---
|
||||
|
||||
<objective>
|
||||
Create the one-time SQLite-to-PostgreSQL data migration script that reads from an existing SQLite database and writes all data into PostgreSQL with proper type conversions.
|
||||
|
||||
Purpose: Existing users need to migrate their data from SQLite to Postgres without data loss (DB-04). This is a standalone script run once during the upgrade.
|
||||
Output: scripts/migrate-sqlite-to-postgres.ts
|
||||
</objective>
|
||||
|
||||
<execution_context>
|
||||
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
|
||||
@$HOME/.claude/get-shit-done/templates/summary.md
|
||||
</execution_context>
|
||||
|
||||
<context>
|
||||
@.planning/PROJECT.md
|
||||
@.planning/ROADMAP.md
|
||||
@.planning/phases/14-postgresql-migration/14-CONTEXT.md
|
||||
@.planning/phases/14-postgresql-migration/14-RESEARCH.md
|
||||
@.planning/phases/14-postgresql-migration/14-01-SUMMARY.md
|
||||
|
||||
@src/db/schema.ts
|
||||
</context>
|
||||
|
||||
<interfaces>
|
||||
<!-- SQLite schema (current production data format): -->
|
||||
<!-- - Timestamps: stored as unix epoch integers (seconds since 1970) -->
|
||||
<!-- - Booleans: stored as integers (0 = false, 1 = true), only oauthCodes.used -->
|
||||
<!-- - Weights: stored as real (float) -->
|
||||
<!-- - IDs: auto-increment integers -->
|
||||
<!-- - settings: key (text PK) + value (text) -->
|
||||
<!-- - sessions: id (text PK) + userId (int) + expiresAt (int timestamp) -->
|
||||
|
||||
<!-- PostgreSQL schema (target format after Plan 01): -->
|
||||
<!-- - Timestamps: native timestamp type (JS Date objects) -->
|
||||
<!-- - Booleans: native boolean type -->
|
||||
<!-- - Weights: doublePrecision -->
|
||||
<!-- - IDs: serial (auto-increment with sequence) -->
|
||||
|
||||
Tables in dependency order:
|
||||
1. categories, users, settings (no foreign keys to other app tables)
|
||||
2. items, threads, sessions, apiKeys, oauthClients (FK to categories/users)
|
||||
3. threadCandidates, setups, oauthCodes, oauthTokens (FK to threads/etc)
|
||||
4. setupItems (FK to setups + items)
|
||||
</interfaces>
|
||||
|
||||
<tasks>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 1: Create SQLite-to-Postgres migration script</name>
|
||||
<files>scripts/migrate-sqlite-to-postgres.ts</files>
|
||||
<read_first>src/db/schema.ts</read_first>
|
||||
<action>
|
||||
Create `scripts/migrate-sqlite-to-postgres.ts` per D-04, D-05, D-06.
|
||||
|
||||
```typescript
|
||||
// scripts/migrate-sqlite-to-postgres.ts
|
||||
import { Database } from "bun:sqlite";
|
||||
import { drizzle } from "drizzle-orm/postgres-js";
|
||||
import postgres from "postgres";
|
||||
import * as schema from "../src/db/schema.ts";
|
||||
```
|
||||
|
||||
**Environment variables:**
|
||||
- `SQLITE_PATH` -- path to SQLite database file (default: `"gearbox.db"`)
|
||||
- `DATABASE_URL` -- PostgreSQL connection string (required)
|
||||
|
||||
**Structure:**
|
||||
1. Open SQLite database read-only
|
||||
2. Connect to PostgreSQL via postgres.js + drizzle
|
||||
3. Migrate tables in dependency order (parents before children)
|
||||
4. Reset all serial sequences after migration
|
||||
5. Close both connections
|
||||
6. Print summary
|
||||
|
||||
**Type conversion functions:**
|
||||
```typescript
|
||||
function unixToDate(unix: number | null): Date | null {
|
||||
if (unix === null || unix === undefined) return null;
|
||||
return new Date(unix * 1000); // Unix seconds to JS milliseconds
|
||||
}
|
||||
|
||||
function intToBool(val: number | null): boolean {
|
||||
return val === 1;
|
||||
}
|
||||
```
|
||||
|
||||
**Migration order and transform functions for each table:**
|
||||
|
||||
1. **categories** -- `id` (serial), `name`, `icon`, `createdAt` (unixToDate)
|
||||
2. **users** -- `id` (serial), `username`, `passwordHash`, `createdAt` (unixToDate)
|
||||
3. **settings** -- `key`, `value` (no transforms needed, text PK)
|
||||
4. **items** -- `id` (serial), `name`, `weightGrams`, `priceCents`, `categoryId`, `notes`, `productUrl`, `imageFilename`, `imageSourceUrl`, `quantity`, `createdAt` (unixToDate), `updatedAt` (unixToDate)
|
||||
5. **threads** -- `id` (serial), `name`, `status`, `resolvedCandidateId`, `categoryId`, `createdAt` (unixToDate), `updatedAt` (unixToDate)
|
||||
6. **sessions** -- `id` (text PK), `userId`, `expiresAt` (unixToDate)
|
||||
7. **apiKeys** -- `id` (serial), `name`, `keyHash`, `keyPrefix`, `createdAt` (unixToDate)
|
||||
8. **oauthClients** -- `id` (serial), `clientId`, `clientName`, `redirectUris`, `createdAt` (unixToDate)
|
||||
9. **threadCandidates** -- `id` (serial), all fields, `createdAt`/`updatedAt` (unixToDate), `sortOrder` (keep as number)
|
||||
10. **setups** -- `id` (serial), `name`, `createdAt`/`updatedAt` (unixToDate)
|
||||
11. **oauthCodes** -- `id` (serial), all fields, `expiresAt` (unixToDate), `used` (intToBool)
|
||||
12. **oauthTokens** -- `id` (serial), all fields, `expiresAt`/`refreshExpiresAt`/`createdAt` (unixToDate)
|
||||
13. **setupItems** -- `id` (serial), `setupId`, `itemId`, `classification`
|
||||
|
||||
**For each table, use this pattern:**
|
||||
```typescript
|
||||
async function migrateTable(tableName: string, pgTable: any, transform: (row: any) => any) {
|
||||
const rows = sqlite.query(`SELECT * FROM ${tableName}`).all();
|
||||
console.log(` ${tableName}: ${rows.length} rows`);
|
||||
if (rows.length === 0) return;
|
||||
|
||||
for (const row of rows) {
|
||||
await db.insert(pgTable).values(transform(row));
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Sequence reset after all data is migrated:**
|
||||
```typescript
|
||||
async function resetSequences() {
|
||||
const tablesWithSerial = [
|
||||
"categories", "items", "threads", "thread_candidates",
|
||||
"setups", "setup_items", "users", "api_keys",
|
||||
"oauth_clients", "oauth_codes", "oauth_tokens"
|
||||
];
|
||||
|
||||
for (const table of tablesWithSerial) {
|
||||
await sql`SELECT setval(pg_get_serial_sequence('${sql.raw(table)}', 'id'), COALESCE((SELECT MAX(id) FROM ${sql.raw(table)}), 0))`;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Note: Use `db.execute(sql\`...\`)` from drizzle-orm for raw SQL, or use `pg\`...\`` from the postgres.js client directly. The `sql.raw()` helper is needed for dynamic table names.
|
||||
|
||||
**Main function:**
|
||||
```typescript
|
||||
async function main() {
|
||||
const sqlitePath = process.env.SQLITE_PATH || "gearbox.db";
|
||||
const databaseUrl = process.env.DATABASE_URL;
|
||||
|
||||
if (!databaseUrl) {
|
||||
console.error("ERROR: DATABASE_URL environment variable is required");
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
console.log(`Migrating from SQLite (${sqlitePath}) to PostgreSQL...`);
|
||||
|
||||
const sqlite = new Database(sqlitePath, { readonly: true });
|
||||
const pg = postgres(databaseUrl);
|
||||
const db = drizzle(pg, { schema });
|
||||
|
||||
// ... migrate all tables in order ...
|
||||
// ... reset sequences ...
|
||||
|
||||
await pg.end();
|
||||
sqlite.close();
|
||||
|
||||
console.log("Migration complete!");
|
||||
}
|
||||
|
||||
main().catch((err) => {
|
||||
console.error("Migration failed:", err);
|
||||
process.exit(1);
|
||||
});
|
||||
```
|
||||
|
||||
**Error handling per table:** Wrap each table migration in try/catch, log which table failed and which row (by ID if available), then re-throw. This aids debugging partial migrations.
|
||||
|
||||
**Add to package.json scripts:**
|
||||
```json
|
||||
"db:migrate-from-sqlite": "bun run scripts/migrate-sqlite-to-postgres.ts"
|
||||
```
|
||||
</action>
|
||||
<verify>
|
||||
<automated>test -f scripts/migrate-sqlite-to-postgres.ts && grep -q "bun:sqlite" scripts/migrate-sqlite-to-postgres.ts && grep -q "postgres" scripts/migrate-sqlite-to-postgres.ts && grep -q "setval" scripts/migrate-sqlite-to-postgres.ts && grep -q "unixToDate\|unix.*Date\|\\* 1000" scripts/migrate-sqlite-to-postgres.ts && bun run lint 2>&1 | tail -3 && echo "PASS" || echo "FAIL"</automated>
|
||||
</verify>
|
||||
<acceptance_criteria>
|
||||
- scripts/migrate-sqlite-to-postgres.ts exists
|
||||
- File imports from `bun:sqlite` (read-only) and `drizzle-orm/postgres-js` and `postgres`
|
||||
- File imports schema from `../src/db/schema.ts`
|
||||
- File contains a unix-to-Date conversion function (multiplies by 1000)
|
||||
- File contains an integer-to-boolean conversion for `used` field
|
||||
- File migrates all 13 tables in dependency order (categories and users before items and threads, etc.)
|
||||
- File contains `setval` calls to reset serial sequences after migration
|
||||
- File reads `DATABASE_URL` from environment and exits with error if missing
|
||||
- File reads `SQLITE_PATH` from environment with default `"gearbox.db"`
|
||||
- File opens SQLite in readonly mode
|
||||
- package.json contains `"db:migrate-from-sqlite"` script
|
||||
</acceptance_criteria>
|
||||
<done>Migration script reads SQLite, writes to Postgres with type conversions, resets sequences. All IDs and FK relationships preserved per D-06.</done>
|
||||
</task>
|
||||
|
||||
</tasks>
|
||||
|
||||
<verification>
|
||||
- `bun run scripts/migrate-sqlite-to-postgres.ts --help` or similar does not crash on syntax errors (will fail on missing DATABASE_URL, which is expected)
|
||||
- Script contains all 13 table migrations
|
||||
- Script resets sequences for all tables with serial IDs
|
||||
- `bun run lint` passes
|
||||
</verification>
|
||||
|
||||
<success_criteria>
|
||||
One-time migration script exists, handles all type conversions (timestamps, booleans), preserves IDs, resets sequences. Can be run with `DATABASE_URL=... SQLITE_PATH=... bun run scripts/migrate-sqlite-to-postgres.ts`. Lint passes.
|
||||
</success_criteria>
|
||||
|
||||
<output>
|
||||
After completion, create `.planning/phases/14-postgresql-migration/14-05-SUMMARY.md`
|
||||
</output>
|
||||
261
.planning/phases/14-postgresql-migration/14-06-PLAN.md
Normal file
261
.planning/phases/14-postgresql-migration/14-06-PLAN.md
Normal file
@@ -0,0 +1,261 @@
|
||||
---
|
||||
phase: 14-postgresql-migration
|
||||
plan: 06
|
||||
type: execute
|
||||
wave: 3
|
||||
depends_on: [14-01, 14-03, 14-04]
|
||||
files_modified:
|
||||
- tests/services/item.service.test.ts
|
||||
- tests/services/category.service.test.ts
|
||||
- tests/services/thread.service.test.ts
|
||||
- tests/services/setup.service.test.ts
|
||||
- tests/services/auth.service.test.ts
|
||||
- tests/services/oauth.service.test.ts
|
||||
- tests/services/csv.service.test.ts
|
||||
- tests/services/image.service.test.ts
|
||||
- tests/services/totals.test.ts
|
||||
- tests/routes/items.test.ts
|
||||
- tests/routes/categories.test.ts
|
||||
- tests/routes/threads.test.ts
|
||||
- tests/routes/setups.test.ts
|
||||
- tests/routes/auth.test.ts
|
||||
- tests/routes/oauth.test.ts
|
||||
- tests/routes/images.test.ts
|
||||
- tests/routes/params.test.ts
|
||||
- tests/mcp/tools.test.ts
|
||||
autonomous: true
|
||||
requirements: [DB-02, DB-03]
|
||||
must_haves:
|
||||
truths:
|
||||
- "All 18 test files use async createTestDb() in beforeEach"
|
||||
- "All test assertions await async service/route calls"
|
||||
- "bun test tests/ passes with zero failures"
|
||||
- "No test file imports from bun:sqlite or drizzle-orm/bun-sqlite"
|
||||
artifacts:
|
||||
- path: "tests/services/item.service.test.ts"
|
||||
provides: "Async item service tests"
|
||||
contains: "await createTestDb"
|
||||
- path: "tests/routes/items.test.ts"
|
||||
provides: "Async item route tests"
|
||||
contains: "await createTestDb"
|
||||
- path: "tests/mcp/tools.test.ts"
|
||||
provides: "Async MCP tools tests"
|
||||
contains: "await createTestDb"
|
||||
key_links:
|
||||
- from: "tests/**/*.test.ts"
|
||||
to: "tests/helpers/db.ts"
|
||||
via: "import { createTestDb }"
|
||||
pattern: "createTestDb"
|
||||
- from: "tests/services/*.test.ts"
|
||||
to: "src/server/services/*.ts"
|
||||
via: "import service functions"
|
||||
pattern: "from.*services/"
|
||||
---
|
||||
|
||||
<objective>
|
||||
Convert all 18 test files to async: await createTestDb(), await all service/route calls, await all assertions involving DB operations. Run the full test suite to confirm everything passes on PGlite.
|
||||
|
||||
Purpose: This is the final verification that the entire stack works on PostgreSQL. Tests must pass on PGlite (DB-03) and confirm async operations work correctly (DB-02).
|
||||
Output: All tests green. Full `bun test tests/` passes.
|
||||
</objective>
|
||||
|
||||
<execution_context>
|
||||
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
|
||||
@$HOME/.claude/get-shit-done/templates/summary.md
|
||||
</execution_context>
|
||||
|
||||
<context>
|
||||
@.planning/PROJECT.md
|
||||
@.planning/ROADMAP.md
|
||||
@.planning/phases/14-postgresql-migration/14-CONTEXT.md
|
||||
@.planning/phases/14-postgresql-migration/14-RESEARCH.md
|
||||
@.planning/phases/14-postgresql-migration/14-01-SUMMARY.md
|
||||
@.planning/phases/14-postgresql-migration/14-03-SUMMARY.md
|
||||
|
||||
@tests/helpers/db.ts
|
||||
</context>
|
||||
|
||||
<interfaces>
|
||||
<!-- Test helper (from Plan 01): -->
|
||||
<!-- export async function createTestDb() { ... } -->
|
||||
<!-- Returns: PGlite-backed Drizzle instance (same query API, but async) -->
|
||||
|
||||
<!-- Db type issue (Pitfall 8 from research): -->
|
||||
<!-- Production uses PostgresJsDatabase<typeof schema> from drizzle-orm/postgres-js -->
|
||||
<!-- Tests use PgliteDatabase<typeof schema> from drizzle-orm/pglite -->
|
||||
<!-- These types may not be directly compatible for the `Db` type parameter in services -->
|
||||
<!-- Solution: Use `any` cast when passing test db to service functions, OR define a shared type -->
|
||||
<!-- Simplest: `const db = await createTestDb() as any` if type errors occur -->
|
||||
|
||||
Conversion rules for ALL test files:
|
||||
1. `beforeEach(() => { db = createTestDb(); })` -> `beforeEach(async () => { db = await createTestDb(); })`
|
||||
2. Every service call in tests: add `await` (they are now async)
|
||||
3. Every direct DB call in tests (inserts for setup, selects for assertions): add `await`, remove `.all()/.get()/.run()`
|
||||
4. Route tests: if using `app.request()`, those are already async. But ensure the test app factory is also async.
|
||||
5. If `type Db = typeof prodDb` causes type mismatch with PGlite db, use `as any` cast
|
||||
</interfaces>
|
||||
|
||||
<tasks>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 1: Convert all 9 service test files to async</name>
|
||||
<files>tests/services/item.service.test.ts, tests/services/category.service.test.ts, tests/services/thread.service.test.ts, tests/services/setup.service.test.ts, tests/services/auth.service.test.ts, tests/services/oauth.service.test.ts, tests/services/csv.service.test.ts, tests/services/image.service.test.ts, tests/services/totals.test.ts</files>
|
||||
<read_first>tests/services/item.service.test.ts, tests/services/category.service.test.ts, tests/services/thread.service.test.ts, tests/services/setup.service.test.ts, tests/services/auth.service.test.ts, tests/services/oauth.service.test.ts, tests/services/csv.service.test.ts, tests/services/image.service.test.ts, tests/services/totals.test.ts</read_first>
|
||||
<action>
|
||||
For EACH of the 9 service test files, apply these changes:
|
||||
|
||||
**1. Make beforeEach async:**
|
||||
```typescript
|
||||
// BEFORE:
|
||||
let db: any;
|
||||
beforeEach(() => {
|
||||
db = createTestDb();
|
||||
});
|
||||
|
||||
// AFTER:
|
||||
let db: any;
|
||||
beforeEach(async () => {
|
||||
db = await createTestDb();
|
||||
});
|
||||
```
|
||||
|
||||
**2. Add `await` to every service function call in test bodies:**
|
||||
```typescript
|
||||
// BEFORE:
|
||||
const items = getAllItems(db);
|
||||
const item = createItem(db, { name: "Test", categoryId: 1 });
|
||||
|
||||
// AFTER:
|
||||
const items = await getAllItems(db);
|
||||
const item = await createItem(db, { name: "Test", categoryId: 1 });
|
||||
```
|
||||
|
||||
**3. Add `await` to direct DB calls used for test setup/assertions:**
|
||||
```typescript
|
||||
// BEFORE:
|
||||
db.insert(schema.items).values({ ... }).run();
|
||||
const [cat] = db.select().from(schema.categories).all();
|
||||
|
||||
// AFTER:
|
||||
await db.insert(schema.items).values({ ... });
|
||||
const [cat] = await db.select().from(schema.categories);
|
||||
```
|
||||
|
||||
**4. Make test callbacks async if not already:**
|
||||
```typescript
|
||||
// BEFORE:
|
||||
it("should return all items", () => {
|
||||
|
||||
// AFTER:
|
||||
it("should return all items", async () => {
|
||||
```
|
||||
|
||||
**5. Handle Db type compatibility:**
|
||||
If TypeScript complains about passing PGlite db to service functions that expect `PostgresJsDatabase`, use `as any` on the db variable:
|
||||
```typescript
|
||||
let db: any; // Use any to accommodate PGlite/postgres-js type difference
|
||||
```
|
||||
|
||||
**6. OAuth tests -- boolean conversion:**
|
||||
If any OAuth test checks `used === 0` or `used === 1`, change to `used === false` or `used === true`.
|
||||
|
||||
After converting each file, run it individually:
|
||||
```bash
|
||||
bun test tests/services/item.service.test.ts
|
||||
```
|
||||
Fix any issues before moving to the next file.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>bun test tests/services/ 2>&1; [ $? -eq 0 ] && echo "PASS" || echo "FAIL"</automated>
|
||||
</verify>
|
||||
<acceptance_criteria>
|
||||
- Every service test file has `beforeEach(async () => { db = await createTestDb(); })`
|
||||
- Every test callback (`it(...)`) that calls service functions or DB is `async`
|
||||
- No test file contains `.all()`, `.get()`, or `.run()` on db objects
|
||||
- No test file imports from `bun:sqlite` or `drizzle-orm/bun-sqlite`
|
||||
- `bun test tests/services/` exits 0 with all tests passing
|
||||
</acceptance_criteria>
|
||||
<done>All 9 service test files converted to async and passing on PGlite.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 2: Convert all route tests + MCP test to async, run full suite</name>
|
||||
<files>tests/routes/items.test.ts, tests/routes/categories.test.ts, tests/routes/threads.test.ts, tests/routes/setups.test.ts, tests/routes/auth.test.ts, tests/routes/oauth.test.ts, tests/routes/images.test.ts, tests/routes/params.test.ts, tests/mcp/tools.test.ts</files>
|
||||
<read_first>tests/routes/items.test.ts, tests/routes/categories.test.ts, tests/routes/threads.test.ts, tests/routes/setups.test.ts, tests/routes/auth.test.ts, tests/routes/oauth.test.ts, tests/routes/images.test.ts, tests/routes/params.test.ts, tests/mcp/tools.test.ts</read_first>
|
||||
<action>
|
||||
Route tests typically create a test app with a test database injected. The pattern is usually:
|
||||
|
||||
```typescript
|
||||
// Common route test pattern:
|
||||
function createTestApp() {
|
||||
const db = createTestDb();
|
||||
// ... create Hono app with db injected
|
||||
return { app, db };
|
||||
}
|
||||
```
|
||||
|
||||
This must become:
|
||||
```typescript
|
||||
async function createTestApp() {
|
||||
const db = await createTestDb();
|
||||
// ... create Hono app with db injected
|
||||
return { app, db };
|
||||
}
|
||||
```
|
||||
|
||||
**For each of the 8 route test files + 1 MCP test file:**
|
||||
|
||||
1. Make the test app factory `async` and `await createTestDb()`
|
||||
2. Make `beforeEach` async if it calls the factory
|
||||
3. Route tests use `app.request()` which returns a Promise -- these should already be awaited. Verify each test awaits the response.
|
||||
4. If any test does direct DB calls for setup/assertions, apply same async conversion as service tests
|
||||
5. Make all test callbacks async
|
||||
|
||||
**MCP test (tests/mcp/tools.test.ts):**
|
||||
- Same pattern: async createTestDb, await all MCP tool calls
|
||||
- MCP tools internally call services which are now async
|
||||
|
||||
**After all files converted, run the FULL test suite:**
|
||||
```bash
|
||||
bun test tests/
|
||||
```
|
||||
|
||||
This is the gate check. ALL tests must pass. If any test fails:
|
||||
1. Read the error message carefully
|
||||
2. Common issues: missing `await`, `.get()` not removed, type mismatch
|
||||
3. Fix and re-run
|
||||
|
||||
**Also verify no SQLite references remain anywhere in test files:**
|
||||
```bash
|
||||
grep -rn "bun:sqlite\|drizzle-orm/bun-sqlite\|\.all()\|\.get()\|\.run()" tests/
|
||||
```
|
||||
Should return NO matches (except possibly string literals in test descriptions).
|
||||
</action>
|
||||
<verify>
|
||||
<automated>bun test tests/ 2>&1; [ $? -eq 0 ] && echo "PASS" || echo "FAIL"</automated>
|
||||
</verify>
|
||||
<acceptance_criteria>
|
||||
- Every route test file has async `createTestApp` or async `beforeEach` with `await createTestDb()`
|
||||
- Every test callback is `async`
|
||||
- tests/mcp/tools.test.ts uses `await createTestDb()`
|
||||
- `grep -rn "bun:sqlite\|drizzle-orm/bun-sqlite" tests/` returns NO matches
|
||||
- `bun test tests/` exits 0 with ALL tests passing (zero failures)
|
||||
</acceptance_criteria>
|
||||
<done>All 18 test files pass on PGlite. Full test suite green. No SQLite test infrastructure remains.</done>
|
||||
</task>
|
||||
|
||||
</tasks>
|
||||
|
||||
<verification>
|
||||
- `bun test tests/` -- ALL tests pass (exit code 0)
|
||||
- `grep -rn "bun:sqlite\|drizzle-orm/bun-sqlite" tests/` -- NO matches
|
||||
- `grep -rn "\.all()\b" tests/ | grep -v "describe\|it(" ` -- NO matches on DB calls (may appear in test descriptions)
|
||||
</verification>
|
||||
|
||||
<success_criteria>
|
||||
All 18 test files converted to async PGlite. Full test suite (`bun test tests/`) passes with zero failures. No SQLite test infrastructure remains anywhere in the tests/ directory.
|
||||
</success_criteria>
|
||||
|
||||
<output>
|
||||
After completion, create `.planning/phases/14-postgresql-migration/14-06-SUMMARY.md`
|
||||
</output>
|
||||
113
.planning/phases/14-postgresql-migration/14-CONTEXT.md
Normal file
113
.planning/phases/14-postgresql-migration/14-CONTEXT.md
Normal file
@@ -0,0 +1,113 @@
|
||||
# Phase 14: PostgreSQL Migration - Context
|
||||
|
||||
**Gathered:** 2026-04-04
|
||||
**Status:** Ready for planning
|
||||
|
||||
<domain>
|
||||
## Phase Boundary
|
||||
|
||||
Replace SQLite with PostgreSQL as the sole database. Make all database operations async. Establish PGlite-based test infrastructure. Provide a one-time data migration script and Docker Compose for local Postgres development.
|
||||
|
||||
</domain>
|
||||
|
||||
<decisions>
|
||||
## Implementation Decisions
|
||||
|
||||
### Migration Strategy
|
||||
- **D-01:** Clean rewrite of `src/db/schema.ts` using `drizzle-orm/pg-core` (pgTable, serial, text, numeric, timestamp, etc.) — not a conversion of the SQLite schema
|
||||
- **D-02:** Start fresh Postgres migration history in a new directory (e.g., `drizzle-pg/`) — keep existing `drizzle/` SQLite migrations archived for reference
|
||||
- **D-03:** `src/db/index.ts` switches from `bun:sqlite` + `drizzle-orm/bun-sqlite` to `drizzle-orm/node-postgres` (or `drizzle-orm/postgres-js`) with async connection
|
||||
|
||||
### Data Migration Script
|
||||
- **D-04:** Standalone TypeScript script (e.g., `scripts/migrate-sqlite-to-postgres.ts`) that reads from SQLite file and writes to Postgres — not a Drizzle migration
|
||||
- **D-05:** Script handles type conversions: integer timestamps → proper Postgres `timestamp` columns, `real` weight → `numeric` or `double precision`, text → text
|
||||
- **D-06:** Script preserves all IDs and foreign key relationships — no ID remapping
|
||||
|
||||
### Test Infrastructure
|
||||
- **D-07:** `createTestDb()` returns an async PGlite-backed Drizzle instance — same API shape as current, but async
|
||||
- **D-08:** Per-test fresh PGlite instance with migrations applied (matches current in-memory SQLite pattern, avoids test pollution)
|
||||
- **D-09:** All service and route tests updated from sync to async database operations
|
||||
|
||||
### Docker Compose
|
||||
- **D-10:** Separate `docker-compose.dev.yml` for development with Postgres service — keep existing `docker-compose.yml` for production (updated to include Postgres)
|
||||
- **D-11:** PostgreSQL 16 (latest stable)
|
||||
- **D-12:** Environment variable `DATABASE_URL` for Postgres connection string (replaces `DATABASE_PATH` for SQLite)
|
||||
|
||||
### Claude's Discretion
|
||||
- Drizzle Postgres driver choice (`node-postgres` vs `postgres-js`) — pick based on Bun compatibility and async performance
|
||||
- PGlite configuration details (version, extensions)
|
||||
- Column type mapping specifics beyond the ones called out (e.g., whether to use `serial` vs `integer().primaryKey()`)
|
||||
- Migration script error handling and progress reporting
|
||||
- Whether to use `drizzle-orm/pglite` driver or generic pg driver for tests
|
||||
|
||||
</decisions>
|
||||
|
||||
<canonical_refs>
|
||||
## Canonical References
|
||||
|
||||
**Downstream agents MUST read these before planning or implementing.**
|
||||
|
||||
### Database Schema & Config
|
||||
- `src/db/schema.ts` — Current SQLite schema (source of truth for tables/columns to migrate)
|
||||
- `src/db/index.ts` — Current database initialization (bun:sqlite + drizzle)
|
||||
- `drizzle.config.ts` — Current Drizzle Kit config (sqlite dialect)
|
||||
- `drizzle/` — Existing SQLite migration files (10 migrations, reference only)
|
||||
|
||||
### Test Infrastructure
|
||||
- `tests/helpers/db.ts` — Current test database helper (in-memory SQLite, migration application, seed)
|
||||
|
||||
### Services (all need sync → async)
|
||||
- `src/server/services/*.ts` — 9 service files that use synchronous Drizzle operations
|
||||
- `src/server/routes/*.ts` — 9 route files that call services
|
||||
|
||||
### Tests (all need updating)
|
||||
- `tests/services/*.test.ts` — 9 service test files
|
||||
- `tests/routes/*.test.ts` — 8 route test files
|
||||
- `tests/mcp/tools.test.ts` — MCP tools test
|
||||
|
||||
### Docker
|
||||
- `docker-compose.yml` — Current production compose (SQLite volumes, no Postgres)
|
||||
|
||||
</canonical_refs>
|
||||
|
||||
<code_context>
|
||||
## Existing Code Insights
|
||||
|
||||
### Reusable Assets
|
||||
- Drizzle ORM already in use — schema definition pattern transfers directly to pg-core
|
||||
- Service layer architecture with DI (db as first param) — makes swapping the db instance straightforward
|
||||
- Zod schemas in `src/shared/schemas.ts` — validation layer is database-agnostic, no changes needed
|
||||
- TanStack Query hooks — frontend is fully decoupled from database, no changes needed
|
||||
|
||||
### Established Patterns
|
||||
- **Service DI pattern**: All services take `db` as first parameter — this means swapping SQLite for Postgres only requires changing what `db` is, not how services use it
|
||||
- **Sync Drizzle calls**: Current code uses `.run()`, `.get()`, `.all()` synchronously — Postgres requires `.execute()` / await on all queries
|
||||
- **Test pattern**: `createTestDb()` creates isolated DB, applies migrations, seeds — same pattern works with PGlite
|
||||
- **Timestamps as integers**: `{ mode: "timestamp" }` on integer columns — Postgres can use native `timestamp` type
|
||||
|
||||
### Integration Points
|
||||
- `src/db/index.ts` — Single point of database creation (good: only one file to change for connection)
|
||||
- `src/server/index.ts` — Where db is provided to Hono context via middleware
|
||||
- `tests/helpers/db.ts` — Single test DB factory (good: only one file to change for test infra)
|
||||
- `drizzle.config.ts` — Needs dialect change from sqlite to postgresql
|
||||
|
||||
</code_context>
|
||||
|
||||
<specifics>
|
||||
## Specific Ideas
|
||||
|
||||
No specific requirements — open to standard approaches for SQLite-to-Postgres migration with Drizzle ORM.
|
||||
|
||||
</specifics>
|
||||
|
||||
<deferred>
|
||||
## Deferred Ideas
|
||||
|
||||
None — discussion stayed within phase scope
|
||||
|
||||
</deferred>
|
||||
|
||||
---
|
||||
|
||||
*Phase: 14-postgresql-migration*
|
||||
*Context gathered: 2026-04-04*
|
||||
@@ -0,0 +1,90 @@
|
||||
# Phase 14: PostgreSQL Migration - Discussion Log
|
||||
|
||||
> **Audit trail only.** Do not use as input to planning, research, or execution agents.
|
||||
> Decisions are captured in CONTEXT.md — this log preserves the alternatives considered.
|
||||
|
||||
**Date:** 2026-04-04
|
||||
**Phase:** 14-postgresql-migration
|
||||
**Areas discussed:** Migration strategy, Data migration script, Test infrastructure, Docker Compose layout
|
||||
**Mode:** --auto (all decisions auto-selected as recommended defaults)
|
||||
|
||||
---
|
||||
|
||||
## Migration Strategy
|
||||
|
||||
| Option | Description | Selected |
|
||||
|--------|-------------|----------|
|
||||
| Clean schema rewrite | Rewrite schema.ts using drizzle-orm/pg-core with fresh migration history | ✓ |
|
||||
| Convert existing migrations | Transform SQLite migrations to Postgres equivalents | |
|
||||
| Dual-dialect schema | Maintain both SQLite and Postgres schema definitions | |
|
||||
|
||||
**User's choice:** [auto] Clean schema rewrite (recommended default)
|
||||
**Notes:** SQLite and Postgres dialects differ enough (type system, auto-increment vs serial, pragma vs native features) that converting migrations is error-prone. Fresh start is cleaner.
|
||||
|
||||
| Option | Description | Selected |
|
||||
|--------|-------------|----------|
|
||||
| Fresh Postgres migration history | New directory, archive SQLite migrations | ✓ |
|
||||
| Convert SQLite migrations | Rewrite each .sql file for Postgres | |
|
||||
|
||||
**User's choice:** [auto] Fresh Postgres migration history (recommended default)
|
||||
**Notes:** 10 existing SQLite migrations would need manual conversion. Starting fresh avoids dialect translation bugs.
|
||||
|
||||
---
|
||||
|
||||
## Data Migration Script
|
||||
|
||||
| Option | Description | Selected |
|
||||
|--------|-------------|----------|
|
||||
| Standalone TypeScript script | Reads SQLite, writes Postgres, one-time use | ✓ |
|
||||
| Drizzle migration | Built into the migration pipeline | |
|
||||
| SQL dump + import | pg_dump-style approach | |
|
||||
|
||||
**User's choice:** [auto] Standalone TypeScript script (recommended default)
|
||||
**Notes:** One-time operation that doesn't belong in the migration pipeline. Script can handle type conversions explicitly.
|
||||
|
||||
---
|
||||
|
||||
## Test Infrastructure
|
||||
|
||||
| Option | Description | Selected |
|
||||
|--------|-------------|----------|
|
||||
| Per-test PGlite instance | Fresh database per test, migrations applied each time | ✓ |
|
||||
| Shared PGlite with transaction rollback | One instance, wrap each test in a rolled-back transaction | |
|
||||
| Shared PGlite with cleanup | One instance, truncate tables between tests | |
|
||||
|
||||
**User's choice:** [auto] Per-test PGlite instance (recommended default)
|
||||
**Notes:** Matches current in-memory SQLite pattern. Avoids test pollution. PGlite is lightweight enough for per-test instances.
|
||||
|
||||
---
|
||||
|
||||
## Docker Compose Layout
|
||||
|
||||
| Option | Description | Selected |
|
||||
|--------|-------------|----------|
|
||||
| Separate dev compose file | docker-compose.dev.yml with Postgres for development | ✓ |
|
||||
| Single compose with profiles | Use Docker Compose profiles for dev vs prod | |
|
||||
| Extend existing compose | Add Postgres to the single docker-compose.yml | |
|
||||
|
||||
**User's choice:** [auto] Separate dev compose file (recommended default)
|
||||
**Notes:** Separation of concerns. Production compose will also need Postgres eventually but with different configuration.
|
||||
|
||||
| Option | Description | Selected |
|
||||
|--------|-------------|----------|
|
||||
| PostgreSQL 16 | Latest stable release | ✓ |
|
||||
| PostgreSQL 15 | Previous stable | |
|
||||
|
||||
**User's choice:** [auto] PostgreSQL 16 (recommended default)
|
||||
|
||||
---
|
||||
|
||||
## Claude's Discretion
|
||||
|
||||
- Drizzle Postgres driver choice (node-postgres vs postgres-js)
|
||||
- PGlite configuration details
|
||||
- Column type mapping specifics
|
||||
- Migration script error handling
|
||||
- Test driver choice for PGlite
|
||||
|
||||
## Deferred Ideas
|
||||
|
||||
None
|
||||
574
.planning/phases/14-postgresql-migration/14-RESEARCH.md
Normal file
574
.planning/phases/14-postgresql-migration/14-RESEARCH.md
Normal file
@@ -0,0 +1,574 @@
|
||||
# Phase 14: PostgreSQL Migration - Research
|
||||
|
||||
**Researched:** 2026-04-04
|
||||
**Domain:** Database migration (SQLite to PostgreSQL), Drizzle ORM, PGlite testing
|
||||
**Confidence:** HIGH
|
||||
|
||||
## Summary
|
||||
|
||||
This phase replaces the SQLite database with PostgreSQL across the entire stack: schema definitions, database driver, all service/route code (sync to async), test infrastructure (PGlite), data migration script, and Docker Compose for local development.
|
||||
|
||||
The core migration is well-supported by Drizzle ORM, which has first-class drivers for both PostgreSQL (via `postgres` package) and PGlite (for testing). The schema rewrite from `drizzle-orm/sqlite-core` to `drizzle-orm/pg-core` is straightforward -- column type mapping is direct. The bulk of the work is mechanical: adding `await` to ~82 sync `.all()/.get()/.run()` calls across 9 service files, updating 4 transaction usages to async, and updating all 18 test files to use async PGlite-backed databases.
|
||||
|
||||
**Primary recommendation:** Use `postgres` (postgres.js) as the production driver for best Bun compatibility and connection pooling. Use `@electric-sql/pglite` with `drizzle-orm/pglite` for tests. Apply schema in tests via `migrate()` from generated migrations (not `pushSchema`) to match production behavior.
|
||||
|
||||
<user_constraints>
|
||||
## User Constraints (from CONTEXT.md)
|
||||
|
||||
### Locked Decisions
|
||||
- **D-01:** Clean rewrite of `src/db/schema.ts` using `drizzle-orm/pg-core` (pgTable, serial, text, numeric, timestamp, etc.) -- not a conversion of the SQLite schema
|
||||
- **D-02:** Start fresh Postgres migration history in a new directory (e.g., `drizzle-pg/`) -- keep existing `drizzle/` SQLite migrations archived for reference
|
||||
- **D-03:** `src/db/index.ts` switches from `bun:sqlite` + `drizzle-orm/bun-sqlite` to `drizzle-orm/node-postgres` (or `drizzle-orm/postgres-js`) with async connection
|
||||
- **D-04:** Standalone TypeScript script (e.g., `scripts/migrate-sqlite-to-postgres.ts`) that reads from SQLite file and writes to Postgres -- not a Drizzle migration
|
||||
- **D-05:** Script handles type conversions: integer timestamps to proper Postgres `timestamp` columns, `real` weight to `numeric` or `double precision`, text to text
|
||||
- **D-06:** Script preserves all IDs and foreign key relationships -- no ID remapping
|
||||
- **D-07:** `createTestDb()` returns an async PGlite-backed Drizzle instance -- same API shape as current, but async
|
||||
- **D-08:** Per-test fresh PGlite instance with migrations applied (matches current in-memory SQLite pattern, avoids test pollution)
|
||||
- **D-09:** All service and route tests updated from sync to async database operations
|
||||
- **D-10:** Separate `docker-compose.dev.yml` for development with Postgres service -- keep existing `docker-compose.yml` for production (updated to include Postgres)
|
||||
- **D-11:** PostgreSQL 16 (latest stable)
|
||||
- **D-12:** Environment variable `DATABASE_URL` for Postgres connection string (replaces `DATABASE_PATH` for SQLite)
|
||||
|
||||
### Claude's Discretion
|
||||
- Drizzle Postgres driver choice (`node-postgres` vs `postgres-js`) -- pick based on Bun compatibility and async performance
|
||||
- PGlite configuration details (version, extensions)
|
||||
- Column type mapping specifics beyond the ones called out (e.g., whether to use `serial` vs `integer().primaryKey()`)
|
||||
- Migration script error handling and progress reporting
|
||||
- Whether to use `drizzle-orm/pglite` driver or generic pg driver for tests
|
||||
|
||||
### Deferred Ideas (OUT OF SCOPE)
|
||||
None -- discussion stayed within phase scope
|
||||
</user_constraints>
|
||||
|
||||
<phase_requirements>
|
||||
## Phase Requirements
|
||||
|
||||
| ID | Description | Research Support |
|
||||
|----|-------------|------------------|
|
||||
| DB-01 | Application runs on PostgreSQL instead of SQLite | Schema rewrite (pg-core), driver swap (postgres.js), async service layer |
|
||||
| DB-02 | All service functions use async database operations | 82 sync calls across 9 services need `await`; 4 transactions need async conversion |
|
||||
| DB-03 | Test infrastructure uses PGlite instead of bun:sqlite in-memory databases | `@electric-sql/pglite` + `drizzle-orm/pglite` with per-test instances |
|
||||
| DB-04 | Existing SQLite data can be migrated to Postgres via a one-time script | Standalone script reads SQLite via `bun:sqlite`, writes to Postgres with type conversion |
|
||||
| DB-05 | Docker Compose provides Postgres for local development | `docker-compose.dev.yml` with PostgreSQL 16, `docker-compose.yml` updated for production |
|
||||
</phase_requirements>
|
||||
|
||||
## Standard Stack
|
||||
|
||||
### Core
|
||||
| Library | Version | Purpose | Why Standard |
|
||||
|---------|---------|---------|--------------|
|
||||
| drizzle-orm | 0.45.2 | ORM (already installed, update minor) | Already in use; pg-core module provides PostgreSQL schema/query support |
|
||||
| drizzle-kit | 0.31.10 | Migration generation (already installed, update minor) | Already in use; supports `postgresql` dialect for migration generation |
|
||||
| postgres | 3.4.8 | PostgreSQL driver (postgres.js) | Best Bun compatibility, built-in connection pooling, no native bindings needed |
|
||||
| @electric-sql/pglite | 0.4.3 | In-process WASM Postgres for testing | Real Postgres SQL execution without Docker; per-test isolation in milliseconds |
|
||||
|
||||
### Supporting
|
||||
| Library | Version | Purpose | When to Use |
|
||||
|---------|---------|---------|-------------|
|
||||
| bun:sqlite (built-in) | N/A | Read-only in migration script | Only used by data migration script to read existing SQLite data |
|
||||
|
||||
### Alternatives Considered
|
||||
| Instead of | Could Use | Tradeoff |
|
||||
|------------|-----------|----------|
|
||||
| postgres (postgres.js) | pg (node-postgres) | pg requires `@types/pg`, has native binding option but no benefit on Bun; postgres.js has cleaner API |
|
||||
| postgres (postgres.js) | bun:sql (Bun SQL) | Bun SQL has known drizzle-kit compatibility issues (push/migrate don't work); not yet mature enough |
|
||||
| @electric-sql/pglite | Docker Postgres for tests | Docker adds latency, setup complexity; PGlite is zero-config, sub-millisecond startup |
|
||||
|
||||
**Driver recommendation: `postgres` (postgres.js)**
|
||||
- No native bindings (works on Bun without build tools)
|
||||
- Built-in connection pooling
|
||||
- Prepared statements by default
|
||||
- Drizzle ORM has first-class `drizzle-orm/postgres-js` driver
|
||||
- Bun SQL driver was considered but drizzle-kit does not fully support it for push/migrate commands yet
|
||||
|
||||
**Installation:**
|
||||
```bash
|
||||
bun add postgres @electric-sql/pglite
|
||||
bun remove better-sqlite3 @types/better-sqlite3
|
||||
```
|
||||
|
||||
Note: `bun:sqlite` is built-in and does not need to be uninstalled -- it remains available for the migration script. `better-sqlite3` and its types are dev dependencies that can be removed since they are no longer needed.
|
||||
|
||||
## Architecture Patterns
|
||||
|
||||
### Recommended Project Structure
|
||||
```
|
||||
src/db/
|
||||
schema.ts # Rewritten with drizzle-orm/pg-core (pgTable, serial, text, timestamp, etc.)
|
||||
index.ts # postgres.js connection + drizzle initialization
|
||||
migrate.ts # Async migration runner for production startup
|
||||
seed.ts # Async seed function
|
||||
drizzle-pg/ # New PostgreSQL migration directory (D-02)
|
||||
drizzle/ # Archived SQLite migrations (kept for reference)
|
||||
drizzle.config.ts # Updated: dialect "postgresql", out "./drizzle-pg"
|
||||
scripts/
|
||||
migrate-sqlite-to-postgres.ts # One-time data migration script (D-04)
|
||||
tests/helpers/
|
||||
db.ts # Rewritten: async createTestDb() with PGlite
|
||||
docker-compose.dev.yml # New: Postgres for local dev
|
||||
docker-compose.yml # Updated: Postgres for production
|
||||
```
|
||||
|
||||
### Pattern 1: PostgreSQL Schema Definition
|
||||
**What:** Rewrite all tables using `drizzle-orm/pg-core` types
|
||||
**When to use:** The one-time schema rewrite
|
||||
|
||||
```typescript
|
||||
// src/db/schema.ts
|
||||
import { doublePrecision, integer, pgTable, serial, text, timestamp } from "drizzle-orm/pg-core";
|
||||
|
||||
export const categories = pgTable("categories", {
|
||||
id: serial("id").primaryKey(),
|
||||
name: text("name").notNull().unique(),
|
||||
icon: text("icon").notNull().default("package"),
|
||||
createdAt: timestamp("created_at").notNull().defaultNow(),
|
||||
});
|
||||
|
||||
export const items = pgTable("items", {
|
||||
id: serial("id").primaryKey(),
|
||||
name: text("name").notNull(),
|
||||
weightGrams: doublePrecision("weight_grams"),
|
||||
priceCents: integer("price_cents"),
|
||||
categoryId: integer("category_id").notNull().references(() => categories.id),
|
||||
notes: text("notes"),
|
||||
productUrl: text("product_url"),
|
||||
imageFilename: text("image_filename"),
|
||||
imageSourceUrl: text("image_source_url"),
|
||||
quantity: integer("quantity").notNull().default(1),
|
||||
createdAt: timestamp("created_at").notNull().defaultNow(),
|
||||
updatedAt: timestamp("updated_at").notNull().defaultNow(),
|
||||
});
|
||||
```
|
||||
|
||||
### Pattern 2: Async Database Connection
|
||||
**What:** Production database initialization with postgres.js
|
||||
**When to use:** `src/db/index.ts`
|
||||
|
||||
```typescript
|
||||
// src/db/index.ts
|
||||
import { drizzle } from "drizzle-orm/postgres-js";
|
||||
import postgres from "postgres";
|
||||
import * as schema from "./schema.ts";
|
||||
|
||||
const queryClient = postgres(process.env.DATABASE_URL!);
|
||||
export const db = drizzle(queryClient, { schema });
|
||||
```
|
||||
|
||||
### Pattern 3: Async Service Functions
|
||||
**What:** Convert sync Drizzle calls to async with await
|
||||
**When to use:** All 9 service files
|
||||
|
||||
```typescript
|
||||
// BEFORE (SQLite sync):
|
||||
export function getAllItems(db: Db = prodDb) {
|
||||
return db.select().from(items).innerJoin(categories, eq(items.categoryId, categories.id)).all();
|
||||
}
|
||||
|
||||
// AFTER (PostgreSQL async):
|
||||
export async function getAllItems(db: Db = prodDb) {
|
||||
return await db.select().from(items).innerJoin(categories, eq(items.categoryId, categories.id));
|
||||
}
|
||||
```
|
||||
|
||||
Key differences:
|
||||
- `.all()` is removed -- Postgres driver returns arrays directly from `await`
|
||||
- `.get()` is replaced with indexing: `const [result] = await db.select()...` or using `.limit(1)` then `[0]`
|
||||
- `.run()` is removed -- `await db.delete()...` / `await db.insert()...` is sufficient
|
||||
- `.returning().get()` becomes `const [result] = await db.insert()...returning()`
|
||||
- `db.transaction(() => { ... })` becomes `await db.transaction(async (tx) => { ... })` with await inside
|
||||
|
||||
### Pattern 4: PGlite Test Database
|
||||
**What:** Per-test Postgres instance using PGlite
|
||||
**When to use:** `tests/helpers/db.ts`
|
||||
|
||||
```typescript
|
||||
// tests/helpers/db.ts
|
||||
import { drizzle } from "drizzle-orm/pglite";
|
||||
import { migrate } from "drizzle-orm/pglite/migrator";
|
||||
import * as schema from "../../src/db/schema.ts";
|
||||
|
||||
export async function createTestDb() {
|
||||
const db = drizzle({ schema });
|
||||
|
||||
// Apply migrations from the new PostgreSQL migration directory
|
||||
await migrate(db, { migrationsFolder: "./drizzle-pg" });
|
||||
|
||||
// Seed default category
|
||||
await db.insert(schema.categories).values({ name: "Uncategorized", icon: "package" });
|
||||
|
||||
return db;
|
||||
}
|
||||
```
|
||||
|
||||
### Pattern 5: Async Transaction
|
||||
**What:** Convert sync transactions to async
|
||||
**When to use:** 4 transaction sites (category delete, setup update, thread resolve/unresolve)
|
||||
|
||||
```typescript
|
||||
// BEFORE (SQLite sync):
|
||||
db.transaction(() => {
|
||||
db.update(items).set({ categoryId: 1 }).where(eq(items.categoryId, id)).run();
|
||||
db.delete(categories).where(eq(categories.id, id)).run();
|
||||
});
|
||||
|
||||
// AFTER (PostgreSQL async):
|
||||
await db.transaction(async (tx) => {
|
||||
await tx.update(items).set({ categoryId: 1 }).where(eq(items.categoryId, id));
|
||||
await tx.delete(categories).where(eq(categories.id, id));
|
||||
});
|
||||
```
|
||||
|
||||
### Pattern 6: Drizzle Config for PostgreSQL
|
||||
**What:** Updated drizzle.config.ts
|
||||
**When to use:** One-time config update
|
||||
|
||||
```typescript
|
||||
// drizzle.config.ts
|
||||
import { defineConfig } from "drizzle-kit";
|
||||
|
||||
export default defineConfig({
|
||||
out: "./drizzle-pg",
|
||||
schema: "./src/db/schema.ts",
|
||||
dialect: "postgresql",
|
||||
dbCredentials: {
|
||||
url: process.env.DATABASE_URL || "postgresql://gearbox:gearbox@localhost:5432/gearbox",
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
### Anti-Patterns to Avoid
|
||||
- **Mixing sync and async:** Do not leave any `.all()`, `.get()`, `.run()` calls -- they are SQLite-only methods
|
||||
- **Forgetting await:** Every database call must be awaited; missing awaits will return Promise objects instead of data
|
||||
- **Using `pushSchema` for tests:** While faster, `pushSchema` from `drizzle-kit/api` does not match production migration behavior -- use `migrate()` to catch migration issues early
|
||||
- **Integer timestamps in Postgres:** Do not carry over `integer("col", { mode: "timestamp" })` -- use native `timestamp()` type
|
||||
- **Keeping `bun:sqlite` imports in production code:** Only the migration script should import `bun:sqlite`
|
||||
|
||||
## Don't Hand-Roll
|
||||
|
||||
| Problem | Don't Build | Use Instead | Why |
|
||||
|---------|-------------|-------------|-----|
|
||||
| Connection pooling | Custom pool manager | `postgres` built-in pooling | Handles connection limits, idle timeout, reconnection |
|
||||
| In-memory test DB | Docker Postgres containers | PGlite | Zero setup, sub-ms startup, real Postgres SQL |
|
||||
| Schema migrations | Manual SQL files | `drizzle-kit generate` | Generates correct DDL from schema diff |
|
||||
| Data type conversion | Manual column-by-column casting | Drizzle schema + postgres driver auto-coercion | Driver handles JS Date <-> Postgres timestamp, number <-> integer |
|
||||
|
||||
**Key insight:** Drizzle ORM abstracts the SQLite/PostgreSQL differences at the query builder level. The schema definition and driver are the only things that change -- service query logic (select, where, join, insert, etc.) stays identical except for removing sync-only methods.
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
### Pitfall 1: Missing Await on Database Calls
|
||||
**What goes wrong:** Route handlers return `Promise<Item>` instead of `Item`, leading to empty/broken JSON responses
|
||||
**Why it happens:** Mechanical conversion misses an `await` in a handler that was previously sync
|
||||
**How to avoid:** Make route handlers `async` if not already; TypeScript will flag return type mismatches if return types are annotated
|
||||
**Warning signs:** Tests pass but return `{}` or undefined fields; API returns `{}`
|
||||
|
||||
### Pitfall 2: `.get()` Does Not Exist on PostgreSQL Drizzle
|
||||
**What goes wrong:** Runtime error: `.get is not a function`
|
||||
**Why it happens:** `.get()` is a SQLite-only convenience method that returns a single row
|
||||
**How to avoid:** Replace `.get()` with array destructuring: `const [row] = await db.select()...`; replace `.returning().get()` with `const [row] = await db.insert()...returning()`
|
||||
**Warning signs:** TypeScript type errors if using strict mode
|
||||
|
||||
### Pitfall 3: `serial` Auto-Increment Behavior in Postgres
|
||||
**What goes wrong:** Data migration script inserts rows with explicit IDs but the `serial` sequence is not advanced, causing conflicts on next insert
|
||||
**Why it happens:** PostgreSQL `serial` is backed by a sequence that is only auto-incremented on default inserts -- explicit ID inserts do not update the sequence
|
||||
**How to avoid:** After data migration, reset sequences: `SELECT setval('table_id_seq', (SELECT MAX(id) FROM table))`
|
||||
**Warning signs:** Duplicate key errors after migration when creating new records
|
||||
|
||||
### Pitfall 4: Boolean Columns (OAuth `used` Field)
|
||||
**What goes wrong:** SQLite uses `integer` for boolean (`0`/`1`); Postgres has native `boolean` type
|
||||
**Why it happens:** Direct schema port without type adjustment
|
||||
**How to avoid:** Use `boolean("used").notNull().default(false)` in pg-core schema; migration script must convert `0/1` to `false/true`
|
||||
**Warning signs:** Type errors in OAuth code that checks `=== 0` or `=== 1`
|
||||
|
||||
### Pitfall 5: Transaction Callback Must Be Async
|
||||
**What goes wrong:** Transaction body runs sync but database calls inside return unresolved promises
|
||||
**Why it happens:** Forgetting to make the transaction callback `async` and `await` internal operations
|
||||
**How to avoid:** `await db.transaction(async (tx) => { await tx.update()... })`
|
||||
**Warning signs:** Empty/partial data writes, no errors thrown
|
||||
|
||||
### Pitfall 6: `createdAt` Default Function Mismatch
|
||||
**What goes wrong:** `$defaultFn(() => new Date())` in SQLite schema is a JS-side default; Postgres `defaultNow()` is SQL-side
|
||||
**Why it happens:** Different default mechanisms
|
||||
**How to avoid:** Use `.defaultNow()` for all timestamp columns in pg-core schema (server-side default is more reliable)
|
||||
**Warning signs:** Null timestamps when inserting without explicit values
|
||||
|
||||
### Pitfall 7: Test `createTestDb()` Becomes Async
|
||||
**What goes wrong:** All `beforeEach` blocks that call `createTestDb()` break
|
||||
**Why it happens:** `createTestDb()` returns a Promise instead of a Drizzle instance
|
||||
**How to avoid:** `beforeEach(async () => { db = await createTestDb(); })` in all 18 test files
|
||||
**Warning signs:** `db.select is not a function` errors in every test
|
||||
|
||||
### Pitfall 8: `Db` Type Changes
|
||||
**What goes wrong:** `type Db = typeof prodDb` in services no longer matches PGlite-created instances in tests
|
||||
**Why it happens:** `drizzle-orm/postgres-js` and `drizzle-orm/pglite` return different Drizzle instance types
|
||||
**How to avoid:** Use a shared type or use the generic `PostgresJsDatabase<typeof schema>` type that both drivers satisfy. Alternatively, use `ReturnType<typeof drizzle>` from pglite driver which is compatible.
|
||||
**Warning signs:** TypeScript errors when passing test DB to service functions
|
||||
|
||||
## Code Examples
|
||||
|
||||
### Data Migration Script Structure
|
||||
```typescript
|
||||
// scripts/migrate-sqlite-to-postgres.ts
|
||||
import { Database } from "bun:sqlite";
|
||||
import { drizzle } from "drizzle-orm/postgres-js";
|
||||
import postgres from "postgres";
|
||||
import * as schema from "../src/db/schema.ts";
|
||||
|
||||
const sqlite = new Database(process.env.SQLITE_PATH || "gearbox.db");
|
||||
const pg = postgres(process.env.DATABASE_URL!);
|
||||
const db = drizzle(pg, { schema });
|
||||
|
||||
async function migrateTable<T>(
|
||||
tableName: string,
|
||||
pgTable: any,
|
||||
transform: (row: any) => T
|
||||
) {
|
||||
const rows = sqlite.query(`SELECT * FROM ${tableName}`).all();
|
||||
console.log(`Migrating ${rows.length} ${tableName}...`);
|
||||
|
||||
if (rows.length === 0) return;
|
||||
|
||||
for (const row of rows) {
|
||||
await db.insert(pgTable).values(transform(row as any));
|
||||
}
|
||||
}
|
||||
|
||||
async function resetSequences() {
|
||||
const tables = ["categories", "items", "threads", "thread_candidates",
|
||||
"setups", "setup_items", "users", "api_keys",
|
||||
"oauth_clients", "oauth_codes", "oauth_tokens"];
|
||||
for (const table of tables) {
|
||||
await pg`SELECT setval('${pg(table)}_id_seq', COALESCE((SELECT MAX(id) FROM ${pg(table)}), 0))`;
|
||||
}
|
||||
}
|
||||
|
||||
async function main() {
|
||||
// Migrate tables in dependency order (parents before children)
|
||||
// 1. categories, users, settings
|
||||
// 2. items, threads, sessions, api_keys, oauth_clients
|
||||
// 3. thread_candidates, setups
|
||||
// 4. setup_items
|
||||
// Convert: unix timestamps -> Date objects, integer booleans -> booleans
|
||||
|
||||
await resetSequences();
|
||||
await pg.end();
|
||||
sqlite.close();
|
||||
console.log("Migration complete!");
|
||||
}
|
||||
|
||||
main().catch(console.error);
|
||||
```
|
||||
|
||||
### Docker Compose Development
|
||||
```yaml
|
||||
# docker-compose.dev.yml
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:16-alpine
|
||||
environment:
|
||||
POSTGRES_USER: gearbox
|
||||
POSTGRES_PASSWORD: gearbox
|
||||
POSTGRES_DB: gearbox
|
||||
ports:
|
||||
- "5432:5432"
|
||||
volumes:
|
||||
- pgdata-dev:/var/lib/postgresql/data
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U gearbox"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 5
|
||||
|
||||
volumes:
|
||||
pgdata-dev:
|
||||
```
|
||||
|
||||
### Docker Compose Production (updated)
|
||||
```yaml
|
||||
# docker-compose.yml
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:16-alpine
|
||||
environment:
|
||||
POSTGRES_USER: gearbox
|
||||
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
|
||||
POSTGRES_DB: gearbox
|
||||
volumes:
|
||||
- pgdata:/var/lib/postgresql/data
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U gearbox"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
|
||||
app:
|
||||
image: gearbox:latest
|
||||
environment:
|
||||
DATABASE_URL: postgresql://gearbox:${POSTGRES_PASSWORD}@postgres:5432/gearbox
|
||||
GEARBOX_URL: ${GEARBOX_URL}
|
||||
ports:
|
||||
- "3000:3000"
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
volumes:
|
||||
- uploads:/app/uploads
|
||||
|
||||
volumes:
|
||||
pgdata:
|
||||
uploads:
|
||||
```
|
||||
|
||||
### Updated Migration Runner
|
||||
```typescript
|
||||
// src/db/migrate.ts
|
||||
import { drizzle } from "drizzle-orm/postgres-js";
|
||||
import { migrate } from "drizzle-orm/postgres-js/migrator";
|
||||
import postgres from "postgres";
|
||||
|
||||
const migrationClient = postgres(process.env.DATABASE_URL!, { max: 1 });
|
||||
const db = drizzle(migrationClient);
|
||||
|
||||
await migrate(db, { migrationsFolder: "./drizzle-pg" });
|
||||
await migrationClient.end();
|
||||
|
||||
console.log("Migrations applied successfully");
|
||||
```
|
||||
|
||||
## Column Type Mapping
|
||||
|
||||
| SQLite Column | pg-core Column | Notes |
|
||||
|---------------|----------------|-------|
|
||||
| `integer("id").primaryKey({ autoIncrement: true })` | `serial("id").primaryKey()` | `serial` = auto-incrementing 4-byte int |
|
||||
| `text("name")` | `text("name")` | Identical |
|
||||
| `real("weight_grams")` | `doublePrecision("weight_grams")` | 8-byte float, matches SQLite `real` precision |
|
||||
| `integer("price_cents")` | `integer("price_cents")` | Identical |
|
||||
| `integer("col", { mode: "timestamp" })` | `timestamp("col")` | Native Postgres timestamp; Drizzle returns JS Date |
|
||||
| `integer("used").default(0)` | `boolean("used").default(false)` | Proper boolean type |
|
||||
| `real("sort_order")` | `doublePrecision("sort_order")` | Or `real()` (4-byte) -- either works |
|
||||
| `text("id").primaryKey()` (sessions) | `text("id").primaryKey()` | Identical |
|
||||
| `text("key").primaryKey()` (settings) | `text("key").primaryKey()` | Identical |
|
||||
|
||||
## State of the Art
|
||||
|
||||
| Old Approach | Current Approach | When Changed | Impact |
|
||||
|--------------|------------------|--------------|--------|
|
||||
| `bun:sqlite` sync driver | `postgres` (postgres.js) async driver | This migration | All DB calls become async |
|
||||
| `drizzle-orm/bun-sqlite` | `drizzle-orm/postgres-js` | This migration | Driver swap in one file |
|
||||
| In-memory SQLite for tests | PGlite WASM Postgres for tests | This migration | Tests run real Postgres SQL |
|
||||
| `drizzle-orm/bun-sql` (Bun native) | `postgres` (postgres.js) | N/A | Bun SQL has drizzle-kit incompatibilities; postgres.js is mature |
|
||||
|
||||
## Scope of Change
|
||||
|
||||
Summary of files that need modification:
|
||||
|
||||
| Category | Files | Change Type |
|
||||
|----------|-------|-------------|
|
||||
| Schema | `src/db/schema.ts` | Full rewrite (sqlite-core to pg-core) |
|
||||
| DB config | `src/db/index.ts` | Full rewrite (bun:sqlite to postgres.js) |
|
||||
| Migrations | `src/db/migrate.ts` | Full rewrite (async, postgres migrator) |
|
||||
| Seed | `src/db/seed.ts` | Async conversion |
|
||||
| Drizzle config | `drizzle.config.ts` | Dialect + output path change |
|
||||
| Services | 9 files in `src/server/services/` | Add async/await to all DB calls (~82 call sites) |
|
||||
| Routes | 9 files in `src/server/routes/` | Add await to service calls, make handlers async |
|
||||
| Server entry | `src/server/index.ts` | Async seed call |
|
||||
| Test helper | `tests/helpers/db.ts` | Full rewrite (PGlite) |
|
||||
| Service tests | 9 files in `tests/services/` | Async beforeEach + await all assertions |
|
||||
| Route tests | 8 files in `tests/routes/` | Async createTestApp + await |
|
||||
| MCP tests | `tests/mcp/tools.test.ts` | Async test DB |
|
||||
| Docker | `docker-compose.dev.yml` (new), `docker-compose.yml` (new) | Postgres service definitions |
|
||||
| Dockerfile | `Dockerfile` | Update: copy `drizzle-pg/`, remove SQLite-specific steps |
|
||||
| Migration script | `scripts/migrate-sqlite-to-postgres.ts` (new) | Data migration |
|
||||
| Package.json | `package.json` | Add `postgres`, `@electric-sql/pglite`; remove `better-sqlite3` |
|
||||
|
||||
**Total: ~40 files touched, ~2 new files created**
|
||||
|
||||
## Validation Architecture
|
||||
|
||||
### Test Framework
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| Framework | Bun test runner (built-in) |
|
||||
| Config file | None (uses bun defaults) |
|
||||
| Quick run command | `bun test tests/services/item.service.test.ts` |
|
||||
| Full suite command | `bun test tests/` |
|
||||
|
||||
### Phase Requirements to Test Map
|
||||
| Req ID | Behavior | Test Type | Automated Command | File Exists? |
|
||||
|--------|----------|-----------|-------------------|-------------|
|
||||
| DB-01 | App runs on PostgreSQL | integration | `bun test tests/` (all tests use PGlite) | Existing (updated) |
|
||||
| DB-02 | Async database operations | unit | `bun test tests/services/` | Existing (updated) |
|
||||
| DB-03 | PGlite test infrastructure | unit | `bun test tests/services/item.service.test.ts -x` | Existing (updated) |
|
||||
| DB-04 | SQLite data migration script | integration | `bun run scripts/migrate-sqlite-to-postgres.ts` | New (Wave 0) |
|
||||
| DB-05 | Docker Compose Postgres | smoke | `docker compose -f docker-compose.dev.yml up -d && bun test tests/` | Manual verification |
|
||||
|
||||
### Sampling Rate
|
||||
- **Per task commit:** `bun test tests/services/item.service.test.ts -x` (fast single-file check)
|
||||
- **Per wave merge:** `bun test tests/` (full suite)
|
||||
- **Phase gate:** Full suite green + manual Docker Compose smoke test
|
||||
|
||||
### Wave 0 Gaps
|
||||
- [ ] `tests/helpers/db.ts` -- must be rewritten to PGlite before any other tests can run
|
||||
- [ ] Migration files in `drizzle-pg/` -- must be generated before test helper can apply them
|
||||
- [ ] `scripts/migrate-sqlite-to-postgres.ts` -- new file, needs at least a basic test or manual verification plan
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. **PGlite + Bun test runner performance**
|
||||
- What we know: PGlite works well with Vitest; Bun test runner is compatible
|
||||
- What's unclear: Whether Bun's test runner parallel mode causes issues with PGlite WASM initialization
|
||||
- Recommendation: Start with sequential tests; if slow, investigate parallelization
|
||||
|
||||
2. **`Db` type compatibility between postgres.js and PGlite drivers**
|
||||
- What we know: Both return Drizzle instances but with different generic type parameters
|
||||
- What's unclear: Whether the types are structurally compatible without explicit casting
|
||||
- Recommendation: Define a shared `AppDb` type alias; if types diverge, use a minimal interface or `any` for the DI parameter with runtime compatibility
|
||||
|
||||
3. **Sequence reset in migration script**
|
||||
- What we know: Explicit ID inserts do not advance Postgres sequences
|
||||
- What's unclear: Exact syntax for `setval` with dynamic table names via postgres.js
|
||||
- Recommendation: Use raw SQL via `postgres.unsafe()` or `db.execute(sql\`...\`)` for sequence resets
|
||||
|
||||
## Environment Availability
|
||||
|
||||
| Dependency | Required By | Available | Version | Fallback |
|
||||
|------------|------------|-----------|---------|----------|
|
||||
| Docker | Docker Compose dev/prod | Yes | 29.0.0 | -- |
|
||||
| Docker Compose | Local Postgres service | Yes | 2.40.3 | -- |
|
||||
| Bun | Runtime | Yes | 1.3.10 | -- |
|
||||
| PostgreSQL (via Docker) | DB-01, DB-05 | Via Docker | 16-alpine (to pull) | -- |
|
||||
| psql CLI | Debug/manual verification | No | -- | Use Docker exec or skip |
|
||||
|
||||
**Missing dependencies with no fallback:** None
|
||||
|
||||
**Missing dependencies with fallback:**
|
||||
- psql CLI not installed locally -- use `docker exec` into Postgres container for manual queries
|
||||
|
||||
## Sources
|
||||
|
||||
### Primary (HIGH confidence)
|
||||
- [Drizzle ORM PGlite docs](https://orm.drizzle.team/docs/connect-pglite) - Connection setup, migration API
|
||||
- [Drizzle ORM PostgreSQL docs](https://orm.drizzle.team/docs/get-started-postgresql) - postgres.js and node-postgres driver setup
|
||||
- [Drizzle ORM pg-core column types](https://orm.drizzle.team/docs/column-types/pg) - Column type definitions
|
||||
- [Drizzle ORM migrations](https://orm.drizzle.team/docs/migrations) - Programmatic migration execution
|
||||
- [Drizzle ORM Bun SQL](https://orm.drizzle.team/docs/connect-bun-sql) - Bun SQL driver (evaluated, not recommended)
|
||||
- Project codebase: `src/db/schema.ts`, `src/db/index.ts`, `tests/helpers/db.ts`, all service files
|
||||
|
||||
### Secondary (MEDIUM confidence)
|
||||
- [Bun + PostgreSQL compatibility](https://github.com/oven-sh/bun/issues/6555) - Historical postgres.js issues (resolved)
|
||||
- [drizzle-kit Bun SQL issue #4122](https://github.com/drizzle-team/drizzle-orm/issues/4122) - drizzle-kit push incompatibility with Bun SQL
|
||||
- [npm registry](https://www.npmjs.com) - Current package versions verified 2026-04-04
|
||||
|
||||
### Tertiary (LOW confidence)
|
||||
- [PGlite + Drizzle testing patterns](https://dev.to/benjamindaniel/how-to-test-your-nodejs-postgres-app-using-drizzle-pglite-4fb3) - Community patterns (Vitest-focused, may need adaptation for Bun test runner)
|
||||
|
||||
## Metadata
|
||||
|
||||
**Confidence breakdown:**
|
||||
- Standard stack: HIGH - Drizzle ORM pg-core and postgres.js are mature, well-documented, verified against official docs
|
||||
- Architecture: HIGH - Schema mapping is direct; async conversion is mechanical; DI pattern makes driver swap clean
|
||||
- Pitfalls: HIGH - Based on known SQLite-to-Postgres differences and verified Drizzle API differences
|
||||
- Testing (PGlite + Bun): MEDIUM - PGlite is well-documented with Vitest; Bun test runner compatibility is inferred but not directly verified
|
||||
|
||||
**Research date:** 2026-04-04
|
||||
**Valid until:** 2026-05-04 (stable domain, Drizzle ORM and PGlite are mature)
|
||||
78
.planning/phases/14-postgresql-migration/14-VALIDATION.md
Normal file
78
.planning/phases/14-postgresql-migration/14-VALIDATION.md
Normal file
@@ -0,0 +1,78 @@
|
||||
---
|
||||
phase: 14
|
||||
slug: postgresql-migration
|
||||
status: draft
|
||||
nyquist_compliant: false
|
||||
wave_0_complete: false
|
||||
created: 2026-04-04
|
||||
---
|
||||
|
||||
# Phase 14 — Validation Strategy
|
||||
|
||||
> Per-phase validation contract for feedback sampling during execution.
|
||||
|
||||
---
|
||||
|
||||
## Test Infrastructure
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| **Framework** | bun test |
|
||||
| **Config file** | none — uses bun built-in test runner |
|
||||
| **Quick run command** | `bun test tests/` |
|
||||
| **Full suite command** | `bun test tests/` |
|
||||
| **Estimated runtime** | ~10 seconds |
|
||||
|
||||
---
|
||||
|
||||
## Sampling Rate
|
||||
|
||||
- **After every task commit:** Run `bun test tests/`
|
||||
- **After every plan wave:** Run `bun test tests/`
|
||||
- **Before `/gsd:verify-work`:** Full suite must be green
|
||||
- **Max feedback latency:** 10 seconds
|
||||
|
||||
---
|
||||
|
||||
## Per-Task Verification Map
|
||||
|
||||
| Task ID | Plan | Wave | Requirement | Test Type | Automated Command | File Exists | Status |
|
||||
|---------|------|------|-------------|-----------|-------------------|-------------|--------|
|
||||
| TBD | TBD | TBD | DB-01 | integration | `bun test tests/` | ❌ W0 | ⬜ pending |
|
||||
| TBD | TBD | TBD | DB-02 | integration | `bun test tests/` | ❌ W0 | ⬜ pending |
|
||||
| TBD | TBD | TBD | DB-03 | integration | `bun test tests/` | ❌ W0 | ⬜ pending |
|
||||
| TBD | TBD | TBD | DB-04 | integration | `bun test tests/` | ❌ W0 | ⬜ pending |
|
||||
| TBD | TBD | TBD | DB-05 | smoke | `docker compose -f docker-compose.dev.yml up -d` | ❌ W0 | ⬜ pending |
|
||||
|
||||
*Status: <20><> pending · ✅ green · ❌ red · ⚠️ flaky*
|
||||
|
||||
---
|
||||
|
||||
## Wave 0 Requirements
|
||||
|
||||
- [ ] `tests/helpers/db.ts` — Rewrite to use PGlite instead of bun:sqlite
|
||||
- [ ] Existing test files updated from sync to async patterns
|
||||
|
||||
*Test infrastructure exists but needs migration from SQLite to PGlite.*
|
||||
|
||||
---
|
||||
|
||||
## Manual-Only Verifications
|
||||
|
||||
| Behavior | Requirement | Why Manual | Test Instructions |
|
||||
|----------|-------------|------------|-------------------|
|
||||
| SQLite data migration preserves all records | DB-04 | One-time script, not automatable in CI | Run migration script against test SQLite DB, verify row counts match |
|
||||
| Docker Compose starts Postgres | DB-05 | Requires Docker runtime | Run `docker compose -f docker-compose.dev.yml up -d`, verify `pg_isready` |
|
||||
|
||||
---
|
||||
|
||||
## Validation Sign-Off
|
||||
|
||||
- [ ] All tasks have `<automated>` verify or Wave 0 dependencies
|
||||
- [ ] Sampling continuity: no 3 consecutive tasks without automated verify
|
||||
- [ ] Wave 0 covers all MISSING references
|
||||
- [ ] No watch-mode flags
|
||||
- [ ] Feedback latency < 10s
|
||||
- [ ] `nyquist_compliant: true` set in frontmatter
|
||||
|
||||
**Approval:** pending
|
||||
Reference in New Issue
Block a user