GearBox/.planning/phases/14-postgresql-migration/14-05-PLAN.md

---
phase: 14-postgresql-migration
plan: 05
type: execute
wave: 2
depends_on: [14-01]
files_modified:
  - scripts/migrate-sqlite-to-postgres.ts
autonomous: true
requirements: [DB-04]
must_haves:
  truths:
    - "Script reads all data from SQLite file and writes it to PostgreSQL"
    - "Integer timestamps are converted to Date objects for Postgres timestamp columns"
    - "Boolean integers (0/1) are converted to true/false for Postgres boolean columns"
    - "All IDs and foreign key relationships are preserved"
    - "Serial sequences are reset after data migration to avoid duplicate key errors"
  artifacts:
    - path: "scripts/migrate-sqlite-to-postgres.ts"
      provides: "One-time SQLite to Postgres data migration"
      contains: "migrate-sqlite-to-postgres"
  key_links:
    - from: "scripts/migrate-sqlite-to-postgres.ts"
      to: "src/db/schema.ts"
      via: "import table definitions for typed inserts"
      pattern: "import.*schema"
---

<objective>
Create the one-time SQLite-to-PostgreSQL data migration script that reads from an existing SQLite database and writes all data into PostgreSQL with proper type conversions.

Purpose: Existing users need to migrate their data from SQLite to Postgres without data loss (DB-04). This is a standalone script run once during the upgrade.
Output: scripts/migrate-sqlite-to-postgres.ts
</objective>

<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/phases/14-postgresql-migration/14-CONTEXT.md
@.planning/phases/14-postgresql-migration/14-RESEARCH.md
@.planning/phases/14-postgresql-migration/14-01-SUMMARY.md

@src/db/schema.ts
</context>

<interfaces>
<!-- SQLite schema (current production data format): -->
<!-- - Timestamps: stored as unix epoch integers (seconds since 1970) -->
<!-- - Booleans: stored as integers (0 = false, 1 = true), only oauthCodes.used -->
<!-- - Weights: stored as real (float) -->
<!-- - IDs: auto-increment integers -->
<!-- - settings: key (text PK) + value (text) -->
<!-- - sessions: id (text PK) + userId (int) + expiresAt (int timestamp) -->

<!-- PostgreSQL schema (target format after Plan 01): -->
<!-- - Timestamps: native timestamp type (JS Date objects) -->
<!-- - Booleans: native boolean type -->
<!-- - Weights: doublePrecision -->
<!-- - IDs: serial (auto-increment with sequence) -->

Tables in dependency order:
1. categories, users, settings (no foreign keys to other app tables)
2. items, threads, sessions, apiKeys, oauthClients (FK to categories/users)
3. threadCandidates, setups, oauthCodes, oauthTokens (FK to threads/etc)
4. setupItems (FK to setups + items)
</interfaces>

<tasks>

<task type="auto">
  <name>Task 1: Create SQLite-to-Postgres migration script</name>
  <files>scripts/migrate-sqlite-to-postgres.ts</files>
  <read_first>src/db/schema.ts</read_first>
  <action>
Create `scripts/migrate-sqlite-to-postgres.ts` per D-04, D-05, D-06.

```typescript
// scripts/migrate-sqlite-to-postgres.ts
import { Database } from "bun:sqlite";
import { drizzle } from "drizzle-orm/postgres-js";
import postgres from "postgres";
import * as schema from "../src/db/schema.ts";
```

**Environment variables:**
- `SQLITE_PATH` -- path to SQLite database file (default: `"gearbox.db"`)
- `DATABASE_URL` -- PostgreSQL connection string (required)

**Structure:**
1. Open SQLite database read-only
2. Connect to PostgreSQL via postgres.js + drizzle
3. Migrate tables in dependency order (parents before children)
4. Reset all serial sequences after migration
5. Close both connections
6. Print summary

**Type conversion functions:**
```typescript
function unixToDate(unix: number | null): Date | null {
  if (unix === null || unix === undefined) return null;
  return new Date(unix * 1000); // Unix seconds to JS milliseconds
}

function intToBool(val: number | null): boolean {
  return val === 1;
}
```

**Migration order and transform functions for each table:**

1. **categories** -- `id` (serial), `name`, `icon`, `createdAt` (unixToDate)
2. **users** -- `id` (serial), `username`, `passwordHash`, `createdAt` (unixToDate)
3. **settings** -- `key`, `value` (no transforms needed, text PK)
4. **items** -- `id` (serial), `name`, `weightGrams`, `priceCents`, `categoryId`, `notes`, `productUrl`, `imageFilename`, `imageSourceUrl`, `quantity`, `createdAt` (unixToDate), `updatedAt` (unixToDate)
5. **threads** -- `id` (serial), `name`, `status`, `resolvedCandidateId`, `categoryId`, `createdAt` (unixToDate), `updatedAt` (unixToDate)
6. **sessions** -- `id` (text PK), `userId`, `expiresAt` (unixToDate)
7. **apiKeys** -- `id` (serial), `name`, `keyHash`, `keyPrefix`, `createdAt` (unixToDate)
8. **oauthClients** -- `id` (serial), `clientId`, `clientName`, `redirectUris`, `createdAt` (unixToDate)
9. **threadCandidates** -- `id` (serial), all fields, `createdAt`/`updatedAt` (unixToDate), `sortOrder` (keep as number)
10. **setups** -- `id` (serial), `name`, `createdAt`/`updatedAt` (unixToDate)
11. **oauthCodes** -- `id` (serial), all fields, `expiresAt` (unixToDate), `used` (intToBool)
12. **oauthTokens** -- `id` (serial), all fields, `expiresAt`/`refreshExpiresAt`/`createdAt` (unixToDate)
13. **setupItems** -- `id` (serial), `setupId`, `itemId`, `classification`

**For each table, use this pattern:**
```typescript
async function migrateTable(tableName: string, pgTable: any, transform: (row: any) => any) {
  const rows = sqlite.query(`SELECT * FROM ${tableName}`).all();
  console.log(`  ${tableName}: ${rows.length} rows`);
  if (rows.length === 0) return;
  
  for (const row of rows) {
    await db.insert(pgTable).values(transform(row));
  }
}
```

**Sequence reset after all data is migrated:**
```typescript
async function resetSequences() {
  const tablesWithSerial = [
    "categories", "items", "threads", "thread_candidates",
    "setups", "setup_items", "users", "api_keys",
    "oauth_clients", "oauth_codes", "oauth_tokens"
  ];
  
  for (const table of tablesWithSerial) {
    await sql`SELECT setval(pg_get_serial_sequence('${sql.raw(table)}', 'id'), COALESCE((SELECT MAX(id) FROM ${sql.raw(table)}), 0))`;
  }
}
```

Note: Use `db.execute(sql\`...\`)` from drizzle-orm for raw SQL, or use `pg\`...\`` from the postgres.js client directly. The `sql.raw()` helper is needed for dynamic table names.

**Main function:**
```typescript
async function main() {
  const sqlitePath = process.env.SQLITE_PATH || "gearbox.db";
  const databaseUrl = process.env.DATABASE_URL;
  
  if (!databaseUrl) {
    console.error("ERROR: DATABASE_URL environment variable is required");
    process.exit(1);
  }
  
  console.log(`Migrating from SQLite (${sqlitePath}) to PostgreSQL...`);
  
  const sqlite = new Database(sqlitePath, { readonly: true });
  const pg = postgres(databaseUrl);
  const db = drizzle(pg, { schema });
  
  // ... migrate all tables in order ...
  // ... reset sequences ...
  
  await pg.end();
  sqlite.close();
  
  console.log("Migration complete!");
}

main().catch((err) => {
  console.error("Migration failed:", err);
  process.exit(1);
});
```

**Error handling per table:** Wrap each table migration in try/catch, log which table failed and which row (by ID if available), then re-throw. This aids debugging partial migrations.

**Add to package.json scripts:**
```json
"db:migrate-from-sqlite": "bun run scripts/migrate-sqlite-to-postgres.ts"
```
  </action>
  <verify>
    <automated>test -f scripts/migrate-sqlite-to-postgres.ts && grep -q "bun:sqlite" scripts/migrate-sqlite-to-postgres.ts && grep -q "postgres" scripts/migrate-sqlite-to-postgres.ts && grep -q "setval" scripts/migrate-sqlite-to-postgres.ts && grep -q "unixToDate\|unix.*Date\|\\* 1000" scripts/migrate-sqlite-to-postgres.ts && bun run lint 2>&1 | tail -3 && echo "PASS" || echo "FAIL"</automated>
  </verify>
  <acceptance_criteria>
    - scripts/migrate-sqlite-to-postgres.ts exists
    - File imports from `bun:sqlite` (read-only) and `drizzle-orm/postgres-js` and `postgres`
    - File imports schema from `../src/db/schema.ts`
    - File contains a unix-to-Date conversion function (multiplies by 1000)
    - File contains an integer-to-boolean conversion for `used` field
    - File migrates all 13 tables in dependency order (categories and users before items and threads, etc.)
    - File contains `setval` calls to reset serial sequences after migration
    - File reads `DATABASE_URL` from environment and exits with error if missing
    - File reads `SQLITE_PATH` from environment with default `"gearbox.db"`
    - File opens SQLite in readonly mode
    - package.json contains `"db:migrate-from-sqlite"` script
  </acceptance_criteria>
  <done>Migration script reads SQLite, writes to Postgres with type conversions, resets sequences. All IDs and FK relationships preserved per D-06.</done>
</task>

</tasks>

<verification>
- `bun run scripts/migrate-sqlite-to-postgres.ts --help` or similar does not crash on syntax errors (will fail on missing DATABASE_URL, which is expected)
- Script contains all 13 table migrations
- Script resets sequences for all tables with serial IDs
- `bun run lint` passes
</verification>

<success_criteria>
One-time migration script exists, handles all type conversions (timestamps, booleans), preserves IDs, resets sequences. Can be run with `DATABASE_URL=... SQLITE_PATH=... bun run scripts/migrate-sqlite-to-postgres.ts`. Lint passes.
</success_criteria>

<output>
After completion, create `.planning/phases/14-postgresql-migration/14-05-SUMMARY.md`
</output>