GearBox/.planning/phases/14-postgresql-migration/14-05-PLAN.md

phase, plan, type, wave, depends_on, files_modified, autonomous, requirements, must_haves

phase	plan	type	wave	depends_on	files_modified	autonomous	requirements	must_haves
14-postgresql-migration	05	execute	2	14-01	scripts/migrate-sqlite-to-postgres.ts	true	DB-04	truths artifacts key_links Script reads all data from SQLite file and writes it to PostgreSQL Integer timestamps are converted to Date objects for Postgres timestamp columns Boolean integers (0/1) are converted to true/false for Postgres boolean columns All IDs and foreign key relationships are preserved Serial sequences are reset after data migration to avoid duplicate key errors path provides contains scripts/migrate-sqlite-to-postgres.ts One-time SQLite to Postgres data migration migrate-sqlite-to-postgres from to via pattern scripts/migrate-sqlite-to-postgres.ts src/db/schema.ts import table definitions for typed inserts import.*schema

Create the one-time SQLite-to-PostgreSQL data migration script that reads from an existing SQLite database and writes all data into PostgreSQL with proper type conversions.

Purpose: Existing users need to migrate their data from SQLite to Postgres without data loss (DB-04). This is a standalone script run once during the upgrade. Output: scripts/migrate-sqlite-to-postgres.ts

<execution_context> @$HOME/.claude/get-shit-done/workflows/execute-plan.md @$HOME/.claude/get-shit-done/templates/summary.md </execution_context>

@.planning/PROJECT.md @.planning/ROADMAP.md @.planning/phases/14-postgresql-migration/14-CONTEXT.md @.planning/phases/14-postgresql-migration/14-RESEARCH.md @.planning/phases/14-postgresql-migration/14-01-SUMMARY.md

@src/db/schema.ts

Tables in dependency order:

1. categories, users, settings (no foreign keys to other app tables)
2. items, threads, sessions, apiKeys, oauthClients (FK to categories/users)
3. threadCandidates, setups, oauthCodes, oauthTokens (FK to threads/etc)
4. setupItems (FK to setups + items)

Task 1: Create SQLite-to-Postgres migration script scripts/migrate-sqlite-to-postgres.ts src/db/schema.ts Create `scripts/migrate-sqlite-to-postgres.ts` per D-04, D-05, D-06.

// scripts/migrate-sqlite-to-postgres.ts
import { Database } from "bun:sqlite";
import { drizzle } from "drizzle-orm/postgres-js";
import postgres from "postgres";
import * as schema from "../src/db/schema.ts";

Environment variables:

• SQLITE_PATH -- path to SQLite database file (default: "gearbox.db")
• DATABASE_URL -- PostgreSQL connection string (required)

Structure:

1. Open SQLite database read-only
2. Connect to PostgreSQL via postgres.js + drizzle
3. Migrate tables in dependency order (parents before children)
4. Reset all serial sequences after migration
5. Close both connections
6. Print summary

Type conversion functions:

function unixToDate(unix: number | null): Date | null {
  if (unix === null || unix === undefined) return null;
  return new Date(unix * 1000); // Unix seconds to JS milliseconds
}

function intToBool(val: number | null): boolean {
  return val === 1;
}

Migration order and transform functions for each table:

1. categories -- id (serial), name, icon, createdAt (unixToDate)
2. users -- id (serial), username, passwordHash, createdAt (unixToDate)
3. settings -- key, value (no transforms needed, text PK)
4. items -- id (serial), name, weightGrams, priceCents, categoryId, notes, productUrl, imageFilename, imageSourceUrl, quantity, createdAt (unixToDate), updatedAt (unixToDate)
5. threads -- id (serial), name, status, resolvedCandidateId, categoryId, createdAt (unixToDate), updatedAt (unixToDate)
6. sessions -- id (text PK), userId, expiresAt (unixToDate)
7. apiKeys -- id (serial), name, keyHash, keyPrefix, createdAt (unixToDate)
8. oauthClients -- id (serial), clientId, clientName, redirectUris, createdAt (unixToDate)
9. threadCandidates -- id (serial), all fields, createdAt/updatedAt (unixToDate), sortOrder (keep as number)
10. setups -- id (serial), name, createdAt/updatedAt (unixToDate)
11. oauthCodes -- id (serial), all fields, expiresAt (unixToDate), used (intToBool)
12. oauthTokens -- id (serial), all fields, expiresAt/refreshExpiresAt/createdAt (unixToDate)
13. setupItems -- id (serial), setupId, itemId, classification

For each table, use this pattern:

async function migrateTable(tableName: string, pgTable: any, transform: (row: any) => any) {
  const rows = sqlite.query(`SELECT * FROM ${tableName}`).all();
  console.log(`  ${tableName}: ${rows.length} rows`);
  if (rows.length === 0) return;
  
  for (const row of rows) {
    await db.insert(pgTable).values(transform(row));
  }
}

Sequence reset after all data is migrated:

async function resetSequences() {
  const tablesWithSerial = [
    "categories", "items", "threads", "thread_candidates",
    "setups", "setup_items", "users", "api_keys",
    "oauth_clients", "oauth_codes", "oauth_tokens"
  ];
  
  for (const table of tablesWithSerial) {
    await sql`SELECT setval(pg_get_serial_sequence('${sql.raw(table)}', 'id'), COALESCE((SELECT MAX(id) FROM ${sql.raw(table)}), 0))`;
  }
}

Note: Use db.execute(sql\...`)from drizzle-orm for raw SQL, or usepg`...`from the postgres.js client directly. Thesql.raw()` helper is needed for dynamic table names.

Main function:

async function main() {
  const sqlitePath = process.env.SQLITE_PATH || "gearbox.db";
  const databaseUrl = process.env.DATABASE_URL;
  
  if (!databaseUrl) {
    console.error("ERROR: DATABASE_URL environment variable is required");
    process.exit(1);
  }
  
  console.log(`Migrating from SQLite (${sqlitePath}) to PostgreSQL...`);
  
  const sqlite = new Database(sqlitePath, { readonly: true });
  const pg = postgres(databaseUrl);
  const db = drizzle(pg, { schema });
  
  // ... migrate all tables in order ...
  // ... reset sequences ...
  
  await pg.end();
  sqlite.close();
  
  console.log("Migration complete!");
}

main().catch((err) => {
  console.error("Migration failed:", err);
  process.exit(1);
});

Error handling per table: Wrap each table migration in try/catch, log which table failed and which row (by ID if available), then re-throw. This aids debugging partial migrations.

Add to package.json scripts:

"db:migrate-from-sqlite": "bun run scripts/migrate-sqlite-to-postgres.ts"

test -f scripts/migrate-sqlite-to-postgres.ts && grep -q "bun:sqlite" scripts/migrate-sqlite-to-postgres.ts && grep -q "postgres" scripts/migrate-sqlite-to-postgres.ts && grep -q "setval" scripts/migrate-sqlite-to-postgres.ts && grep -q "unixToDate\|unix.*Date\|\\* 1000" scripts/migrate-sqlite-to-postgres.ts && bun run lint 2>&1 | tail -3 && echo "PASS" || echo "FAIL" - scripts/migrate-sqlite-to-postgres.ts exists - File imports from `bun:sqlite` (read-only) and `drizzle-orm/postgres-js` and `postgres` - File imports schema from `../src/db/schema.ts` - File contains a unix-to-Date conversion function (multiplies by 1000) - File contains an integer-to-boolean conversion for `used` field - File migrates all 13 tables in dependency order (categories and users before items and threads, etc.) - File contains `setval` calls to reset serial sequences after migration - File reads `DATABASE_URL` from environment and exits with error if missing - File reads `SQLITE_PATH` from environment with default `"gearbox.db"` - File opens SQLite in readonly mode - package.json contains `"db:migrate-from-sqlite"` script Migration script reads SQLite, writes to Postgres with type conversions, resets sequences. All IDs and FK relationships preserved per D-06. - `bun run scripts/migrate-sqlite-to-postgres.ts --help` or similar does not crash on syntax errors (will fail on missing DATABASE_URL, which is expected) - Script contains all 13 table migrations - Script resets sequences for all tables with serial IDs - `bun run lint` passes

<success_criteria> One-time migration script exists, handles all type conversions (timestamps, booleans), preserves IDs, resets sequences. Can be run with DATABASE_URL=... SQLITE_PATH=... bun run scripts/migrate-sqlite-to-postgres.ts. Lint passes. </success_criteria>

After completion, create `.planning/phases/14-postgresql-migration/14-05-SUMMARY.md`