Files
GearBox/.planning/phases/14-postgresql-migration/14-RESEARCH.md

28 KiB

Phase 14: PostgreSQL Migration - Research

Researched: 2026-04-04 Domain: Database migration (SQLite to PostgreSQL), Drizzle ORM, PGlite testing Confidence: HIGH

Summary

This phase replaces the SQLite database with PostgreSQL across the entire stack: schema definitions, database driver, all service/route code (sync to async), test infrastructure (PGlite), data migration script, and Docker Compose for local development.

The core migration is well-supported by Drizzle ORM, which has first-class drivers for both PostgreSQL (via postgres package) and PGlite (for testing). The schema rewrite from drizzle-orm/sqlite-core to drizzle-orm/pg-core is straightforward -- column type mapping is direct. The bulk of the work is mechanical: adding await to ~82 sync .all()/.get()/.run() calls across 9 service files, updating 4 transaction usages to async, and updating all 18 test files to use async PGlite-backed databases.

Primary recommendation: Use postgres (postgres.js) as the production driver for best Bun compatibility and connection pooling. Use @electric-sql/pglite with drizzle-orm/pglite for tests. Apply schema in tests via migrate() from generated migrations (not pushSchema) to match production behavior.

<user_constraints>

User Constraints (from CONTEXT.md)

Locked Decisions

  • D-01: Clean rewrite of src/db/schema.ts using drizzle-orm/pg-core (pgTable, serial, text, numeric, timestamp, etc.) -- not a conversion of the SQLite schema
  • D-02: Start fresh Postgres migration history in a new directory (e.g., drizzle-pg/) -- keep existing drizzle/ SQLite migrations archived for reference
  • D-03: src/db/index.ts switches from bun:sqlite + drizzle-orm/bun-sqlite to drizzle-orm/node-postgres (or drizzle-orm/postgres-js) with async connection
  • D-04: Standalone TypeScript script (e.g., scripts/migrate-sqlite-to-postgres.ts) that reads from SQLite file and writes to Postgres -- not a Drizzle migration
  • D-05: Script handles type conversions: integer timestamps to proper Postgres timestamp columns, real weight to numeric or double precision, text to text
  • D-06: Script preserves all IDs and foreign key relationships -- no ID remapping
  • D-07: createTestDb() returns an async PGlite-backed Drizzle instance -- same API shape as current, but async
  • D-08: Per-test fresh PGlite instance with migrations applied (matches current in-memory SQLite pattern, avoids test pollution)
  • D-09: All service and route tests updated from sync to async database operations
  • D-10: Separate docker-compose.dev.yml for development with Postgres service -- keep existing docker-compose.yml for production (updated to include Postgres)
  • D-11: PostgreSQL 16 (latest stable)
  • D-12: Environment variable DATABASE_URL for Postgres connection string (replaces DATABASE_PATH for SQLite)

Claude's Discretion

  • Drizzle Postgres driver choice (node-postgres vs postgres-js) -- pick based on Bun compatibility and async performance
  • PGlite configuration details (version, extensions)
  • Column type mapping specifics beyond the ones called out (e.g., whether to use serial vs integer().primaryKey())
  • Migration script error handling and progress reporting
  • Whether to use drizzle-orm/pglite driver or generic pg driver for tests

Deferred Ideas (OUT OF SCOPE)

None -- discussion stayed within phase scope </user_constraints>

<phase_requirements>

Phase Requirements

ID Description Research Support
DB-01 Application runs on PostgreSQL instead of SQLite Schema rewrite (pg-core), driver swap (postgres.js), async service layer
DB-02 All service functions use async database operations 82 sync calls across 9 services need await; 4 transactions need async conversion
DB-03 Test infrastructure uses PGlite instead of bun:sqlite in-memory databases @electric-sql/pglite + drizzle-orm/pglite with per-test instances
DB-04 Existing SQLite data can be migrated to Postgres via a one-time script Standalone script reads SQLite via bun:sqlite, writes to Postgres with type conversion
DB-05 Docker Compose provides Postgres for local development docker-compose.dev.yml with PostgreSQL 16, docker-compose.yml updated for production
</phase_requirements>

Standard Stack

Core

Library Version Purpose Why Standard
drizzle-orm 0.45.2 ORM (already installed, update minor) Already in use; pg-core module provides PostgreSQL schema/query support
drizzle-kit 0.31.10 Migration generation (already installed, update minor) Already in use; supports postgresql dialect for migration generation
postgres 3.4.8 PostgreSQL driver (postgres.js) Best Bun compatibility, built-in connection pooling, no native bindings needed
@electric-sql/pglite 0.4.3 In-process WASM Postgres for testing Real Postgres SQL execution without Docker; per-test isolation in milliseconds

Supporting

Library Version Purpose When to Use
bun:sqlite (built-in) N/A Read-only in migration script Only used by data migration script to read existing SQLite data

Alternatives Considered

Instead of Could Use Tradeoff
postgres (postgres.js) pg (node-postgres) pg requires @types/pg, has native binding option but no benefit on Bun; postgres.js has cleaner API
postgres (postgres.js) bun:sql (Bun SQL) Bun SQL has known drizzle-kit compatibility issues (push/migrate don't work); not yet mature enough
@electric-sql/pglite Docker Postgres for tests Docker adds latency, setup complexity; PGlite is zero-config, sub-millisecond startup

Driver recommendation: postgres (postgres.js)

  • No native bindings (works on Bun without build tools)
  • Built-in connection pooling
  • Prepared statements by default
  • Drizzle ORM has first-class drizzle-orm/postgres-js driver
  • Bun SQL driver was considered but drizzle-kit does not fully support it for push/migrate commands yet

Installation:

bun add postgres @electric-sql/pglite
bun remove better-sqlite3 @types/better-sqlite3

Note: bun:sqlite is built-in and does not need to be uninstalled -- it remains available for the migration script. better-sqlite3 and its types are dev dependencies that can be removed since they are no longer needed.

Architecture Patterns

src/db/
  schema.ts          # Rewritten with drizzle-orm/pg-core (pgTable, serial, text, timestamp, etc.)
  index.ts           # postgres.js connection + drizzle initialization
  migrate.ts         # Async migration runner for production startup
  seed.ts            # Async seed function
drizzle-pg/          # New PostgreSQL migration directory (D-02)
drizzle/             # Archived SQLite migrations (kept for reference)
drizzle.config.ts    # Updated: dialect "postgresql", out "./drizzle-pg"
scripts/
  migrate-sqlite-to-postgres.ts  # One-time data migration script (D-04)
tests/helpers/
  db.ts              # Rewritten: async createTestDb() with PGlite
docker-compose.dev.yml  # New: Postgres for local dev
docker-compose.yml      # Updated: Postgres for production

Pattern 1: PostgreSQL Schema Definition

What: Rewrite all tables using drizzle-orm/pg-core types When to use: The one-time schema rewrite

// src/db/schema.ts
import { doublePrecision, integer, pgTable, serial, text, timestamp } from "drizzle-orm/pg-core";

export const categories = pgTable("categories", {
  id: serial("id").primaryKey(),
  name: text("name").notNull().unique(),
  icon: text("icon").notNull().default("package"),
  createdAt: timestamp("created_at").notNull().defaultNow(),
});

export const items = pgTable("items", {
  id: serial("id").primaryKey(),
  name: text("name").notNull(),
  weightGrams: doublePrecision("weight_grams"),
  priceCents: integer("price_cents"),
  categoryId: integer("category_id").notNull().references(() => categories.id),
  notes: text("notes"),
  productUrl: text("product_url"),
  imageFilename: text("image_filename"),
  imageSourceUrl: text("image_source_url"),
  quantity: integer("quantity").notNull().default(1),
  createdAt: timestamp("created_at").notNull().defaultNow(),
  updatedAt: timestamp("updated_at").notNull().defaultNow(),
});

Pattern 2: Async Database Connection

What: Production database initialization with postgres.js When to use: src/db/index.ts

// src/db/index.ts
import { drizzle } from "drizzle-orm/postgres-js";
import postgres from "postgres";
import * as schema from "./schema.ts";

const queryClient = postgres(process.env.DATABASE_URL!);
export const db = drizzle(queryClient, { schema });

Pattern 3: Async Service Functions

What: Convert sync Drizzle calls to async with await When to use: All 9 service files

// BEFORE (SQLite sync):
export function getAllItems(db: Db = prodDb) {
  return db.select().from(items).innerJoin(categories, eq(items.categoryId, categories.id)).all();
}

// AFTER (PostgreSQL async):
export async function getAllItems(db: Db = prodDb) {
  return await db.select().from(items).innerJoin(categories, eq(items.categoryId, categories.id));
}

Key differences:

  • .all() is removed -- Postgres driver returns arrays directly from await
  • .get() is replaced with indexing: const [result] = await db.select()... or using .limit(1) then [0]
  • .run() is removed -- await db.delete()... / await db.insert()... is sufficient
  • .returning().get() becomes const [result] = await db.insert()...returning()
  • db.transaction(() => { ... }) becomes await db.transaction(async (tx) => { ... }) with await inside

Pattern 4: PGlite Test Database

What: Per-test Postgres instance using PGlite When to use: tests/helpers/db.ts

// tests/helpers/db.ts
import { drizzle } from "drizzle-orm/pglite";
import { migrate } from "drizzle-orm/pglite/migrator";
import * as schema from "../../src/db/schema.ts";

export async function createTestDb() {
  const db = drizzle({ schema });

  // Apply migrations from the new PostgreSQL migration directory
  await migrate(db, { migrationsFolder: "./drizzle-pg" });

  // Seed default category
  await db.insert(schema.categories).values({ name: "Uncategorized", icon: "package" });

  return db;
}

Pattern 5: Async Transaction

What: Convert sync transactions to async When to use: 4 transaction sites (category delete, setup update, thread resolve/unresolve)

// BEFORE (SQLite sync):
db.transaction(() => {
  db.update(items).set({ categoryId: 1 }).where(eq(items.categoryId, id)).run();
  db.delete(categories).where(eq(categories.id, id)).run();
});

// AFTER (PostgreSQL async):
await db.transaction(async (tx) => {
  await tx.update(items).set({ categoryId: 1 }).where(eq(items.categoryId, id));
  await tx.delete(categories).where(eq(categories.id, id));
});

Pattern 6: Drizzle Config for PostgreSQL

What: Updated drizzle.config.ts When to use: One-time config update

// drizzle.config.ts
import { defineConfig } from "drizzle-kit";

export default defineConfig({
  out: "./drizzle-pg",
  schema: "./src/db/schema.ts",
  dialect: "postgresql",
  dbCredentials: {
    url: process.env.DATABASE_URL || "postgresql://gearbox:gearbox@localhost:5432/gearbox",
  },
});

Anti-Patterns to Avoid

  • Mixing sync and async: Do not leave any .all(), .get(), .run() calls -- they are SQLite-only methods
  • Forgetting await: Every database call must be awaited; missing awaits will return Promise objects instead of data
  • Using pushSchema for tests: While faster, pushSchema from drizzle-kit/api does not match production migration behavior -- use migrate() to catch migration issues early
  • Integer timestamps in Postgres: Do not carry over integer("col", { mode: "timestamp" }) -- use native timestamp() type
  • Keeping bun:sqlite imports in production code: Only the migration script should import bun:sqlite

Don't Hand-Roll

Problem Don't Build Use Instead Why
Connection pooling Custom pool manager postgres built-in pooling Handles connection limits, idle timeout, reconnection
In-memory test DB Docker Postgres containers PGlite Zero setup, sub-ms startup, real Postgres SQL
Schema migrations Manual SQL files drizzle-kit generate Generates correct DDL from schema diff
Data type conversion Manual column-by-column casting Drizzle schema + postgres driver auto-coercion Driver handles JS Date <-> Postgres timestamp, number <-> integer

Key insight: Drizzle ORM abstracts the SQLite/PostgreSQL differences at the query builder level. The schema definition and driver are the only things that change -- service query logic (select, where, join, insert, etc.) stays identical except for removing sync-only methods.

Common Pitfalls

Pitfall 1: Missing Await on Database Calls

What goes wrong: Route handlers return Promise<Item> instead of Item, leading to empty/broken JSON responses Why it happens: Mechanical conversion misses an await in a handler that was previously sync How to avoid: Make route handlers async if not already; TypeScript will flag return type mismatches if return types are annotated Warning signs: Tests pass but return {} or undefined fields; API returns {}

Pitfall 2: .get() Does Not Exist on PostgreSQL Drizzle

What goes wrong: Runtime error: .get is not a function Why it happens: .get() is a SQLite-only convenience method that returns a single row How to avoid: Replace .get() with array destructuring: const [row] = await db.select()...; replace .returning().get() with const [row] = await db.insert()...returning() Warning signs: TypeScript type errors if using strict mode

Pitfall 3: serial Auto-Increment Behavior in Postgres

What goes wrong: Data migration script inserts rows with explicit IDs but the serial sequence is not advanced, causing conflicts on next insert Why it happens: PostgreSQL serial is backed by a sequence that is only auto-incremented on default inserts -- explicit ID inserts do not update the sequence How to avoid: After data migration, reset sequences: SELECT setval('table_id_seq', (SELECT MAX(id) FROM table)) Warning signs: Duplicate key errors after migration when creating new records

Pitfall 4: Boolean Columns (OAuth used Field)

What goes wrong: SQLite uses integer for boolean (0/1); Postgres has native boolean type Why it happens: Direct schema port without type adjustment How to avoid: Use boolean("used").notNull().default(false) in pg-core schema; migration script must convert 0/1 to false/true Warning signs: Type errors in OAuth code that checks === 0 or === 1

Pitfall 5: Transaction Callback Must Be Async

What goes wrong: Transaction body runs sync but database calls inside return unresolved promises Why it happens: Forgetting to make the transaction callback async and await internal operations How to avoid: await db.transaction(async (tx) => { await tx.update()... }) Warning signs: Empty/partial data writes, no errors thrown

Pitfall 6: createdAt Default Function Mismatch

What goes wrong: $defaultFn(() => new Date()) in SQLite schema is a JS-side default; Postgres defaultNow() is SQL-side Why it happens: Different default mechanisms How to avoid: Use .defaultNow() for all timestamp columns in pg-core schema (server-side default is more reliable) Warning signs: Null timestamps when inserting without explicit values

Pitfall 7: Test createTestDb() Becomes Async

What goes wrong: All beforeEach blocks that call createTestDb() break Why it happens: createTestDb() returns a Promise instead of a Drizzle instance How to avoid: beforeEach(async () => { db = await createTestDb(); }) in all 18 test files Warning signs: db.select is not a function errors in every test

Pitfall 8: Db Type Changes

What goes wrong: type Db = typeof prodDb in services no longer matches PGlite-created instances in tests Why it happens: drizzle-orm/postgres-js and drizzle-orm/pglite return different Drizzle instance types How to avoid: Use a shared type or use the generic PostgresJsDatabase<typeof schema> type that both drivers satisfy. Alternatively, use ReturnType<typeof drizzle> from pglite driver which is compatible. Warning signs: TypeScript errors when passing test DB to service functions

Code Examples

Data Migration Script Structure

// scripts/migrate-sqlite-to-postgres.ts
import { Database } from "bun:sqlite";
import { drizzle } from "drizzle-orm/postgres-js";
import postgres from "postgres";
import * as schema from "../src/db/schema.ts";

const sqlite = new Database(process.env.SQLITE_PATH || "gearbox.db");
const pg = postgres(process.env.DATABASE_URL!);
const db = drizzle(pg, { schema });

async function migrateTable<T>(
  tableName: string,
  pgTable: any,
  transform: (row: any) => T
) {
  const rows = sqlite.query(`SELECT * FROM ${tableName}`).all();
  console.log(`Migrating ${rows.length} ${tableName}...`);
  
  if (rows.length === 0) return;
  
  for (const row of rows) {
    await db.insert(pgTable).values(transform(row as any));
  }
}

async function resetSequences() {
  const tables = ["categories", "items", "threads", "thread_candidates", 
                   "setups", "setup_items", "users", "api_keys", 
                   "oauth_clients", "oauth_codes", "oauth_tokens"];
  for (const table of tables) {
    await pg`SELECT setval('${pg(table)}_id_seq', COALESCE((SELECT MAX(id) FROM ${pg(table)}), 0))`;
  }
}

async function main() {
  // Migrate tables in dependency order (parents before children)
  // 1. categories, users, settings
  // 2. items, threads, sessions, api_keys, oauth_clients
  // 3. thread_candidates, setups
  // 4. setup_items
  // Convert: unix timestamps -> Date objects, integer booleans -> booleans
  
  await resetSequences();
  await pg.end();
  sqlite.close();
  console.log("Migration complete!");
}

main().catch(console.error);

Docker Compose Development

# docker-compose.dev.yml
services:
  postgres:
    image: postgres:16-alpine
    environment:
      POSTGRES_USER: gearbox
      POSTGRES_PASSWORD: gearbox
      POSTGRES_DB: gearbox
    ports:
      - "5432:5432"
    volumes:
      - pgdata-dev:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U gearbox"]
      interval: 5s
      timeout: 3s
      retries: 5

volumes:
  pgdata-dev:

Docker Compose Production (updated)

# docker-compose.yml
services:
  postgres:
    image: postgres:16-alpine
    environment:
      POSTGRES_USER: gearbox
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
      POSTGRES_DB: gearbox
    volumes:
      - pgdata:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U gearbox"]
      interval: 10s
      timeout: 5s
      retries: 5

  app:
    image: gearbox:latest
    environment:
      DATABASE_URL: postgresql://gearbox:${POSTGRES_PASSWORD}@postgres:5432/gearbox
      GEARBOX_URL: ${GEARBOX_URL}
    ports:
      - "3000:3000"
    depends_on:
      postgres:
        condition: service_healthy
    volumes:
      - uploads:/app/uploads

volumes:
  pgdata:
  uploads:

Updated Migration Runner

// src/db/migrate.ts
import { drizzle } from "drizzle-orm/postgres-js";
import { migrate } from "drizzle-orm/postgres-js/migrator";
import postgres from "postgres";

const migrationClient = postgres(process.env.DATABASE_URL!, { max: 1 });
const db = drizzle(migrationClient);

await migrate(db, { migrationsFolder: "./drizzle-pg" });
await migrationClient.end();

console.log("Migrations applied successfully");

Column Type Mapping

SQLite Column pg-core Column Notes
integer("id").primaryKey({ autoIncrement: true }) serial("id").primaryKey() serial = auto-incrementing 4-byte int
text("name") text("name") Identical
real("weight_grams") doublePrecision("weight_grams") 8-byte float, matches SQLite real precision
integer("price_cents") integer("price_cents") Identical
integer("col", { mode: "timestamp" }) timestamp("col") Native Postgres timestamp; Drizzle returns JS Date
integer("used").default(0) boolean("used").default(false) Proper boolean type
real("sort_order") doublePrecision("sort_order") Or real() (4-byte) -- either works
text("id").primaryKey() (sessions) text("id").primaryKey() Identical
text("key").primaryKey() (settings) text("key").primaryKey() Identical

State of the Art

Old Approach Current Approach When Changed Impact
bun:sqlite sync driver postgres (postgres.js) async driver This migration All DB calls become async
drizzle-orm/bun-sqlite drizzle-orm/postgres-js This migration Driver swap in one file
In-memory SQLite for tests PGlite WASM Postgres for tests This migration Tests run real Postgres SQL
drizzle-orm/bun-sql (Bun native) postgres (postgres.js) N/A Bun SQL has drizzle-kit incompatibilities; postgres.js is mature

Scope of Change

Summary of files that need modification:

Category Files Change Type
Schema src/db/schema.ts Full rewrite (sqlite-core to pg-core)
DB config src/db/index.ts Full rewrite (bun:sqlite to postgres.js)
Migrations src/db/migrate.ts Full rewrite (async, postgres migrator)
Seed src/db/seed.ts Async conversion
Drizzle config drizzle.config.ts Dialect + output path change
Services 9 files in src/server/services/ Add async/await to all DB calls (~82 call sites)
Routes 9 files in src/server/routes/ Add await to service calls, make handlers async
Server entry src/server/index.ts Async seed call
Test helper tests/helpers/db.ts Full rewrite (PGlite)
Service tests 9 files in tests/services/ Async beforeEach + await all assertions
Route tests 8 files in tests/routes/ Async createTestApp + await
MCP tests tests/mcp/tools.test.ts Async test DB
Docker docker-compose.dev.yml (new), docker-compose.yml (new) Postgres service definitions
Dockerfile Dockerfile Update: copy drizzle-pg/, remove SQLite-specific steps
Migration script scripts/migrate-sqlite-to-postgres.ts (new) Data migration
Package.json package.json Add postgres, @electric-sql/pglite; remove better-sqlite3

Total: ~40 files touched, ~2 new files created

Validation Architecture

Test Framework

Property Value
Framework Bun test runner (built-in)
Config file None (uses bun defaults)
Quick run command bun test tests/services/item.service.test.ts
Full suite command bun test tests/

Phase Requirements to Test Map

Req ID Behavior Test Type Automated Command File Exists?
DB-01 App runs on PostgreSQL integration bun test tests/ (all tests use PGlite) Existing (updated)
DB-02 Async database operations unit bun test tests/services/ Existing (updated)
DB-03 PGlite test infrastructure unit bun test tests/services/item.service.test.ts -x Existing (updated)
DB-04 SQLite data migration script integration bun run scripts/migrate-sqlite-to-postgres.ts New (Wave 0)
DB-05 Docker Compose Postgres smoke docker compose -f docker-compose.dev.yml up -d && bun test tests/ Manual verification

Sampling Rate

  • Per task commit: bun test tests/services/item.service.test.ts -x (fast single-file check)
  • Per wave merge: bun test tests/ (full suite)
  • Phase gate: Full suite green + manual Docker Compose smoke test

Wave 0 Gaps

  • tests/helpers/db.ts -- must be rewritten to PGlite before any other tests can run
  • Migration files in drizzle-pg/ -- must be generated before test helper can apply them
  • scripts/migrate-sqlite-to-postgres.ts -- new file, needs at least a basic test or manual verification plan

Open Questions

  1. PGlite + Bun test runner performance

    • What we know: PGlite works well with Vitest; Bun test runner is compatible
    • What's unclear: Whether Bun's test runner parallel mode causes issues with PGlite WASM initialization
    • Recommendation: Start with sequential tests; if slow, investigate parallelization
  2. Db type compatibility between postgres.js and PGlite drivers

    • What we know: Both return Drizzle instances but with different generic type parameters
    • What's unclear: Whether the types are structurally compatible without explicit casting
    • Recommendation: Define a shared AppDb type alias; if types diverge, use a minimal interface or any for the DI parameter with runtime compatibility
  3. Sequence reset in migration script

    • What we know: Explicit ID inserts do not advance Postgres sequences
    • What's unclear: Exact syntax for setval with dynamic table names via postgres.js
    • Recommendation: Use raw SQL via postgres.unsafe() or db.execute(sql\...`)` for sequence resets

Environment Availability

Dependency Required By Available Version Fallback
Docker Docker Compose dev/prod Yes 29.0.0 --
Docker Compose Local Postgres service Yes 2.40.3 --
Bun Runtime Yes 1.3.10 --
PostgreSQL (via Docker) DB-01, DB-05 Via Docker 16-alpine (to pull) --
psql CLI Debug/manual verification No -- Use Docker exec or skip

Missing dependencies with no fallback: None

Missing dependencies with fallback:

  • psql CLI not installed locally -- use docker exec into Postgres container for manual queries

Sources

Primary (HIGH confidence)

Secondary (MEDIUM confidence)

Tertiary (LOW confidence)

Metadata

Confidence breakdown:

  • Standard stack: HIGH - Drizzle ORM pg-core and postgres.js are mature, well-documented, verified against official docs
  • Architecture: HIGH - Schema mapping is direct; async conversion is mechanical; DI pattern makes driver swap clean
  • Pitfalls: HIGH - Based on known SQLite-to-Postgres differences and verified Drizzle API differences
  • Testing (PGlite + Bun): MEDIUM - PGlite is well-documented with Vitest; Bun test runner compatibility is inferred but not directly verified

Research date: 2026-04-04 Valid until: 2026-05-04 (stable domain, Drizzle ORM and PGlite are mature)