docs(03): research PostgreSQL support phase
This commit is contained in:
575
.planning/phases/03-postgresql-support/03-RESEARCH.md
Normal file
575
.planning/phases/03-postgresql-support/03-RESEARCH.md
Normal file
@@ -0,0 +1,575 @@
|
|||||||
|
# Phase 3: PostgreSQL Support - Research
|
||||||
|
|
||||||
|
**Researched:** 2026-03-24
|
||||||
|
**Domain:** Go database/sql with pgx/v5 + golang-migrate PostgreSQL dialect
|
||||||
|
**Confidence:** HIGH
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
Phase 3 adds PostgreSQL as an alternative backend alongside SQLite. The Store interface and all HTTP handlers are already dialect-neutral (Phase 2 delivered this). The work is entirely in three areas: (1) a new `PostgresStore` struct that implements the existing `Store` interface using PostgreSQL SQL syntax, (2) a separate migration runner for PostgreSQL using `golang-migrate`'s dedicated `pgx/v5` database driver, and (3) wiring in `main.go` to branch on `DATABASE_URL`.
|
||||||
|
|
||||||
|
The critical dialect difference is `CreateTag`: PostgreSQL does not support `LastInsertId()` via `pgx/stdlib`. The `PostgresStore.CreateTag` method must use `QueryRow` with `RETURNING id` instead of `Exec` + `LastInsertId`. Every other SQL translation is mechanical (positional params, `NOW()`, `SERIAL`, `ON CONFLICT ... DO UPDATE` instead of `INSERT OR REPLACE`).
|
||||||
|
|
||||||
|
The golang-migrate ecosystem ships a dedicated `database/pgx/v5` sub-package that wraps a `*sql.DB` opened via `pgx/v5/stdlib`. This fits the established pattern in `migrate.go` exactly — a new `RunPostgresMigrations(db *sql.DB) error` function using the same `iofs` source with an embedded `migrations/postgres` directory.
|
||||||
|
|
||||||
|
**Primary recommendation:** Follow the locked decisions in CONTEXT.md verbatim. The implementation is a straightforward port of `SQLiteStore` with dialect adjustments; the only non-obvious trap is the `LastInsertId` incompatibility in `CreateTag`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
<user_constraints>
|
||||||
|
## User Constraints (from CONTEXT.md)
|
||||||
|
|
||||||
|
### Locked Decisions
|
||||||
|
|
||||||
|
**D-01:** Use `pgx/v5/stdlib` as the database/sql adapter — matches SQLiteStore's `*sql.DB` pattern so PostgresStore has the same constructor signature (`*sql.DB` in, Store out)
|
||||||
|
**D-02:** Do NOT use pgx native interface directly — keeping both stores on `database/sql` means the Store interface stays unchanged and `NewServer(store Store, ...)` works identically
|
||||||
|
**D-03:** Each store implementation has its own raw SQL — no runtime dialect switching, no query builder, no shared SQL templates
|
||||||
|
**D-04:** PostgreSQL-specific syntax differences handled in PostgresStore methods:
|
||||||
|
- `SERIAL` instead of `INTEGER PRIMARY KEY AUTOINCREMENT` for tags.id
|
||||||
|
- `$1, $2, $3` positional params instead of `?` placeholders
|
||||||
|
- `NOW()` or `CURRENT_TIMESTAMP` instead of `datetime('now')` for acknowledged_at
|
||||||
|
- `ON CONFLICT ... DO UPDATE SET` syntax is compatible (PostgreSQL 9.5+)
|
||||||
|
- `INSERT ... ON CONFLICT DO UPDATE` for UPSERT (same pattern, different param style)
|
||||||
|
- `INSERT ... ON CONFLICT` for tag assignments instead of `INSERT OR REPLACE`
|
||||||
|
**D-05:** PostgresStore does NOT use a mutex — PostgreSQL handles concurrent writes natively
|
||||||
|
**D-06:** Use `database/sql` default pool settings with sensible overrides: `MaxOpenConns(25)`, `MaxIdleConns(5)`, `ConnMaxLifetime(5 * time.Minute)`
|
||||||
|
**D-07:** `DATABASE_URL` env var present → PostgreSQL; absent → SQLite with `DB_PATH`
|
||||||
|
**D-08:** No separate `DB_DRIVER` variable — the presence of `DATABASE_URL` is the switch
|
||||||
|
**D-09:** Startup log clearly indicates which backend is active: `"Using PostgreSQL database"` vs `"Using SQLite database at {path}"`
|
||||||
|
**D-10:** Separate migration directories: `migrations/sqlite/` (exists) and `migrations/postgres/` (new)
|
||||||
|
**D-11:** PostgreSQL baseline migration `0001_initial_schema.up.sql` creates the same 3 tables with PostgreSQL-native types
|
||||||
|
**D-12:** `RunMigrations` becomes dialect-aware or split into `RunSQLiteMigrations`/`RunPostgresMigrations` — researcher should determine best approach (see Architecture Patterns below)
|
||||||
|
**D-13:** PostgreSQL migrations embedded via separate `//go:embed migrations/postgres` directive
|
||||||
|
**D-14:** Use Docker Compose profiles — `docker compose --profile postgres up` activates the postgres service
|
||||||
|
**D-15:** Default compose (no profile) remains SQLite-only for simple deploys
|
||||||
|
**D-16:** Compose file includes a `postgres` service with health check, and the app service gets `DATABASE_URL` when the profile is active
|
||||||
|
**D-17:** PostgresStore integration tests use a `//go:build postgres` build tag — they only run when a PostgreSQL instance is available
|
||||||
|
**D-18:** CI can optionally run `-tags postgres` with a postgres service container; SQLite tests always run
|
||||||
|
**D-19:** Test helper `NewTestPostgresServer()` creates a test database and runs migrations, similar to `NewTestServer()` for SQLite
|
||||||
|
|
||||||
|
### Claude's Discretion
|
||||||
|
- Exact PostgreSQL connection pool tuning beyond the defaults in D-06
|
||||||
|
- Whether to split RunMigrations into two functions or use a dialect parameter
|
||||||
|
- Error message formatting for PostgreSQL connection failures
|
||||||
|
- Whether to add a health check endpoint that verifies database connectivity
|
||||||
|
|
||||||
|
### Deferred Ideas (OUT OF SCOPE)
|
||||||
|
None — discussion stayed within phase scope.
|
||||||
|
</user_constraints>
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
<phase_requirements>
|
||||||
|
## Phase Requirements
|
||||||
|
|
||||||
|
| ID | Description | Research Support |
|
||||||
|
|----|-------------|------------------|
|
||||||
|
| DB-01 | PostgreSQL is supported as an alternative to SQLite via pgx v5 driver | `pgx/v5/stdlib` confirmed pure-Go, `*sql.DB` compatible; `PostgresStore` implements all 9 Store methods |
|
||||||
|
| DB-02 | Database backend is selected via DATABASE_URL env var (present = PostgreSQL, absent = SQLite with DB_PATH) | main.go branching pattern documented; driver registration names confirmed: `"sqlite"` and `"pgx"` |
|
||||||
|
| DB-03 | Existing SQLite users can upgrade without data loss (baseline migration represents current schema) | SQLite migration already uses `CREATE TABLE IF NOT EXISTS`; PostgreSQL migration is a fresh baseline for new deployments; no cross-dialect migration needed |
|
||||||
|
</phase_requirements>
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Standard Stack
|
||||||
|
|
||||||
|
### Core
|
||||||
|
| Library | Version | Purpose | Why Standard |
|
||||||
|
|---------|---------|---------|--------------|
|
||||||
|
| `github.com/jackc/pgx/v5` | v5.9.1 (Mar 22 2026) | PostgreSQL driver + `database/sql` adapter via `pgx/v5/stdlib` | De-facto standard Go PostgreSQL driver; pure Go (no CGO); actively maintained; 8,394 packages import it |
|
||||||
|
| `github.com/golang-migrate/migrate/v4/database/pgx/v5` | v4.19.1 (same module as existing golang-migrate) | golang-migrate database driver for pgx v5 | Already in project; dedicated pgx/v5 sub-package fits existing `migrate.go` pattern exactly |
|
||||||
|
|
||||||
|
### Supporting
|
||||||
|
| Library | Version | Purpose | When to Use |
|
||||||
|
|---------|---------|---------|-------------|
|
||||||
|
| `github.com/golang-migrate/migrate/v4/source/iofs` | v4.19.1 (already imported) | Serve embedded FS migration files | Reuse existing pattern from `migrate.go` |
|
||||||
|
|
||||||
|
### Alternatives Considered
|
||||||
|
| Instead of | Could Use | Tradeoff |
|
||||||
|
|------------|-----------|----------|
|
||||||
|
| `pgx/v5/stdlib` (`database/sql`) | pgx native interface | Native pgx is faster but breaks `Store` interface — rejected by D-02 |
|
||||||
|
| `golang-migrate database/pgx/v5` | `golang-migrate database/postgres` | `database/postgres` uses `lib/pq` internally; `database/pgx/v5` uses pgx consistently — use pgx/v5 sub-package |
|
||||||
|
| Two separate `RunMigrations` functions | Single function with dialect param | Two functions is simpler, avoids string-switch, each can be `go:embed`-scoped independently — use two functions (see Architecture) |
|
||||||
|
|
||||||
|
**Installation:**
|
||||||
|
```bash
|
||||||
|
go get github.com/jackc/pgx/v5@v5.9.1
|
||||||
|
go get github.com/golang-migrate/migrate/v4/database/pgx/v5
|
||||||
|
```
|
||||||
|
|
||||||
|
Note: `golang-migrate/migrate/v4` is already in `go.mod` at v4.19.1. Adding the `database/pgx/v5` sub-package pulls from the same module version — no module version conflict.
|
||||||
|
|
||||||
|
**Version verification (current as of 2026-03-24):**
|
||||||
|
- `pgx/v5`: v5.9.1 — verified via pkg.go.dev versions tab
|
||||||
|
- `golang-migrate/v4`: v4.19.1 — already in go.mod
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Architecture Patterns
|
||||||
|
|
||||||
|
### Recommended Project Structure
|
||||||
|
```
|
||||||
|
pkg/diunwebhook/
|
||||||
|
├── store.go # Store interface (unchanged)
|
||||||
|
├── sqlite_store.go # SQLiteStore (unchanged)
|
||||||
|
├── postgres_store.go # PostgresStore (new)
|
||||||
|
├── migrate.go # Split: RunSQLiteMigrations + RunPostgresMigrations
|
||||||
|
├── migrations/
|
||||||
|
│ ├── sqlite/
|
||||||
|
│ │ ├── 0001_initial_schema.up.sql (exists)
|
||||||
|
│ │ └── 0001_initial_schema.down.sql (exists)
|
||||||
|
│ └── postgres/
|
||||||
|
│ ├── 0001_initial_schema.up.sql (new)
|
||||||
|
│ └── 0001_initial_schema.down.sql (new)
|
||||||
|
├── diunwebhook.go (unchanged)
|
||||||
|
└── export_test.go # Add NewTestPostgresServer (build-tagged)
|
||||||
|
cmd/diunwebhook/
|
||||||
|
└── main.go # Add DATABASE_URL branching
|
||||||
|
compose.yml # Add postgres profile
|
||||||
|
compose.dev.yml # Add postgres profile
|
||||||
|
```
|
||||||
|
|
||||||
|
### Pattern 1: PostgresStore Constructor (no mutex, pool config)
|
||||||
|
|
||||||
|
**What:** Constructor opens pool, sets sensible limits, no mutex (PostgreSQL serializes writes natively).
|
||||||
|
**When to use:** Called from `main.go` when `DATABASE_URL` is present.
|
||||||
|
|
||||||
|
```go
|
||||||
|
// Source: CONTEXT.md D-05, D-06 + established SQLiteStore pattern in sqlite_store.go
|
||||||
|
package diunwebhook
|
||||||
|
|
||||||
|
import (
|
||||||
|
"database/sql"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
type PostgresStore struct {
|
||||||
|
db *sql.DB
|
||||||
|
}
|
||||||
|
|
||||||
|
func NewPostgresStore(db *sql.DB) *PostgresStore {
|
||||||
|
db.SetMaxOpenConns(25)
|
||||||
|
db.SetMaxIdleConns(5)
|
||||||
|
db.SetConnMaxLifetime(5 * time.Minute)
|
||||||
|
return &PostgresStore{db: db}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Pattern 2: RunPostgresMigrations (separate function, separate embed)
|
||||||
|
|
||||||
|
**What:** A dedicated migration runner for PostgreSQL using `golang-migrate`'s `database/pgx/v5` driver. Mirrors `RunMigrations` (which becomes `RunSQLiteMigrations`) exactly.
|
||||||
|
**When to use:** Called from `main.go` after `sql.Open("pgx", databaseURL)` when `DATABASE_URL` is set.
|
||||||
|
|
||||||
|
Decision D-12 leaves the split-vs-param choice to researcher. **Recommendation: two separate functions** (`RunSQLiteMigrations` and `RunPostgresMigrations`). Rationale: each function has its own `//go:embed` scope, there's no shared logic to deduplicate, and a string-switch approach adds a code path that can fail at runtime. Rename the existing `RunMigrations` to `RunSQLiteMigrations` for symmetry.
|
||||||
|
|
||||||
|
```go
|
||||||
|
// Source: migrate.go (existing pattern) + golang-migrate pgx/v5 docs
|
||||||
|
//go:embed migrations/postgres
|
||||||
|
var postgresMigrations embed.FS
|
||||||
|
|
||||||
|
func RunPostgresMigrations(db *sql.DB) error {
|
||||||
|
src, err := iofs.New(postgresMigrations, "migrations/postgres")
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
driver, err := pgxmigrate.WithInstance(db, &pgxmigrate.Config{})
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
m, err := migrate.NewWithInstance("iofs", src, "pgx5", driver)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if err := m.Up(); err != nil && !errors.Is(err, migrate.ErrNoChange) {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Import alias: `pgxmigrate "github.com/golang-migrate/migrate/v4/database/pgx/v5"`.
|
||||||
|
Driver name string for `NewWithInstance` is `"pgx5"` (matches the registration name in the pgx/v5 driver).
|
||||||
|
|
||||||
|
### Pattern 3: CreateTag — RETURNING id (CRITICAL)
|
||||||
|
|
||||||
|
**What:** PostgreSQL's pgx driver does not support `LastInsertId()`. `CreateTag` must use `QueryRow` with `RETURNING id`.
|
||||||
|
**When to use:** In every `PostgresStore.CreateTag` implementation — this is the most error-prone difference from SQLiteStore.
|
||||||
|
|
||||||
|
```go
|
||||||
|
// Source: pgx issue #1483 + pkg.go.dev pgx/v5/stdlib docs
|
||||||
|
func (s *PostgresStore) CreateTag(name string) (Tag, error) {
|
||||||
|
var id int
|
||||||
|
err := s.db.QueryRow(
|
||||||
|
`INSERT INTO tags (name) VALUES ($1) RETURNING id`, name,
|
||||||
|
).Scan(&id)
|
||||||
|
if err != nil {
|
||||||
|
return Tag{}, err
|
||||||
|
}
|
||||||
|
return Tag{ID: id, Name: name}, nil
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Pattern 4: AssignTag — ON CONFLICT DO UPDATE (replaces INSERT OR REPLACE)
|
||||||
|
|
||||||
|
**What:** PostgreSQL does not have `INSERT OR REPLACE`. Use `INSERT ... ON CONFLICT (image) DO UPDATE SET tag_id = EXCLUDED.tag_id`.
|
||||||
|
**When to use:** `PostgresStore.AssignTag`.
|
||||||
|
|
||||||
|
```go
|
||||||
|
// Source: CONTEXT.md D-04
|
||||||
|
_, err := s.db.Exec(
|
||||||
|
`INSERT INTO tag_assignments (image, tag_id) VALUES ($1, $2)
|
||||||
|
ON CONFLICT (image) DO UPDATE SET tag_id = EXCLUDED.tag_id`,
|
||||||
|
image, tagID,
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Pattern 5: main.go DATABASE_URL branching
|
||||||
|
|
||||||
|
```go
|
||||||
|
// Source: CONTEXT.md D-07, D-08, D-09
|
||||||
|
databaseURL := os.Getenv("DATABASE_URL")
|
||||||
|
var store diun.Store
|
||||||
|
if databaseURL != "" {
|
||||||
|
db, err := sql.Open("pgx", databaseURL)
|
||||||
|
if err != nil {
|
||||||
|
log.Fatalf("sql.Open postgres: %v", err)
|
||||||
|
}
|
||||||
|
if err := diun.RunPostgresMigrations(db); err != nil {
|
||||||
|
log.Fatalf("RunPostgresMigrations: %v", err)
|
||||||
|
}
|
||||||
|
store = diun.NewPostgresStore(db)
|
||||||
|
log.Println("Using PostgreSQL database")
|
||||||
|
} else {
|
||||||
|
dbPath := os.Getenv("DB_PATH")
|
||||||
|
if dbPath == "" {
|
||||||
|
dbPath = "./diun.db"
|
||||||
|
}
|
||||||
|
db, err := sql.Open("sqlite", dbPath)
|
||||||
|
if err != nil {
|
||||||
|
log.Fatalf("sql.Open sqlite: %v", err)
|
||||||
|
}
|
||||||
|
if err := diun.RunSQLiteMigrations(db); err != nil {
|
||||||
|
log.Fatalf("RunSQLiteMigrations: %v", err)
|
||||||
|
}
|
||||||
|
store = diun.NewSQLiteStore(db)
|
||||||
|
log.Printf("Using SQLite database at %s", dbPath)
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Add `_ "github.com/jackc/pgx/v5/stdlib"` import to `main.go` (blank import registers the `"pgx"` driver name).
|
||||||
|
|
||||||
|
### Pattern 6: Docker Compose postgres profile
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
# compose.yml — adds postgres profile without breaking default SQLite deploy
|
||||||
|
services:
|
||||||
|
app:
|
||||||
|
image: gitea.jeanlucmakiola.de/makiolaj/diundashboard:latest
|
||||||
|
ports:
|
||||||
|
- "8080:8080"
|
||||||
|
environment:
|
||||||
|
- WEBHOOK_SECRET=${WEBHOOK_SECRET:-}
|
||||||
|
- PORT=${PORT:-8080}
|
||||||
|
- DB_PATH=/data/diun.db
|
||||||
|
- DATABASE_URL=${DATABASE_URL:-}
|
||||||
|
volumes:
|
||||||
|
- diun-data:/data
|
||||||
|
restart: unless-stopped
|
||||||
|
depends_on:
|
||||||
|
postgres:
|
||||||
|
condition: service_healthy
|
||||||
|
required: false # only enforced when postgres profile is active
|
||||||
|
|
||||||
|
postgres:
|
||||||
|
image: postgres:17-alpine
|
||||||
|
profiles:
|
||||||
|
- postgres
|
||||||
|
environment:
|
||||||
|
POSTGRES_USER: ${POSTGRES_USER:-diun}
|
||||||
|
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-diun}
|
||||||
|
POSTGRES_DB: ${POSTGRES_DB:-diundashboard}
|
||||||
|
volumes:
|
||||||
|
- postgres-data:/var/lib/postgresql/data
|
||||||
|
healthcheck:
|
||||||
|
test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER:-diun}"]
|
||||||
|
interval: 5s
|
||||||
|
timeout: 5s
|
||||||
|
retries: 5
|
||||||
|
start_period: 10s
|
||||||
|
restart: unless-stopped
|
||||||
|
|
||||||
|
volumes:
|
||||||
|
diun-data:
|
||||||
|
postgres-data:
|
||||||
|
profiles:
|
||||||
|
- postgres
|
||||||
|
```
|
||||||
|
|
||||||
|
Activate with: `docker compose --profile postgres up -d`
|
||||||
|
|
||||||
|
### Pattern 7: Build-tagged PostgreSQL integration tests
|
||||||
|
|
||||||
|
```go
|
||||||
|
// Source: CONTEXT.md D-17, D-19 + export_test.go pattern
|
||||||
|
//go:build postgres
|
||||||
|
|
||||||
|
package diunwebhook
|
||||||
|
|
||||||
|
import (
|
||||||
|
"database/sql"
|
||||||
|
"os"
|
||||||
|
_ "github.com/jackc/pgx/v5/stdlib"
|
||||||
|
)
|
||||||
|
|
||||||
|
func NewTestPostgresServer() (*Server, error) {
|
||||||
|
databaseURL := os.Getenv("TEST_DATABASE_URL")
|
||||||
|
if databaseURL == "" {
|
||||||
|
databaseURL = "postgres://diun:diun@localhost:5432/diundashboard_test?sslmode=disable"
|
||||||
|
}
|
||||||
|
db, err := sql.Open("pgx", databaseURL)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if err := RunPostgresMigrations(db); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
store := NewPostgresStore(db)
|
||||||
|
return NewServer(store, ""), nil
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Anti-Patterns to Avoid
|
||||||
|
- **Using `res.LastInsertId()` after `db.Exec`**: pgx does not implement this — returns an error at runtime. Use `QueryRow(...).Scan(&id)` with `RETURNING id` instead.
|
||||||
|
- **Sharing the mutex with PostgresStore**: PostgreSQL handles concurrent writes; adding a mutex is unnecessary and hurts performance.
|
||||||
|
- **Using `INSERT OR REPLACE`**: Not valid PostgreSQL syntax. Use `INSERT ... ON CONFLICT ... DO UPDATE SET`.
|
||||||
|
- **Using `datetime('now')`**: SQLite function — not valid in PostgreSQL. Use `NOW()` or `CURRENT_TIMESTAMP`.
|
||||||
|
- **Using `?` placeholders**: Not valid in PostgreSQL. Use `$1`, `$2`, etc.
|
||||||
|
- **Using `INTEGER PRIMARY KEY AUTOINCREMENT`**: Not valid in PostgreSQL. Use `SERIAL` or `BIGSERIAL`.
|
||||||
|
- **Forgetting `//go:build postgres` on test files**: Without the build tag, the test file will be compiled for all builds — `pgx/v5/stdlib` import will fail on SQLite-only CI runs.
|
||||||
|
- **Calling `RunSQLiteMigrations` on a PostgreSQL connection**: The sqlite migration driver will fail to initialize against a PostgreSQL database.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Don't Hand-Roll
|
||||||
|
|
||||||
|
| Problem | Don't Build | Use Instead | Why |
|
||||||
|
|---------|-------------|-------------|-----|
|
||||||
|
| PostgreSQL migration tracking | Custom `schema_version` table | `golang-migrate/v4/database/pgx/v5` | Handles dirty state, locking, version history, rollbacks — all already solved |
|
||||||
|
| Connection pooling | Custom pool implementation | `database/sql` built-in pool + `pgx/v5/stdlib` | `database/sql` pool is production-grade; pgx stdlib wraps it correctly |
|
||||||
|
| Connection string parsing | Custom URL parser | Pass `DATABASE_URL` directly to `sql.Open("pgx", url)` | pgx parses standard PostgreSQL URI format natively |
|
||||||
|
| Dialect detection at runtime | Inspect driver name at query time | Separate store structs with their own SQL | Runtime dialect switching creates test surface, runtime failures; two structs is simpler |
|
||||||
|
|
||||||
|
**Key insight:** The existing `Store` interface already separates the concern — `PostgresStore` is just another implementation. There is nothing to invent.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Common Pitfalls
|
||||||
|
|
||||||
|
### Pitfall 1: LastInsertId on PostgreSQL
|
||||||
|
**What goes wrong:** `CreateTag` calls `res.LastInsertId()` — pgx returns `ErrNoLastInsertId` at runtime, not compile time.
|
||||||
|
**Why it happens:** The `database/sql` `Result` interface defines `LastInsertId()` but pgx does not support it. SQLite does.
|
||||||
|
**How to avoid:** In `PostgresStore.CreateTag`, use `QueryRow(...RETURNING id...).Scan(&id)` instead of `Exec` + `LastInsertId`.
|
||||||
|
**Warning signs:** Test passes compile, panics or returns error at runtime on tag creation.
|
||||||
|
|
||||||
|
### Pitfall 2: golang-migrate driver name mismatch
|
||||||
|
**What goes wrong:** Passing the wrong database name string to `migrate.NewWithInstance` causes "unknown driver" errors.
|
||||||
|
**Why it happens:** The `golang-migrate/database/pgx/v5` driver registers as `"pgx5"`, not `"pgx"` or `"postgres"`.
|
||||||
|
**How to avoid:** Use `"pgx5"` as the database name arg to `migrate.NewWithInstance("iofs", src, "pgx5", driver)`.
|
||||||
|
**Warning signs:** `migrate.NewWithInstance` returns an error mentioning an unknown driver.
|
||||||
|
|
||||||
|
### Pitfall 3: pgx/v5/stdlib import not registered
|
||||||
|
**What goes wrong:** `sql.Open("pgx", url)` fails with `"unknown driver pgx"`.
|
||||||
|
**Why it happens:** The `"pgx"` driver is only registered when `pgx/v5/stdlib` is imported (blank import side effect).
|
||||||
|
**How to avoid:** Add `_ "github.com/jackc/pgx/v5/stdlib"` to `main.go` and to any test files that open a `"pgx"` connection.
|
||||||
|
**Warning signs:** Runtime error "unknown driver pgx" despite pgx being in go.mod.
|
||||||
|
|
||||||
|
### Pitfall 4: SQLite `migrate.go` import conflict
|
||||||
|
**What goes wrong:** Adding the pgx/v5 migrate driver import to `migrate.go` introduces pgx as a dependency of the SQLite migration path.
|
||||||
|
**Why it happens:** Go imports are file-scoped; putting both drivers in one file compiles both.
|
||||||
|
**How to avoid:** Put `RunSQLiteMigrations` and `RunPostgresMigrations` in separate files, or at minimum keep the blank driver import for pgx only in the PostgreSQL branch. Alternatively, keep both in `migrate.go` — both drivers are compiled into the binary regardless; this is a binary size trade-off, not a correctness issue.
|
||||||
|
**Warning signs:** `modernc.org/sqlite` and `pgx` both appear in a file that should only need one.
|
||||||
|
|
||||||
|
### Pitfall 5: Docker Compose `required: false` on depends_on
|
||||||
|
**What goes wrong:** `app` service fails to start when postgres profile is inactive because `depends_on.postgres` is unconditional.
|
||||||
|
**Why it happens:** `depends_on` without `required: false` makes the dependency mandatory even when the postgres profile is not active.
|
||||||
|
**How to avoid:** Use `depends_on.postgres.required: false` so the health check dependency is only enforced when the postgres service is actually started. Requires Docker Compose v2.20+.
|
||||||
|
**Warning signs:** `docker compose up` (no profile) fails with "service postgres not found".
|
||||||
|
|
||||||
|
### Pitfall 6: GetUpdates timestamp scanning differences
|
||||||
|
**What goes wrong:** `GetUpdates` scans `received_at` and `created` as strings (`createdStr`, `receivedStr`) and then calls `time.Parse(time.RFC3339, ...)`. In the PostgreSQL schema these columns are `TEXT` (by design), so scanning behaves the same. If someone types them as `TIMESTAMPTZ` instead, scanning into a string breaks.
|
||||||
|
**Why it happens:** The SQLiteStore scans timestamps as strings because SQLite stores them as TEXT. If the PostgreSQL migration uses `TEXT` for these columns (matching the SQLite schema), the existing scan logic works unchanged in `PostgresStore`.
|
||||||
|
**How to avoid:** Use `TEXT NOT NULL` for `received_at`, `acknowledged_at`, and `created` in the PostgreSQL migration, mirroring the SQLite schema exactly. Do not use `TIMESTAMPTZ` unless you also update the scan/format logic.
|
||||||
|
**Warning signs:** `sql: Scan error ... converting driver.Value type time.Time into *string`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Code Examples
|
||||||
|
|
||||||
|
### PostgreSQL baseline migration (0001_initial_schema.up.sql)
|
||||||
|
```sql
|
||||||
|
-- Source: sqlite/0001_initial_schema.up.sql translated to PostgreSQL dialect
|
||||||
|
CREATE TABLE IF NOT EXISTS updates (
|
||||||
|
image TEXT PRIMARY KEY,
|
||||||
|
diun_version TEXT NOT NULL DEFAULT '',
|
||||||
|
hostname TEXT NOT NULL DEFAULT '',
|
||||||
|
status TEXT NOT NULL DEFAULT '',
|
||||||
|
provider TEXT NOT NULL DEFAULT '',
|
||||||
|
hub_link TEXT NOT NULL DEFAULT '',
|
||||||
|
mime_type TEXT NOT NULL DEFAULT '',
|
||||||
|
digest TEXT NOT NULL DEFAULT '',
|
||||||
|
created TEXT NOT NULL DEFAULT '',
|
||||||
|
platform TEXT NOT NULL DEFAULT '',
|
||||||
|
ctn_name TEXT NOT NULL DEFAULT '',
|
||||||
|
ctn_id TEXT NOT NULL DEFAULT '',
|
||||||
|
ctn_state TEXT NOT NULL DEFAULT '',
|
||||||
|
ctn_status TEXT NOT NULL DEFAULT '',
|
||||||
|
received_at TEXT NOT NULL,
|
||||||
|
acknowledged_at TEXT
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS tags (
|
||||||
|
id SERIAL PRIMARY KEY,
|
||||||
|
name TEXT NOT NULL UNIQUE
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS tag_assignments (
|
||||||
|
image TEXT PRIMARY KEY,
|
||||||
|
tag_id INTEGER NOT NULL REFERENCES tags(id) ON DELETE CASCADE
|
||||||
|
);
|
||||||
|
```
|
||||||
|
|
||||||
|
Key differences from SQLite version:
|
||||||
|
- `SERIAL PRIMARY KEY` replaces `INTEGER PRIMARY KEY AUTOINCREMENT`
|
||||||
|
- All other columns are identical (`TEXT` type used throughout)
|
||||||
|
- `ON DELETE CASCADE` is the same — PostgreSQL enforces FK constraints by default (no equivalent of `PRAGMA foreign_keys = ON` needed)
|
||||||
|
|
||||||
|
### PostgreSQL down migration (0001_initial_schema.down.sql)
|
||||||
|
```sql
|
||||||
|
DROP TABLE IF EXISTS tag_assignments;
|
||||||
|
DROP TABLE IF EXISTS tags;
|
||||||
|
DROP TABLE IF EXISTS updates;
|
||||||
|
```
|
||||||
|
Identical to SQLite version.
|
||||||
|
|
||||||
|
### UpsertEvent (PostgreSQL)
|
||||||
|
```go
|
||||||
|
// Positional params $1..$15, acknowledged_at reset to NULL on conflict
|
||||||
|
_, err := s.db.Exec(`
|
||||||
|
INSERT INTO updates (
|
||||||
|
image, diun_version, hostname, status, provider,
|
||||||
|
hub_link, mime_type, digest, created, platform,
|
||||||
|
ctn_name, ctn_id, ctn_state, ctn_status,
|
||||||
|
received_at, acknowledged_at
|
||||||
|
) VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,NULL)
|
||||||
|
ON CONFLICT(image) DO UPDATE SET
|
||||||
|
diun_version = EXCLUDED.diun_version,
|
||||||
|
hostname = EXCLUDED.hostname,
|
||||||
|
status = EXCLUDED.status,
|
||||||
|
provider = EXCLUDED.provider,
|
||||||
|
hub_link = EXCLUDED.hub_link,
|
||||||
|
mime_type = EXCLUDED.mime_type,
|
||||||
|
digest = EXCLUDED.digest,
|
||||||
|
created = EXCLUDED.created,
|
||||||
|
platform = EXCLUDED.platform,
|
||||||
|
ctn_name = EXCLUDED.ctn_name,
|
||||||
|
ctn_id = EXCLUDED.ctn_id,
|
||||||
|
ctn_state = EXCLUDED.ctn_state,
|
||||||
|
ctn_status = EXCLUDED.ctn_status,
|
||||||
|
received_at = EXCLUDED.received_at,
|
||||||
|
acknowledged_at = NULL`,
|
||||||
|
event.Image, event.DiunVersion, ...
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
### AcknowledgeUpdate (PostgreSQL)
|
||||||
|
```go
|
||||||
|
// NOW() replaces datetime('now'), $1 replaces ?
|
||||||
|
res, err := s.db.Exec(`UPDATE updates SET acknowledged_at = NOW() WHERE image = $1`, image)
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## State of the Art
|
||||||
|
|
||||||
|
| Old Approach | Current Approach | When Changed | Impact |
|
||||||
|
|--------------|------------------|--------------|--------|
|
||||||
|
| `lib/pq` (archived) | `pgx/v5/stdlib` | pgx v4→v5, lib/pq archived ~2023 | pgx is now the consensus standard Go PostgreSQL driver |
|
||||||
|
| `golang-migrate database/postgres` (uses lib/pq) | `golang-migrate database/pgx/v5` | golang-migrate added pgx/v5 sub-package | Use the pgx-native driver to avoid a lib/pq dependency |
|
||||||
|
| Single global `RunMigrations` | Separate `RunSQLiteMigrations` / `RunPostgresMigrations` | This phase | Each function owns its embed directive and driver import |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Open Questions
|
||||||
|
|
||||||
|
1. **Rename `RunMigrations` to `RunSQLiteMigrations`**
|
||||||
|
- What we know: `RunMigrations` is only called in `main.go` and `export_test.go`. Renaming breaks two call sites.
|
||||||
|
- What's unclear: Whether to rename (consistency) or keep old name and add a new `RunPostgresMigrations` (backward compatible for hypothetical external callers).
|
||||||
|
- Recommendation: Rename to `RunSQLiteMigrations` — this is internal-only code and symmetry aids comprehension. Update the two call sites.
|
||||||
|
|
||||||
|
2. **`depends_on.required: false` Docker Compose version requirement**
|
||||||
|
- What we know: `required: false` under `depends_on` was added in Docker Compose v2.20.
|
||||||
|
- What's unclear: Whether the target deployment environment has Compose v2.20+. Docker 29.0.0 (confirmed present) ships with Compose v2.29+ — this is not a concern for the dev machine. Production deployments depend on the user's Docker version.
|
||||||
|
- Recommendation: Use `required: false`; document minimum Docker Compose v2.20 in compose.yml comment.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Environment Availability
|
||||||
|
|
||||||
|
| Dependency | Required By | Available | Version | Fallback |
|
||||||
|
|------------|------------|-----------|---------|----------|
|
||||||
|
| Docker | Compose postgres profile, integration tests | ✓ | 29.0.0 | — |
|
||||||
|
| PostgreSQL server | Integration test execution (`-tags postgres`) | ✗ | — | Tests skip via build tag; Docker Compose spins up postgres for CI |
|
||||||
|
| `pg_isready` / psql client | Health check inside postgres container | ✗ (host) | — | `pg_isready` is inside the `postgres:17-alpine` image — not needed on host |
|
||||||
|
| Go 1.26 | Build | Not directly measurable from this shell | go.mod specifies 1.26 | — |
|
||||||
|
|
||||||
|
**Missing dependencies with no fallback:**
|
||||||
|
- None that block development. PostgreSQL integration tests require a live server but are gated behind `//go:build postgres`.
|
||||||
|
|
||||||
|
**Missing dependencies with fallback:**
|
||||||
|
- PostgreSQL server (host): not installed, but not required — tests use build tags, Docker Compose provides the server for integration runs.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Project Constraints (from CLAUDE.md)
|
||||||
|
|
||||||
|
Directives the planner must verify compliance with:
|
||||||
|
|
||||||
|
- **No CGO**: `CGO_ENABLED=0` in Dockerfile Stage 2. `pgx/v5` is pure Go — this constraint is satisfied. Verify that adding `pgx/v5` does not transitively pull in any CGO package.
|
||||||
|
- **Pure Go SQLite driver**: `modernc.org/sqlite` must remain. Adding pgx does not replace it — both coexist.
|
||||||
|
- **Database must support both SQLite and PostgreSQL**: This is exactly what Phase 3 delivers via the Store interface.
|
||||||
|
- **`database/sql` abstraction**: Both stores use `*sql.DB`. No pgx native interface in handlers.
|
||||||
|
- **`net/http` only, no router framework**: No impact from this phase.
|
||||||
|
- **`gofmt` enforced**: All new `.go` files must be `gofmt`-clean.
|
||||||
|
- **Naming conventions**: New file `postgres_store.go`, new type `PostgresStore`, new constructor `NewPostgresStore`. Test helper `NewTestPostgresServer`. Functions `RunSQLiteMigrations` / `RunPostgresMigrations`.
|
||||||
|
- **Error handling**: `http.Error(w, ..., status)` with lowercase messages. Not directly affected — PostgresStore is storage-layer only. `log.Fatalf` in `main.go` for connection/migration failures (matches existing pattern).
|
||||||
|
- **No global state**: `PostgresStore` holds `*sql.DB` as struct field, no package-level vars — consistent with Phase 2 refactor.
|
||||||
|
- **GSD workflow**: Do not make direct edits outside a GSD phase.
|
||||||
|
- **Module name**: `awesomeProject` (in go.mod). Import as `diun "awesomeProject/pkg/diunwebhook"` in main.go.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Sources
|
||||||
|
|
||||||
|
### Primary (HIGH confidence)
|
||||||
|
- pkg.go.dev/github.com/jackc/pgx/v5 — version confirmed v5.9.1 (Mar 22 2026), stdlib package import path, driver name `"pgx"`, pure Go confirmed
|
||||||
|
- pkg.go.dev/github.com/jackc/pgx/v5/stdlib — `sql.Open("pgx", url)` pattern, `LastInsertId` not supported
|
||||||
|
- pkg.go.dev/github.com/golang-migrate/migrate/v4/database/pgx/v5 — `WithInstance(*sql.DB, *Config)`, driver registers as `"pgx5"`, v4.19.1
|
||||||
|
- github.com/golang-migrate/migrate/blob/master/database/pgx/v5/pgx.go — confirmed `database.Register("pgx5", &db)` registration name
|
||||||
|
- Existing codebase: `store.go`, `sqlite_store.go`, `migrate.go`, `export_test.go`, `main.go` — all read directly
|
||||||
|
|
||||||
|
### Secondary (MEDIUM confidence)
|
||||||
|
- github.com/jackc/pgx/issues/1483 — `LastInsertId` not supported by pgx, confirmed by multiple sources
|
||||||
|
- Docker Compose docs (docs.docker.com/reference/compose-file/services/) — profiles syntax, depends_on with required: false
|
||||||
|
|
||||||
|
### Tertiary (LOW confidence)
|
||||||
|
- WebSearch results re: Docker Compose `required: false` version requirement — states Compose v2.20; not independently verified against official changelog. However, Docker 29.0.0 (installed) ships Compose v2.29+, so this is moot for the dev machine.
|
||||||
|
|
||||||
|
## Metadata
|
||||||
|
|
||||||
|
**Confidence breakdown:**
|
||||||
|
- Standard stack: HIGH — versions verified via pkg.go.dev on 2026-03-24
|
||||||
|
- Architecture: HIGH — based on existing codebase patterns + confirmed library APIs
|
||||||
|
- Pitfalls: HIGH for LastInsertId, driver name, import registration (all verified via official sources); MEDIUM for Docker Compose `required: false` version boundary
|
||||||
|
|
||||||
|
**Research date:** 2026-03-24
|
||||||
|
**Valid until:** 2026-05-24 (stable ecosystem; pgx and golang-migrate release infrequently)
|
||||||
Reference in New Issue
Block a user