Files

Jean-Luc Makiola e4d59d4788 docs: complete project research

2026-03-23 19:45:06 +01:00

19 KiB

Raw Blame History

Architecture Patterns

Domain: Container update dashboard with dual-database support Project: DiunDashboard Researched: 2026-03-23 Confidence: HIGH (based on direct codebase analysis + established Go patterns)

Current Architecture (Before Milestone)

The app is a single monolithic package (pkg/diunwebhook/diunwebhook.go) where database logic and HTTP handlers live in the same file and share package-level globals:

cmd/diunwebhook/main.go
    └── pkg/diunwebhook/diunwebhook.go
            ├── package-level var db *sql.DB      ← global, opaque
            ├── package-level var mu sync.Mutex   ← global, opaque
            ├── InitDB(), UpdateEvent(), GetUpdates() ← storage functions
            └── WebhookHandler, UpdatesHandler, ... ← handlers call db directly

The problem for dual-database support: SQL is written inline in handler functions and storage functions using SQLite-specific syntax:

INSERT OR REPLACE (SQLite only; PostgreSQL uses INSERT ... ON CONFLICT DO UPDATE)
datetime('now') (SQLite only; PostgreSQL uses NOW())
AUTOINCREMENT (SQLite only; PostgreSQL uses SERIAL or GENERATED ALWAYS AS IDENTITY)
PRAGMA foreign_keys = ON (SQLite only; PostgreSQL enforces FKs by default)
modernc.org/sqlite driver import (SQLite only)

There is no abstraction layer. Adding PostgreSQL directly to the current code would mean if dialect == "postgres" branches scattered across 380 lines — unmaintainable.

Recommended Architecture

Core Pattern: Repository Interface

Extract all database operations behind a Go interface. Each database backend implements the interface. The HTTP handlers receive the interface, not a concrete *sql.DB.

cmd/diunwebhook/main.go
    ├── reads DB_DRIVER env var ("sqlite" | "postgres")
    ├── constructs concrete store (SQLiteStore or PostgresStore)
    └── passes store to Server struct

pkg/diunwebhook/
    ├── store.go        ← Store interface definition
    ├── sqlite.go       ← SQLiteStore implements Store
    ├── postgres.go     ← PostgresStore implements Store
    ├── server.go       ← Server struct holds Store, secret; methods = handlers
    ├── handlers.go     ← HTTP handler methods on Server (no direct DB access)
    └── models.go       ← DiunEvent, UpdateEntry, Tag structs

The Store Interface

// pkg/diunwebhook/store.go

type Store interface {
    // Lifecycle
    Close() error

    // Updates
    UpsertEvent(ctx context.Context, event DiunEvent) error
    GetAllUpdates(ctx context.Context) (map[string]UpdateEntry, error)
    AcknowledgeUpdate(ctx context.Context, image string) (found bool, err error)
    AcknowledgeAll(ctx context.Context) error
    AcknowledgeByTag(ctx context.Context, tagID int) error

    // Tags
    ListTags(ctx context.Context) ([]Tag, error)
    CreateTag(ctx context.Context, name string) (Tag, error)
    DeleteTag(ctx context.Context, id int) (found bool, err error)

    // Tag assignments
    AssignTag(ctx context.Context, image string, tagID int) error
    UnassignTag(ctx context.Context, image string) error
}

Why this interface boundary:

Handlers never import a database driver — they only call Store methods.
Tests inject a fake/in-memory implementation with no database.
Adding a third backend (e.g., MySQL) requires implementing the interface, not modifying handlers.
The interface expresses domain intent (AcknowledgeUpdate) not SQL mechanics (UPDATE SET acknowledged_at).

Server Struct (Replaces Package Globals)

// pkg/diunwebhook/server.go

type Server struct {
    store  Store
    secret string
}

func NewServer(store Store, secret string) *Server {
    return &Server{store: store, secret: secret}
}

// Handler methods: func (s *Server) WebhookHandler(w http.ResponseWriter, r *http.Request)

This addresses the "global mutable state" concern in CONCERNS.md. Multiple instances can coexist (useful for tests). Tests construct NewServer(fakeStore, "") without touching a real database.

Component Boundaries

Component	Responsibility	Communicates With	Location
`main.go`	Read env vars, construct store, wire server, run HTTP	`Server`, `SQLiteStore` or `PostgresStore`	`cmd/diunwebhook/`
`Server`	HTTP request lifecycle: parse, validate, delegate, respond	`Store` interface	`pkg/diunwebhook/server.go`
`Store` interface	Contract for all persistence operations	Implemented by `SQLiteStore`, `PostgresStore`	`pkg/diunwebhook/store.go`
`SQLiteStore`	All SQLite-specific SQL, schema init, migrations	`database/sql` + `modernc.org/sqlite`	`pkg/diunwebhook/sqlite.go`
`PostgresStore`	All PostgreSQL-specific SQL, schema init, migrations	`database/sql` + `pgx` stdlib driver	`pkg/diunwebhook/postgres.go`
`models.go`	Shared data structs (`DiunEvent`, `UpdateEntry`, `Tag`)	Imported by all components	`pkg/diunwebhook/models.go`
Frontend SPA	Visual dashboard, REST polling, drag-and-drop	HTTP API only (`/api/*`)	`frontend/src/`

Strict boundary rules:

Server never imports modernc.org/sqlite or pgx — only Store.
SQLiteStore and PostgresStore never import net/http.
main.go is the only place that chooses which backend to construct.
models.go has zero imports beyond stdlib.

Data Flow

Webhook Ingestion

DIUN (external)
  POST /webhook
    → Server.WebhookHandler
        → validate auth header (constant-time compare)
        → decode JSON into DiunEvent
        → store.UpsertEvent(ctx, event)
            → SQLiteStore: INSERT INTO updates ... ON CONFLICT(image) DO UPDATE SET ...
              OR
            → PostgresStore: INSERT INTO updates ... ON CONFLICT (image) DO UPDATE SET ...
        → 200 OK

Both backends use standard SQL UPSERT syntax (fixing the current INSERT OR REPLACE bug). The SQL differs only in timestamp functions and driver-specific syntax, isolated to each store file.

Dashboard Polling

Browser (every 5s)
  GET /api/updates
    → Server.UpdatesHandler
        → store.GetAllUpdates(ctx)
            → SQLiteStore: SELECT ... LEFT JOIN ... (SQLite datetime handling)
              OR
            → PostgresStore: SELECT ... LEFT JOIN ... (PostgreSQL timestamp handling)
        → encode map[string]UpdateEntry as JSON
        → 200 OK with body

Acknowledge Flow

Browser click
  PATCH /api/updates/{image}
    → Server.DismissHandler
        → extract image from URL path
        → store.AcknowledgeUpdate(ctx, image)
            → SQLiteStore: UPDATE ... SET acknowledged_at = datetime('now') WHERE image = ?
              OR
            → PostgresStore: UPDATE ... SET acknowledged_at = NOW() WHERE image = $1
        → if not found: 404; else 204

Startup / Initialization

main()
  → read DB_DRIVER env var ("sqlite" default, "postgres" opt-in)
  → if sqlite:  NewSQLiteStore(DB_PATH)   → opens modernc/sqlite, runs migrations
  → if postgres: NewPostgresStore(DSN)     → opens pgx driver, runs migrations
  → NewServer(store, WEBHOOK_SECRET)
  → register handler methods on mux
  → srv.ListenAndServe()

Migration Strategy: Dual Schema Management

Each store manages its own schema independently. No shared migration runner.

SQLiteStore migrations

func (s *SQLiteStore) migrate() error {
    // Enable FK enforcement (fixes current bug)
    s.db.Exec("PRAGMA foreign_keys = ON")
    // Create tables with IF NOT EXISTS
    // Apply ALTER TABLE migrations with error-ignore for idempotency
    // Future: schema_version table for tracked migrations
}

PostgresStore migrations

func (s *PostgresStore) migrate() error {
    // CREATE TABLE IF NOT EXISTS with PostgreSQL syntax
    // SERIAL or IDENTITY for auto-increment
    // FK enforcement is on by default — no PRAGMA needed
    // Timestamp columns as TIMESTAMPTZ not TEXT
    // Future: schema_version table for tracked migrations
}

Key difference: SQLite stores timestamps as RFC3339 TEXT (current behavior, must be preserved for backward compatibility). PostgreSQL stores timestamps as TIMESTAMPTZ. Each store handles its own serialization/deserialization of time.Time.

Patterns to Follow

Pattern 1: Constructor-Injected Store

What: NewServer(store Store, secret string) — store is a parameter, not a global.

When: Always. This replaces var db *sql.DB and var mu sync.Mutex package globals.

Why: Enables parallel test execution (each test creates its own Server with its own store). Eliminates the "single instance per process" constraint documented in CONCERNS.md.

Pattern 2: Context Propagation

What: All Store interface methods accept context.Context as first argument.

When: From the initial Store interface design — do not add it later.

Why: Enables request cancellation and timeout propagation. PostgreSQL's pgx driver uses context natively. Without context, long-running queries cannot be cancelled when the client disconnects.

Pattern 3: Driver-Specific SQL Isolated in Store Files

What: Each store file contains all SQL for that backend. No SQL strings in handlers.

When: Any time a handler needs to read or write data — call a Store method instead.

Why: SQLite uses ? placeholders; PostgreSQL uses $1, $2. SQLite uses datetime('now'); PostgreSQL uses NOW(). SQLite uses INTEGER PRIMARY KEY AUTOINCREMENT; PostgreSQL uses BIGSERIAL. Mixing these in handler code creates unmaintainable conditional branches.

Pattern 4: Idempotent Schema Creation

What: Both store constructors run schema setup on every startup via CREATE TABLE IF NOT EXISTS.

When: In NewSQLiteStore() and NewPostgresStore() constructors.

Why: Preserves current behavior where existing databases are safely upgraded. No external migration tool required for the current scope.

Anti-Patterns to Avoid

Anti-Pattern 1: Dialect Switches in Handlers

What: if s.dialect == "postgres" { query = "..." } else { query = "..." } inside handler methods.

Why bad: Handlers become aware of database internals. Every handler must be updated when adding a new backend. Tests must cover both branches per handler.

Instead: Move all dialect differences into the Store implementation. Handlers call store.AcknowledgeUpdate(ctx, image) — they never see SQL.

Anti-Pattern 2: Shared `database/sql` Pool Exposed to Handlers

What: Passing *sql.DB directly to handlers (as the current package globals effectively do).

Why bad: Handlers can write arbitrary SQL, bypassing any abstraction. Type system cannot enforce the boundary.

Instead: Expose only the Store interface to Server. The *sql.DB is a private field of SQLiteStore / PostgresStore.

Anti-Pattern 3: Single Store File for Both Backends

What: One store.go file with SQLite and PostgreSQL implementations side by side.

Why bad: The two implementations use different drivers, different SQL syntax, different connection setup. Merging them creates a large file with low cohesion.

Instead: sqlite.go for SQLiteStore, postgres.go for PostgresStore. Both in pkg/diunwebhook/ package. Build tags are not needed since both compile — main.go chooses at runtime.

Anti-Pattern 4: Reusing the Mutex from the Current Code

What: Keeping var mu sync.Mutex as a package global once the Store abstraction is introduced.

Why bad: SQLiteStore needs its own mutex (SQLite single-writer limitation). PostgresStore does not — PostgreSQL has its own concurrency control. Sharing a mutex across backends is wrong for Postgres and forces a false constraint.

Instead: SQLiteStore embeds sync.Mutex as a private field. PostgresStore does not use a mutex — it relies on pgx's connection pool.

Suggested Build Order

The dependency graph dictates this order. Each step must complete before the next.

Step 1: Fix Current SQLite Bugs (prerequisite)

Fix INSERT OR REPLACE → proper UPSERT, add PRAGMA foreign_keys = ON. These bugs exist independent of the refactor and will be harder to fix correctly after the abstraction layer is introduced. Do this on the current flat code, with tests confirming the fix.

Rationale: Existing users rely on SQLite working correctly. The refactor must not change behavior — fixing bugs before refactoring means the tests that pass after bugfix become the regression suite for the refactor.

Step 2: Extract Models

Move DiunEvent, UpdateEntry, Tag into models.go. No logic changes. This is a safe mechanical split — confirms the package compiles and tests pass after file reorganization.

Rationale: Models are referenced by both Store implementations and by Server. Extracting them first removes a coupling that would otherwise force all files to reference a single monolith.

Step 3: Define Store Interface + SQLiteStore

Define the Store interface in store.go. Implement SQLiteStore in sqlite.go by moving all SQL from the current monolith into SQLiteStore methods. All existing tests must still pass with zero behavior changes. This step does not add PostgreSQL — it only restructures.

Rationale: Restructuring and new backend introduction must be separate commits. If tests break, the cause is isolated to the refactor, not the PostgreSQL code.

Step 4: Introduce Server Struct

Refactor pkg/diunwebhook/ to a struct-based design: Server with injected Store. Update main.go to construct NewServer(store, secret) and register s.WebhookHandler etc. on the mux. All existing tests must still pass.

Rationale: This decouples handler tests from database initialization. Tests can now construct a Server with a stub Store — faster, no filesystem I/O, parallelisable.

Step 5: Implement PostgresStore

Add postgres.go with PostgresStore implementing the Store interface. Add pgx (github.com/jackc/pgx/v5) as a dependency using its database/sql compatibility shim (pgx/v5/stdlib) to avoid changing the *sql.DB usage pattern in SQLiteStore. Add DB_DRIVER env var to main.go — "sqlite" (default) or "postgres". Add DATABASE_URL env var for PostgreSQL DSN. Update compose.dev.yml and deployment docs.

Rationale: pgx/v5/stdlib registers as a database/sql driver, so PostgresStore can use the same *sql.DB API as SQLiteStore. This minimizes the interface surface difference between the two implementations.

Step 6: Update Docker Compose and Configuration Docs

Update compose.dev.yml with a postgres service profile. Update deployment documentation for PostgreSQL setup. This is explicitly the last step — infrastructure follows working code.

Scalability Considerations

Concern	SQLite (current)	PostgreSQL (new)
Concurrent writes	Serialized by mutex + `SetMaxOpenConns(1)`	Connection pool, DB-level locking
Multiple server instances	Not possible (file lock)	Supported via shared DSN
Read performance	`LEFT JOIN` on every poll	Same query; can add indexes
Data retention	Unbounded growth	Same; retention policy deferred
Connection management	Single connection	`pgx` pool (default 5 conns)

For the self-hosted single-user target audience, both backends are more than sufficient. PostgreSQL is recommended when the user already runs a PostgreSQL instance (common in Coolify deployments) to avoid volume-mounting complexity and SQLite file permission issues.

Component Interaction Diagram

┌─────────────────────────────────────────────────────────┐
│  cmd/diunwebhook/main.go                                │
│                                                         │
│  DB_DRIVER=sqlite  → NewSQLiteStore(DB_PATH)           │
│  DB_DRIVER=postgres → NewPostgresStore(DATABASE_URL)   │
│                          │                              │
│  NewServer(store, secret)│                              │
└──────────────────────────┼──────────────────────────────┘
                           │
                           ▼
┌──────────────────────────────────────────┐
│  Server (pkg/diunwebhook/server.go)      │
│                                          │
│  store Store  ◄──── interface boundary  │
│  secret string                           │
│                                          │
│  .WebhookHandler()                       │
│  .UpdatesHandler()                       │
│  .DismissHandler()                       │
│  .TagsHandler()                          │
│  .TagByIDHandler()                       │
│  .TagAssignmentHandler()                 │
└──────────────┬───────────────────────────┘
               │ calls Store methods only
               ▼
┌──────────────────────────────────────────┐
│  Store interface (store.go)              │
│  UpsertEvent / GetAllUpdates /           │
│  AcknowledgeUpdate / ListTags / ...      │
└────────────┬─────────────────┬───────────┘
             │                 │
             ▼                 ▼
┌────────────────────┐  ┌──────────────────────┐
│  SQLiteStore       │  │  PostgresStore        │
│  (sqlite.go)       │  │  (postgres.go)        │
│                    │  │                       │
│  modernc.org/sqlite│  │  pgx/v5/stdlib        │
│  *sql.DB           │  │  *sql.DB              │
│  sync.Mutex        │  │  (no mutex needed)    │
│  SQLite SQL syntax │  │  PostgreSQL SQL syntax│
└────────────────────┘  └──────────────────────┘

Sources

Direct analysis of pkg/diunwebhook/diunwebhook.go (current monolith) — HIGH confidence
Direct analysis of cmd/diunwebhook/main.go (entry point) — HIGH confidence
.planning/codebase/CONCERNS.md (identified tech debt) — HIGH confidence
.planning/PROJECT.md (constraints: no CGO, backward compat, dual DB) — HIGH confidence
Go database/sql standard library interface pattern — HIGH confidence (well-established Go idiom)
pgx/v5/stdlib compatibility layer for database/sql — MEDIUM confidence (standard approach, verify exact import path during implementation)

Architecture research: 2026-03-23

19 KiB Raw Blame History