Jean-Luc Makiola
9dbf019466
- SUMMARY.md: attribution columns, upsert service, Zod schemas - STATE.md: advance to plan 2, add decisions - ROADMAP.md: update phase 25 progress - REQUIREMENTS.md: mark CATL-01, CATL-02, CATL-05 complete
133 lines
5.9 KiB
Markdown
133 lines
5.9 KiB
Markdown
---
|
|
phase: 25-catalog-enrichment-agent-tools
|
|
plan: 01
|
|
subsystem: database
|
|
tags: [drizzle, postgres, zod, catalog, upsert, attribution]
|
|
|
|
# Dependency graph
|
|
requires: []
|
|
provides:
|
|
- globalItems table with sourceUrl, imageCredit, imageSourceUrl attribution columns
|
|
- unique constraint on (brand, model) in globalItems table
|
|
- migration 0003_loving_serpent_society.sql
|
|
- upsertGlobalItemSchema and bulkUpsertGlobalItemsSchema Zod schemas
|
|
- UpsertGlobalItemInput and BulkUpsertGlobalItemsInput TypeScript types
|
|
- upsertGlobalItem service function with tag sync
|
|
- bulkUpsertGlobalItems service function with transaction atomicity
|
|
affects:
|
|
- 25-02 (HTTP routes and MCP tools will call these service functions)
|
|
|
|
# Tech tracking
|
|
tech-stack:
|
|
added: []
|
|
patterns:
|
|
- onConflictDoUpdate with multi-column target for brand+model upsert
|
|
- syncGlobalItemTags helper using delete-then-insert in transaction
|
|
- tag create-if-not-exists via onConflictDoUpdate on tags.name
|
|
|
|
key-files:
|
|
created:
|
|
- drizzle-pg/0003_loving_serpent_society.sql
|
|
modified:
|
|
- src/db/schema.ts
|
|
- src/shared/schemas.ts
|
|
- src/shared/types.ts
|
|
- src/server/services/global-item.service.ts
|
|
- tests/services/global-item.service.test.ts
|
|
|
|
key-decisions:
|
|
- "Unique constraint on (brand, model): enables safe ON CONFLICT DO UPDATE for catalog enrichment"
|
|
- "Tags sync: undefined=leave untouched, []=clear all, [names]=replace — three-way tag handling"
|
|
- "Migration 0003 fixed: drizzle-kit generated spurious duplicate DDL; trimmed to only new changes"
|
|
|
|
patterns-established:
|
|
- "upsertGlobalItem pattern: check existence before upsert to track created vs updated"
|
|
- "syncGlobalItemTags: delete existing links, then create-if-not-exists tags and insert links"
|
|
|
|
requirements-completed: [CATL-01, CATL-02, CATL-05]
|
|
|
|
# Metrics
|
|
duration: 3min
|
|
completed: 2026-04-10
|
|
---
|
|
|
|
# Phase 25 Plan 01: Catalog Enrichment Data Layer Summary
|
|
|
|
**globalItems attribution columns (sourceUrl, imageCredit, imageSourceUrl) with unique(brand, model) constraint, upsertGlobalItem/bulkUpsertGlobalItems service functions, and Zod schemas — 21 tests passing**
|
|
|
|
## Performance
|
|
|
|
- **Duration:** ~3 min
|
|
- **Started:** 2026-04-10T08:55:26Z
|
|
- **Completed:** 2026-04-10T08:58:39Z
|
|
- **Tasks:** 2
|
|
- **Files modified:** 5
|
|
|
|
## Accomplishments
|
|
|
|
- Added three attribution columns to globalItems table with unique(brand, model) constraint and generated migration
|
|
- Implemented upsertGlobalItem with onConflictDoUpdate, three-way tag sync, and created/updated tracking
|
|
- Implemented bulkUpsertGlobalItems processing arrays in a single atomic transaction
|
|
- Defined upsertGlobalItemSchema and bulkUpsertGlobalItemsSchema Zod validation schemas
|
|
- All 21 tests pass (13 pre-existing + 8 new upsert operation tests)
|
|
|
|
## Task Commits
|
|
|
|
Each task was committed atomically:
|
|
|
|
1. **Task 1: Schema migration — attribution columns + unique constraint** - `39ef9cc` (feat)
|
|
2. **Task 2: TDD RED — failing tests** - `9093a2c` (test)
|
|
3. **Task 2: TDD GREEN — Zod schemas, service functions, tests passing** - `c8ebbf8` (feat)
|
|
|
|
_Note: TDD tasks have multiple commits (test RED → feat GREEN)_
|
|
|
|
## Files Created/Modified
|
|
|
|
- `src/db/schema.ts` - Added sourceUrl, imageCredit, imageSourceUrl columns and unique().on(brand, model) constraint to globalItems
|
|
- `drizzle-pg/0003_loving_serpent_society.sql` - Migration adding 3 columns + unique constraint
|
|
- `src/shared/schemas.ts` - Added upsertGlobalItemSchema and bulkUpsertGlobalItemsSchema
|
|
- `src/shared/types.ts` - Added UpsertGlobalItemInput and BulkUpsertGlobalItemsInput types
|
|
- `src/server/services/global-item.service.ts` - Added upsertGlobalItem, bulkUpsertGlobalItems, syncGlobalItemTags
|
|
- `tests/services/global-item.service.test.ts` - Added 8 upsert operation tests
|
|
|
|
## Decisions Made
|
|
|
|
- **Three-way tag handling**: `undefined` leaves existing tags untouched, `[]` clears all tags, `[names]` replaces tags. This allows callers to selectively update tags without clobbering existing data.
|
|
- **Unique constraint on (brand, model)**: Required for ON CONFLICT DO UPDATE semantics. Without it, duplicate inserts would fail rather than update.
|
|
- **Created/updated tracking via pre-check**: The service checks for existing row before upsert to accurately report created vs updated counts, since ON CONFLICT DO UPDATE doesn't distinguish via returning rows alone.
|
|
|
|
## Deviations from Plan
|
|
|
|
### Auto-fixed Issues
|
|
|
|
**1. [Rule 1 - Bug] Fixed drizzle-kit generated spurious duplicate DDL in migration 0003**
|
|
- **Found during:** Task 1 (schema migration)
|
|
- **Issue:** drizzle-kit generated a migration that re-created global_item_tags, re-added FKs on items and thread_candidates, and re-added oauth_codes.user_id — all already present in migration 0002. PGlite tests failed with "relation already exists".
|
|
- **Fix:** Trimmed migration 0003 to only include the three new ALTER TABLE ADD COLUMN statements and the unique constraint.
|
|
- **Files modified:** drizzle-pg/0003_loving_serpent_society.sql
|
|
- **Verification:** 21 tests pass including all new upsert tests
|
|
- **Committed in:** c8ebbf8 (Task 2 feat commit)
|
|
|
|
---
|
|
|
|
**Total deviations:** 1 auto-fixed (Rule 1 - bug in generated migration)
|
|
**Impact on plan:** Fix was necessary for test correctness. No scope creep.
|
|
|
|
## Issues Encountered
|
|
|
|
- drizzle-kit migration generation included duplicate DDL from prior migrations — likely a state tracking issue in the drizzle-kit snapshots. Fixed by manually editing the migration to contain only the new changes.
|
|
|
|
## User Setup Required
|
|
|
|
None - no external service configuration required. Production database push (`bun run db:push`) will apply the migration when the database is available.
|
|
|
|
## Next Phase Readiness
|
|
|
|
- Data layer complete: globalItems has attribution columns, unique constraint, and upsert service functions
|
|
- Plan 02 (HTTP routes + MCP tools) can now import upsertGlobalItem and bulkUpsertGlobalItems directly
|
|
- No blockers
|
|
|
|
---
|
|
*Phase: 25-catalog-enrichment-agent-tools*
|
|
*Completed: 2026-04-10*
|