docs(25-01): complete catalog enrichment data layer plan
- SUMMARY.md: attribution columns, upsert service, Zod schemas - STATE.md: advance to plan 2, add decisions - ROADMAP.md: update phase 25 progress - REQUIREMENTS.md: mark CATL-01, CATL-02, CATL-05 complete
This commit is contained in:
@@ -0,0 +1,132 @@
|
||||
---
|
||||
phase: 25-catalog-enrichment-agent-tools
|
||||
plan: 01
|
||||
subsystem: database
|
||||
tags: [drizzle, postgres, zod, catalog, upsert, attribution]
|
||||
|
||||
# Dependency graph
|
||||
requires: []
|
||||
provides:
|
||||
- globalItems table with sourceUrl, imageCredit, imageSourceUrl attribution columns
|
||||
- unique constraint on (brand, model) in globalItems table
|
||||
- migration 0003_loving_serpent_society.sql
|
||||
- upsertGlobalItemSchema and bulkUpsertGlobalItemsSchema Zod schemas
|
||||
- UpsertGlobalItemInput and BulkUpsertGlobalItemsInput TypeScript types
|
||||
- upsertGlobalItem service function with tag sync
|
||||
- bulkUpsertGlobalItems service function with transaction atomicity
|
||||
affects:
|
||||
- 25-02 (HTTP routes and MCP tools will call these service functions)
|
||||
|
||||
# Tech tracking
|
||||
tech-stack:
|
||||
added: []
|
||||
patterns:
|
||||
- onConflictDoUpdate with multi-column target for brand+model upsert
|
||||
- syncGlobalItemTags helper using delete-then-insert in transaction
|
||||
- tag create-if-not-exists via onConflictDoUpdate on tags.name
|
||||
|
||||
key-files:
|
||||
created:
|
||||
- drizzle-pg/0003_loving_serpent_society.sql
|
||||
modified:
|
||||
- src/db/schema.ts
|
||||
- src/shared/schemas.ts
|
||||
- src/shared/types.ts
|
||||
- src/server/services/global-item.service.ts
|
||||
- tests/services/global-item.service.test.ts
|
||||
|
||||
key-decisions:
|
||||
- "Unique constraint on (brand, model): enables safe ON CONFLICT DO UPDATE for catalog enrichment"
|
||||
- "Tags sync: undefined=leave untouched, []=clear all, [names]=replace — three-way tag handling"
|
||||
- "Migration 0003 fixed: drizzle-kit generated spurious duplicate DDL; trimmed to only new changes"
|
||||
|
||||
patterns-established:
|
||||
- "upsertGlobalItem pattern: check existence before upsert to track created vs updated"
|
||||
- "syncGlobalItemTags: delete existing links, then create-if-not-exists tags and insert links"
|
||||
|
||||
requirements-completed: [CATL-01, CATL-02, CATL-05]
|
||||
|
||||
# Metrics
|
||||
duration: 3min
|
||||
completed: 2026-04-10
|
||||
---
|
||||
|
||||
# Phase 25 Plan 01: Catalog Enrichment Data Layer Summary
|
||||
|
||||
**globalItems attribution columns (sourceUrl, imageCredit, imageSourceUrl) with unique(brand, model) constraint, upsertGlobalItem/bulkUpsertGlobalItems service functions, and Zod schemas — 21 tests passing**
|
||||
|
||||
## Performance
|
||||
|
||||
- **Duration:** ~3 min
|
||||
- **Started:** 2026-04-10T08:55:26Z
|
||||
- **Completed:** 2026-04-10T08:58:39Z
|
||||
- **Tasks:** 2
|
||||
- **Files modified:** 5
|
||||
|
||||
## Accomplishments
|
||||
|
||||
- Added three attribution columns to globalItems table with unique(brand, model) constraint and generated migration
|
||||
- Implemented upsertGlobalItem with onConflictDoUpdate, three-way tag sync, and created/updated tracking
|
||||
- Implemented bulkUpsertGlobalItems processing arrays in a single atomic transaction
|
||||
- Defined upsertGlobalItemSchema and bulkUpsertGlobalItemsSchema Zod validation schemas
|
||||
- All 21 tests pass (13 pre-existing + 8 new upsert operation tests)
|
||||
|
||||
## Task Commits
|
||||
|
||||
Each task was committed atomically:
|
||||
|
||||
1. **Task 1: Schema migration — attribution columns + unique constraint** - `39ef9cc` (feat)
|
||||
2. **Task 2: TDD RED — failing tests** - `9093a2c` (test)
|
||||
3. **Task 2: TDD GREEN — Zod schemas, service functions, tests passing** - `c8ebbf8` (feat)
|
||||
|
||||
_Note: TDD tasks have multiple commits (test RED → feat GREEN)_
|
||||
|
||||
## Files Created/Modified
|
||||
|
||||
- `src/db/schema.ts` - Added sourceUrl, imageCredit, imageSourceUrl columns and unique().on(brand, model) constraint to globalItems
|
||||
- `drizzle-pg/0003_loving_serpent_society.sql` - Migration adding 3 columns + unique constraint
|
||||
- `src/shared/schemas.ts` - Added upsertGlobalItemSchema and bulkUpsertGlobalItemsSchema
|
||||
- `src/shared/types.ts` - Added UpsertGlobalItemInput and BulkUpsertGlobalItemsInput types
|
||||
- `src/server/services/global-item.service.ts` - Added upsertGlobalItem, bulkUpsertGlobalItems, syncGlobalItemTags
|
||||
- `tests/services/global-item.service.test.ts` - Added 8 upsert operation tests
|
||||
|
||||
## Decisions Made
|
||||
|
||||
- **Three-way tag handling**: `undefined` leaves existing tags untouched, `[]` clears all tags, `[names]` replaces tags. This allows callers to selectively update tags without clobbering existing data.
|
||||
- **Unique constraint on (brand, model)**: Required for ON CONFLICT DO UPDATE semantics. Without it, duplicate inserts would fail rather than update.
|
||||
- **Created/updated tracking via pre-check**: The service checks for existing row before upsert to accurately report created vs updated counts, since ON CONFLICT DO UPDATE doesn't distinguish via returning rows alone.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
### Auto-fixed Issues
|
||||
|
||||
**1. [Rule 1 - Bug] Fixed drizzle-kit generated spurious duplicate DDL in migration 0003**
|
||||
- **Found during:** Task 1 (schema migration)
|
||||
- **Issue:** drizzle-kit generated a migration that re-created global_item_tags, re-added FKs on items and thread_candidates, and re-added oauth_codes.user_id — all already present in migration 0002. PGlite tests failed with "relation already exists".
|
||||
- **Fix:** Trimmed migration 0003 to only include the three new ALTER TABLE ADD COLUMN statements and the unique constraint.
|
||||
- **Files modified:** drizzle-pg/0003_loving_serpent_society.sql
|
||||
- **Verification:** 21 tests pass including all new upsert tests
|
||||
- **Committed in:** c8ebbf8 (Task 2 feat commit)
|
||||
|
||||
---
|
||||
|
||||
**Total deviations:** 1 auto-fixed (Rule 1 - bug in generated migration)
|
||||
**Impact on plan:** Fix was necessary for test correctness. No scope creep.
|
||||
|
||||
## Issues Encountered
|
||||
|
||||
- drizzle-kit migration generation included duplicate DDL from prior migrations — likely a state tracking issue in the drizzle-kit snapshots. Fixed by manually editing the migration to contain only the new changes.
|
||||
|
||||
## User Setup Required
|
||||
|
||||
None - no external service configuration required. Production database push (`bun run db:push`) will apply the migration when the database is available.
|
||||
|
||||
## Next Phase Readiness
|
||||
|
||||
- Data layer complete: globalItems has attribution columns, unique constraint, and upsert service functions
|
||||
- Plan 02 (HTTP routes + MCP tools) can now import upsertGlobalItem and bulkUpsertGlobalItems directly
|
||||
- No blockers
|
||||
|
||||
---
|
||||
*Phase: 25-catalog-enrichment-agent-tools*
|
||||
*Completed: 2026-04-10*
|
||||
Reference in New Issue
Block a user