docs(25-01): complete catalog enrichment data layer plan
- SUMMARY.md: attribution columns, upsert service, Zod schemas - STATE.md: advance to plan 2, add decisions - ROADMAP.md: update phase 25 progress - REQUIREMENTS.md: mark CATL-01, CATL-02, CATL-05 complete
This commit is contained in:
@@ -25,11 +25,11 @@ Requirements for Public Discovery milestone. Each maps to roadmap phases.
|
||||
|
||||
### Catalog Enrichment
|
||||
|
||||
- [ ] **CATL-01**: Global items have attribution fields (sourceUrl, manufacturer, imageCredit, imageSourceUrl)
|
||||
- [ ] **CATL-02**: Global items have a unique constraint on (brand, model) preventing duplicates
|
||||
- [x] **CATL-01**: Global items have attribution fields (sourceUrl, manufacturer, imageCredit, imageSourceUrl)
|
||||
- [x] **CATL-02**: Global items have a unique constraint on (brand, model) preventing duplicates
|
||||
- [ ] **CATL-03**: Catalog detail pages display image attribution with credit and source link
|
||||
- [ ] **CATL-04**: Bulk import API endpoint accepts multiple catalog items in one request
|
||||
- [ ] **CATL-05**: Bulk import uses upsert semantics (ON CONFLICT update, not fail)
|
||||
- [x] **CATL-05**: Bulk import uses upsert semantics (ON CONFLICT update, not fail)
|
||||
|
||||
### Agent Seeding Tools
|
||||
|
||||
@@ -122,11 +122,11 @@ Which phases cover which requirements. Updated during roadmap creation.
|
||||
| PUBL-04 | Phase 24 | Complete |
|
||||
| PUBL-05 | Phase 24 | Complete |
|
||||
| INFR-01 | Phase 24 | Complete |
|
||||
| CATL-01 | Phase 25 | Pending |
|
||||
| CATL-02 | Phase 25 | Pending |
|
||||
| CATL-01 | Phase 25 | Complete |
|
||||
| CATL-02 | Phase 25 | Complete |
|
||||
| CATL-03 | Phase 25 | Pending |
|
||||
| CATL-04 | Phase 25 | Pending |
|
||||
| CATL-05 | Phase 25 | Pending |
|
||||
| CATL-05 | Phase 25 | Complete |
|
||||
| SEED-01 | Phase 25 | Pending |
|
||||
| SEED-02 | Phase 25 | Pending |
|
||||
| SEED-03 | Phase 25 | Pending |
|
||||
|
||||
@@ -105,7 +105,7 @@ Plans:
|
||||
**Plans**: 2 plans
|
||||
|
||||
Plans:
|
||||
- [ ] 25-01-PLAN.md — Schema migration (attribution columns + unique constraint) and upsert service layer
|
||||
- [x] 25-01-PLAN.md — Schema migration (attribution columns + unique constraint) and upsert service layer
|
||||
- [ ] 25-02-PLAN.md — HTTP upsert routes, MCP catalog tools, and client attribution display
|
||||
|
||||
### Phase 26: Discovery Landing Page
|
||||
@@ -149,7 +149,7 @@ Plans:
|
||||
| 22. Add-from-Catalog & Thread Integration | v2.0 | 2/2 | Complete | 2026-04-06 |
|
||||
| 23. Manual Entry Fallback | v2.0 | 1/1 | Complete | 2026-04-06 |
|
||||
| 24. Public Access & Infrastructure | v2.1 | 2/2 | Complete | 2026-04-10 |
|
||||
| 25. Catalog Enrichment & Agent Tools | v2.1 | 0/2 | Not started | - |
|
||||
| 25. Catalog Enrichment & Agent Tools | v2.1 | 1/2 | In Progress| |
|
||||
| 26. Discovery Landing Page | v2.1 | 0/TBD | Not started | - |
|
||||
|
||||
## Backlog
|
||||
|
||||
@@ -2,15 +2,15 @@
|
||||
gsd_state_version: 1.0
|
||||
milestone: v2.1
|
||||
milestone_name: Public Discovery
|
||||
status: verifying
|
||||
stopped_at: Phase 25 context gathered
|
||||
last_updated: "2026-04-10T08:33:11.968Z"
|
||||
status: executing
|
||||
stopped_at: Completed 25-01-PLAN.md
|
||||
last_updated: "2026-04-10T08:59:46.905Z"
|
||||
last_activity: 2026-04-10
|
||||
progress:
|
||||
total_phases: 6
|
||||
completed_phases: 1
|
||||
total_plans: 2
|
||||
completed_plans: 2
|
||||
total_plans: 4
|
||||
completed_plans: 3
|
||||
percent: 0
|
||||
---
|
||||
|
||||
@@ -21,13 +21,13 @@ progress:
|
||||
See: .planning/PROJECT.md (updated 2026-04-09)
|
||||
|
||||
**Core value:** Help people make better gear decisions — discover what others use, compare real-world data, and see how a potential buy affects your setup before committing.
|
||||
**Current focus:** Phase 24 — public-access-infrastructure
|
||||
**Current focus:** Phase 25 — catalog-enrichment-agent-tools
|
||||
|
||||
## Current Position
|
||||
|
||||
Phase: 999.1
|
||||
Plan: Not started
|
||||
Status: Phase complete — ready for verification
|
||||
Phase: 25 (catalog-enrichment-agent-tools) — EXECUTING
|
||||
Plan: 2 of 2
|
||||
Status: Ready to execute
|
||||
Last activity: 2026-04-10
|
||||
|
||||
Progress: [░░░░░░░░░░] 0%
|
||||
@@ -64,6 +64,8 @@ v2.1 decisions:
|
||||
- [Phase 24-public-access-infrastructure]: Browse tier 120/min, detail tier 60/min — same limits for auth and anon users
|
||||
- [Phase 24]: Both auth prompt CTAs go to /login — Logto handles sign-in and sign-up at the same OIDC endpoint
|
||||
- [Phase 24]: Soft navigate() replaces hard window.location.href for private route redirect — defers until auth resolves
|
||||
- [Phase 25-catalog-enrichment-agent-tools]: Three-way tag sync: undefined=leave untouched, []=clear all, [names]=replace — enables selective tag updates from catalog agents
|
||||
- [Phase 25-catalog-enrichment-agent-tools]: unique(brand, model) constraint on globalItems: enables safe ON CONFLICT DO UPDATE for catalog enrichment agents
|
||||
|
||||
### Pending Todos
|
||||
|
||||
@@ -75,6 +77,6 @@ None.
|
||||
|
||||
## Session Continuity
|
||||
|
||||
Last session: 2026-04-10T08:33:11.966Z
|
||||
Stopped at: Phase 25 context gathered
|
||||
Resume file: .planning/phases/25-catalog-enrichment-agent-tools/25-CONTEXT.md
|
||||
Last session: 2026-04-10T08:59:46.903Z
|
||||
Stopped at: Completed 25-01-PLAN.md
|
||||
Resume file: None
|
||||
|
||||
@@ -0,0 +1,132 @@
|
||||
---
|
||||
phase: 25-catalog-enrichment-agent-tools
|
||||
plan: 01
|
||||
subsystem: database
|
||||
tags: [drizzle, postgres, zod, catalog, upsert, attribution]
|
||||
|
||||
# Dependency graph
|
||||
requires: []
|
||||
provides:
|
||||
- globalItems table with sourceUrl, imageCredit, imageSourceUrl attribution columns
|
||||
- unique constraint on (brand, model) in globalItems table
|
||||
- migration 0003_loving_serpent_society.sql
|
||||
- upsertGlobalItemSchema and bulkUpsertGlobalItemsSchema Zod schemas
|
||||
- UpsertGlobalItemInput and BulkUpsertGlobalItemsInput TypeScript types
|
||||
- upsertGlobalItem service function with tag sync
|
||||
- bulkUpsertGlobalItems service function with transaction atomicity
|
||||
affects:
|
||||
- 25-02 (HTTP routes and MCP tools will call these service functions)
|
||||
|
||||
# Tech tracking
|
||||
tech-stack:
|
||||
added: []
|
||||
patterns:
|
||||
- onConflictDoUpdate with multi-column target for brand+model upsert
|
||||
- syncGlobalItemTags helper using delete-then-insert in transaction
|
||||
- tag create-if-not-exists via onConflictDoUpdate on tags.name
|
||||
|
||||
key-files:
|
||||
created:
|
||||
- drizzle-pg/0003_loving_serpent_society.sql
|
||||
modified:
|
||||
- src/db/schema.ts
|
||||
- src/shared/schemas.ts
|
||||
- src/shared/types.ts
|
||||
- src/server/services/global-item.service.ts
|
||||
- tests/services/global-item.service.test.ts
|
||||
|
||||
key-decisions:
|
||||
- "Unique constraint on (brand, model): enables safe ON CONFLICT DO UPDATE for catalog enrichment"
|
||||
- "Tags sync: undefined=leave untouched, []=clear all, [names]=replace — three-way tag handling"
|
||||
- "Migration 0003 fixed: drizzle-kit generated spurious duplicate DDL; trimmed to only new changes"
|
||||
|
||||
patterns-established:
|
||||
- "upsertGlobalItem pattern: check existence before upsert to track created vs updated"
|
||||
- "syncGlobalItemTags: delete existing links, then create-if-not-exists tags and insert links"
|
||||
|
||||
requirements-completed: [CATL-01, CATL-02, CATL-05]
|
||||
|
||||
# Metrics
|
||||
duration: 3min
|
||||
completed: 2026-04-10
|
||||
---
|
||||
|
||||
# Phase 25 Plan 01: Catalog Enrichment Data Layer Summary
|
||||
|
||||
**globalItems attribution columns (sourceUrl, imageCredit, imageSourceUrl) with unique(brand, model) constraint, upsertGlobalItem/bulkUpsertGlobalItems service functions, and Zod schemas — 21 tests passing**
|
||||
|
||||
## Performance
|
||||
|
||||
- **Duration:** ~3 min
|
||||
- **Started:** 2026-04-10T08:55:26Z
|
||||
- **Completed:** 2026-04-10T08:58:39Z
|
||||
- **Tasks:** 2
|
||||
- **Files modified:** 5
|
||||
|
||||
## Accomplishments
|
||||
|
||||
- Added three attribution columns to globalItems table with unique(brand, model) constraint and generated migration
|
||||
- Implemented upsertGlobalItem with onConflictDoUpdate, three-way tag sync, and created/updated tracking
|
||||
- Implemented bulkUpsertGlobalItems processing arrays in a single atomic transaction
|
||||
- Defined upsertGlobalItemSchema and bulkUpsertGlobalItemsSchema Zod validation schemas
|
||||
- All 21 tests pass (13 pre-existing + 8 new upsert operation tests)
|
||||
|
||||
## Task Commits
|
||||
|
||||
Each task was committed atomically:
|
||||
|
||||
1. **Task 1: Schema migration — attribution columns + unique constraint** - `39ef9cc` (feat)
|
||||
2. **Task 2: TDD RED — failing tests** - `9093a2c` (test)
|
||||
3. **Task 2: TDD GREEN — Zod schemas, service functions, tests passing** - `c8ebbf8` (feat)
|
||||
|
||||
_Note: TDD tasks have multiple commits (test RED → feat GREEN)_
|
||||
|
||||
## Files Created/Modified
|
||||
|
||||
- `src/db/schema.ts` - Added sourceUrl, imageCredit, imageSourceUrl columns and unique().on(brand, model) constraint to globalItems
|
||||
- `drizzle-pg/0003_loving_serpent_society.sql` - Migration adding 3 columns + unique constraint
|
||||
- `src/shared/schemas.ts` - Added upsertGlobalItemSchema and bulkUpsertGlobalItemsSchema
|
||||
- `src/shared/types.ts` - Added UpsertGlobalItemInput and BulkUpsertGlobalItemsInput types
|
||||
- `src/server/services/global-item.service.ts` - Added upsertGlobalItem, bulkUpsertGlobalItems, syncGlobalItemTags
|
||||
- `tests/services/global-item.service.test.ts` - Added 8 upsert operation tests
|
||||
|
||||
## Decisions Made
|
||||
|
||||
- **Three-way tag handling**: `undefined` leaves existing tags untouched, `[]` clears all tags, `[names]` replaces tags. This allows callers to selectively update tags without clobbering existing data.
|
||||
- **Unique constraint on (brand, model)**: Required for ON CONFLICT DO UPDATE semantics. Without it, duplicate inserts would fail rather than update.
|
||||
- **Created/updated tracking via pre-check**: The service checks for existing row before upsert to accurately report created vs updated counts, since ON CONFLICT DO UPDATE doesn't distinguish via returning rows alone.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
### Auto-fixed Issues
|
||||
|
||||
**1. [Rule 1 - Bug] Fixed drizzle-kit generated spurious duplicate DDL in migration 0003**
|
||||
- **Found during:** Task 1 (schema migration)
|
||||
- **Issue:** drizzle-kit generated a migration that re-created global_item_tags, re-added FKs on items and thread_candidates, and re-added oauth_codes.user_id — all already present in migration 0002. PGlite tests failed with "relation already exists".
|
||||
- **Fix:** Trimmed migration 0003 to only include the three new ALTER TABLE ADD COLUMN statements and the unique constraint.
|
||||
- **Files modified:** drizzle-pg/0003_loving_serpent_society.sql
|
||||
- **Verification:** 21 tests pass including all new upsert tests
|
||||
- **Committed in:** c8ebbf8 (Task 2 feat commit)
|
||||
|
||||
---
|
||||
|
||||
**Total deviations:** 1 auto-fixed (Rule 1 - bug in generated migration)
|
||||
**Impact on plan:** Fix was necessary for test correctness. No scope creep.
|
||||
|
||||
## Issues Encountered
|
||||
|
||||
- drizzle-kit migration generation included duplicate DDL from prior migrations — likely a state tracking issue in the drizzle-kit snapshots. Fixed by manually editing the migration to contain only the new changes.
|
||||
|
||||
## User Setup Required
|
||||
|
||||
None - no external service configuration required. Production database push (`bun run db:push`) will apply the migration when the database is available.
|
||||
|
||||
## Next Phase Readiness
|
||||
|
||||
- Data layer complete: globalItems has attribution columns, unique constraint, and upsert service functions
|
||||
- Plan 02 (HTTP routes + MCP tools) can now import upsertGlobalItem and bulkUpsertGlobalItems directly
|
||||
- No blockers
|
||||
|
||||
---
|
||||
*Phase: 25-catalog-enrichment-agent-tools*
|
||||
*Completed: 2026-04-10*
|
||||
Reference in New Issue
Block a user