type: user

role: user

content:
Design a custom geocoder index patching system. I'll provide the full context.

## Goal
Two tools:
- `geocoder-diff <old-dir> <new-dir> -o patch.gcpatch` — compares two independently-built indexes, produces a compact semantic patch
- `geocoder-patch <current-dir> patch.gcpatch -o <new-dir>` — applies patch, produces new index

Requirements from the user:
1. Daily builds are completely independent (no dependency on previous builds)
2. Diff tool compares full output of today vs yesterday
3. Patches must be SMALL (target: ~5-15 MiB for Europe, vs 7 GiB full dataset)
4. Users can apply patches in sequence: day1 + patch_1→2 = day2, then day2 + patch_2→3 = day3

## Key Challenge: Sequential Patching
Two independent builds of the same day have completely different file layouts (different string offsets, different record orderings, different entry offsets). A patch generated from fresh_day2 vs fresh_day3 can't be applied to a file produced by patching day1→day2, because the byte layout differs.

**Solution**: The patch applicator always produces DETERMINISTIC (canonical) output. Same semantic data → same bytes. This means:
- `patch(canonical_day1, diff_1→2)` produces `canonical_day2`
- `canonicalize(fresh_day2)` also produces `canonical_day2`
- Therefore `diff_2→3` generated from `canonical_day2` vs `fresh_day3` is applicable to the client's `canonical_day2`

The build server pipeline becomes:
1. Fresh build → output/
2. Canonicalize output/ → canonical/ (deterministic format)
3. Diff previous canonical/ vs new canonical/ → patch

## Binary Format (14 files per index)

**Cell index files** (sorted by cell_id, binary search):
- geo_cells.bin: [uint64 cell_id, uint32 street_off, uint32 addr_off, uint32 interp_off] × N (20 bytes/entry)
- admin_cells.bin: [uint64 cell_id, uint32 entry_off] × N (12 bytes/entry)

**Entry files** (variable-length, referenced by byte offset from cell files):
- street_entries.bin, addr_entries.bin, interp_entries.bin, admin_entries.bin: [uint16 count][uint32 id] × count

**Data arrays** (fixed-size records, indexed by ID from entries):
- street_ways.bin: [uint32 node_offset, uint8 node_count, uint32 name_id] (9 bytes, NO padding)
- addr_points.bin: [float lat, float lng, uint32 housenumber_id, uint32 street_id] (16 bytes)
- interp_ways.bin: [uint32 node_offset, uint8 node_count, uint32 street_id, uint32 start_num, uint32 end_num, uint8 interp_type] (18 bytes, struct-packed with padding)
- admin_polygons.bin: [uint32 vertex_offset, uint32 vertex_count, uint32 name_id, uint8 admin_level, float area, uint16 country_code] (with padding)

**Coordinate arrays** (flat, 8 bytes/entry):
- street_nodes.bin, interp_nodes.bin, admin_vertices.bin: [float lat, float lng]

**String pool**: strings.bin: null-terminated strings concatenated, referenced by byte offset

**Server constraints**: Files are mmap'd. Cell files MUST be sorted by cell_id. Data arrays are accessed by direct indexing. No validation of IDs or offsets.

## Existing Comparison Infrastructure
`compare_indexes.cpp` already implements semantic fingerprinting:
- way_fingerprint: "name|node_count|sorted_grid_coords" (1e5 precision)
- addr_fingerprint: "grid_lat,grid_lng|housenumber|street"
- admin_fingerprint: "name|admin_level|country|sorted_grid_vertices"
- interp_fingerprint: "street|start-end|interp_type|sorted_grid_nodes"

## Design the diff tool, patch format, and applicator

Key considerations:
1. The diff tool compares two builds by semantic fingerprinting
2. The patch contains: added records (full data + S2 cell assignments), removed record fingerprints
3. The applicator rebuilds all files deterministically (canonical format)
4. No S2 dependency on client — cell assignments for new records come from the patch
5. Kept records carry their cell assignments from the old build's entries files
6. The canonical format: alphabetically sorted string pool, records sorted by canonical key

Please design:
1. The patch binary format (sections, encoding)
2. The diff tool algorithm (matching, cell assignment extraction)
3. The applicator algorithm (how it rebuilds each of the 14 files)
4. The canonical record ordering (what key to sort by for each record type)
5. Memory and performance estimates
6. How to handle admin data (polygons + vertices are linked)

The implementation will be in C++, in builder/tools/. It can reuse types from builder/src/types.h and the string pool from builder/src/string_pool.h.

userType: external

entrypoint: cli

gitBranch: main

slug: concurrent-humming-sonnet


type: assistant

model: <synthetic>

role: assistant

type: message

type: text

text: You've hit your limit · resets 4am (UTC)