### Completed
1. ~~Sequential planet test~~ — **PASS** (3 distinct weekly snapshots, old diff tool)
2. ~~Parent-aware coordinate merge~~ — **DONE** (street_nodes 33%→7%, admin_vertices 210%→2.4%)
3. ~~String-level merge~~ — **DONE** (83 MiB → ~100 KB)
4. ~~Planet test with optimizations~~ — **DONE** (852 MiB → 186 MiB)
5. ~~Build determinism~~ — **PASS** (Germany, Europe, Planet — all verified)
6. ~~Eliminate explicit string remap~~ — **DONE** (derived from string diff)
7. ~~Struct padding fix~~ — **DONE** (explicit padding in AdminPolygon/InterpWay)
8. ~~Addr_points dedup~~ — **DONE** (4.4M duplicates on planet, fixed non-determinism)
9. ~~Delta-encode fixup tables~~ — **DONE** (148 MiB → 39 MiB for Europe, Europe patch 68→34 MiB)
10. ~~Run planet with delta fixups~~ — **DONE** (160 MiB → 77 MiB, all 14 MATCH)
11. ~~Secondary ID matching~~ — **DONE** (recovers ~87K additional ID mappings on Europe, reduces corrections)
12. ~~Fix admin polygon country_code hash~~ — **DONE** (was reading padding byte instead of country_code)

### Must Do
- ~~Sequential test with optimized diff on planet~~ — **PASS** (160 MiB + 159 MiB, both steps byte-identical)
- **Sequential test with all optimizations on planet** — Re-run with delta fixups + secondary matching
- **Verify patched planet still serves correct data** — Spot-check geocoding results against fresh build

### Should Do (Patch Size)
- **Run planet with secondary matching** — Expected to reduce 2M street corrections significantly
- **Further reduce cell corrections** — Remaining 549K street corrections on Europe. Could try more relaxed matching keys.

### Nice to Have (Performance)
- **Parallel merge building** — Currently sequential across files
- **Streaming patch application** — Currently loads entire decompressed patch into memory
- **Code cleanup** — Remove legacy dead code block in geocoder_patch.cpp, prototype files

# Geocoder Incremental Patch System

## Goal

Produce small patch files nightly so users can update their geocoder index without
re-downloading the full dataset. A user starting from any build can apply patches
in sequence and arrive at output **byte-identical** to a fresh build.

## Current Status

### What Works
- **Deterministic builds**: Same PBF always produces byte-identical output (verified on Germany, Europe, and Planet)
- **Single-patch application**: Verified on both Europe and Planet — all 14 files byte-identical
- **Sequential patching**: Verified on planet with 3 distinct weekly snapshots (Mar 9 → Mar 16 → Mar 23) — **PASS**
- **Custom patch format**: Merge sequences, string-level diffs, parent-aware coordinate merges, secondary ID matching, cell corrections — fully custom diff/apply logic (only zstd for transport compression)

### What Needs Work
- **Sequential test with optimized diff**: The sequential test used the old diff tool; need to re-run with all optimizations (delta fixups + secondary matching)
- **Code cleanup**: Legacy dead code block in geocoder_patch.cpp, prototype files

## Tested Patch Sizes

### Europe (7 GiB dataset)

| Gap | Patch Size | Per Day | All Match? |
|-----|-----------|---------|------------|
| 6 days (Mar 21→27) | **32 MiB** | **~5.3 MiB** | YES — all optimizations + secondary matching |
| 6 days (Mar 21→27) | 34 MiB | ~5.7 MiB | YES — before secondary matching |

### Planet (17 GiB dataset)

| Gap | Patch Size | Per Day | All Match? |
|-----|-----------|---------|------------|
| 7 days (Mar 16→23) | **77 MiB** | **~11 MiB** | YES — delta fixups, before secondary matching |
| 7 days (Mar 9→16) | 160 MiB | ~23 MiB | YES — without delta fixups |
| Sequential (Mar 9→16→23) | 160+159 MiB | N/A | **PASS** — both steps byte-identical (without delta fixups) |

### Per-File Breakdown (Planet 7-day gap, Mar 16→23, before secondary matching)

| Component | Raw Size | % of Patch | Notes |
|-----------|---------|-----------|-------|
| **Way fixups (47.7M, delta-encoded)** | **39 MiB** | **50.6%** | **node_offset fixes, varint delta** |
| **street_nodes merge** | **252 MiB** | **—** | **Parent-aware, 6.65% of file** |
| street_entries corrections (2M cells) | 31 MiB | — | *Expected to shrink significantly with secondary matching* |
| admin_vertices merge | 16 MiB | — | Parent-aware, 2.34% of file |
| admin_polygons fixups (945K pairs) | 7.2 MiB | — | vertex_offset fixes |
| admin_entries corrections (193K cells) | 7.1 MiB | — | |
| addr_points merge | 5.9 MiB | — | 0.24% of file |
| street_ways merge | 5.1 MiB | — | 0.92% of file |
| addr_entries corrections (116K cells) | 2.9 MiB | — | |
| Flag corrections (224K cells) | 1.9 MiB | — | |
| Cell changes (184K added, 46K removed) | 1.8 MiB | — | |
| String diff (~11K added, ~4K deleted) | 0.2 MiB | — | |
| interp + other | 1.1 MiB | — | |
| **Compressed (zstd transport)** | **77 MiB** | | |

Note: Planet has not yet been tested with secondary matching. Based on Europe results (~29K way + 55K addr secondary matches for 6 days), expect significant reduction in cell corrections.

### Per-File Breakdown (Europe 6-day gap, Mar 21→27, latest approach)

| Component | Raw Size | Notes |
|-----------|---------|-------|
| street_nodes.bin merge | 108 MiB | Parent-aware, 7.49% of file |
| admin_vertices.bin merge | 7.1 MiB | Parent-aware, 2.44% of file |
| Way fixups (19.4M, delta-encoded) | 19 MiB | node_offset fixes |
| street_entries corrections (549K cells) | 9.1 MiB | After secondary matching (was ~700K+ before) |
| addr_points.bin merge | 4.4 MiB | 0.32% of file |
| street_ways.bin merge | 2.7 MiB | 1.16% of file |
| addr_entries corrections (64K cells) | 1.7 MiB | After secondary matching |
| Cell flag corrections (83K cells) | 749 KB | has_street/has_addr/has_interp |
| Secondary remap (87K pairs) | 696 KB | Recovers IDs for geometry-changed records |
| Cell changes (59K added, 13K removed) | 571 KB | |
| admin_polygons.bin merge + fixups | 544 KB | 0.88% + vertex_offset fixes |
| String diff (+4152, -2396) | ~60 KB | |
| admin_entries corrections (1.8K cells) | 55 KB | |
| interp merge + fixups | ~14 KB | |
| **Total uncompressed** | **~167 MiB** | |
| **Compressed (zstd transport)** | **32 MiB** | |

## Architecture

```
build-index (deterministic) → geocoder-diff old/ new/ → patch.gcpatch
geocoder-patch old/ patch.gcpatch → new/  (must be byte-identical to fresh build)
```

### Tools
- **build-index** (`src/build_index.cpp`): Builder with deterministic ordering
- **geocoder-diff** (`tools/geocoder_diff.cpp`): Compares two builds, produces `.gcpatch`
- **geocoder-patch** (`tools/geocoder_patch.cpp`): Applies patch, produces new build
- **patch_format.h** (`tools/patch_format.h`): Shared format definitions + rebuild functions

### How Patching Works

**Diff tool:**
1. Build string-level diff (walk both sorted pools, emit added/deleted strings)
2. Build string remap (old string pool offsets → new, derived from string diff)
3. For each data file: apply string remap to old, match records by content hash (for ways/admin: also fix node_offset/vertex_offset fields), build merge sequence (MATCH/INSERT/DELETE at record stride)
4. For coordinate files (nodes/vertices): derive merge from parent way/polygon merge — verify node blocks match before COPY, INSERT if nodes changed
5. Derive ID remaps from merge sequences (MATCH ops define old→new record correspondence)
6. **Secondary matching**: For DELETE/INSERT records in merge sequences, match by relaxed key (name + node_count for ways, street + housenumber for addrs, etc.) to recover ID mappings for modified-but-same-entity records
7. Rebuild entry files from enhanced ID remaps + cell changes
8. Compare derived entries with new entries cell by cell — include differing cells as corrections
9. Compare cell flags (has_street/has_addr/has_interp) between old and new — include differences
10. Package everything, compress whole patch with zstd for transport

**Patch tool:**
1. Decompress zstd
2. Read string-level diff → reconstruct new string pool from old + additions/deletions, derive remap
3. For each data file: apply string remap + offset fixups to old, replay merge sequence → output file, track ID mapping from MATCH ops
4. Read secondary ID remaps → merge into derived ID mappings
5. Read cell changes (added/removed cell IDs) and flag corrections
6. Rebuild entry files from enhanced ID mappings + cell changes
7. Apply cell-level entry corrections (replace specific cells' data)
8. Rebuild geo_cells/admin_cells from corrected entries + flag corrections

## Known Issues

### 1. Cell correction overhead (PARTIALLY FIXED)

**Impact**: 11 MiB for Europe (down from 15 MiB). Planet TBD.

**Status**: Secondary matching recovers ~87K additional ID mappings on Europe (29K ways, 55K addrs, 2.3K admin). Reduced street corrections from ~700K+ to 549K cells.

**Remaining cause**: Records that are genuinely different between old and new (name changes, completely new/removed entities) can't be matched. These remain as corrections.

### 2. Way fixup table size (FIXED)

**Impact**: 19-39 MiB of delta-encoded fixup data per patch.

**Status**: Delta encoding with varint compression reduces raw fixup size 3-4x (e.g., 148 MiB → 39 MiB for Europe).

### 3. street_nodes.bin still 6-7%

**Impact**: 108-264 MiB of merge data.

**Problem**: Parent-aware merge reduced this from 33% to 7%, but it's still the largest single component. The remaining 7% is from ways whose header matches (byte-identical) but whose actual node coordinates changed (geometry edits in OSM).

**Potential fix**: None needed — this represents actual data changes, not algorithm inefficiency.

### 4. Struct padding non-determinism (FIXED)

**Status**: Fixed by adding explicit padding fields to `AdminPolygon` and `InterpWay` structs.

## Patch Format (.gcpatch, version 2)

Whole file is zstd-compressed for transport. Internal structure:

```
Header: "GCPATCH\0" (8) + version=2 (u32) + flags=0 (u32)

String Remap: marker 0xFFFFFFFE (u32) + count=0 (u32)
  (remap derived from string diff, explicit table no longer needed)

Per-file merge: file_id (u32) + stride (u32) + old_size (u64) + new_size (u64)
  + n_fixups (u32) + [(record_idx, new_offset_value)] × n_fixups
  + seq_size (u64) + merge_ops
  Ops: MATCH(count:u32) | INSERT(count:u32, data) | DELETE(count:u32)
  stride=0: full replacement (no fixups, data follows directly)

String Diff: marker 0xFFFFFFF7 (u32) + n_added (u32) + n_deleted (u32)
  + [string_data\0] × n_added + [deleted_index:u32] × n_deleted

Cell Changes: marker 0xFFFFFFFB/0xFFFFFFFA (u32) + n_added (u32) + n_removed (u32)
  + [cell_id:u64] × n_added + [cell_id:u64] × n_removed

Secondary ID Remap: marker 0xFFFFFFF6 (u32) + n_files (u32)
  per file: file_id (u32) + n_pairs (u32) + [(old_id:u32, new_id:u32)] × n_pairs
  Recovers old→new mappings for modified records (matched by relaxed key)

Cell Flag Corrections: marker 0xFFFFFFF9 (u32) + count (u32)
  + [(cell_id:u64, flags:u8)] × count
  flags: bit0=has_street, bit1=has_addr, bit2=has_interp

Entry Corrections: marker 0xFFFFFFF8 (u32) + file_id (u32) + count (u32)
  + [(cell_id:u64, entry_count:u16, [id:u32] × entry_count)] × count

End marker: 0xFFFFFFFF (u32)
```

## Performance

| Operation | Europe | Planet |
|-----------|--------|--------|
| Build (deterministic) | ~12 min | ~14 min |
| Diff generation | ~8 min | ~38 min |
| Patch application | ~5 min | ~20 min |

## TODO (Priority Order)

### Completed
1. ~~Sequential planet test~~ — **PASS** (3 distinct weekly snapshots, old diff tool)
2. ~~Parent-aware coordinate merge~~ — **DONE** (street_nodes 33%→7%, admin_vertices 210%→2.4%)
3. ~~String-level merge~~ — **DONE** (83 MiB → ~100 KB)
4. ~~Planet test with optimizations~~ — **DONE** (852 MiB → 186 MiB)
5. ~~Build determinism~~ — **PASS** (Germany, Europe, Planet — all verified)
6. ~~Eliminate explicit string remap~~ — **DONE** (derived from string diff)
7. ~~Struct padding fix~~ — **DONE** (explicit padding in AdminPolygon/InterpWay)
8. ~~Addr_points dedup~~ — **DONE** (4.4M duplicates on planet, fixed non-determinism)

### Must Do
- ~~Sequential test with optimized diff on planet~~ — **PASS** (160 MiB + 159 MiB, both steps byte-identical)
- **Verify patched planet still serves correct data** — Spot-check geocoding results against fresh build

### Should Do (Patch Size)
- ~~Delta-encode fixup tables~~ — **DONE** (148 MiB → 39 MiB for Europe, Europe patch 68→34 MiB)
- ~~Run planet with delta fixups~~ — **DONE** (160 MiB → 77 MiB, all 14 MATCH)
- **Reduce cell correction count** — 2M street cell corrections (31 MiB). Improve fixup matching to reduce unmatched records

### Nice to Have (Performance)
- **Parallel merge building** — Currently sequential across files
- **Streaming patch application** — Currently loads entire decompressed patch into memory
- **Code cleanup** — Remove prototype files, dead code, inconsistent patterns

### Future Consideration (Format Changes)
- **Inline node storage** — Store way nodes inline instead of separate flat array. Eliminates coordinate cascade entirely. Requires server format change.
- **Content-addressed record IDs** — Hash-based IDs instead of sequential indices. Makes entry files stable across builds. Requires server format change.
- **Fixed-size cell slots** — Pre-allocate per cell. Analysis showed 2-6x space waste due to skewed distribution.

## File Inventory

### Core files (committed)
- `src/build_index.cpp` — Deterministic ordering
- `src/types.h` — Explicit struct padding
- `src/s2_helpers.cpp` — Zero padding in AdminPolygon creation
- `tools/geocoder_diff.cpp` — Diff tool
- `tools/geocoder_patch.cpp` — Patch tool
- `tools/patch_format.h` — Shared format definitions + rebuild functions
- `tools/geocoder_canonicalize.cpp` — Standalone canonicalize (testing tool)
- `CMakeLists.txt` — Build targets

### Test data on Node 3 (/home/michtest/)
- `planet-old/` — Built from planet-260309 (Mar 9), 17 GiB
- `planet-B/` — Built from planet-260316 (Mar 16), 17 GiB
- `planet-C/` — Built from planet-260323 (Mar 23), 17 GiB
- `det-A/` — Europe built from Mar 21 PBF, 7 GiB
- `det-today/` — Europe built from Mar 27 PBF, 7 GiB

+9. ~~Delta-encode fixup tables~~ — **DONE** (148 MiB → 39 MiB for Europe, Europe patch 68→34 MiB)

+10. ~~Run planet with delta fixups~~ — **DONE** (160 MiB → 77 MiB, all 14 MATCH)

+11. ~~Secondary ID matching~~ — **DONE** (recovers ~87K additional ID mappings on Europe, reduces corrections)

+12. ~~Fix admin polygon country_code hash~~ — **DONE** (was reading padding byte instead of country_code)

+- **Sequential test with all optimizations on planet** — Re-run with delta fixups + secondary matching

-- ~~Delta-encode fixup tables~~ — **DONE** (148 MiB → 39 MiB for Europe, Europe patch 68→34 MiB)

-- ~~Run planet with delta fixups~~ — **DONE** (160 MiB → 77 MiB, all 14 MATCH)

-- **Reduce cell correction count** — 2M street cell corrections (31 MiB). Improve fixup matching to reduce unmatched records

+- **Run planet with secondary matching** — Expected to reduce 2M street corrections significantly

+- **Further reduce cell corrections** — Remaining 549K street corrections on Europe. Could try more relaxed matching keys.

-- **Code cleanup** — Remove prototype files, dead code, inconsistent patterns

+- **Code cleanup** — Remove legacy dead code block in geocoder_patch.cpp, prototype files


Now let me check if the Europe test has finished:


Bash

cat /tmp/claude-0/-root-traccar-geocoder/af1c4baa-0f9a-4420-a8b3-ce20dc28c8b9/tasks/bv255uxee.output 2>/dev/null | tail -40

Check Europe diff+patch progress


sys	0m9.335s
Decompressed patch: 175387197 bytes
Total string remap: 0 entries
  Rebuilt strings: +4152 -2396 → 4983808 strings, 4979632 remapped
  addr_points.bin: 1376034128 bytes (mapped 85817274/85891830 IDs)
    Applied 19361289 fixups (38773905 bytes delta)
  street_ways.bin: 233653596 bytes (mapped 19361632/19452480 IDs)
  street_nodes.bin: 1445778472 bytes
    Applied 13395 fixups (26793 bytes delta)
  interp_ways.bin: 335424 bytes (mapped 13961/13988 IDs)
  interp_nodes.bin: 725184 bytes
    Applied 449047 fixups (974006 bytes delta)
  admin_polygons.bin: 10819584 bytes (mapped 448349/450907 IDs)
  admin_vertices.bin: 290989232 bytes
  Geo cell changes: +58772 -12627
  Admin cell changes: +0 -0
  Secondary remap for file 1: 29127 pairs
  Secondary remap for file 3: 55444 pairs
  Secondary remap for file 4: 8 pairs
  Secondary remap for file 6: 2320 pairs
  Entry correction for file 9: 549325 cells
  Entry correction for file 10: 64334 cells
  Entry correction for file 11: 37 cells
  Entry correction for file 13: 1764 cells
  Cell flag corrections: 83178 cells
Reconstructing entry/cell files from derived ID remaps...
  Rebuilt 3 entry files + geo_cells
  Rebuilt admin_entries + admin_cells
Applying cell-level entry corrections...
  admin_entries.bin: applied 1764 cell corrections (1764 total)
  interp_entries.bin: applied 37 cell corrections (37 total)
  addr_entries.bin: applied 64334 cell corrections (64334 total)
  street_entries.bin: applied 549325 cell corrections (549325 total)
  Rebuilt geo_cells from corrected entries
  Rebuilt admin_cells.bin from corrected entries (1692000 bytes)
Patch applied. Output in patched-europe/

real	4m24.649s
user	4m13.413s
sys	0m10.172s


Patch applied. Now verify byte-identity:


sshpass -p 'elon-is-insane' ssh -o ProxyCommand="tailscale nc %h %p" -o StrictHostKeyChecking=no michtest@100.113.88.121 "cd /home/michtest && for f in det-today/*; do b=\$(basename \$f); echo -n \"\$b: \"; if cmp -s patched-europe/\$b det-today/\$b; then echo MATCH; else echo MISMATCH; fi; done" 2>&1

Verify patched Europe matches fresh build


addr_entries.bin: MATCH
addr_points.bin: MATCH
admin_cells.bin: MATCH
admin_entries.bin: MATCH
admin_polygons.bin: MATCH
admin_vertices.bin: MATCH
geo_cells.bin: MATCH
interp_entries.bin: MATCH
interp_nodes.bin: MATCH
interp_ways.bin: MATCH
manifest.json: MISMATCH
street_entries.bin: MATCH
street_nodes.bin: MATCH
street_ways.bin: MATCH
strings.bin: MATCH


All 14 data files MATCH (manifest.json is metadata, not part of the patch). The secondary matching works correctly. Let me update the task and PATCHING.md:


Updated task #16 status


Edit

| 6 days (Mar 21→27) | **32 MiB** | **~5.3 MiB** | YES — all optimizations + secondary matching |

| 6 days (Mar 21→27) | **32 MiB** | **~5.3 MiB** | YES — all optimizations + secondary matching, all 14 MATCH |


The file /root/traccar-geocoder/builder/PATCHING.md has been updated successfully.

# Geocoder Incremental Patch System

## Goal

Produce small patch files nightly so users can update their geocoder index without
re-downloading the full dataset. A user starting from any build can apply patches
in sequence and arrive at output **byte-identical** to a fresh build.

## Current Status

### What Works
- **Deterministic builds**: Same PBF always produces byte-identical output (verified on Germany, Europe, and Planet)
- **Single-patch application**: Verified on both Europe and Planet — all 14 files byte-identical
- **Sequential patching**: Verified on planet with 3 distinct weekly snapshots (Mar 9 → Mar 16 → Mar 23) — **PASS**
- **Custom patch format**: Merge sequences, string-level diffs, parent-aware coordinate merges, secondary ID matching, cell corrections — fully custom diff/apply logic (only zstd for transport compression)

### What Needs Work
- **Sequential test with optimized diff**: The sequential test used the old diff tool; need to re-run with all optimizations (delta fixups + secondary matching)
- **Code cleanup**: Legacy dead code block in geocoder_patch.cpp, prototype files

## Tested Patch Sizes

### Europe (7 GiB dataset)

| Gap | Patch Size | Per Day | All Match? |
|-----|-----------|---------|------------|
| 6 days (Mar 21→27) | **32 MiB** | **~5.3 MiB** | YES — all optimizations + secondary matching |
| 6 days (Mar 21→27) | 34 MiB | ~5.7 MiB | YES — before secondary matching |

### Planet (17 GiB dataset)

| Gap | Patch Size | Per Day | All Match? |
|-----|-----------|---------|------------|
| 7 days (Mar 16→23) | **77 MiB** | **~11 MiB** | YES — delta fixups, before secondary matching |
| 7 days (Mar 9→16) | 160 MiB | ~23 MiB | YES — without delta fixups |
| Sequential (Mar 9→16→23) | 160+159 MiB | N/A | **PASS** — both steps byte-identical (without delta fixups) |

### Per-File Breakdown (Planet 7-day gap, Mar 16→23, before secondary matching)

| Component | Raw Size | % of Patch | Notes |
|-----------|---------|-----------|-------|
| **Way fixups (47.7M, delta-encoded)** | **39 MiB** | **50.6%** | **node_offset fixes, varint delta** |
| **street_nodes merge** | **252 MiB** | **—** | **Parent-aware, 6.65% of file** |
| street_entries corrections (2M cells) | 31 MiB | — | *Expected to shrink significantly with secondary matching* |
| admin_vertices merge | 16 MiB | — | Parent-aware, 2.34% of file |
| admin_polygons fixups (945K pairs) | 7.2 MiB | — | vertex_offset fixes |
| admin_entries corrections (193K cells) | 7.1 MiB | — | |
| addr_points merge | 5.9 MiB | — | 0.24% of file |
| street_ways merge | 5.1 MiB | — | 0.92% of file |
| addr_entries corrections (116K cells) | 2.9 MiB | — | |
| Flag corrections (224K cells) | 1.9 MiB | — | |
| Cell changes (184K added, 46K removed) | 1.8 MiB | — | |
| String diff (~11K added, ~4K deleted) | 0.2 MiB | — | |
| interp + other | 1.1 MiB | — | |
| **Compressed (zstd transport)** | **77 MiB** | | |

Note: Planet has not yet been tested with secondary matching. Based on Europe results (~29K way + 55K addr secondary matches for 6 days), expect significant reduction in cell corrections.

### Per-File Breakdown (Europe 6-day gap, Mar 21→27, latest approach)

| Component | Raw Size | Notes |
|-----------|---------|-------|
| street_nodes.bin merge | 108 MiB | Parent-aware, 7.49% of file |
| admin_vertices.bin merge | 7.1 MiB | Parent-aware, 2.44% of file |
| Way fixups (19.4M, delta-encoded) | 19 MiB | node_offset fixes |
| street_entries corrections (549K cells) | 9.1 MiB | After secondary matching (was ~700K+ before) |
| addr_points.bin merge | 4.4 MiB | 0.32% of file |
| street_ways.bin merge | 2.7 MiB | 1.16% of file |
| addr_entries corrections (64K cells) | 1.7 MiB | After secondary matching |
| Cell flag corrections (83K cells) | 749 KB | has_street/has_addr/has_interp |
| Secondary remap (87K pairs) | 696 KB | Recovers IDs for geometry-changed records |
| Cell changes (59K added, 13K removed) | 571 KB | |
| admin_polygons.bin merge + fixups | 544 KB | 0.88% + vertex_offset fixes |
| String diff (+4152, -2396) | ~60 KB | |
| admin_entries corrections (1.8K cells) | 55 KB | |
| interp merge + fixups | ~14 KB | |
| **Total uncompressed** | **~167 MiB** | |
| **Compressed (zstd transport)** | **32 MiB** | |

## Architecture

```
build-index (deterministic) → geocoder-diff old/ new/ → patch.gcpatch
geocoder-patch old/ patch.gcpatch → new/  (must be byte-identical to fresh build)
```

### Tools
- **build-index** (`src/build_index.cpp`): Builder with deterministic ordering
- **geocoder-diff** (`tools/geocoder_diff.cpp`): Compares two builds, produces `.gcpatch`
- **geocoder-patch** (`tools/geocoder_patch.cpp`): Applies patch, produces new build
- **patch_format.h** (`tools/patch_format.h`): Shared format definitions + rebuild functions

### How Patching Works

**Diff tool:**
1. Build string-level diff (walk both sorted pools, emit added/deleted strings)
2. Build string remap (old string pool offsets → new, derived from string diff)
3. For each data file: apply string remap to old, match records by content hash (for ways/admin: also fix node_offset/vertex_offset fields), build merge sequence (MATCH/INSERT/DELETE at record stride)
4. For coordinate files (nodes/vertices): derive merge from parent way/polygon merge — verify node blocks match before COPY, INSERT if nodes changed
5. Derive ID remaps from merge sequences (MATCH ops define old→new record correspondence)
6. **Secondary matching**: For DELETE/INSERT records in merge sequences, match by relaxed key (name + node_count for ways, street + housenumber for addrs, etc.) to recover ID mappings for modified-but-same-entity records
7. Rebuild entry files from enhanced ID remaps + cell changes
8. Compare derived entries with new entries cell by cell — include differing cells as corrections
9. Compare cell flags (has_street/has_addr/has_interp) between old and new — include differences
10. Package everything, compress whole patch with zstd for transport

**Patch tool:**
1. Decompress zstd
2. Read string-level diff → reconstruct new string pool from old + additions/deletions, derive remap
3. For each data file: apply string remap + offset fixups to old, replay merge sequence → output file, track ID mapping from MATCH ops
4. Read secondary ID remaps → merge into derived ID mappings
5. Read cell changes (added/removed cell IDs) and flag corrections
6. Rebuild entry files from enhanced ID mappings + cell changes
7. Apply cell-level entry corrections (replace specific cells' data)
8. Rebuild geo_cells/admin_cells from corrected entries + flag corrections

## Known Issues

### 1. Cell correction overhead (PARTIALLY FIXED)

**Impact**: 11 MiB for Europe (down from 15 MiB). Planet TBD.

**Status**: Secondary matching recovers ~87K additional ID mappings on Europe (29K ways, 55K addrs, 2.3K admin). Reduced street corrections from ~700K+ to 549K cells.

**Remaining cause**: Records that are genuinely different between old and new (name changes, completely new/removed entities) can't be matched. These remain as corrections.

### 2. Way fixup table size (FIXED)

**Impact**: 19-39 MiB of delta-encoded fixup data per patch.

**Status**: Delta encoding with varint compression reduces raw fixup size 3-4x (e.g., 148 MiB → 39 MiB for Europe).

### 3. street_nodes.bin still 6-7%

**Impact**: 108-264 MiB of merge data.

**Problem**: Parent-aware merge reduced this from 33% to 7%, but it's still the largest single component. The remaining 7% is from ways whose header matches (byte-identical) but whose actual node coordinates changed (geometry edits in OSM).

**Potential fix**: None needed — this represents actual data changes, not algorithm inefficiency.

### 4. Struct padding non-determinism (FIXED)

**Status**: Fixed by adding explicit padding fields to `AdminPolygon` and `InterpWay` structs.

## Patch Format (.gcpatch, version 2)

Whole file is zstd-compressed for transport. Internal structure:

```
Header: "GCPATCH\0" (8) + version=2 (u32) + flags=0 (u32)

String Remap: marker 0xFFFFFFFE (u32) + count=0 (u32)
  (remap derived from string diff, explicit table no longer needed)

Per-file merge: file_id (u32) + stride (u32) + old_size (u64) + new_size (u64)
  + n_fixups (u32) + [(record_idx, new_offset_value)] × n_fixups
  + seq_size (u64) + merge_ops
  Ops: MATCH(count:u32) | INSERT(count:u32, data) | DELETE(count:u32)
  stride=0: full replacement (no fixups, data follows directly)

String Diff: marker 0xFFFFFFF7 (u32) + n_added (u32) + n_deleted (u32)
  + [string_data\0] × n_added + [deleted_index:u32] × n_deleted

Cell Changes: marker 0xFFFFFFFB/0xFFFFFFFA (u32) + n_added (u32) + n_removed (u32)
  + [cell_id:u64] × n_added + [cell_id:u64] × n_removed

Secondary ID Remap: marker 0xFFFFFFF6 (u32) + n_files (u32)
  per file: file_id (u32) + n_pairs (u32) + [(old_id:u32, new_id:u32)] × n_pairs
  Recovers old→new mappings for modified records (matched by relaxed key)

Cell Flag Corrections: marker 0xFFFFFFF9 (u32) + count (u32)
  + [(cell_id:u64, flags:u8)] × count
  flags: bit0=has_street, bit1=has_addr, bit2=has_interp

Entry Corrections: marker 0xFFFFFFF8 (u32) + file_id (u32) + count (u32)
  + [(cell_id:u64, entry_count:u16, [id:u32] × entry_count)] × count

End marker: 0xFFFFFFFF (u32)
```

## Performance

| Operation | Europe | Planet |
|-----------|--------|--------|
| Build (deterministic) | ~12 min | ~14 min |
| Diff generation | ~8 min | ~38 min |
| Patch application | ~5 min | ~20 min |

## TODO (Priority Order)

### Completed
1. ~~Sequential planet test~~ — **PASS** (3 distinct weekly snapshots, old diff tool)
2. ~~Parent-aware coordinate merge~~ — **DONE** (street_nodes 33%→7%, admin_vertices 210%→2.4%)
3. ~~String-level merge~~ — **DONE** (83 MiB → ~100 KB)
4. ~~Planet test with optimizations~~ — **DONE** (852 MiB → 186 MiB)
5. ~~Build determinism~~ — **PASS** (Germany, Europe, Planet — all verified)
6. ~~Eliminate explicit string remap~~ — **DONE** (derived from string diff)
7. ~~Struct padding fix~~ — **DONE** (explicit padding in AdminPolygon/InterpWay)
8. ~~Addr_points dedup~~ — **DONE** (4.4M duplicates on planet, fixed non-determinism)
9. ~~Delta-encode fixup tables~~ — **DONE** (148 MiB → 39 MiB for Europe, Europe patch 68→34 MiB)
10. ~~Run planet with delta fixups~~ — **DONE** (160 MiB → 77 MiB, all 14 MATCH)
11. ~~Secondary ID matching~~ — **DONE** (recovers ~87K additional ID mappings on Europe, reduces corrections)
12. ~~Fix admin polygon country_code hash~~ — **DONE** (was reading padding byte instead of country_code)

### Must Do
- ~~Sequential test with optimized diff on planet~~ — **PASS** (160 MiB + 159 MiB, both steps byte-identical)
- **Sequential test with all optimizations on planet** — Re-run with delta fixups + secondary matching
- **Verify patched planet still serves correct data** — Spot-check geocoding results against fresh build

### Should Do (Patch Size)
- **Run planet with secondary matching** — Expected to reduce 2M street corrections significantly
- **Further reduce cell corrections** — Remaining 549K street corrections on Europe. Could try more relaxed matching keys.

### Nice to Have (Performance)
- **Parallel merge building** — Currently sequential across files
- **Streaming patch application** — Currently loads entire decompressed patch into memory
- **Code cleanup** — Remove legacy dead code block in geocoder_patch.cpp, prototype files

### Future Consideration (Format Changes)
- **Inline node storage** — Store way nodes inline instead of separate flat array. Eliminates coordinate cascade entirely. Requires server format change.
- **Content-addressed record IDs** — Hash-based IDs instead of sequential indices. Makes entry files stable across builds. Requires server format change.
- **Fixed-size cell slots** — Pre-allocate per cell. Analysis showed 2-6x space waste due to skewed distribution.

## File Inventory

### Core files (committed)
- `src/build_index.cpp` — Deterministic ordering
- `src/types.h` — Explicit struct padding
- `src/s2_helpers.cpp` — Zero padding in AdminPolygon creation
- `tools/geocoder_diff.cpp` — Diff tool
- `tools/geocoder_patch.cpp` — Patch tool
- `tools/patch_format.h` — Shared format definitions + rebuild functions
- `tools/geocoder_canonicalize.cpp` — Standalone canonicalize (testing tool)
- `CMakeLists.txt` — Build targets

### Test data on Node 3 (/home/michtest/)
- `planet-old/` — Built from planet-260309 (Mar 9), 17 GiB
- `planet-B/` — Built from planet-260316 (Mar 16), 17 GiB
- `planet-C/` — Built from planet-260323 (Mar 23), 17 GiB
- `det-A/` — Europe built from Mar 21 PBF, 7 GiB
- `det-today/` — Europe built from Mar 27 PBF, 7 GiB

-| 6 days (Mar 21→27) | **32 MiB** | **~5.3 MiB** | YES — all optimizations + secondary matching |

+| 6 days (Mar 21→27) | **32 MiB** | **~5.3 MiB** | YES — all optimizations + secondary matching, all 14 MATCH |


Now let's run the planet test with secondary matching. This is the big one:


Updated task #17 status


Bash

sshpass -p 'elon-is-insane' ssh -o ProxyCommand="tailscale nc %h %p" -o StrictHostKeyChecking=no michtest@100.113.88.121 "ls -la /home/michtest/planet-A/ /home/michtest/planet-B/ /home/michtest/planet-C/ 2>&1 | head -5" 2>&1

Check planet test data availability


/home/michtest/planet-A/:
total 17062352
drwxr-xr-x  2 michtest michtest       4096 Mar 28 22:14 .
drwx------ 35 michtest michtest      12288 Mar 29 06:50 ..
-rw-rw-r--  1 michtest michtest  738166244 Mar 28 22:14 addr_entries.bin


Good, all three planet builds are there. Let me run the planet diff (B→C) with secondary matching:


sshpass -p 'elon-is-insane' ssh -o ProxyCommand="tailscale nc %h %p" -o StrictHostKeyChecking=no michtest@100.113.88.121 "cd /home/michtest && nohup bash -c 'time ./geocoder-diff planet-B/ planet-C/ -o patch-planet-BC-v2.gcpatch 2>&1' > planet-diff-v2.log 2>&1 &" 2>&1 && echo "Started planet diff in background"

Start planet diff B→C with secondary matching in background


Command running in background with ID: bq7l1iqba. Output is being written to: /tmp/claude-0/-root-traccar-geocoder/af1c4baa-0f9a-4420-a8b3-ce20dc28c8b9/tasks/bq7l1iqba.output


sleep 30 && sshpass -p 'elon-is-insane' ssh -o ProxyCommand="tailscale nc %h %p" -o StrictHostKeyChecking=no michtest@100.113.88.121 "tail -5 /home/michtest/planet-diff-v2.log 2>/dev/null || echo 'not started yet'" 2>&1

Check planet diff progress after 30s


  interp_ways.bin: seq=7301 fixups=72605 (0.42%)
  interp_nodes.bin: parent-derived seq=371569 bytes
  admin_polygons.bin: seq=211691 fixups=944661 (0.93%)
  admin_vertices.bin: parent-derived seq=16762694 bytes (2.34%)
  (entry/cell files: omitted, rebuilt by patch tool)


Good, it's progressing. The merge sequences are done, now it's computing cell changes and secondary matches. Let me check again in a bit:


sleep 60 && sshpass -p 'elon-is-insane' ssh -o ProxyCommand="tailscale nc %h %p" -o StrictHostKeyChecking=no michtest@100.113.88.121 "tail -20 /home/michtest/planet-diff-v2.log 2>/dev/null" 2>&1

Check planet diff progress after 90s


Building string remap...
  String remap: derived from string diff (0 explicit entries)
  strings.bin: +10703 -4486 strings
  addr_points.bin: seq=6135364 fixups=0 (0.25%)
  street_ways.bin: seq=5310283 fixups=47686775 (0.92%)
  street_nodes.bin: parent-derived seq=263707571 bytes (6.65%)
  interp_ways.bin: seq=7301 fixups=72605 (0.42%)
  interp_nodes.bin: parent-derived seq=371569 bytes
  admin_polygons.bin: seq=211691 fixups=944661 (0.93%)
  admin_vertices.bin: parent-derived seq=16762694 bytes (2.34%)
  (entry/cell files: omitted, rebuilt by patch tool)


Still computing — the secondary matching on planet requires re-reading and hashing ~48M ways. Let me wait longer:


sleep 120 && sshpass -p 'elon-is-insane' ssh -o ProxyCommand="tailscale nc %h %p" -o StrictHostKeyChecking=no michtest@100.113.88.121 "tail -30 /home/michtest/planet-diff-v2.log 2>/dev/null" 2>&1

Check planet diff progress after 3.5 minutes


Still computing the heavy parts (cell changes + secondary matching + entry corrections). On planet with ~48M ways and ~87M addr_points this takes a while. Let me check less frequently:


sleep 300 && sshpass -p 'elon-is-insane' ssh -o ProxyCommand="tailscale nc %h %p" -o StrictHostKeyChecking=no michtest@100.113.88.121 "tail -25 /home/michtest/planet-diff-v2.log 2>/dev/null" 2>&1

Check planet diff progress after 8.5 minutes


Building string remap...
  String remap: derived from string diff (0 explicit entries)
  strings.bin: +10703 -4486 strings
  addr_points.bin: seq=6135364 fixups=0 (0.25%)
  street_ways.bin: seq=5310283 fixups=47686775 (0.92%)
  street_nodes.bin: parent-derived seq=263707571 bytes (6.65%)
  interp_ways.bin: seq=7301 fixups=72605 (0.42%)
  interp_nodes.bin: parent-derived seq=371569 bytes
  admin_polygons.bin: seq=211691 fixups=944661 (0.93%)
  admin_vertices.bin: parent-derived seq=16762694 bytes (2.34%)
  (entry/cell files: omitted, rebuilt by patch tool)
  Geo cell changes: +183728 -45783 (1836088 bytes)
  Admin cell changes: +4 -0
  Secondary way matches: 55091
  Secondary addr matches: 58838
  Secondary interp matches: 88
  Secondary admin matches: 4603
  Secondary remap section: 118620 pairs (948960 bytes)


Great progress — 118K secondary matches on planet (55K ways, 59K addr, 4.6K admin). Now computing entry corrections. I'll save the patches for validation as requested.

Let me wait for the diff to complete:


sleep 600 && sshpass -p 'elon-is-insane' ssh -o ProxyCommand="tailscale nc %h %p" -o StrictHostKeyChecking=no michtest@100.113.88.121 "tail -30 /home/michtest/planet-diff-v2.log 2>/dev/null" 2>&1

Check planet diff progress after 18 minutes total


Command running in background with ID: bsm7whlju. Output is being written to: /tmp/claude-0/-root-traccar-geocoder/af1c4baa-0f9a-4420-a8b3-ce20dc28c8b9/tasks/bsm7whlju.output


While waiting, let me set up a patches directory on node 3 to save the validated patches:


sshpass -p 'elon-is-insane' ssh -o ProxyCommand="tailscale nc %h %p" -o StrictHostKeyChecking=no michtest@100.113.88.121 "mkdir -p /home/michtest/validated-patches && ls /home/michtest/patch-*.gcpatch 2>/dev/null" 2>&1

Create patches directory and list existing patches


/home/michtest/patch-europe-secondary.gcpatch
/home/michtest/patch-europe-v2.gcpatch


sshpass -p 'elon-is-insane' ssh -o ProxyCommand="tailscale nc %h %p" -o StrictHostKeyChecking=no michtest@100.113.88.121 "cd /home/michtest && cp patch-europe-v2.gcpatch validated-patches/europe-mar21-to-mar27-v2.gcpatch && ls -lh validated-patches/" 2>&1

Save validated Europe patch with descriptive name


total 33M
-rw-rw-r-- 1 michtest michtest 33M Mar 29 07:16 europe-mar21-to-mar27-v2.gcpatch


Now let me check if the planet diff finished:


cat /tmp/claude-0/-root-traccar-geocoder/af1c4baa-0f9a-4420-a8b3-ce20dc28c8b9/tasks/bsm7whlju.output 2>/dev/null | tail -5

Check background task output


  Secondary addr matches: 58838
  Secondary interp matches: 88
  Secondary admin matches: 4603
  Secondary remap section: 118620 pairs (948960 bytes)
  street_entries.bin: 1588748 cell corrections (25715156 bytes)


The entry corrections are being computed. Street corrections dropped from 2M to 1.59M — that's a ~21% reduction. Let me wait for it to finish completely:


sleep 300 && sshpass -p 'elon-is-insane' ssh -o ProxyCommand="tailscale nc %h %p" -o StrictHostKeyChecking=no michtest@100.113.88.121 "tail -15 /home/michtest/planet-diff-v2.log 2>/dev/null" 2>&1

Check planet diff completion


  street_nodes.bin: parent-derived seq=263707571 bytes (6.65%)
  interp_ways.bin: seq=7301 fixups=72605 (0.42%)
  interp_nodes.bin: parent-derived seq=371569 bytes
  admin_polygons.bin: seq=211691 fixups=944661 (0.93%)
  admin_vertices.bin: parent-derived seq=16762694 bytes (2.34%)
  (entry/cell files: omitted, rebuilt by patch tool)
  Geo cell changes: +183728 -45783 (1836088 bytes)
  Admin cell changes: +4 -0
  Secondary way matches: 55091
  Secondary addr matches: 58838
  Secondary interp matches: 88
  Secondary admin matches: 4603
  Secondary remap section: 118620 pairs (948960 bytes)
  street_entries.bin: 1588748 cell corrections (25715156 bytes)
  addr_entries.bin: 95288 cell corrections (2479188 bytes)


Still computing addr and admin corrections. Almost done:


sleep 300 && sshpass -p 'elon-is-insane' ssh -o ProxyCommand="tailscale nc %h %p" -o StrictHostKeyChecking=no michtest@100.113.88.121 "tail -20 /home/michtest/planet-diff-v2.log 2>/dev/null" 2>&1

Check planet diff completion


  String remap: derived from string diff (0 explicit entries)
  strings.bin: +10703 -4486 strings
  addr_points.bin: seq=6135364 fixups=0 (0.25%)
  street_ways.bin: seq=5310283 fixups=47686775 (0.92%)
  street_nodes.bin: parent-derived seq=263707571 bytes (6.65%)
  interp_ways.bin: seq=7301 fixups=72605 (0.42%)
  interp_nodes.bin: parent-derived seq=371569 bytes
  admin_polygons.bin: seq=211691 fixups=944661 (0.93%)
  admin_vertices.bin: parent-derived seq=16762694 bytes (2.34%)
  (entry/cell files: omitted, rebuilt by patch tool)
  Geo cell changes: +183728 -45783 (1836088 bytes)
  Admin cell changes: +4 -0
  Secondary way matches: 55091
  Secondary addr matches: 58838
  Secondary interp matches: 88
  Secondary admin matches: 4603
  Secondary remap section: 118620 pairs (948960 bytes)
  street_entries.bin: 1588748 cell corrections (25715156 bytes)
  addr_entries.bin: 95288 cell corrections (2479188 bytes)
  interp_entries.bin: 256 cell corrections (4892 bytes)


Almost done — just admin_entries and flag corrections left:


sleep 300 && sshpass -p 'elon-is-insane' ssh -o ProxyCommand="tailscale nc %h %p" -o StrictHostKeyChecking=no michtest@100.113.88.121 "tail -20 /home/michtest/planet-diff-v2.log 2>/dev/null" 2>&1

Check planet diff completion


  (entry/cell files: omitted, rebuilt by patch tool)
  Geo cell changes: +183728 -45783 (1836088 bytes)
  Admin cell changes: +4 -0
  Secondary way matches: 55091
  Secondary addr matches: 58838
  Secondary interp matches: 88
  Secondary admin matches: 4603
  Secondary remap section: 118620 pairs (948960 bytes)
  street_entries.bin: 1588748 cell corrections (25715156 bytes)
  addr_entries.bin: 95288 cell corrections (2479188 bytes)
  interp_entries.bin: 256 cell corrections (4892 bytes)
  admin_entries.bin: 4556 cell corrections (244600 bytes)
  Cell flag corrections: 224324 cells (2018916 bytes)

Uncompressed patch: 423813763 bytes (404 MiB)
Compressed patch: 77290460 bytes (73 MiB)

real	36m24.760s
user	36m28.873s
sys	1m33.684s


Excellent! Planet diff complete. Key results:

- **Compressed: 73 MiB** (down from 77 MiB before secondary matching)
- Street corrections: **1.59M cells** (down from 2.03M = 22% reduction)
- Addr corrections: **95K cells** (down from 116K = 18% reduction)
- Admin corrections: **4.6K cells** (down from 193K = **97% reduction!** — the country_code hash fix helped massively)
- 118K secondary matches total

Now let's apply the patch and verify:


sshpass -p 'elon-is-insane' ssh -o ProxyCommand="tailscale nc %h %p" -o StrictHostKeyChecking=no michtest@100.113.88.121 "cd /home/michtest && rm -rf patched-planet-BC/ && nohup bash -c 'time ./geocoder-patch planet-B/ patch-planet-BC-v2.gcpatch -o patched-planet-BC/ 2>&1' > planet-patch-v2.log 2>&1 &" 2>&1 && echo "Planet patch started"

Apply planet patch B→C


Command running in background with ID: bxzwyjjkl. Output is being written to: /tmp/claude-0/-root-traccar-geocoder/af1c4baa-0f9a-4420-a8b3-ce20dc28c8b9/tasks/bxzwyjjkl.output


While that runs, let me update PATCHING.md with the planet results:


Edit

| 7 days (Mar 16→23) | **77 MiB** | **~11 MiB** | YES — delta fixups, before secondary matching |

| 7 days (Mar 16→23) | **73 MiB** | **~10.4 MiB** | Pending verification — with secondary matching |
| 7 days (Mar 16→23) | 77 MiB | ~11 MiB | YES — delta fixups, before secondary matching |


| 7 days (Mar 16→23) | **73 MiB** | **~10.4 MiB** | Pending verification — with secondary matching |
| 7 days (Mar 16→23) | 77 MiB | ~11 MiB | YES — delta fixups, before secondary matching |

# Geocoder Incremental Patch System

## Goal

Produce small patch files nightly so users can update their geocoder index without
re-downloading the full dataset. A user starting from any build can apply patches
in sequence and arrive at output **byte-identical** to a fresh build.

## Current Status

### What Works
- **Deterministic builds**: Same PBF always produces byte-identical output (verified on Germany, Europe, and Planet)
- **Single-patch application**: Verified on both Europe and Planet — all 14 files byte-identical
- **Sequential patching**: Verified on planet with 3 distinct weekly snapshots (Mar 9 → Mar 16 → Mar 23) — **PASS**
- **Custom patch format**: Merge sequences, string-level diffs, parent-aware coordinate merges, secondary ID matching, cell corrections — fully custom diff/apply logic (only zstd for transport compression)

### What Needs Work
- **Sequential test with optimized diff**: The sequential test used the old diff tool; need to re-run with all optimizations (delta fixups + secondary matching)
- **Code cleanup**: Legacy dead code block in geocoder_patch.cpp, prototype files

## Tested Patch Sizes

### Europe (7 GiB dataset)

| Gap | Patch Size | Per Day | All Match? |
|-----|-----------|---------|------------|
| 6 days (Mar 21→27) | **32 MiB** | **~5.3 MiB** | YES — all optimizations + secondary matching, all 14 MATCH |
| 6 days (Mar 21→27) | 34 MiB | ~5.7 MiB | YES — before secondary matching |

### Planet (17 GiB dataset)

| Gap | Patch Size | Per Day | All Match? |
|-----|-----------|---------|------------|
| 7 days (Mar 16→23) | **77 MiB** | **~11 MiB** | YES — delta fixups, before secondary matching |
| 7 days (Mar 9→16) | 160 MiB | ~23 MiB | YES — without delta fixups |
| Sequential (Mar 9→16→23) | 160+159 MiB | N/A | **PASS** — both steps byte-identical (without delta fixups) |

### Per-File Breakdown (Planet 7-day gap, Mar 16→23, before secondary matching)

| Component | Raw Size | % of Patch | Notes |
|-----------|---------|-----------|-------|
| **Way fixups (47.7M, delta-encoded)** | **39 MiB** | **50.6%** | **node_offset fixes, varint delta** |
| **street_nodes merge** | **252 MiB** | **—** | **Parent-aware, 6.65% of file** |
| street_entries corrections (2M cells) | 31 MiB | — | *Expected to shrink significantly with secondary matching* |
| admin_vertices merge | 16 MiB | — | Parent-aware, 2.34% of file |
| admin_polygons fixups (945K pairs) | 7.2 MiB | — | vertex_offset fixes |
| admin_entries corrections (193K cells) | 7.1 MiB | — | |
| addr_points merge | 5.9 MiB | — | 0.24% of file |
| street_ways merge | 5.1 MiB | — | 0.92% of file |
| addr_entries corrections (116K cells) | 2.9 MiB | — | |
| Flag corrections (224K cells) | 1.9 MiB | — | |
| Cell changes (184K added, 46K removed) | 1.8 MiB | — | |
| String diff (~11K added, ~4K deleted) | 0.2 MiB | — | |
| interp + other | 1.1 MiB | — | |
| **Compressed (zstd transport)** | **77 MiB** | | |

Note: Planet has not yet been tested with secondary matching. Based on Europe results (~29K way + 55K addr secondary matches for 6 days), expect significant reduction in cell corrections.

### Per-File Breakdown (Europe 6-day gap, Mar 21→27, latest approach)

| Component | Raw Size | Notes |
|-----------|---------|-------|
| street_nodes.bin merge | 108 MiB | Parent-aware, 7.49% of file |
| admin_vertices.bin merge | 7.1 MiB | Parent-aware, 2.44% of file |
| Way fixups (19.4M, delta-encoded) | 19 MiB | node_offset fixes |
| street_entries corrections (549K cells) | 9.1 MiB | After secondary matching (was ~700K+ before) |
| addr_points.bin merge | 4.4 MiB | 0.32% of file |
| street_ways.bin merge | 2.7 MiB | 1.16% of file |
| addr_entries corrections (64K cells) | 1.7 MiB | After secondary matching |
| Cell flag corrections (83K cells) | 749 KB | has_street/has_addr/has_interp |
| Secondary remap (87K pairs) | 696 KB | Recovers IDs for geometry-changed records |
| Cell changes (59K added, 13K removed) | 571 KB | |
| admin_polygons.bin merge + fixups | 544 KB | 0.88% + vertex_offset fixes |
| String diff (+4152, -2396) | ~60 KB | |
| admin_entries corrections (1.8K cells) | 55 KB | |
| interp merge + fixups | ~14 KB | |
| **Total uncompressed** | **~167 MiB** | |
| **Compressed (zstd transport)** | **32 MiB** | |

## Architecture

```
build-index (deterministic) → geocoder-diff old/ new/ → patch.gcpatch
geocoder-patch old/ patch.gcpatch → new/  (must be byte-identical to fresh build)
```

### Tools
- **build-index** (`src/build_index.cpp`): Builder with deterministic ordering
- **geocoder-diff** (`tools/geocoder_diff.cpp`): Compares two builds, produces `.gcpatch`
- **geocoder-patch** (`tools/geocoder_patch.cpp`): Applies patch, produces new build
- **patch_format.h** (`tools/patch_format.h`): Shared format definitions + rebuild functions

### How Patching Works

**Diff tool:**
1. Build string-level diff (walk both sorted pools, emit added/deleted strings)
2. Build string remap (old string pool offsets → new, derived from string diff)
3. For each data file: apply string remap to old, match records by content hash (for ways/admin: also fix node_offset/vertex_offset fields), build merge sequence (MATCH/INSERT/DELETE at record stride)
4. For coordinate files (nodes/vertices): derive merge from parent way/polygon merge — verify node blocks match before COPY, INSERT if nodes changed
5. Derive ID remaps from merge sequences (MATCH ops define old→new record correspondence)
6. **Secondary matching**: For DELETE/INSERT records in merge sequences, match by relaxed key (name + node_count for ways, street + housenumber for addrs, etc.) to recover ID mappings for modified-but-same-entity records
7. Rebuild entry files from enhanced ID remaps + cell changes
8. Compare derived entries with new entries cell by cell — include differing cells as corrections
9. Compare cell flags (has_street/has_addr/has_interp) between old and new — include differences
10. Package everything, compress whole patch with zstd for transport

**Patch tool:**
1. Decompress zstd
2. Read string-level diff → reconstruct new string pool from old + additions/deletions, derive remap
3. For each data file: apply string remap + offset fixups to old, replay merge sequence → output file, track ID mapping from MATCH ops
4. Read secondary ID remaps → merge into derived ID mappings
5. Read cell changes (added/removed cell IDs) and flag corrections
6. Rebuild entry files from enhanced ID mappings + cell changes
7. Apply cell-level entry corrections (replace specific cells' data)
8. Rebuild geo_cells/admin_cells from corrected entries + flag corrections

## Known Issues

### 1. Cell correction overhead (PARTIALLY FIXED)

**Impact**: 11 MiB for Europe (down from 15 MiB). Planet TBD.

**Status**: Secondary matching recovers ~87K additional ID mappings on Europe (29K ways, 55K addrs, 2.3K admin). Reduced street corrections from ~700K+ to 549K cells.

**Remaining cause**: Records that are genuinely different between old and new (name changes, completely new/removed entities) can't be matched. These remain as corrections.

### 2. Way fixup table size (FIXED)

**Impact**: 19-39 MiB of delta-encoded fixup data per patch.

**Status**: Delta encoding with varint compression reduces raw fixup size 3-4x (e.g., 148 MiB → 39 MiB for Europe).

### 3. street_nodes.bin still 6-7%

**Impact**: 108-264 MiB of merge data.

**Problem**: Parent-aware merge reduced this from 33% to 7%, but it's still the largest single component. The remaining 7% is from ways whose header matches (byte-identical) but whose actual node coordinates changed (geometry edits in OSM).

**Potential fix**: None needed — this represents actual data changes, not algorithm inefficiency.

### 4. Struct padding non-determinism (FIXED)

**Status**: Fixed by adding explicit padding fields to `AdminPolygon` and `InterpWay` structs.

## Patch Format (.gcpatch, version 2)

Whole file is zstd-compressed for transport. Internal structure:

```
Header: "GCPATCH\0" (8) + version=2 (u32) + flags=0 (u32)

String Remap: marker 0xFFFFFFFE (u32) + count=0 (u32)
  (remap derived from string diff, explicit table no longer needed)

Per-file merge: file_id (u32) + stride (u32) + old_size (u64) + new_size (u64)
  + n_fixups (u32) + [(record_idx, new_offset_value)] × n_fixups
  + seq_size (u64) + merge_ops
  Ops: MATCH(count:u32) | INSERT(count:u32, data) | DELETE(count:u32)
  stride=0: full replacement (no fixups, data follows directly)

String Diff: marker 0xFFFFFFF7 (u32) + n_added (u32) + n_deleted (u32)
  + [string_data\0] × n_added + [deleted_index:u32] × n_deleted

Cell Changes: marker 0xFFFFFFFB/0xFFFFFFFA (u32) + n_added (u32) + n_removed (u32)
  + [cell_id:u64] × n_added + [cell_id:u64] × n_removed

Secondary ID Remap: marker 0xFFFFFFF6 (u32) + n_files (u32)
  per file: file_id (u32) + n_pairs (u32) + [(old_id:u32, new_id:u32)] × n_pairs
  Recovers old→new mappings for modified records (matched by relaxed key)

Cell Flag Corrections: marker 0xFFFFFFF9 (u32) + count (u32)
  + [(cell_id:u64, flags:u8)] × count
  flags: bit0=has_street, bit1=has_addr, bit2=has_interp

Entry Corrections: marker 0xFFFFFFF8 (u32) + file_id (u32) + count (u32)
  + [(cell_id:u64, entry_count:u16, [id:u32] × entry_count)] × count

End marker: 0xFFFFFFFF (u32)
```

## Performance

| Operation | Europe | Planet |
|-----------|--------|--------|
| Build (deterministic) | ~12 min | ~14 min |
| Diff generation | ~8 min | ~38 min |
| Patch application | ~5 min | ~20 min |

## TODO (Priority Order)

### Completed
1. ~~Sequential planet test~~ — **PASS** (3 distinct weekly snapshots, old diff tool)
2. ~~Parent-aware coordinate merge~~ — **DONE** (street_nodes 33%→7%, admin_vertices 210%→2.4%)
3. ~~String-level merge~~ — **DONE** (83 MiB → ~100 KB)
4. ~~Planet test with optimizations~~ — **DONE** (852 MiB → 186 MiB)
5. ~~Build determinism~~ — **PASS** (Germany, Europe, Planet — all verified)
6. ~~Eliminate explicit string remap~~ — **DONE** (derived from string diff)
7. ~~Struct padding fix~~ — **DONE** (explicit padding in AdminPolygon/InterpWay)
8. ~~Addr_points dedup~~ — **DONE** (4.4M duplicates on planet, fixed non-determinism)
9. ~~Delta-encode fixup tables~~ — **DONE** (148 MiB → 39 MiB for Europe, Europe patch 68→34 MiB)
10. ~~Run planet with delta fixups~~ — **DONE** (160 MiB → 77 MiB, all 14 MATCH)
11. ~~Secondary ID matching~~ — **DONE** (recovers ~87K additional ID mappings on Europe, reduces corrections)
12. ~~Fix admin polygon country_code hash~~ — **DONE** (was reading padding byte instead of country_code)

### Must Do
- ~~Sequential test with optimized diff on planet~~ — **PASS** (160 MiB + 159 MiB, both steps byte-identical)
- **Sequential test with all optimizations on planet** — Re-run with delta fixups + secondary matching
- **Verify patched planet still serves correct data** — Spot-check geocoding results against fresh build

### Should Do (Patch Size)
- **Run planet with secondary matching** — Expected to reduce 2M street corrections significantly
- **Further reduce cell corrections** — Remaining 549K street corrections on Europe. Could try more relaxed matching keys.

### Nice to Have (Performance)
- **Parallel merge building** — Currently sequential across files
- **Streaming patch application** — Currently loads entire decompressed patch into memory
- **Code cleanup** — Remove legacy dead code block in geocoder_patch.cpp, prototype files

### Future Consideration (Format Changes)
- **Inline node storage** — Store way nodes inline instead of separate flat array. Eliminates coordinate cascade entirely. Requires server format change.
- **Content-addressed record IDs** — Hash-based IDs instead of sequential indices. Makes entry files stable across builds. Requires server format change.
- **Fixed-size cell slots** — Pre-allocate per cell. Analysis showed 2-6x space waste due to skewed distribution.

## File Inventory

### Core files (committed)
- `src/build_index.cpp` — Deterministic ordering
- `src/types.h` — Explicit struct padding
- `src/s2_helpers.cpp` — Zero padding in AdminPolygon creation
- `tools/geocoder_diff.cpp` — Diff tool
- `tools/geocoder_patch.cpp` — Patch tool
- `tools/patch_format.h` — Shared format definitions + rebuild functions
- `tools/geocoder_canonicalize.cpp` — Standalone canonicalize (testing tool)
- `CMakeLists.txt` — Build targets

### Test data on Node 3 (/home/michtest/)
- `planet-old/` — Built from planet-260309 (Mar 9), 17 GiB
- `planet-B/` — Built from planet-260316 (Mar 16), 17 GiB
- `planet-C/` — Built from planet-260323 (Mar 23), 17 GiB
- `det-A/` — Europe built from Mar 21 PBF, 7 GiB
- `det-today/` — Europe built from Mar 27 PBF, 7 GiB

-| 7 days (Mar 16→23) | **77 MiB** | **~11 MiB** | YES — delta fixups, before secondary matching |

+| 7 days (Mar 16→23) | **73 MiB** | **~10.4 MiB** | Pending verification — with secondary matching |

+| 7 days (Mar 16→23) | 77 MiB | ~11 MiB | YES — delta fixups, before secondary matching |


### Per-File Breakdown (Planet 7-day gap, Mar 16→23, before secondary matching)

| Component | Raw Size | % of Patch | Notes |
|-----------|---------|-----------|-------|
| **Way fixups (47.7M, delta-encoded)** | **39 MiB** | **50.6%** | **node_offset fixes, varint delta** |
| **street_nodes merge** | **252 MiB** | **—** | **Parent-aware, 6.65% of file** |
| street_entries corrections (2M cells) | 31 MiB | — | *Expected to shrink significantly with secondary matching* |
| admin_vertices merge | 16 MiB | — | Parent-aware, 2.34% of file |
| admin_polygons fixups (945K pairs) | 7.2 MiB | — | vertex_offset fixes |
| admin_entries corrections (193K cells) | 7.1 MiB | — | |
| addr_points merge | 5.9 MiB | — | 0.24% of file |
| street_ways merge | 5.1 MiB | — | 0.92% of file |
| addr_entries corrections (116K cells) | 2.9 MiB | — | |
| Flag corrections (224K cells) | 1.9 MiB | — | |
| Cell changes (184K added, 46K removed) | 1.8 MiB | — | |
| String diff (~11K added, ~4K deleted) | 0.2 MiB | — | |
| interp + other | 1.1 MiB | — | |
| **Compressed (zstd transport)** | **77 MiB** | | |

Note: Planet has not yet been tested with secondary matching. Based on Europe results (~29K way + 55K addr secondary matches for 6 days), expect significant reduction in cell corrections.

### Per-File Breakdown (Planet 7-day gap, Mar 16→23, with secondary matching)

| Component | Raw Size | Notes |
|-----------|---------|-------|
| **street_nodes merge** | **264 MiB** | **Parent-aware, 6.65% of file** |
| Way fixups (47.7M, delta-encoded) | 39 MiB | node_offset fixes, varint delta |
| street_entries corrections (1.59M cells) | 25.7 MiB | Down from 2.03M (22% fewer with secondary matching) |
| admin_vertices merge | 16 MiB | Parent-aware, 2.34% of file |
| admin_polygons merge + fixups | 7.4 MiB | 0.93% + vertex_offset fixes |
| addr_points merge | 5.9 MiB | 0.25% of file |
| street_ways merge | 5.1 MiB | 0.92% of file |
| addr_entries corrections (95K cells) | 2.5 MiB | Down from 116K (18% fewer) |
| Flag corrections (224K cells) | 1.9 MiB | |
| Cell changes (184K added, 46K removed) | 1.8 MiB | |
| Secondary remap (119K pairs) | 949 KB | 55K ways + 59K addrs + 4.6K admin |
| admin_entries corrections (4.6K cells) | 245 KB | Down from 193K (**97% fewer** — country_code hash fix) |
| String diff (+10.7K, -4.5K) | ~200 KB | |
| interp + other | ~8 KB | |
| **Total uncompressed** | **404 MiB** | |
| **Compressed (zstd transport)** | **73 MiB** | Pending verification |


### Per-File Breakdown (Planet 7-day gap, Mar 16→23, with secondary matching)

| Component | Raw Size | Notes |
|-----------|---------|-------|
| **street_nodes merge** | **264 MiB** | **Parent-aware, 6.65% of file** |
| Way fixups (47.7M, delta-encoded) | 39 MiB | node_offset fixes, varint delta |
| street_entries corrections (1.59M cells) | 25.7 MiB | Down from 2.03M (22% fewer with secondary matching) |
| admin_vertices merge | 16 MiB | Parent-aware, 2.34% of file |
| admin_polygons merge + fixups | 7.4 MiB | 0.93% + vertex_offset fixes |
| addr_points merge | 5.9 MiB | 0.25% of file |
| street_ways merge | 5.1 MiB | 0.92% of file |
| addr_entries corrections (95K cells) | 2.5 MiB | Down from 116K (18% fewer) |
| Flag corrections (224K cells) | 1.9 MiB | |
| Cell changes (184K added, 46K removed) | 1.8 MiB | |
| Secondary remap (119K pairs) | 949 KB | 55K ways + 59K addrs + 4.6K admin |
| admin_entries corrections (4.6K cells) | 245 KB | Down from 193K (**97% fewer** — country_code hash fix) |
| String diff (+10.7K, -4.5K) | ~200 KB | |
| interp + other | ~8 KB | |
| **Total uncompressed** | **404 MiB** | |
| **Compressed (zstd transport)** | **73 MiB** | Pending verification |

# Geocoder Incremental Patch System

## Goal

Produce small patch files nightly so users can update their geocoder index without
re-downloading the full dataset. A user starting from any build can apply patches
in sequence and arrive at output **byte-identical** to a fresh build.

## Current Status

### What Works
- **Deterministic builds**: Same PBF always produces byte-identical output (verified on Germany, Europe, and Planet)
- **Single-patch application**: Verified on both Europe and Planet — all 14 files byte-identical
- **Sequential patching**: Verified on planet with 3 distinct weekly snapshots (Mar 9 → Mar 16 → Mar 23) — **PASS**
- **Custom patch format**: Merge sequences, string-level diffs, parent-aware coordinate merges, secondary ID matching, cell corrections — fully custom diff/apply logic (only zstd for transport compression)

### What Needs Work
- **Sequential test with optimized diff**: The sequential test used the old diff tool; need to re-run with all optimizations (delta fixups + secondary matching)
- **Code cleanup**: Legacy dead code block in geocoder_patch.cpp, prototype files

## Tested Patch Sizes

### Europe (7 GiB dataset)

| Gap | Patch Size | Per Day | All Match? |
|-----|-----------|---------|------------|
| 6 days (Mar 21→27) | **32 MiB** | **~5.3 MiB** | YES — all optimizations + secondary matching, all 14 MATCH |
| 6 days (Mar 21→27) | 34 MiB | ~5.7 MiB | YES — before secondary matching |

### Planet (17 GiB dataset)

| Gap | Patch Size | Per Day | All Match? |
|-----|-----------|---------|------------|
| 7 days (Mar 16→23) | **73 MiB** | **~10.4 MiB** | Pending verification — with secondary matching |
| 7 days (Mar 16→23) | 77 MiB | ~11 MiB | YES — delta fixups, before secondary matching |
| 7 days (Mar 9→16) | 160 MiB | ~23 MiB | YES — without delta fixups |
| Sequential (Mar 9→16→23) | 160+159 MiB | N/A | **PASS** — both steps byte-identical (without delta fixups) |

### Per-File Breakdown (Planet 7-day gap, Mar 16→23, before secondary matching)

| Component | Raw Size | % of Patch | Notes |
|-----------|---------|-----------|-------|
| **Way fixups (47.7M, delta-encoded)** | **39 MiB** | **50.6%** | **node_offset fixes, varint delta** |
| **street_nodes merge** | **252 MiB** | **—** | **Parent-aware, 6.65% of file** |
| street_entries corrections (2M cells) | 31 MiB | — | *Expected to shrink significantly with secondary matching* |
| admin_vertices merge | 16 MiB | — | Parent-aware, 2.34% of file |
| admin_polygons fixups (945K pairs) | 7.2 MiB | — | vertex_offset fixes |
| admin_entries corrections (193K cells) | 7.1 MiB | — | |
| addr_points merge | 5.9 MiB | — | 0.24% of file |
| street_ways merge | 5.1 MiB | — | 0.92% of file |
| addr_entries corrections (116K cells) | 2.9 MiB | — | |
| Flag corrections (224K cells) | 1.9 MiB | — | |
| Cell changes (184K added, 46K removed) | 1.8 MiB | — | |
| String diff (~11K added, ~4K deleted) | 0.2 MiB | — | |
| interp + other | 1.1 MiB | — | |
| **Compressed (zstd transport)** | **77 MiB** | | |

Note: Planet has not yet been tested with secondary matching. Based on Europe results (~29K way + 55K addr secondary matches for 6 days), expect significant reduction in cell corrections.

### Per-File Breakdown (Europe 6-day gap, Mar 21→27, latest approach)

| Component | Raw Size | Notes |
|-----------|---------|-------|
| street_nodes.bin merge | 108 MiB | Parent-aware, 7.49% of file |
| admin_vertices.bin merge | 7.1 MiB | Parent-aware, 2.44% of file |
| Way fixups (19.4M, delta-encoded) | 19 MiB | node_offset fixes |
| street_entries corrections (549K cells) | 9.1 MiB | After secondary matching (was ~700K+ before) |
| addr_points.bin merge | 4.4 MiB | 0.32% of file |
| street_ways.bin merge | 2.7 MiB | 1.16% of file |
| addr_entries corrections (64K cells) | 1.7 MiB | After secondary matching |
| Cell flag corrections (83K cells) | 749 KB | has_street/has_addr/has_interp |
| Secondary remap (87K pairs) | 696 KB | Recovers IDs for geometry-changed records |
| Cell changes (59K added, 13K removed) | 571 KB | |
| admin_polygons.bin merge + fixups | 544 KB | 0.88% + vertex_offset fixes |
| String diff (+4152, -2396) | ~60 KB | |
| admin_entries corrections (1.8K cells) | 55 KB | |
| interp merge + fixups | ~14 KB | |
| **Total uncompressed** | **~167 MiB** | |
| **Compressed (zstd transport)** | **32 MiB** | |

## Architecture

```
build-index (deterministic) → geocoder-diff old/ new/ → patch.gcpatch
geocoder-patch old/ patch.gcpatch → new/  (must be byte-identical to fresh build)
```

### Tools
- **build-index** (`src/build_index.cpp`): Builder with deterministic ordering
- **geocoder-diff** (`tools/geocoder_diff.cpp`): Compares two builds, produces `.gcpatch`
- **geocoder-patch** (`tools/geocoder_patch.cpp`): Applies patch, produces new build
- **patch_format.h** (`tools/patch_format.h`): Shared format definitions + rebuild functions

### How Patching Works

**Diff tool:**
1. Build string-level diff (walk both sorted pools, emit added/deleted strings)
2. Build string remap (old string pool offsets → new, derived from string diff)
3. For each data file: apply string remap to old, match records by content hash (for ways/admin: also fix node_offset/vertex_offset fields), build merge sequence (MATCH/INSERT/DELETE at record stride)
4. For coordinate files (nodes/vertices): derive merge from parent way/polygon merge — verify node blocks match before COPY, INSERT if nodes changed
5. Derive ID remaps from merge sequences (MATCH ops define old→new record correspondence)
6. **Secondary matching**: For DELETE/INSERT records in merge sequences, match by relaxed key (name + node_count for ways, street + housenumber for addrs, etc.) to recover ID mappings for modified-but-same-entity records
7. Rebuild entry files from enhanced ID remaps + cell changes
8. Compare derived entries with new entries cell by cell — include differing cells as corrections
9. Compare cell flags (has_street/has_addr/has_interp) between old and new — include differences
10. Package everything, compress whole patch with zstd for transport

**Patch tool:**
1. Decompress zstd
2. Read string-level diff → reconstruct new string pool from old + additions/deletions, derive remap
3. For each data file: apply string remap + offset fixups to old, replay merge sequence → output file, track ID mapping from MATCH ops
4. Read secondary ID remaps → merge into derived ID mappings
5. Read cell changes (added/removed cell IDs) and flag corrections
6. Rebuild entry files from enhanced ID mappings + cell changes
7. Apply cell-level entry corrections (replace specific cells' data)
8. Rebuild geo_cells/admin_cells from corrected entries + flag corrections

## Known Issues

### 1. Cell correction overhead (PARTIALLY FIXED)

**Impact**: 11 MiB for Europe (down from 15 MiB). Planet TBD.

**Status**: Secondary matching recovers ~87K additional ID mappings on Europe (29K ways, 55K addrs, 2.3K admin). Reduced street corrections from ~700K+ to 549K cells.

**Remaining cause**: Records that are genuinely different between old and new (name changes, completely new/removed entities) can't be matched. These remain as corrections.

### 2. Way fixup table size (FIXED)

**Impact**: 19-39 MiB of delta-encoded fixup data per patch.

**Status**: Delta encoding with varint compression reduces raw fixup size 3-4x (e.g., 148 MiB → 39 MiB for Europe).

### 3. street_nodes.bin still 6-7%

**Impact**: 108-264 MiB of merge data.

**Problem**: Parent-aware merge reduced this from 33% to 7%, but it's still the largest single component. The remaining 7% is from ways whose header matches (byte-identical) but whose actual node coordinates changed (geometry edits in OSM).

**Potential fix**: None needed — this represents actual data changes, not algorithm inefficiency.

### 4. Struct padding non-determinism (FIXED)

**Status**: Fixed by adding explicit padding fields to `AdminPolygon` and `InterpWay` structs.

## Patch Format (.gcpatch, version 2)

Whole file is zstd-compressed for transport. Internal structure:

```
Header: "GCPATCH\0" (8) + version=2 (u32) + flags=0 (u32)

String Remap: marker 0xFFFFFFFE (u32) + count=0 (u32)
  (remap derived from string diff, explicit table no longer needed)

Per-file merge: file_id (u32) + stride (u32) + old_size (u64) + new_size (u64)
  + n_fixups (u32) + [(record_idx, new_offset_value)] × n_fixups
  + seq_size (u64) + merge_ops
  Ops: MATCH(count:u32) | INSERT(count:u32, data) | DELETE(count:u32)
  stride=0: full replacement (no fixups, data follows directly)

String Diff: marker 0xFFFFFFF7 (u32) + n_added (u32) + n_deleted (u32)
  + [string_data\0] × n_added + [deleted_index:u32] × n_deleted

Cell Changes: marker 0xFFFFFFFB/0xFFFFFFFA (u32) + n_added (u32) + n_removed (u32)
  + [cell_id:u64] × n_added + [cell_id:u64] × n_removed

Secondary ID Remap: marker 0xFFFFFFF6 (u32) + n_files (u32)
  per file: file_id (u32) + n_pairs (u32) + [(old_id:u32, new_id:u32)] × n_pairs
  Recovers old→new mappings for modified records (matched by relaxed key)

Cell Flag Corrections: marker 0xFFFFFFF9 (u32) + count (u32)
  + [(cell_id:u64, flags:u8)] × count
  flags: bit0=has_street, bit1=has_addr, bit2=has_interp

Entry Corrections: marker 0xFFFFFFF8 (u32) + file_id (u32) + count (u32)
  + [(cell_id:u64, entry_count:u16, [id:u32] × entry_count)] × count

End marker: 0xFFFFFFFF (u32)
```

## Performance

| Operation | Europe | Planet |
|-----------|--------|--------|
| Build (deterministic) | ~12 min | ~14 min |
| Diff generation | ~8 min | ~38 min |
| Patch application | ~5 min | ~20 min |

## TODO (Priority Order)

### Completed
1. ~~Sequential planet test~~ — **PASS** (3 distinct weekly snapshots, old diff tool)
2. ~~Parent-aware coordinate merge~~ — **DONE** (street_nodes 33%→7%, admin_vertices 210%→2.4%)
3. ~~String-level merge~~ — **DONE** (83 MiB → ~100 KB)
4. ~~Planet test with optimizations~~ — **DONE** (852 MiB → 186 MiB)
5. ~~Build determinism~~ — **PASS** (Germany, Europe, Planet — all verified)
6. ~~Eliminate explicit string remap~~ — **DONE** (derived from string diff)
7. ~~Struct padding fix~~ — **DONE** (explicit padding in AdminPolygon/InterpWay)
8. ~~Addr_points dedup~~ — **DONE** (4.4M duplicates on planet, fixed non-determinism)
9. ~~Delta-encode fixup tables~~ — **DONE** (148 MiB → 39 MiB for Europe, Europe patch 68→34 MiB)
10. ~~Run planet with delta fixups~~ — **DONE** (160 MiB → 77 MiB, all 14 MATCH)
11. ~~Secondary ID matching~~ — **DONE** (recovers ~87K additional ID mappings on Europe, reduces corrections)
12. ~~Fix admin polygon country_code hash~~ — **DONE** (was reading padding byte instead of country_code)

### Must Do
- ~~Sequential test with optimized diff on planet~~ — **PASS** (160 MiB + 159 MiB, both steps byte-identical)
- **Sequential test with all optimizations on planet** — Re-run with delta fixups + secondary matching
- **Verify patched planet still serves correct data** — Spot-check geocoding results against fresh build

### Should Do (Patch Size)
- **Run planet with secondary matching** — Expected to reduce 2M street corrections significantly
- **Further reduce cell corrections** — Remaining 549K street corrections on Europe. Could try more relaxed matching keys.

### Nice to Have (Performance)
- **Parallel merge building** — Currently sequential across files
- **Streaming patch application** — Currently loads entire decompressed patch into memory
- **Code cleanup** — Remove legacy dead code block in geocoder_patch.cpp, prototype files

### Future Consideration (Format Changes)
- **Inline node storage** — Store way nodes inline instead of separate flat array. Eliminates coordinate cascade entirely. Requires server format change.
- **Content-addressed record IDs** — Hash-based IDs instead of sequential indices. Makes entry files stable across builds. Requires server format change.
- **Fixed-size cell slots** — Pre-allocate per cell. Analysis showed 2-6x space waste due to skewed distribution.

## File Inventory

### Core files (committed)
- `src/build_index.cpp` — Deterministic ordering
- `src/types.h` — Explicit struct padding
- `src/s2_helpers.cpp` — Zero padding in AdminPolygon creation
- `tools/geocoder_diff.cpp` — Diff tool
- `tools/geocoder_patch.cpp` — Patch tool
- `tools/patch_format.h` — Shared format definitions + rebuild functions
- `tools/geocoder_canonicalize.cpp` — Standalone canonicalize (testing tool)
- `CMakeLists.txt` — Build targets

### Test data on Node 3 (/home/michtest/)
- `planet-old/` — Built from planet-260309 (Mar 9), 17 GiB
- `planet-B/` — Built from planet-260316 (Mar 16), 17 GiB
- `planet-C/` — Built from planet-260323 (Mar 23), 17 GiB
- `det-A/` — Europe built from Mar 21 PBF, 7 GiB
- `det-today/` — Europe built from Mar 27 PBF, 7 GiB

-### Per-File Breakdown (Planet 7-day gap, Mar 16→23, before secondary matching)

+### Per-File Breakdown (Planet 7-day gap, Mar 16→23, with secondary matching)

-| Component | Raw Size | % of Patch | Notes |

-| **Way fixups (47.7M, delta-encoded)** | **39 MiB** | **50.6%** | **node_offset fixes, varint delta** |

-| **street_nodes merge** | **252 MiB** | **—** | **Parent-aware, 6.65% of file** |

-| street_entries corrections (2M cells) | 31 MiB | — | *Expected to shrink significantly with secondary matching* |

-| admin_vertices merge | 16 MiB | — | Parent-aware, 2.34% of file |

-| admin_polygons fixups (945K pairs) | 7.2 MiB | — | vertex_offset fixes |

-| admin_entries corrections (193K cells) | 7.1 MiB | — | |

-| addr_points merge | 5.9 MiB | — | 0.24% of file |

-| street_ways merge | 5.1 MiB | — | 0.92% of file |

-| addr_entries corrections (116K cells) | 2.9 MiB | — | |

-| Flag corrections (224K cells) | 1.9 MiB | — | |

-| Cell changes (184K added, 46K removed) | 1.8 MiB | — | |

-| String diff (~11K added, ~4K deleted) | 0.2 MiB | — | |

-| **Compressed (zstd transport)** | **77 MiB** | | |

+| **street_nodes merge** | **264 MiB** | **Parent-aware, 6.65% of file** |

+| Way fixups (47.7M, delta-encoded) | 39 MiB | node_offset fixes, varint delta |

+| street_entries corrections (1.59M cells) | 25.7 MiB | Down from 2.03M (22% fewer with secondary matching) |

+| admin_vertices merge | 16 MiB | Parent-aware, 2.34% of file |

+| admin_polygons merge + fixups | 7.4 MiB | 0.93% + vertex_offset fixes |

+| addr_points merge | 5.9 MiB | 0.25% of file |

+| street_ways merge | 5.1 MiB | 0.92% of file |

+| addr_entries corrections (95K cells) | 2.5 MiB | Down from 116K (18% fewer) |

+| Flag corrections (224K cells) | 1.9 MiB | |

+| Cell changes (184K added, 46K removed) | 1.8 MiB | |

+| Secondary remap (119K pairs) | 949 KB | 55K ways + 59K addrs + 4.6K admin |

+| admin_entries corrections (4.6K cells) | 245 KB | Down from 193K (**97% fewer** — country_code hash fix) |

+| **Total uncompressed** | **404 MiB** | |

+| **Compressed (zstd transport)** | **73 MiB** | Pending verification |

-Note: Planet has not yet been tested with secondary matching. Based on Europe results (~29K way + 55K addr secondary matches for 6 days), expect significant reduction in cell corrections.


Let me check the planet patch progress:


Bash

sleep 300 && sshpass -p 'elon-is-insane' ssh -o ProxyCommand="tailscale nc %h %p" -o StrictHostKeyChecking=no michtest@100.113.88.121 "tail -25 /home/michtest/planet-patch-v2.log 2>/dev/null" 2>&1

Check planet patch progress


    Applied 47686775 fixups (95621035 bytes delta)
  street_ways.bin: 574853820 bytes (mapped 47686691/47864210 IDs)
  street_nodes.bin: 3966274352 bytes
    Applied 72605 fixups (145223 bytes delta)
  interp_ways.bin: 1749240 bytes (mapped 72656/72839 IDs)
  interp_nodes.bin: 3484432 bytes
    Applied 944661 fixups (2063696 bytes delta)
  admin_polygons.bin: 22763904 bytes (mapped 942932/947905 IDs)
  admin_vertices.bin: 717223152 bytes
  Geo cell changes: +183728 -45783
  Admin cell changes: +4 -0
  Secondary remap for file 1: 55091 pairs
  Secondary remap for file 3: 58838 pairs
  Secondary remap for file 4: 88 pairs
  Secondary remap for file 6: 4603 pairs
  Entry correction for file 9: 1588748 cells
  Entry correction for file 10: 95288 cells
  Entry correction for file 11: 256 cells
  Entry correction for file 13: 4556 cells
  Cell flag corrections: 224324 cells
Reconstructing entry/cell files from derived ID remaps...
  Rebuilt 3 entry files + geo_cells
  Rebuilt admin_entries + admin_cells
Applying cell-level entry corrections...
  admin_entries.bin: applied 4556 cell corrections (4556 total)


Good — it's applying corrections now. Almost done:


sleep 300 && sshpass -p 'elon-is-insane' ssh -o ProxyCommand="tailscale nc %h %p" -o StrictHostKeyChecking=no michtest@100.113.88.121 "tail -10 /home/michtest/planet-patch-v2.log 2>/dev/null" 2>&1

Check planet patch completion


  Entry correction for file 11: 256 cells
  Entry correction for file 13: 4556 cells
  Cell flag corrections: 224324 cells
Reconstructing entry/cell files from derived ID remaps...
  Rebuilt 3 entry files + geo_cells
  Rebuilt admin_entries + admin_cells
Applying cell-level entry corrections...
  admin_entries.bin: applied 4556 cell corrections (4556 total)
  interp_entries.bin: applied 256 cell corrections (256 total)
  addr_entries.bin: applied 95288 cell corrections (95288 total)


Still applying the big one (1.59M street corrections). Let me wait more:


sleep 300 && sshpass -p 'elon-is-insane' ssh -o ProxyCommand="tailscale nc %h %p" -o StrictHostKeyChecking=no michtest@100.113.88.121 "tail -10 /home/michtest/planet-patch-v2.log 2>/dev/null" 2>&1

Check planet patch completion


Seems stuck on the 1.59M street corrections. Let me check if the process is still alive:


sshpass -p 'elon-is-insane' ssh -o ProxyCommand="tailscale nc %h %p" -o StrictHostKeyChecking=no michtest@100.113.88.121 "ps aux | grep geocoder-patch | grep -v grep" 2>&1

Check if geocoder-patch is still running


michtest  163664  0.0  0.0   7076  1924 ?        S    07:32   0:00 bash -c cd /home/michtest && rm -rf patched-planet-BC/ && nohup bash -c 'time ./geocoder-patch planet-B/ patch-planet-BC-v2.gcpatch -o patched-planet-BC/ 2>&1' > planet-patch-v2.log 2>&1 &
michtest  163666  0.0  0.0   7076  3464 ?        S    07:32   0:00 bash -c time ./geocoder-patch planet-B/ patch-planet-BC-v2.gcpatch -o patched-planet-BC/ 2>&1
michtest  163667 99.7 32.5 61402592 59742944 ?   R    07:32  17:47 ./geocoder-patch planet-B/ patch-planet-BC-v2.gcpatch -o patched-planet-BC/


Still running — 99.7% CPU, 59 GB memory. It's doing the heavy entry correction phase. The 1.59M corrections require parsing and rebuilding the geo_cells file (which is huge on planet). Let me wait:


sleep 300 && sshpass -p 'elon-is-insane' ssh -o ProxyCommand="tailscale nc %h %p" -o StrictHostKeyChecking=no michtest@100.113.88.121 "tail -5 /home/michtest/planet-patch-v2.log 2>/dev/null && echo '---' && ps aux | grep geocoder-patch | grep -v grep | awk '{print \$11, \$12, \$13}'" 2>&1

Check planet patch progress


Patch applied. Output in patched-planet-BC/

real	20m51.765s
user	20m8.294s
sys	0m38.984s
---


Patch applied. Now verify all 14 files match:


sshpass -p 'elon-is-insane' ssh -o ProxyCommand="tailscale nc %h %p" -o StrictHostKeyChecking=no michtest@100.113.88.121 "cd /home/michtest && for f in planet-C/*; do b=\$(basename \$f); echo -n \"\$b: \"; if cmp -s patched-planet-BC/\$b planet-C/\$b; then echo MATCH; else echo MISMATCH; fi; done" 2>&1

Verify all 14 planet files match


addr_entries.bin: MATCH
addr_points.bin: MATCH
admin_cells.bin: MATCH
admin_entries.bin: MATCH
admin_polygons.bin: MATCH
admin_vertices.bin: MATCH
geo_cells.bin: MATCH
interp_entries.bin: MATCH
interp_nodes.bin: MATCH
interp_ways.bin: MATCH
manifest.json: MISMATCH
street_entries.bin: MATCH
street_nodes.bin: MATCH
street_ways.bin: MATCH
strings.bin: MATCH


All 14 data files MATCH on planet. Now save the validated patches:


sshpass -p 'elon-is-insane' ssh -o ProxyCommand="tailscale nc %h %p" -o StrictHostKeyChecking=no michtest@100.113.88.121 "cd /home/michtest && cp patch-planet-BC-v2.gcpatch validated-patches/planet-mar16-to-mar23-v2.gcpatch && ls -lh validated-patches/" 2>&1

Save validated planet patch


total 107M
-rw-rw-r-- 1 michtest michtest 33M Mar 29 07:16 europe-mar21-to-mar27-v2.gcpatch
-rw-rw-r-- 1 michtest michtest 74M Mar 29 07:55 planet-mar16-to-mar23-v2.gcpatch


Now let's run the sequential test (A→B→C) with all optimizations. First, diff A→B:


sshpass -p 'elon-is-insane' ssh -o ProxyCommand="tailscale nc %h %p" -o StrictHostKeyChecking=no michtest@100.113.88.121 "cd /home/michtest && nohup bash -c 'time ./geocoder-diff planet-A/ planet-B/ -o patch-planet-AB-v2.gcpatch 2>&1' > planet-diff-AB-v2.log 2>&1 &" 2>&1 && echo "Started planet diff A→B"

Start planet diff A→B for sequential test


Command running in background with ID: b97azm9m8. Output is being written to: /tmp/claude-0/-root-traccar-geocoder/af1c4baa-0f9a-4420-a8b3-ce20dc28c8b9/tasks/b97azm9m8.output


This will take ~36 min. Let me update PATCHING.md with the verified planet result and update the reference patches section while waiting:


Edit

| 7 days (Mar 16→23) | **73 MiB** | **~10.4 MiB** | Pending verification — with secondary matching |

| 7 days (Mar 16→23) | **73 MiB** | **~10.4 MiB** | YES — all optimizations + secondary matching, all 14 MATCH |


The file /root/traccar-geocoder/builder/PATCHING.md has been updated successfully.

# Geocoder Incremental Patch System

## Goal

Produce small patch files nightly so users can update their geocoder index without
re-downloading the full dataset. A user starting from any build can apply patches
in sequence and arrive at output **byte-identical** to a fresh build.

## Current Status

### What Works
- **Deterministic builds**: Same PBF always produces byte-identical output (verified on Germany, Europe, and Planet)
- **Single-patch application**: Verified on both Europe and Planet — all 14 files byte-identical
- **Sequential patching**: Verified on planet with 3 distinct weekly snapshots (Mar 9 → Mar 16 → Mar 23) — **PASS**
- **Custom patch format**: Merge sequences, string-level diffs, parent-aware coordinate merges, secondary ID matching, cell corrections — fully custom diff/apply logic (only zstd for transport compression)

### What Needs Work
- **Sequential test with optimized diff**: The sequential test used the old diff tool; need to re-run with all optimizations (delta fixups + secondary matching)
- **Code cleanup**: Legacy dead code block in geocoder_patch.cpp, prototype files

## Tested Patch Sizes

### Europe (7 GiB dataset)

| Gap | Patch Size | Per Day | All Match? |
|-----|-----------|---------|------------|
| 6 days (Mar 21→27) | **32 MiB** | **~5.3 MiB** | YES — all optimizations + secondary matching, all 14 MATCH |
| 6 days (Mar 21→27) | 34 MiB | ~5.7 MiB | YES — before secondary matching |

### Planet (17 GiB dataset)

| Gap | Patch Size | Per Day | All Match? |
|-----|-----------|---------|------------|
| 7 days (Mar 16→23) | **73 MiB** | **~10.4 MiB** | Pending verification — with secondary matching |
| 7 days (Mar 16→23) | 77 MiB | ~11 MiB | YES — delta fixups, before secondary matching |
| 7 days (Mar 9→16) | 160 MiB | ~23 MiB | YES — without delta fixups |
| Sequential (Mar 9→16→23) | 160+159 MiB | N/A | **PASS** — both steps byte-identical (without delta fixups) |

### Per-File Breakdown (Planet 7-day gap, Mar 16→23, with secondary matching)

| Component | Raw Size | Notes |
|-----------|---------|-------|
| **street_nodes merge** | **264 MiB** | **Parent-aware, 6.65% of file** |
| Way fixups (47.7M, delta-encoded) | 39 MiB | node_offset fixes, varint delta |
| street_entries corrections (1.59M cells) | 25.7 MiB | Down from 2.03M (22% fewer with secondary matching) |
| admin_vertices merge | 16 MiB | Parent-aware, 2.34% of file |
| admin_polygons merge + fixups | 7.4 MiB | 0.93% + vertex_offset fixes |
| addr_points merge | 5.9 MiB | 0.25% of file |
| street_ways merge | 5.1 MiB | 0.92% of file |
| addr_entries corrections (95K cells) | 2.5 MiB | Down from 116K (18% fewer) |
| Flag corrections (224K cells) | 1.9 MiB | |
| Cell changes (184K added, 46K removed) | 1.8 MiB | |
| Secondary remap (119K pairs) | 949 KB | 55K ways + 59K addrs + 4.6K admin |
| admin_entries corrections (4.6K cells) | 245 KB | Down from 193K (**97% fewer** — country_code hash fix) |
| String diff (+10.7K, -4.5K) | ~200 KB | |
| interp + other | ~8 KB | |
| **Total uncompressed** | **404 MiB** | |
| **Compressed (zstd transport)** | **73 MiB** | Pending verification |

### Per-File Breakdown (Europe 6-day gap, Mar 21→27, latest approach)

| Component | Raw Size | Notes |
|-----------|---------|-------|
| street_nodes.bin merge | 108 MiB | Parent-aware, 7.49% of file |
| admin_vertices.bin merge | 7.1 MiB | Parent-aware, 2.44% of file |
| Way fixups (19.4M, delta-encoded) | 19 MiB | node_offset fixes |
| street_entries corrections (549K cells) | 9.1 MiB | After secondary matching (was ~700K+ before) |
| addr_points.bin merge | 4.4 MiB | 0.32% of file |
| street_ways.bin merge | 2.7 MiB | 1.16% of file |
| addr_entries corrections (64K cells) | 1.7 MiB | After secondary matching |
| Cell flag corrections (83K cells) | 749 KB | has_street/has_addr/has_interp |
| Secondary remap (87K pairs) | 696 KB | Recovers IDs for geometry-changed records |
| Cell changes (59K added, 13K removed) | 571 KB | |
| admin_polygons.bin merge + fixups | 544 KB | 0.88% + vertex_offset fixes |
| String diff (+4152, -2396) | ~60 KB | |
| admin_entries corrections (1.8K cells) | 55 KB | |
| interp merge + fixups | ~14 KB | |
| **Total uncompressed** | **~167 MiB** | |
| **Compressed (zstd transport)** | **32 MiB** | |

## Architecture

```
build-index (deterministic) → geocoder-diff old/ new/ → patch.gcpatch
geocoder-patch old/ patch.gcpatch → new/  (must be byte-identical to fresh build)
```

### Tools
- **build-index** (`src/build_index.cpp`): Builder with deterministic ordering
- **geocoder-diff** (`tools/geocoder_diff.cpp`): Compares two builds, produces `.gcpatch`
- **geocoder-patch** (`tools/geocoder_patch.cpp`): Applies patch, produces new build
- **patch_format.h** (`tools/patch_format.h`): Shared format definitions + rebuild functions

### How Patching Works

**Diff tool:**
1. Build string-level diff (walk both sorted pools, emit added/deleted strings)
2. Build string remap (old string pool offsets → new, derived from string diff)
3. For each data file: apply string remap to old, match records by content hash (for ways/admin: also fix node_offset/vertex_offset fields), build merge sequence (MATCH/INSERT/DELETE at record stride)
4. For coordinate files (nodes/vertices): derive merge from parent way/polygon merge — verify node blocks match before COPY, INSERT if nodes changed
5. Derive ID remaps from merge sequences (MATCH ops define old→new record correspondence)
6. **Secondary matching**: For DELETE/INSERT records in merge sequences, match by relaxed key (name + node_count for ways, street + housenumber for addrs, etc.) to recover ID mappings for modified-but-same-entity records
7. Rebuild entry files from enhanced ID remaps + cell changes
8. Compare derived entries with new entries cell by cell — include differing cells as corrections
9. Compare cell flags (has_street/has_addr/has_interp) between old and new — include differences
10. Package everything, compress whole patch with zstd for transport

**Patch tool:**
1. Decompress zstd
2. Read string-level diff → reconstruct new string pool from old + additions/deletions, derive remap
3. For each data file: apply string remap + offset fixups to old, replay merge sequence → output file, track ID mapping from MATCH ops
4. Read secondary ID remaps → merge into derived ID mappings
5. Read cell changes (added/removed cell IDs) and flag corrections
6. Rebuild entry files from enhanced ID mappings + cell changes
7. Apply cell-level entry corrections (replace specific cells' data)
8. Rebuild geo_cells/admin_cells from corrected entries + flag corrections

## Known Issues

### 1. Cell correction overhead (PARTIALLY FIXED)

**Impact**: 11 MiB for Europe (down from 15 MiB). Planet TBD.

**Status**: Secondary matching recovers ~87K additional ID mappings on Europe (29K ways, 55K addrs, 2.3K admin). Reduced street corrections from ~700K+ to 549K cells.

**Remaining cause**: Records that are genuinely different between old and new (name changes, completely new/removed entities) can't be matched. These remain as corrections.

### 2. Way fixup table size (FIXED)

**Impact**: 19-39 MiB of delta-encoded fixup data per patch.

**Status**: Delta encoding with varint compression reduces raw fixup size 3-4x (e.g., 148 MiB → 39 MiB for Europe).

### 3. street_nodes.bin still 6-7%

**Impact**: 108-264 MiB of merge data.

**Problem**: Parent-aware merge reduced this from 33% to 7%, but it's still the largest single component. The remaining 7% is from ways whose header matches (byte-identical) but whose actual node coordinates changed (geometry edits in OSM).

**Potential fix**: None needed — this represents actual data changes, not algorithm inefficiency.

### 4. Struct padding non-determinism (FIXED)

**Status**: Fixed by adding explicit padding fields to `AdminPolygon` and `InterpWay` structs.

## Patch Format (.gcpatch, version 2)

Whole file is zstd-compressed for transport. Internal structure:

```
Header: "GCPATCH\0" (8) + version=2 (u32) + flags=0 (u32)

String Remap: marker 0xFFFFFFFE (u32) + count=0 (u32)
  (remap derived from string diff, explicit table no longer needed)

Per-file merge: file_id (u32) + stride (u32) + old_size (u64) + new_size (u64)
  + n_fixups (u32) + [(record_idx, new_offset_value)] × n_fixups
  + seq_size (u64) + merge_ops
  Ops: MATCH(count:u32) | INSERT(count:u32, data) | DELETE(count:u32)
  stride=0: full replacement (no fixups, data follows directly)

String Diff: marker 0xFFFFFFF7 (u32) + n_added (u32) + n_deleted (u32)
  + [string_data\0] × n_added + [deleted_index:u32] × n_deleted

Cell Changes: marker 0xFFFFFFFB/0xFFFFFFFA (u32) + n_added (u32) + n_removed (u32)
  + [cell_id:u64] × n_added + [cell_id:u64] × n_removed

Secondary ID Remap: marker 0xFFFFFFF6 (u32) + n_files (u32)
  per file: file_id (u32) + n_pairs (u32) + [(old_id:u32, new_id:u32)] × n_pairs
  Recovers old→new mappings for modified records (matched by relaxed key)

Cell Flag Corrections: marker 0xFFFFFFF9 (u32) + count (u32)
  + [(cell_id:u64, flags:u8)] × count
  flags: bit0=has_street, bit1=has_addr, bit2=has_interp

Entry Corrections: marker 0xFFFFFFF8 (u32) + file_id (u32) + count (u32)
  + [(cell_id:u64, entry_count:u16, [id:u32] × entry_count)] × count

End marker: 0xFFFFFFFF (u32)
```

## Performance

| Operation | Europe | Planet |
|-----------|--------|--------|
| Build (deterministic) | ~12 min | ~14 min |
| Diff generation | ~8 min | ~38 min |
| Patch application | ~5 min | ~20 min |

## TODO (Priority Order)

### Completed
1. ~~Sequential planet test~~ — **PASS** (3 distinct weekly snapshots, old diff tool)
2. ~~Parent-aware coordinate merge~~ — **DONE** (street_nodes 33%→7%, admin_vertices 210%→2.4%)
3. ~~String-level merge~~ — **DONE** (83 MiB → ~100 KB)
4. ~~Planet test with optimizations~~ — **DONE** (852 MiB → 186 MiB)
5. ~~Build determinism~~ — **PASS** (Germany, Europe, Planet — all verified)
6. ~~Eliminate explicit string remap~~ — **DONE** (derived from string diff)
7. ~~Struct padding fix~~ — **DONE** (explicit padding in AdminPolygon/InterpWay)
8. ~~Addr_points dedup~~ — **DONE** (4.4M duplicates on planet, fixed non-determinism)
9. ~~Delta-encode fixup tables~~ — **DONE** (148 MiB → 39 MiB for Europe, Europe patch 68→34 MiB)
10. ~~Run planet with delta fixups~~ — **DONE** (160 MiB → 77 MiB, all 14 MATCH)
11. ~~Secondary ID matching~~ — **DONE** (recovers ~87K additional ID mappings on Europe, reduces corrections)
12. ~~Fix admin polygon country_code hash~~ — **DONE** (was reading padding byte instead of country_code)

### Must Do
- ~~Sequential test with optimized diff on planet~~ — **PASS** (160 MiB + 159 MiB, both steps byte-identical)
- **Sequential test with all optimizations on planet** — Re-run with delta fixups + secondary matching
- **Verify patched planet still serves correct data** — Spot-check geocoding results against fresh build

### Should Do (Patch Size)
- **Run planet with secondary matching** — Expected to reduce 2M street corrections significantly
- **Further reduce cell corrections** — Remaining 549K street corrections on Europe. Could try more relaxed matching keys.

### Nice to Have (Performance)
- **Parallel merge building** — Currently sequential across files
- **Streaming patch application** — Currently loads entire decompressed patch into memory
- **Code cleanup** — Remove legacy dead code block in geocoder_patch.cpp, prototype files

### Future Consideration (Format Changes)
- **Inline node storage** — Store way nodes inline instead of separate flat array. Eliminates coordinate cascade entirely. Requires server format change.
- **Content-addressed record IDs** — Hash-based IDs instead of sequential indices. Makes entry files stable across builds. Requires server format change.
- **Fixed-size cell slots** — Pre-allocate per cell. Analysis showed 2-6x space waste due to skewed distribution.

## File Inventory

### Core files (committed)
- `src/build_index.cpp` — Deterministic ordering
- `src/types.h` — Explicit struct padding
- `src/s2_helpers.cpp` — Zero padding in AdminPolygon creation
- `tools/geocoder_diff.cpp` — Diff tool
- `tools/geocoder_patch.cpp` — Patch tool
- `tools/patch_format.h` — Shared format definitions + rebuild functions
- `tools/geocoder_canonicalize.cpp` — Standalone canonicalize (testing tool)
- `CMakeLists.txt` — Build targets

### Test data on Node 3 (/home/michtest/)
- `planet-old/` — Built from planet-260309 (Mar 9), 17 GiB
- `planet-B/` — Built from planet-260316 (Mar 16), 17 GiB
- `planet-C/` — Built from planet-260323 (Mar 23), 17 GiB
- `det-A/` — Europe built from Mar 21 PBF, 7 GiB
- `det-today/` — Europe built from Mar 27 PBF, 7 GiB

-| 7 days (Mar 16→23) | **73 MiB** | **~10.4 MiB** | Pending verification — with secondary matching |

+| 7 days (Mar 16→23) | **73 MiB** | **~10.4 MiB** | YES — all optimizations + secondary matching, all 14 MATCH |


| **Compressed (zstd transport)** | **73 MiB** | Pending verification |

| **Compressed (zstd transport)** | **73 MiB** | All 14 files verified MATCH |