Conversation compacted


This session is being continued from a previous conversation that ran out of context. The summary below covers the earlier portion of the conversation.

Summary:
1. Primary Request and Intent:
   The user requested building a complete incremental patch system for their geocoder index files (~7 GiB Europe, ~17 GiB Planet). Core requirements:
   - Produce small nightly patch files so users can update without re-downloading full datasets
   - Users must be able to apply patches in sequence (day1 + patch_1→2 = day2, day2 + patch_2→3 = day3)
   - Patched output must be **byte-identical** to a fresh build from the same PBF
   - Builds must be deterministic (same PBF → same bytes every time)
   - Custom tooling that understands the binary format (not relying on generic binary diff tools)
   - Only zstd for transport compression of the final patch file
   - Track progress in a PATCHING.md document
   - Test on Europe first for fast iteration, planet only for final validation

2. Key Technical Concepts:
   - **14 binary index files**: geo_cells.bin, street_entries.bin, addr_entries.bin, interp_entries.bin, admin_cells.bin, admin_entries.bin, street_ways.bin, street_nodes.bin, addr_points.bin, interp_ways.bin, interp_nodes.bin, admin_polygons.bin, admin_vertices.bin, strings.bin
   - **Deterministic builds**: Canonical sort order for all data (strings alphabetical, records by content fingerprint with float-bit tiebreakers)
   - **Merge sequences**: MATCH/INSERT/DELETE ops at record stride for data files
   - **String-level diff**: Walk sorted string pools, emit only added/deleted strings (~100KB vs 83-215 MiB full replacement)
   - **Parent-aware coordinate merges**: Derive node/vertex merges from parent way/polygon merges instead of independent 8-byte sequential matching
   - **Content hash matching**: FNV-1a hash for matching records by semantic content (name + coordinates), not byte position
   - **Offset fixup tables**: node_offset/vertex_offset fixes for matched ways/polygons, delta-encoded with varint
   - **Cell-level entry corrections**: Custom cell-by-cell diff of derived vs new entries (no zstd --patch-from)
   - **Cell flag corrections**: has_street/has_addr/has_interp flags for cells that changed
   - **ID remap derivation**: Merge sequence MATCH ops implicitly define old→new record correspondence
   - **Addr_points dedup**: Remove byte-identical duplicate address records (4.4M on planet) for determinism
   - **Struct padding zeroing**: AdminPolygon and InterpWay have compiler-inserted padding that must be explicit/zeroed

3. Files and Code Sections:

   - **builder/src/build_index.cpp** — Main builder with deterministic ordering
     - Added deterministic sort section after dedup phase (~line 1290+)
     - Sort string pool alphabetically, remap all string offsets
     - Sort addr_points by (street_id, housenumber_id, lat_bits, lng_bits) with stable tiebreaker
     - Sort ways by (name_id, node_count, all node coordinates as float bits)
     - Sort interps by (street_id, start_number, end_number, interpolation, node coords)
     - Sort admin polygons by (name_id, admin_level, country_code, vertex_count, vertex coords)
     - Addr_points dedup: collapse consecutive identical records after sort
     ```cpp
     // Dedup consecutive identical records (planet has ~4M duplicates)
     std::vector<uint32_t> dedup_remap(n);
     size_t write_pos = 0;
     for (size_t i = 0; i < n; i++) {
         if (write_pos == 0 || memcmp(&sorted[i], &sorted[write_pos - 1], sizeof(AddrPoint)) != 0) {
             sorted[write_pos] = sorted[i];
             dedup_remap[i] = static_cast<uint32_t>(write_pos);
             write_pos++;
         } else {
             dedup_remap[i] = static_cast<uint32_t>(write_pos - 1);
         }
     }
     ```

   - **builder/src/types.h** — Binary record structs with explicit padding
     ```cpp
     struct AdminPolygon {
         uint32_t vertex_offset;
         uint32_t vertex_count;
         uint32_t name_id;
         uint8_t admin_level;
         uint8_t _pad1 = 0, _pad2 = 0, _pad3 = 0;
         float area;
         uint16_t country_code;
         uint16_t _pad4 = 0;
     };
     struct InterpWay {
         uint32_t node_offset;
         uint8_t node_count;
         uint8_t _pad1 = 0, _pad2 = 0, _pad3 = 0;
         uint32_t street_id;
         uint32_t start_number;
         uint32_t end_number;
         uint8_t interpolation;
         uint8_t _pad4 = 0, _pad5 = 0, _pad6 = 0;
     };
     ```

   - **builder/tools/geocoder_diff.cpp** — Diff tool (~900 lines)
     - Builds string-level diff (added/deleted strings)
     - Builds string remap from diff reconstruction
     - For each data file: string remap + content hash matching + offset fixups + merge sequences
     - Parent-aware coordinate merges (street_nodes from street_ways, admin_vertices from admin_polygons, interp_nodes from interp_ways)
     - Delta-encoded fixup tables (varint)
     - Derives ID remaps from merge sequences
     - Rebuilds entries, compares cell-by-cell, includes corrections
     - Cell flag corrections
     - zstd transport compression of whole patch

   - **builder/tools/geocoder_patch.cpp** — Patch tool (~700 lines)
     - Decompresses zstd transport
     - Processes string-level diff → rebuilds string pool + derives remap
     - Replays merge sequences for each data file, tracking ID mappings
     - Reads cell changes (added/removed) and flag corrections
     - Rebuilds entry files from derived ID mappings
     - Applies cell-level entry corrections
     - Rebuilds geo_cells from corrected entries + flag corrections
     - Has legacy dead code block marked with `if (false) { // REMOVED`

   - **builder/tools/patch_format.h** — Shared definitions (~450 lines)
     - Binary record structs for reading files (PatchWayHeader, PatchAddrPoint, etc.)
     - File I/O helpers (read_file, write_file, get_string, read_ways, read_structs)
     - Patch format constants (GCPATCH_MAGIC, version, PatchFileId enum, markers)
     - Shared rebuild functions: `rebuild_geo_from_remap()` and `rebuild_admin_from_remap()`
     - Varint encoding/decoding for delta-compressed fixups
     - Cell change and flag correction markers

   - **builder/PATCHING.md** — Comprehensive documentation of the system
     - Current status, tested patch sizes, per-file breakdowns
     - Architecture, how patching works, patch format spec
     - Known issues, TODO with completed/remaining items
     - Performance numbers, file inventory

   - **builder/tools/geocoder_canonicalize.cpp** — Standalone canonicalize tool (testing only)

   - **builder/CMakeLists.txt** — Build targets (build-index, geocoder-diff, geocoder-patch, geocoder-canonicalize)

4. Errors and fixes:
   - **WayHeader stride mismatch**: Builder writes 12-byte WayHeaders (with padding), not 9-byte packed. Fixed by detecting stride from file size.
   - **AdminPolygon struct padding non-determinism**: Compiler padding between admin_level and area contained random bytes. Fixed with explicit padding fields + zeroing in diff/patch tools.
   - **InterpWay struct padding**: Same issue. Fixed with explicit padding fields + zeroing.
   - **Addr_points sort non-determinism on planet**: 4.4M duplicate addr_points had identical sort keys. std::sort is unstable for equal elements. Fixed by deduplicating consecutive identical records after sorting.
   - **zstd --patch-from API vs CLI**: The zstd C API's refPrefix didn't replicate --patch-from behavior for large files. Switched to shelling out to zstd CLI, then later eliminated zstd --patch-from entirely for entry corrections.
   - **String diff section ordering**: String diff marker appeared in main section loop as "unknown file" because it was written after the remap section but the patch expected it before. Fixed by handling the marker in the main section loop.
   - **geo_cells overwrite bug**: Reconstruction step always wrote derived geo_cells, overwriting full-replacement data. Fixed with `if (!have_geo)` guard.
   - **Entry corrections applied before reconstruction**: Cell-level corrections need derived entries as reference, but corrections were processed before entries were rebuilt. Fixed by separating entry corrections from cell corrections and applying them in sequence.
   - **Admin cell changes not passed to rebuild**: `rebuild_admin_from_remap()` didn't support added/removed cells. Added parameters for cell changes.
   - **User feedback "I don't want these fallbacks"**: Removed full-replacement fallback for small corrections, replaced with proper custom cell-level entry corrections.
   - **User feedback "Why are we still using zstd patch-from"**: Eliminated all zstd --patch-from usage for entry/cell corrections, replaced with fully custom cell-level diffs.
   - **User feedback "test on Europe first"**: Saved as memory — always test new concepts on Europe before running planet.

5. Problem Solving:
   - **Deterministic builds**: Solved by canonical sorting in build_index.cpp + struct padding + addr_points dedup
   - **String offset instability**: Solved by string-level diff (walk sorted pools, emit added/deleted)
   - **Coordinate file cascade**: Solved by parent-aware merges (derive node merge from way merge)
   - **Entry file reconstruction**: Solved by deriving ID remaps from merge sequences + cell-level corrections
   - **geo_cells reconstruction**: Solved by rebuilding from corrected entries + cell flag corrections
   - **Fixup table size**: Solved by varint delta encoding (8 bytes → ~2 bytes per entry)
   - **Sequential patching**: Verified on planet with 3 distinct weekly snapshots (Mar 9→16→23)
   - **Planet duplicate PBFs**: Discovered planet.osm.pbf was actually planet-260309, not a different snapshot. Downloaded actual planet-260316 and planet-260323 for real testing.

6. All user messages:
   - "This session is being continued from a previous conversation..." (context about binary patch generation)
   - "We should come up with a patch system that works well with the changing indexes and such, if zstd patch doesn't do that, we should figure something custom out and test that"
   - "If we are going to do this in the builds, we should make sure that we aren't going to massively increase the build times"
   - "I specifically said that the offset remapping and such wouldn't work because it would mean that you couldn't apply multiple patches in a row"
   - "Lets rethink this from the beginning. What I want is a system where we can produce small patch files nightly..." (detailed requirements)
   - Plan mode approval
   - Various "keep going", "yep", approval messages
   - "Did you validate applying the patch on top of the older file and then validating that the new patched file matches the file built from source?"
   - "what about the other files?"
   - "These patches are far too big, and changing the build process like this really isn't ideal either..."
   - "keep optimising to get to the target you mentioned before. i want this entire thing to be as small as possible, ideally <50MB per day"
   - "You should keep going on the optimisations for those two things. If zstd isn't working as expected, we can still use it to compress the patch files for download, but we could have our own code to actually restore and apply the patch"
   - "Why are we still using zstd patch-from when I said we should build everything custom?"
   - "I don't want these fallbacks, we should just fix the patching so that it works properly. There is no reason that I can see why we can't get the patching working."
   - "keep working until this is fixed and everything matches, then ideally we test a sequential patch like we did on europe"
   - "we should fix the planet patching so that all files match"
   - "Remember we can test completely new concepts with europe before running full planet to speed up iteration"
   - "Should we consider alternate approaches... we could consider changing the indexes to not be numerical and instead they could be a hash potentially?"
   - "How big is the geo_cells file and how much space is this fixed-size cell slot format going to consume?"
   - "Can we run this for the planet and calculate the size of a 1 day diff?"
   - "Don't run the builds in parallel, they will thrash for memory"
   - "56 MiB seems very strange, can you verify that the patch applies and creates an identical file?"
   - Request for PATCHING.md document with full breakdowns
   - "As you make changes you can commit to this file directly to push those changes up"
   - "Can you verify build determinism on Europe and then the planet?"
   - "The total breakdowns don't make sense for the files, the numbers don't add up for the planet build per file breakdown"
   - "Can we verify that the planet files contain all the data they should still though"
   - "How much space did we save in the index with this change then?"
   - "Keep checking the build"
   - "Check more often so you can act faster when the build finishes"
   - "Check the document, update it, and then keep working through the tasks on there"
   - "Continue monitoring the planet deterministic build"
   - "ALright, seems like you know what to work on... work autonomously until I am back which will be in 8-9 hours"
   - "Awesome, lets continue working on those things now then"
   - Various "keep going" and monitoring instructions
   - There are historical planet pbf files available from osm... Take a look at those.
   - "we should fix the planet patching so that all files match"
   - "ALright, can we verify that the planet files contain all the data they should still though"
   - Various approvals and "keep working" messages

7. Pending Tasks (from PATCHING.md TODO):
   - **Reduce cell correction count** — 2M street cell corrections (31 MiB on planet). Improve fixup matching to reduce unmatched records
   - **Parallel merge building** — Currently sequential across files
   - **Streaming patch application** — Currently loads entire decompressed patch into memory
   - **Code cleanup** — Remove legacy if(false) block in geocoder_patch.cpp
   - **Sequential test with delta fixups on planet** — The sequential test used old diff without delta fixups

8. Current Work:
   The most recent exchange was:
   - User: "Awesome, lets continue working on those things now then" (referring to remaining TODO items from PATCHING.md)
   - I retrieved the planet patch breakdown showing the current composition:
     - street_entries corrections: 2,028,628 cells (32.7 MiB raw)
     - addr_entries corrections: 116,251 cells (3.0 MiB raw)
     - admin_entries corrections: 193,375 cells (7.5 MiB raw)
     - Cell flag corrections: 224,324 cells (2.0 MiB raw)
     - street_nodes parent merge: 263.7 MiB (6.65%)
     - Way fixups: 47.7M entries (now delta-encoded)

   The latest verified results are:
   - Europe 6-day: **34 MiB** (~5.7 MiB/day), all 14 files MATCH
   - Planet 7-day: **77 MiB** (~11 MiB/day), all 14 files MATCH

   Test data on Node 3:
   - `planet-A/` — planet-260309 (Mar 9), built with dedup
   - `planet-B/` — planet-260316 (Mar 16), built with dedup
   - `planet-C/` — planet-260323 (Mar 23), built with dedup
   - `det-A/` — Europe Mar 21
   - `det-today/` — Europe Mar 27

9. Optional Next Step:
   The user said "Awesome, lets continue working on those things now then" — referring to the PATCHING.md TODO items. I had just retrieved the planet patch breakdown to analyze what's most impactful. The next items by priority:

   1. **Reduce cell correction count** — The biggest remaining optimization target. 2M street cell corrections (32.7 MiB raw) are caused by ~173K unmatched ways whose IDs cascade to many cells. Could investigate why these ways don't match (maybe WayHeader padding?) and improve the fixup hash matching.

   2. **Code cleanup** — Remove the legacy `if (false) { // REMOVED` block in geocoder_patch.cpp and any other dead code.

   3. **Sequential test with delta fixups** — The sequential planet test used the old diff tool (160 MiB patches). Should re-verify with delta fixups.

If you need specific details from before compaction (like exact code snippets, error messages, or content you generated), read the full transcript at: /root/.claude/projects/-root-traccar-geocoder/af1c4baa-0f9a-4420-a8b3-ce20dc28c8b9.jsonl
Continue the conversation from where it left off without asking the user any further questions. Resume directly — do not acknowledge the summary, do not recap what was happening, do not preface with "I'll continue" or similar. Pick up the last task as if the break never happened.


Let me start by investigating the 173K unmatched ways that cascade to 2M street cell corrections. I need to understand why the content hash matching misses them.


     1→// geocoder-diff v3: Fully custom patch format using merge-sequence encoding.
     2→//
     3→// For each data file: walk old (string-remapped) and new in parallel,
     4→// match records by content, emit MATCH/INSERT/DELETE operations.
     5→// For entry/cell files: include changed cell entry data directly.
     6→// geo_cells rebuilt by patch tool from entries (not delta-patched).
     7→//
     8→// Patch format: custom binary, zstd-compressed as a whole for transport.
     9→//
    10→// Usage: geocoder-diff <old-dir> <new-dir> -o <patch-file>
    11→
    12→#include <algorithm>
    13→#include <cstdint>
    14→#include <cstdlib>
    15→#include <cstring>
    16→#include <fstream>
    17→#include <functional>
    18→#include <iomanip>
    19→#include <iostream>
    20→#include <string>
    21→#include <unistd.h>
    22→#include <unordered_map>
    23→#include <unordered_set>
    24→#include <vector>
    25→
    26→#include "patch_format.h"
    27→
    28→// --- Merge sequence encoding ---
    29→// Walk two sorted arrays (old remapped, new), matching by record bytes.
    30→// Emit: MATCH(n) = copy n records from old, INSERT(n,data) = add n new records,
    31→//       DELETE(n) = skip n old records.
    32→
    33→enum MergeOp : uint8_t { OP_MATCH_RUN = 0, OP_INSERT_RUN = 1, OP_DELETE_RUN = 2 };
    34→
    35→struct MergeSequence {
    36→    std::vector<char> data; // serialized ops
    37→
    38→    void add_match(uint32_t count) {
    39→        uint8_t op = OP_MATCH_RUN;
    40→        data.insert(data.end(), (char*)&op, (char*)&op + 1);
    41→        data.insert(data.end(), (char*)&count, (char*)&count + 4);
    42→    }
    43→    void add_delete(uint32_t count) {
    44→        uint8_t op = OP_DELETE_RUN;
    45→        data.insert(data.end(), (char*)&op, (char*)&op + 1);
    46→        data.insert(data.end(), (char*)&count, (char*)&count + 4);
    47→    }
    48→    void add_insert(const char* records, uint32_t count, size_t stride) {
    49→        uint8_t op = OP_INSERT_RUN;
    50→        data.insert(data.end(), (char*)&op, (char*)&op + 1);
    51→        data.insert(data.end(), (char*)&count, (char*)&count + 4);
    52→        data.insert(data.end(), records, records + count * stride);
    53→    }
    54→};
    55→
    56→// Build merge sequence for a data file.
    57→// old_data has string remap already applied.
    58→// Records are compared by byte equality (stride bytes).
    59→static MergeSequence build_merge_seq(
    60→    const std::vector<char>& old_data, const std::vector<char>& new_data,
    61→    size_t stride)
    62→{
    63→    size_t old_n = old_data.size() / stride;
    64→    size_t new_n = new_data.size() / stride;
    65→    size_t oi = 0, ni = 0;
    66→    MergeSequence seq;
    67→    uint32_t match_run = 0, del_run = 0;
    68→
    69→    auto flush_match = [&]() { if (match_run > 0) { seq.add_match(match_run); match_run = 0; } };
    70→    auto flush_del = [&]() { if (del_run > 0) { seq.add_delete(del_run); del_run = 0; } };
    71→
    72→    // For small strides (<=8), records aren't unique enough for hash matching.
    73→    // Use simple sequential scan instead.
    74→    bool use_hash = (stride > 8);
    75→
    76→    // Pre-build hash index of new records for fast mismatch resolution
    77→    auto record_hash = [&](const char* p, size_t s) -> uint64_t {
    78→        uint64_t h = 14695981039346656037ULL;
    79→        for (size_t i = 0; i < s; i++) { h ^= (uint8_t)p[i]; h *= 1099511628211ULL; }
    80→        return h;
    81→    };
    82→    std::unordered_multimap<uint64_t, uint32_t> new_hash;
    83→    if (use_hash) {
    84→        new_hash.reserve(new_n);
    85→        for (uint32_t i = 0; i < new_n; i++)
    86→            new_hash.emplace(record_hash(new_data.data() + i * stride, stride), i);
    87→    }
    88→
    89→    while (oi < old_n && ni < new_n) {
    90→        const char* op = old_data.data() + oi * stride;
    91→        const char* np = new_data.data() + ni * stride;
    92→
    93→        if (memcmp(op, np, stride) == 0) {
    94→            flush_del();
    95→            match_run++;
    96→            oi++; ni++;
    97→        } else if (use_hash) {
    98→            // Hash-based mismatch resolution (for records with unique content)
    99→            uint64_t oh = record_hash(op, stride);
   100→            auto range = new_hash.equal_range(oh);

// geocoder-diff v3: Fully custom patch format using merge-sequence encoding.
//
// For each data file: walk old (string-remapped) and new in parallel,
// match records by content, emit MATCH/INSERT/DELETE operations.
// For entry/cell files: include changed cell entry data directly.
// geo_cells rebuilt by patch tool from entries (not delta-patched).
//
// Patch format: custom binary, zstd-compressed as a whole for transport.
//
// Usage: geocoder-diff <old-dir> <new-dir> -o <patch-file>

#include <algorithm>
#include <cstdint>
#include <cstdlib>
#include <cstring>
#include <fstream>
#include <functional>
#include <iomanip>
#include <iostream>
#include <string>
#include <unistd.h>
#include <unordered_map>
#include <unordered_set>
#include <vector>

#include "patch_format.h"

// --- Merge sequence encoding ---
// Walk two sorted arrays (old remapped, new), matching by record bytes.
// Emit: MATCH(n) = copy n records from old, INSERT(n,data) = add n new records,
//       DELETE(n) = skip n old records.

enum MergeOp : uint8_t { OP_MATCH_RUN = 0, OP_INSERT_RUN = 1, OP_DELETE_RUN = 2 };

struct MergeSequence {
    std::vector<char> data; // serialized ops

    void add_match(uint32_t count) {
        uint8_t op = OP_MATCH_RUN;
        data.insert(data.end(), (char*)&op, (char*)&op + 1);
        data.insert(data.end(), (char*)&count, (char*)&count + 4);
    }
    void add_delete(uint32_t count) {
        uint8_t op = OP_DELETE_RUN;
        data.insert(data.end(), (char*)&op, (char*)&op + 1);
        data.insert(data.end(), (char*)&count, (char*)&count + 4);
    }
    void add_insert(const char* records, uint32_t count, size_t stride) {
        uint8_t op = OP_INSERT_RUN;
        data.insert(data.end(), (char*)&op, (char*)&op + 1);
        data.insert(data.end(), (char*)&count, (char*)&count + 4);
        data.insert(data.end(), records, records + count * stride);
    }
};

// Build merge sequence for a data file.
// old_data has string remap already applied.
// Records are compared by byte equality (stride bytes).
static MergeSequence build_merge_seq(
    const std::vector<char>& old_data, const std::vector<char>& new_data,
    size_t stride)
{
    size_t old_n = old_data.size() / stride;
    size_t new_n = new_data.size() / stride;
    size_t oi = 0, ni = 0;
    MergeSequence seq;
    uint32_t match_run = 0, del_run = 0;

    auto flush_match = [&]() { if (match_run > 0) { seq.add_match(match_run); match_run = 0; } };
    auto flush_del = [&]() { if (del_run > 0) { seq.add_delete(del_run); del_run = 0; } };

    // For small strides (<=8), records aren't unique enough for hash matching.
    // Use simple sequential scan instead.
    bool use_hash = (stride > 8);

    // Pre-build hash index of new records for fast mismatch resolution
    auto record_hash = [&](const char* p, size_t s) -> uint64_t {
        uint64_t h = 14695981039346656037ULL;
        for (size_t i = 0; i < s; i++) { h ^= (uint8_t)p[i]; h *= 1099511628211ULL; }
        return h;
    };
    std::unordered_multimap<uint64_t, uint32_t> new_hash;
    if (use_hash) {
        new_hash.reserve(new_n);
        for (uint32_t i = 0; i < new_n; i++)
            new_hash.emplace(record_hash(new_data.data() + i * stride, stride), i);
    }

    while (oi < old_n && ni < new_n) {
        const char* op = old_data.data() + oi * stride;
        const char* np = new_data.data() + ni * stride;

        if (memcmp(op, np, stride) == 0) {
            flush_del();
            match_run++;
            oi++; ni++;
        } else if (use_hash) {
            // Hash-based mismatch resolution (for records with unique content)
            uint64_t oh = record_hash(op, stride);
            auto range = new_hash.equal_range(oh);


     1→// geocoder-patch v3: Apply custom patch format.
     2→//
     3→// Decompresses zstd transport layer, then:
     4→// 1. Reads string remap, applies to old data files
     5→// 2. Replays merge sequences to reconstruct new data files
     6→// 3. Writes full-replacement entry/cell files
     7→//
     8→// Usage: geocoder-patch <current-dir> <patch-file> -o <output-dir>
     9→
    10→#include <cstdint>
    11→#include <cstdlib>
    12→#include <cstring>
    13→#include <fstream>
    14→#include <iostream>
    15→#include <string>
    16→#include <unordered_map>
    17→#include <unistd.h>
    18→#include <vector>
    19→
    20→#include "patch_format.h"
    21→
    22→// Merge ops (must match diff tool)
    23→enum MergeOp : uint8_t { OP_MATCH_RUN = 0, OP_INSERT_RUN = 1, OP_DELETE_RUN = 2 };
    24→
    25→int main(int argc, char* argv[]) {
    26→    if (argc < 5 || std::string(argv[3]) != "-o") {
    27→        std::cerr << "Usage: geocoder-patch <current-dir> <patch-file> -o <output-dir>" << std::endl;
    28→        return 1;
    29→    }
    30→    std::string cur_dir = argv[1], patch_path = argv[2], out_dir = argv[4];
    31→    ensure_dir(out_dir);
    32→    std::string tmpdir = "/tmp/geocoder-patch-" + std::to_string(getpid());
    33→    ensure_dir(tmpdir);
    34→
    35→    // Decompress zstd transport layer
    36→    std::string raw_path = tmpdir + "/patch.raw";
    37→    {
    38→        std::string cmd = "zstd -d '" + patch_path + "' -o '" + raw_path + "' -f --quiet 2>/dev/null";
    39→        if (system(cmd.c_str()) != 0) {
    40→            std::cerr << "Failed to decompress patch" << std::endl; return 1;
    41→        }
    42→    }
    43→    auto patch = read_file(raw_path);
    44→    remove(raw_path.c_str());
    45→    std::cerr << "Decompressed patch: " << patch.size() << " bytes" << std::endl;
    46→
    47→    size_t pos = 0;
    48→    auto read_u32 = [&]() -> uint32_t { uint32_t v; memcpy(&v, patch.data() + pos, 4); pos += 4; return v; };
    49→    auto read_u64 = [&]() -> uint64_t { uint64_t v; memcpy(&v, patch.data() + pos, 8); pos += 8; return v; };
    50→
    51→    // Check header
    52→    if (memcmp(patch.data(), GCPATCH_MAGIC, 8) != 0) {
    53→        std::cerr << "Invalid patch magic" << std::endl; return 1;
    54→    }
    55→    pos = 8;
    56→    uint32_t version = read_u32();
    57→    if (version != 2) { std::cerr << "Expected patch v2, got " << version << std::endl; return 1; }
    58→    uint32_t flags = read_u32();
    59→    (void)flags;
    60→
    61→    // Read string-level diff OR string remap
    62→    std::unordered_map<uint32_t, uint32_t> str_remap;
    63→    {
    64→        uint32_t marker = read_u32();
    65→
    66→        if (marker == 0xFFFFFFF7) {
    67→            // String-level diff: merge new/deleted strings into old pool
    68→            uint32_t n_added = read_u32(), n_deleted = read_u32();
    69→            std::vector<std::string> added;
    70→            for (uint32_t i = 0; i < n_added; i++) {
    71→                const char* s = patch.data() + pos;
    72→                added.push_back(s);
    73→                pos += strlen(s) + 1;
    74→            }
    75→            std::vector<uint32_t> del_idx(n_deleted);
    76→            for (uint32_t i = 0; i < n_deleted; i++) del_idx[i] = read_u32();
    77→            std::unordered_set<uint32_t> del_set(del_idx.begin(), del_idx.end());
    78→
    79→            // Parse old pool
    80→            auto old_pool = read_file(cur_dir + "/strings.bin");
    81→            std::vector<std::pair<uint32_t, std::string>> old_strs;
    82→            size_t sp = 0;
    83→            while (sp < old_pool.size()) {
    84→                const char* s = old_pool.data() + sp;
    85→                old_strs.push_back({(uint32_t)sp, std::string(s)});
    86→                sp += strlen(s) + 1;
    87→            }
    88→
    89→            // Merge: keep non-deleted + add new, sort
    90→            std::vector<std::string> merged;
    91→            for (size_t i = 0; i < old_strs.size(); i++)
    92→                if (!del_set.count(i)) merged.push_back(old_strs[i].second);
    93→            for (auto& s : added) merged.push_back(s);
    94→            std::sort(merged.begin(), merged.end());
    95→
    96→            // Build new pool
    97→            std::vector<char> new_pool;
    98→            std::unordered_map<std::string, uint32_t> new_offs;
    99→            for (auto& s : merged) {
   100→                new_offs[s] = (uint32_t)new_pool.size();

// geocoder-patch v3: Apply custom patch format.
//
// Decompresses zstd transport layer, then:
// 1. Reads string remap, applies to old data files
// 2. Replays merge sequences to reconstruct new data files
// 3. Writes full-replacement entry/cell files
//
// Usage: geocoder-patch <current-dir> <patch-file> -o <output-dir>

#include <cstdint>
#include <cstdlib>
#include <cstring>
#include <fstream>
#include <iostream>
#include <string>
#include <unordered_map>
#include <unistd.h>
#include <vector>

#include "patch_format.h"

// Merge ops (must match diff tool)
enum MergeOp : uint8_t { OP_MATCH_RUN = 0, OP_INSERT_RUN = 1, OP_DELETE_RUN = 2 };

int main(int argc, char* argv[]) {
    if (argc < 5 || std::string(argv[3]) != "-o") {
        std::cerr << "Usage: geocoder-patch <current-dir> <patch-file> -o <output-dir>" << std::endl;
        return 1;
    }
    std::string cur_dir = argv[1], patch_path = argv[2], out_dir = argv[4];
    ensure_dir(out_dir);
    std::string tmpdir = "/tmp/geocoder-patch-" + std::to_string(getpid());
    ensure_dir(tmpdir);

    // Decompress zstd transport layer
    std::string raw_path = tmpdir + "/patch.raw";
    {
        std::string cmd = "zstd -d '" + patch_path + "' -o '" + raw_path + "' -f --quiet 2>/dev/null";
        if (system(cmd.c_str()) != 0) {
            std::cerr << "Failed to decompress patch" << std::endl; return 1;
        }
    }
    auto patch = read_file(raw_path);
    remove(raw_path.c_str());
    std::cerr << "Decompressed patch: " << patch.size() << " bytes" << std::endl;

    size_t pos = 0;
    auto read_u32 = [&]() -> uint32_t { uint32_t v; memcpy(&v, patch.data() + pos, 4); pos += 4; return v; };
    auto read_u64 = [&]() -> uint64_t { uint64_t v; memcpy(&v, patch.data() + pos, 8); pos += 8; return v; };

    // Check header
    if (memcmp(patch.data(), GCPATCH_MAGIC, 8) != 0) {
        std::cerr << "Invalid patch magic" << std::endl; return 1;
    }
    pos = 8;
    uint32_t version = read_u32();
    if (version != 2) { std::cerr << "Expected patch v2, got " << version << std::endl; return 1; }
    uint32_t flags = read_u32();
    (void)flags;

    // Read string-level diff OR string remap
    std::unordered_map<uint32_t, uint32_t> str_remap;
    {
        uint32_t marker = read_u32();

        if (marker == 0xFFFFFFF7) {
            // String-level diff: merge new/deleted strings into old pool
            uint32_t n_added = read_u32(), n_deleted = read_u32();
            std::vector<std::string> added;
            for (uint32_t i = 0; i < n_added; i++) {
                const char* s = patch.data() + pos;
                added.push_back(s);
                pos += strlen(s) + 1;
            }
            std::vector<uint32_t> del_idx(n_deleted);
            for (uint32_t i = 0; i < n_deleted; i++) del_idx[i] = read_u32();
            std::unordered_set<uint32_t> del_set(del_idx.begin(), del_idx.end());

            // Parse old pool
            auto old_pool = read_file(cur_dir + "/strings.bin");
            std::vector<std::pair<uint32_t, std::string>> old_strs;
            size_t sp = 0;
            while (sp < old_pool.size()) {
                const char* s = old_pool.data() + sp;
                old_strs.push_back({(uint32_t)sp, std::string(s)});
                sp += strlen(s) + 1;
            }

            // Merge: keep non-deleted + add new, sort
            std::vector<std::string> merged;
            for (size_t i = 0; i < old_strs.size(); i++)
                if (!del_set.count(i)) merged.push_back(old_strs[i].second);
            for (auto& s : added) merged.push_back(s);
            std::sort(merged.begin(), merged.end());

            // Build new pool
            std::vector<char> new_pool;
            std::unordered_map<std::string, uint32_t> new_offs;
            for (auto& s : merged) {
                new_offs[s] = (uint32_t)new_pool.size();


   100→            auto range = new_hash.equal_range(oh);
   101→            uint32_t best_ni = UINT32_MAX;
   102→            for (auto it = range.first; it != range.second; ++it) {
   103→                if (it->second >= ni && it->second < ni + 10000 &&
   104→                    memcmp(op, new_data.data() + it->second * stride, stride) == 0) {
   105→                    if (it->second < best_ni) best_ni = it->second;
   106→                }
   107→            }
   108→            if (best_ni != UINT32_MAX && best_ni > ni) {
   109→                flush_match(); flush_del();
   110→                seq.add_insert(new_data.data() + ni * stride, best_ni - ni, stride);
   111→                ni = best_ni;
   112→            } else if (best_ni == ni) {
   113→                flush_del(); match_run++; oi++; ni++;
   114→            } else {
   115→                flush_match(); del_run++; oi++;
   116→            }
   117→        } else {
   118→            // Sequential scan (for coordinate files where records aren't unique)
   119→            size_t lookahead = std::min((size_t)200, std::min(old_n - oi, new_n - ni));
   120→            bool found = false;
   121→            for (size_t k = 1; k <= lookahead; k++) {
   122→                if (ni + k < new_n && memcmp(op, new_data.data() + (ni + k) * stride, stride) == 0) {
   123→                    flush_match(); flush_del();
   124→                    seq.add_insert(new_data.data() + ni * stride, k, stride);
   125→                    ni += k; found = true; break;
   126→                }
   127→                if (oi + k < old_n && memcmp(np, old_data.data() + (oi + k) * stride, stride) == 0) {
   128→                    flush_match(); del_run += k; oi += k; found = true; break;
   129→                }
   130→            }
   131→            if (!found) { flush_match(); del_run++; oi++; flush_del(); seq.add_insert(np, 1, stride); ni++; }
   132→        }
   133→    }
   134→
   135→    flush_match(); flush_del();
   136→    // Remaining old records: deletions
   137→    if (oi < old_n) seq.add_delete(old_n - oi);
   138→    // Remaining new records: insertions
   139→    if (ni < new_n) seq.add_insert(new_data.data() + ni * stride, new_n - ni, stride);
   140→
   141→    return seq;
   142→}
   143→
   144→// --- String remap ---
   145→
   146→static std::unordered_map<uint32_t, uint32_t> build_string_remap(
   147→    const std::vector<char>& old_pool, const std::vector<char>& new_pool)
   148→{
   149→    std::unordered_map<std::string, uint32_t> new_idx;
   150→    size_t pos = 0;
   151→    while (pos < new_pool.size()) {
   152→        const char* s = new_pool.data() + pos;
   153→        size_t len = strlen(s);
   154→        new_idx[std::string(s, len)] = static_cast<uint32_t>(pos);
   155→        pos += len + 1;
   156→    }
   157→    std::unordered_map<uint32_t, uint32_t> remap;
   158→    pos = 0;
   159→    while (pos < old_pool.size()) {
   160→        const char* s = old_pool.data() + pos;
   161→        size_t len = strlen(s);
   162→        auto it = new_idx.find(std::string(s, len));
   163→        if (it != new_idx.end())
   164→            remap[static_cast<uint32_t>(pos)] = it->second;
   165→        pos += len + 1;
   166→    }
   167→    return remap;
   168→}
   169→
   170→// --- Remap + fixup helpers ---
   171→
   172→static void remap_addr_points(std::vector<char>& data, const std::unordered_map<uint32_t,uint32_t>& rm) {
   173→    for (size_t i = 0; i + 16 <= data.size(); i += 16)
   174→        for (size_t off : {8, 12}) {
   175→            uint32_t v; memcpy(&v, data.data() + i + off, 4);
   176→            auto it = rm.find(v); if (it != rm.end()) memcpy(data.data() + i + off, &it->second, 4);
   177→        }
   178→}
   179→static void remap_field(std::vector<char>& data, size_t stride, size_t field_off,
   180→                         const std::unordered_map<uint32_t,uint32_t>& rm) {
   181→    for (size_t i = 0; i + stride <= data.size(); i += stride) {
   182→        uint32_t v; memcpy(&v, data.data() + i + field_off, 4);
   183→        auto it = rm.find(v); if (it != rm.end()) memcpy(data.data() + i + field_off, &it->second, 4);
   184→    }
   185→}
   186→
   187→// Content matching for ways (by name + nodes, ignoring node_offset)
   188→static uint64_t fnv_mix(uint64_t h, uint64_t v) { h ^= v; h *= 1099511628211ULL; return h; }
   189→
   190→static void fixup_way_offsets(std::vector<char>& old_ways, const std::vector<char>& old_nodes,
   191→                                const std::vector<char>& new_ways, const std::vector<char>& new_nodes,
   192→                                size_t stride) {
   193→    size_t name_off = (stride == 12) ? 8 : 5;
   194→    size_t old_n = old_ways.size() / stride, new_n = new_ways.size() / stride;
   195→    size_t old_nc = old_nodes.size() / 8, new_nc = new_nodes.size() / 8;
   196→
   197→    auto way_hash = [&](const char* w, const char* nodes, size_t max_n) -> uint64_t {
   198→        uint32_t node_offset, name_id; uint8_t node_count;
   199→        memcpy(&node_offset, w, 4); node_count = (uint8_t)w[4]; memcpy(&name_id, w + name_off, 4);
   200→        uint64_t h = 14695981039346656037ULL;
   201→        h = fnv_mix(h, name_id); h = fnv_mix(h, node_count);
   202→        for (uint8_t j = 0; j < node_count && (node_offset + j) < max_n; j++) {
   203→            float lat, lng;
   204→            memcpy(&lat, nodes + (node_offset + j) * 8, 4);
   205→            memcpy(&lng, nodes + (node_offset + j) * 8 + 4, 4);
   206→            h = fnv_mix(h, to_grid(lat)); h = fnv_mix(h, to_grid(lng));
   207→        }
   208→        return h;
   209→    };
   210→
   211→    std::unordered_multimap<uint64_t, uint32_t> new_map;
   212→    new_map.reserve(new_n);
   213→    for (uint32_t i = 0; i < new_n; i++)
   214→        new_map.emplace(way_hash(new_ways.data() + i * stride, new_nodes.data(), new_nc), i);
   215→
   216→    for (uint32_t i = 0; i < old_n; i++) {
   217→        uint64_t h = way_hash(old_ways.data() + i * stride, old_nodes.data(), old_nc);
   218→        auto it = new_map.find(h);
   219→        if (it != new_map.end()) {
   220→            // Fix node_offset to match new way
   221→            uint32_t new_node_off;
   222→            memcpy(&new_node_off, new_ways.data() + it->second * stride, 4);
   223→            memcpy(old_ways.data() + i * stride, &new_node_off, 4);
   224→            new_map.erase(it);
   225→        }
   226→    }
   227→}
   228→
   229→// Same for admin polygons (vertex_offset)
   230→static void fixup_admin_offsets(std::vector<char>& old_polys, const std::vector<char>& old_verts,
   231→                                  const std::vector<char>& new_polys, const std::vector<char>& new_verts,
   232→                                  size_t stride) {
   233→    size_t old_n = old_polys.size() / stride, new_n = new_polys.size() / stride;
   234→    size_t old_vc = old_verts.size() / 8, new_vc = new_verts.size() / 8;
   235→
   236→    auto poly_hash = [&](const char* p, const char* verts, size_t max_v) -> uint64_t {
   237→        uint32_t vert_offset, vert_count, name_id;
   238→        memcpy(&vert_offset, p, 4); memcpy(&vert_count, p + 4, 4); memcpy(&name_id, p + 8, 4);
   239→        uint8_t level = (uint8_t)p[12]; uint16_t cc; memcpy(&cc, p + stride - 2, 2);
   240→        uint64_t h = 14695981039346656037ULL;
   241→        h = fnv_mix(h, name_id); h = fnv_mix(h, level); h = fnv_mix(h, cc); h = fnv_mix(h, vert_count);
   242→        for (uint32_t j = 0; j < std::min(vert_count, 10u) && (vert_offset + j) < max_v; j++) {
   243→            float lat, lng;
   244→            memcpy(&lat, verts + (vert_offset + j) * 8, 4); memcpy(&lng, verts + (vert_offset + j) * 8 + 4, 4);
   245→            h = fnv_mix(h, to_grid(lat)); h = fnv_mix(h, to_grid(lng));
   246→        }
   247→        return h;
   248→    };
   249→
   250→    std::unordered_multimap<uint64_t, uint32_t> new_map;
   251→    new_map.reserve(new_n);
   252→    for (uint32_t i = 0; i < new_n; i++)
   253→        new_map.emplace(poly_hash(new_polys.data() + i * stride, new_verts.data(), new_vc), i);
   254→
   255→    for (uint32_t i = 0; i < old_n; i++) {
   256→        uint64_t h = poly_hash(old_polys.data() + i * stride, old_verts.data(), old_vc);
   257→        auto it = new_map.find(h);
   258→        if (it != new_map.end()) {
   259→            uint32_t new_vert_off;
   260→            memcpy(&new_vert_off, new_polys.data() + it->second * stride, 4);
   261→            memcpy(old_polys.data() + i * stride, &new_vert_off, 4);
   262→            new_map.erase(it);
   263→        }
   264→    }
   265→}
   266→
   267→// Same for interp ways
   268→static void fixup_interp_offsets(std::vector<char>& old_data, const std::vector<char>& old_nodes,
   269→                                   const std::vector<char>& new_data, const std::vector<char>& new_nodes,
   270→                                   size_t stride) {
   271→    size_t street_off = (stride >= 20) ? 8 : 5;
   272→    size_t old_n = old_data.size() / stride, new_n = new_data.size() / stride;
   273→    size_t old_nc = old_nodes.size() / 8, new_nc = new_nodes.size() / 8;
   274→
   275→    auto ihash = [&](const char* p, const char* nodes, size_t max_n) -> uint64_t {
   276→        uint32_t node_offset, street_id, start, end; uint8_t count, itype;
   277→        memcpy(&node_offset, p, 4); count = (uint8_t)p[4];
   278→        memcpy(&street_id, p + street_off, 4); memcpy(&start, p + street_off + 4, 4);
   279→        memcpy(&end, p + street_off + 8, 4);
   280→        itype = (street_off + 12 < stride) ? (uint8_t)p[street_off + 12] : 0;
   281→        uint64_t h = 14695981039346656037ULL;
   282→        h = fnv_mix(h, street_id); h = fnv_mix(h, start); h = fnv_mix(h, end);
   283→        h = fnv_mix(h, itype); h = fnv_mix(h, count);
   284→        for (uint8_t j = 0; j < count && (node_offset + j) < max_n; j++) {
   285→            float lat, lng;
   286→            memcpy(&lat, nodes + (node_offset + j) * 8, 4); memcpy(&lng, nodes + (node_offset + j) * 8 + 4, 4);
   287→            h = fnv_mix(h, to_grid(lat)); h = fnv_mix(h, to_grid(lng));
   288→        }
   289→        return h;
   290→    };
   291→
   292→    std::unordered_multimap<uint64_t, uint32_t> new_map;
   293→    new_map.reserve(new_n);
   294→    for (uint32_t i = 0; i < new_n; i++)
   295→        new_map.emplace(ihash(new_data.data() + i * stride, new_nodes.data(), new_nc), i);
   296→    for (uint32_t i = 0; i < old_n; i++) {
   297→        uint64_t h = ihash(old_data.data() + i * stride, old_nodes.data(), old_nc);
   298→        auto it = new_map.find(h);
   299→        if (it != new_map.end()) {

            auto range = new_hash.equal_range(oh);
            uint32_t best_ni = UINT32_MAX;
            for (auto it = range.first; it != range.second; ++it) {
                if (it->second >= ni && it->second < ni + 10000 &&
                    memcmp(op, new_data.data() + it->second * stride, stride) == 0) {
                    if (it->second < best_ni) best_ni = it->second;
                }
            }
            if (best_ni != UINT32_MAX && best_ni > ni) {
                flush_match(); flush_del();
                seq.add_insert(new_data.data() + ni * stride, best_ni - ni, stride);
                ni = best_ni;
            } else if (best_ni == ni) {
                flush_del(); match_run++; oi++; ni++;
            } else {
                flush_match(); del_run++; oi++;
            }
        } else {
            // Sequential scan (for coordinate files where records aren't unique)
            size_t lookahead = std::min((size_t)200, std::min(old_n - oi, new_n - ni));
            bool found = false;
            for (size_t k = 1; k <= lookahead; k++) {
                if (ni + k < new_n && memcmp(op, new_data.data() + (ni + k) * stride, stride) == 0) {
                    flush_match(); flush_del();
                    seq.add_insert(new_data.data() + ni * stride, k, stride);
                    ni += k; found = true; break;
                }
                if (oi + k < old_n && memcmp(np, old_data.data() + (oi + k) * stride, stride) == 0) {
                    flush_match(); del_run += k; oi += k; found = true; break;
                }
            }
            if (!found) { flush_match(); del_run++; oi++; flush_del(); seq.add_insert(np, 1, stride); ni++; }
        }
    }

    flush_match(); flush_del();
    // Remaining old records: deletions
    if (oi < old_n) seq.add_delete(old_n - oi);
    // Remaining new records: insertions
    if (ni < new_n) seq.add_insert(new_data.data() + ni * stride, new_n - ni, stride);

    return seq;
}

// --- String remap ---

static std::unordered_map<uint32_t, uint32_t> build_string_remap(
    const std::vector<char>& old_pool, const std::vector<char>& new_pool)
{
    std::unordered_map<std::string, uint32_t> new_idx;
    size_t pos = 0;
    while (pos < new_pool.size()) {
        const char* s = new_pool.data() + pos;
        size_t len = strlen(s);
        new_idx[std::string(s, len)] = static_cast<uint32_t>(pos);
        pos += len + 1;
    }
    std::unordered_map<uint32_t, uint32_t> remap;
    pos = 0;
    while (pos < old_pool.size()) {
        const char* s = old_pool.data() + pos;
        size_t len = strlen(s);
        auto it = new_idx.find(std::string(s, len));
        if (it != new_idx.end())
            remap[static_cast<uint32_t>(pos)] = it->second;
        pos += len + 1;
    }
    return remap;
}

// --- Remap + fixup helpers ---

static void remap_addr_points(std::vector<char>& data, const std::unordered_map<uint32_t,uint32_t>& rm) {
    for (size_t i = 0; i + 16 <= data.size(); i += 16)
        for (size_t off : {8, 12}) {
            uint32_t v; memcpy(&v, data.data() + i + off, 4);
            auto it = rm.find(v); if (it != rm.end()) memcpy(data.data() + i + off, &it->second, 4);
        }
}
static void remap_field(std::vector<char>& data, size_t stride, size_t field_off,
                         const std::unordered_map<uint32_t,uint32_t>& rm) {
    for (size_t i = 0; i + stride <= data.size(); i += stride) {
        uint32_t v; memcpy(&v, data.data() + i + field_off, 4);
        auto it = rm.find(v); if (it != rm.end()) memcpy(data.data() + i + field_off, &it->second, 4);
    }
}

// Content matching for ways (by name + nodes, ignoring node_offset)
static uint64_t fnv_mix(uint64_t h, uint64_t v) { h ^= v; h *= 1099511628211ULL; return h; }

static void fixup_way_offsets(std::vector<char>& old_ways, const std::vector<char>& old_nodes,
                                const std::vector<char>& new_ways, const std::vector<char>& new_nodes,
                                size_t stride) {
    size_t name_off = (stride == 12) ? 8 : 5;
    size_t old_n = old_ways.size() / stride, new_n = new_ways.size() / stride;
    size_t old_nc = old_nodes.size() / 8, new_nc = new_nodes.size() / 8;

    auto way_hash = [&](const char* w, const char* nodes, size_t max_n) -> uint64_t {
        uint32_t node_offset, name_id; uint8_t node_count;
        memcpy(&node_offset, w, 4); node_count = (uint8_t)w[4]; memcpy(&name_id, w + name_off, 4);
        uint64_t h = 14695981039346656037ULL;
        h = fnv_mix(h, name_id); h = fnv_mix(h, node_count);
        for (uint8_t j = 0; j < node_count && (node_offset + j) < max_n; j++) {
            float lat, lng;
            memcpy(&lat, nodes + (node_offset + j) * 8, 4);
            memcpy(&lng, nodes + (node_offset + j) * 8 + 4, 4);
            h = fnv_mix(h, to_grid(lat)); h = fnv_mix(h, to_grid(lng));
        }
        return h;
    };

    std::unordered_multimap<uint64_t, uint32_t> new_map;
    new_map.reserve(new_n);
    for (uint32_t i = 0; i < new_n; i++)
        new_map.emplace(way_hash(new_ways.data() + i * stride, new_nodes.data(), new_nc), i);

    for (uint32_t i = 0; i < old_n; i++) {
        uint64_t h = way_hash(old_ways.data() + i * stride, old_nodes.data(), old_nc);
        auto it = new_map.find(h);
        if (it != new_map.end()) {
            // Fix node_offset to match new way
            uint32_t new_node_off;
            memcpy(&new_node_off, new_ways.data() + it->second * stride, 4);
            memcpy(old_ways.data() + i * stride, &new_node_off, 4);
            new_map.erase(it);
        }
    }
}

// Same for admin polygons (vertex_offset)
static void fixup_admin_offsets(std::vector<char>& old_polys, const std::vector<char>& old_verts,
                                  const std::vector<char>& new_polys, const std::vector<char>& new_verts,
                                  size_t stride) {
    size_t old_n = old_polys.size() / stride, new_n = new_polys.size() / stride;
    size_t old_vc = old_verts.size() / 8, new_vc = new_verts.size() / 8;

    auto poly_hash = [&](const char* p, const char* verts, size_t max_v) -> uint64_t {
        uint32_t vert_offset, vert_count, name_id;
        memcpy(&vert_offset, p, 4); memcpy(&vert_count, p + 4, 4); memcpy(&name_id, p + 8, 4);
        uint8_t level = (uint8_t)p[12]; uint16_t cc; memcpy(&cc, p + stride - 2, 2);
        uint64_t h = 14695981039346656037ULL;
        h = fnv_mix(h, name_id); h = fnv_mix(h, level); h = fnv_mix(h, cc); h = fnv_mix(h, vert_count);
        for (uint32_t j = 0; j < std::min(vert_count, 10u) && (vert_offset + j) < max_v; j++) {
            float lat, lng;
            memcpy(&lat, verts + (vert_offset + j) * 8, 4); memcpy(&lng, verts + (vert_offset + j) * 8 + 4, 4);
            h = fnv_mix(h, to_grid(lat)); h = fnv_mix(h, to_grid(lng));
        }
        return h;
    };

    std::unordered_multimap<uint64_t, uint32_t> new_map;
    new_map.reserve(new_n);
    for (uint32_t i = 0; i < new_n; i++)
        new_map.emplace(poly_hash(new_polys.data() + i * stride, new_verts.data(), new_vc), i);

    for (uint32_t i = 0; i < old_n; i++) {
        uint64_t h = poly_hash(old_polys.data() + i * stride, old_verts.data(), old_vc);
        auto it = new_map.find(h);
        if (it != new_map.end()) {
            uint32_t new_vert_off;
            memcpy(&new_vert_off, new_polys.data() + it->second * stride, 4);
            memcpy(old_polys.data() + i * stride, &new_vert_off, 4);
            new_map.erase(it);
        }
    }
}

// Same for interp ways
static void fixup_interp_offsets(std::vector<char>& old_data, const std::vector<char>& old_nodes,
                                   const std::vector<char>& new_data, const std::vector<char>& new_nodes,
                                   size_t stride) {
    size_t street_off = (stride >= 20) ? 8 : 5;
    size_t old_n = old_data.size() / stride, new_n = new_data.size() / stride;
    size_t old_nc = old_nodes.size() / 8, new_nc = new_nodes.size() / 8;

    auto ihash = [&](const char* p, const char* nodes, size_t max_n) -> uint64_t {
        uint32_t node_offset, street_id, start, end; uint8_t count, itype;
        memcpy(&node_offset, p, 4); count = (uint8_t)p[4];
        memcpy(&street_id, p + street_off, 4); memcpy(&start, p + street_off + 4, 4);
        memcpy(&end, p + street_off + 8, 4);
        itype = (street_off + 12 < stride) ? (uint8_t)p[street_off + 12] : 0;
        uint64_t h = 14695981039346656037ULL;
        h = fnv_mix(h, street_id); h = fnv_mix(h, start); h = fnv_mix(h, end);
        h = fnv_mix(h, itype); h = fnv_mix(h, count);
        for (uint8_t j = 0; j < count && (node_offset + j) < max_n; j++) {
            float lat, lng;
            memcpy(&lat, nodes + (node_offset + j) * 8, 4); memcpy(&lng, nodes + (node_offset + j) * 8 + 4, 4);
            h = fnv_mix(h, to_grid(lat)); h = fnv_mix(h, to_grid(lng));
        }
        return h;
    };

    std::unordered_multimap<uint64_t, uint32_t> new_map;
    new_map.reserve(new_n);
    for (uint32_t i = 0; i < new_n; i++)
        new_map.emplace(ihash(new_data.data() + i * stride, new_nodes.data(), new_nc), i);
    for (uint32_t i = 0; i < old_n; i++) {
        uint64_t h = ihash(old_data.data() + i * stride, old_nodes.data(), old_nc);
        auto it = new_map.find(h);
        if (it != new_map.end()) {


   300→            uint32_t new_off; memcpy(&new_off, new_data.data() + it->second * stride, 4);
   301→            memcpy(old_data.data() + i * stride, &new_off, 4);
   302→            new_map.erase(it);
   303→        }
   304→    }
   305→}
   306→
   307→// --- Derive ID remap from merge sequence (same logic as patch tool) ---
   308→static std::vector<uint32_t> derive_id_remap_from_merge(
   309→    const MergeSequence& seq, size_t old_count, size_t stride)
   310→{
   311→    std::vector<uint32_t> id_map(old_count, 0xFFFFFFFF);
   312→    size_t pos = 0, old_rec = 0, new_rec = 0;
   313→    while (pos < seq.data.size()) {
   314→        uint8_t op = static_cast<uint8_t>(seq.data[pos]); pos++;
   315→        uint32_t count; memcpy(&count, seq.data.data() + pos, 4); pos += 4;
   316→        if (op == OP_MATCH_RUN) {
   317→            for (uint32_t k = 0; k < count; k++)
   318→                if (old_rec + k < id_map.size())
   319→                    id_map[old_rec + k] = static_cast<uint32_t>(new_rec + k);
   320→            old_rec += count; new_rec += count;
   321→        } else if (op == OP_INSERT_RUN) {
   322→            pos += count * stride;
   323→            new_rec += count;
   324→        } else if (op == OP_DELETE_RUN) {
   325→            old_rec += count;
   326→        }
   327→    }
   328→    return id_map;
   329→}
   330→
   331→// --- Write helpers ---
   332→static void wval(std::vector<char>& buf, const void* data, size_t size) {
   333→    buf.insert(buf.end(), (const char*)data, (const char*)data + size);
   334→}
   335→
   336→int main(int argc, char* argv[]) {
   337→    if (argc < 5 || std::string(argv[3]) != "-o") {
   338→        std::cerr << "Usage: geocoder-diff <old-dir> <new-dir> -o <patch-file>" << std::endl;
   339→        return 1;
   340→    }
   341→    std::string old_dir = argv[1], new_dir = argv[2], patch_path = argv[4];
   342→    std::string tmpdir = "/tmp/geocoder-diff-" + std::to_string(getpid());
   343→    ensure_dir(tmpdir);
   344→
   345→    // Build string remap
   346→    std::cerr << "Building string remap..." << std::endl;
   347→    auto old_strings = read_file(old_dir + "/strings.bin");
   348→    auto new_strings = read_file(new_dir + "/strings.bin");
   349→    auto str_remap = build_string_remap(old_strings, new_strings);
   350→
   351→    // Detect strides
   352→    auto detect = [](const std::string& path, std::vector<size_t> cs) -> size_t {
   353→        auto d = read_file(path);
   354→        for (size_t s : cs) if (d.size() % s == 0 && d.size() > 0) return s;
   355→        return cs[0];
   356→    };
   357→    size_t way_stride = detect(old_dir + "/street_ways.bin", {12, 9});
   358→    size_t interp_stride = detect(old_dir + "/interp_ways.bin", {24, 20, 18});
   359→    size_t admin_stride = detect(old_dir + "/admin_polygons.bin", {24, 20, 19});
   360→
   361→    // Build patch data (uncompressed, will be zstd-compressed at the end)
   362→    std::vector<char> patch;
   363→    patch.insert(patch.end(), GCPATCH_MAGIC, GCPATCH_MAGIC + 8);
   364→    uint32_t ver = 2, flags = 0; // version 2 = custom format
   365→    wval(patch, &ver, 4); wval(patch, &flags, 4);
   366→
   367→    // --- Section: String remap ---
   368→    // When string-level diff is present, the remap is derived by the patch tool.
   369→    // Only include explicit remap if string diff is NOT used.
   370→    // For now, always include empty remap (string diff handles everything).
   371→    {
   372→        uint32_t marker = 0xFFFFFFFE, count = 0;
   373→        wval(patch, &marker, 4); wval(patch, &count, 4);
   374→        std::cerr << "  String remap: derived from string diff (0 explicit entries)" << std::endl;
   375→    }
   376→
   377→    // --- Section: Per-file merge sequences ---
   378→    // Track fixups per file (applied before merge, included in patch for patch tool)
   379→    std::unordered_map<uint32_t, std::vector<std::pair<uint32_t,uint32_t>>> file_fixups;
   380→
   381→    // Stored merge sequences for ID remap derivation
   382→    std::unordered_map<uint32_t, MergeSequence> stored_merges;
   383→
   384→    auto write_merge = [&](PatchFileId id, const std::string& name,
   385→                             const std::vector<char>& old_data, const std::vector<char>& new_data,
   386→                             size_t stride) {
   387→        auto seq = build_merge_seq(old_data, new_data, stride);
   388→        stored_merges[static_cast<uint32_t>(id)] = seq; // store for entry rebuild
   389→        uint32_t fid = static_cast<uint32_t>(id);
   390→        uint64_t os = old_data.size(), ns = new_data.size(), ss = seq.data.size();
   391→
   392→        // Check for fixups
   393→        auto fit = file_fixups.find(fid);
   394→        uint32_t n_fixups = fit != file_fixups.end() ? static_cast<uint32_t>(fit->second.size()) : 0;
   395→
   396→        wval(patch, &fid, 4);
   397→        uint32_t st = static_cast<uint32_t>(stride);
   398→        wval(patch, &st, 4);
   399→        wval(patch, &os, 8); wval(patch, &ns, 8);
   400→        // Fixup count + delta-encoded data before merge sequence
   401→        wval(patch, &n_fixups, 4);
   402→        if (n_fixups > 0) {
   403→            std::vector<char> delta_buf;
   404→            uint32_t prev_idx = 0, prev_val = 0;
   405→            for (auto& [idx, val] : fit->second) {
   406→                write_varint(delta_buf, idx - prev_idx);
   407→                write_varint(delta_buf, val - prev_val);
   408→                prev_idx = idx;
   409→                prev_val = val;
   410→            }
   411→            uint32_t delta_size = static_cast<uint32_t>(delta_buf.size());
   412→            wval(patch, &delta_size, 4);
   413→            patch.insert(patch.end(), delta_buf.begin(), delta_buf.end());
   414→        }
   415→        wval(patch, &ss, 8);
   416→        patch.insert(patch.end(), seq.data.begin(), seq.data.end());
   417→        std::cerr << "  " << name << ": seq=" << ss << " fixups=" << n_fixups
   418→                  << " (" << std::fixed << std::setprecision(2)
   419→                  << (ns > 0 ? ss * 100.0 / ns : 0) << "%)" << std::endl;
   420→    };
   421→
   422→    // strings.bin: string-level diff (both pools are sorted alphabetically)
   423→    // Walk both pools, emit new strings to insert and indices of deleted strings.
   424→    // The patch tool merges new strings into old pool alphabetically.
   425→    {
   426→        // Parse both pools into string lists
   427→        std::vector<std::string> old_strs, new_strs;
   428→        {
   429→            size_t p = 0;
   430→            while (p < old_strings.size()) {
   431→                const char* s = old_strings.data() + p;
   432→                old_strs.push_back(s);
   433→                p += strlen(s) + 1;
   434→            }
   435→        }
   436→        {
   437→            size_t p = 0;
   438→            while (p < new_strings.size()) {
   439→                const char* s = new_strings.data() + p;
   440→                new_strs.push_back(s);
   441→                p += strlen(s) + 1;
   442→            }
   443→        }
   444→
   445→        // Merge-walk to find insertions and deletions
   446→        std::vector<std::string> added_strings;
   447→        std::vector<uint32_t> deleted_indices;
   448→        size_t oi = 0, ni = 0;
   449→        while (oi < old_strs.size() && ni < new_strs.size()) {
   450→            int c = old_strs[oi].compare(new_strs[ni]);
   451→            if (c == 0) { oi++; ni++; } // same string
   452→            else if (c < 0) { deleted_indices.push_back(oi); oi++; } // old only → deleted
   453→            else { added_strings.push_back(new_strs[ni]); ni++; } // new only → added
   454→        }
   455→        while (oi < old_strs.size()) { deleted_indices.push_back(oi); oi++; }
   456→        while (ni < new_strs.size()) { added_strings.push_back(new_strs[ni]); ni++; }
   457→
   458→        // Serialize: marker, n_added, n_deleted, added_string_data, deleted_indices
   459→        uint32_t str_diff_marker = 0xFFFFFFF7;
   460→        uint32_t n_added = static_cast<uint32_t>(added_strings.size());
   461→        uint32_t n_deleted = static_cast<uint32_t>(deleted_indices.size());
   462→        wval(patch, &str_diff_marker, 4);
   463→        wval(patch, &n_added, 4);
   464→        wval(patch, &n_deleted, 4);
   465→        for (auto& s : added_strings) {
   466→            patch.insert(patch.end(), s.begin(), s.end());
   467→            patch.push_back('\0');
   468→        }
   469→        for (auto idx : deleted_indices) wval(patch, &idx, 4);
   470→
   471→        size_t str_diff_size = 12 + (n_added > 0 ? new_strings.size() - old_strings.size() + n_deleted * 16 : 0);
   472→        std::cerr << "  strings.bin: +" << n_added << " -" << n_deleted << " strings" << std::endl;
   473→    }
   474→
   475→    // addr_points.bin
   476→    {
   477→        auto old_data = read_file(old_dir + "/addr_points.bin");
   478→        remap_addr_points(old_data, str_remap);
   479→        auto new_data = read_file(new_dir + "/addr_points.bin");
   480→        write_merge(PatchFileId::ADDR_POINTS, "addr_points.bin", old_data, new_data, 16);
   481→    }
   482→
   483→    // street_ways.bin (string remap + offset fixup for node_offset)
   484→    {
   485→        auto old_w = read_file(old_dir + "/street_ways.bin");
   486→        remap_field(old_w, way_stride, way_stride == 12 ? 8 : 5, str_remap);
   487→        auto new_w = read_file(new_dir + "/street_ways.bin");
   488→        auto old_n = read_file(old_dir + "/street_nodes.bin");
   489→        auto new_n = read_file(new_dir + "/street_nodes.bin");
   490→        // Record fixups: (record_index, new_node_offset)
   491→        auto& fixups = file_fixups[static_cast<uint32_t>(PatchFileId::STREET_WAYS)];
   492→        // Save old offsets, apply fixups, then record changes
   493→        size_t wn = old_w.size() / way_stride;
   494→        std::vector<uint32_t> old_offsets(wn);
   495→        for (size_t i = 0; i < wn; i++) memcpy(&old_offsets[i], old_w.data() + i * way_stride, 4);
   496→        fixup_way_offsets(old_w, old_n, new_w, new_n, way_stride);
   497→        for (size_t i = 0; i < wn; i++) {
   498→            uint32_t new_off; memcpy(&new_off, old_w.data() + i * way_stride, 4);
   499→            if (new_off != old_offsets[i]) fixups.push_back({static_cast<uint32_t>(i), new_off});

            uint32_t new_off; memcpy(&new_off, new_data.data() + it->second * stride, 4);
            memcpy(old_data.data() + i * stride, &new_off, 4);
            new_map.erase(it);
        }
    }
}

// --- Derive ID remap from merge sequence (same logic as patch tool) ---
static std::vector<uint32_t> derive_id_remap_from_merge(
    const MergeSequence& seq, size_t old_count, size_t stride)
{
    std::vector<uint32_t> id_map(old_count, 0xFFFFFFFF);
    size_t pos = 0, old_rec = 0, new_rec = 0;
    while (pos < seq.data.size()) {
        uint8_t op = static_cast<uint8_t>(seq.data[pos]); pos++;
        uint32_t count; memcpy(&count, seq.data.data() + pos, 4); pos += 4;
        if (op == OP_MATCH_RUN) {
            for (uint32_t k = 0; k < count; k++)
                if (old_rec + k < id_map.size())
                    id_map[old_rec + k] = static_cast<uint32_t>(new_rec + k);
            old_rec += count; new_rec += count;
        } else if (op == OP_INSERT_RUN) {
            pos += count * stride;
            new_rec += count;
        } else if (op == OP_DELETE_RUN) {
            old_rec += count;
        }
    }
    return id_map;
}

// --- Write helpers ---
static void wval(std::vector<char>& buf, const void* data, size_t size) {
    buf.insert(buf.end(), (const char*)data, (const char*)data + size);
}

int main(int argc, char* argv[]) {
    if (argc < 5 || std::string(argv[3]) != "-o") {
        std::cerr << "Usage: geocoder-diff <old-dir> <new-dir> -o <patch-file>" << std::endl;
        return 1;
    }
    std::string old_dir = argv[1], new_dir = argv[2], patch_path = argv[4];
    std::string tmpdir = "/tmp/geocoder-diff-" + std::to_string(getpid());
    ensure_dir(tmpdir);

    // Build string remap
    std::cerr << "Building string remap..." << std::endl;
    auto old_strings = read_file(old_dir + "/strings.bin");
    auto new_strings = read_file(new_dir + "/strings.bin");
    auto str_remap = build_string_remap(old_strings, new_strings);

    // Detect strides
    auto detect = [](const std::string& path, std::vector<size_t> cs) -> size_t {
        auto d = read_file(path);
        for (size_t s : cs) if (d.size() % s == 0 && d.size() > 0) return s;
        return cs[0];
    };
    size_t way_stride = detect(old_dir + "/street_ways.bin", {12, 9});
    size_t interp_stride = detect(old_dir + "/interp_ways.bin", {24, 20, 18});
    size_t admin_stride = detect(old_dir + "/admin_polygons.bin", {24, 20, 19});

    // Build patch data (uncompressed, will be zstd-compressed at the end)
    std::vector<char> patch;
    patch.insert(patch.end(), GCPATCH_MAGIC, GCPATCH_MAGIC + 8);
    uint32_t ver = 2, flags = 0; // version 2 = custom format
    wval(patch, &ver, 4); wval(patch, &flags, 4);

    // --- Section: String remap ---
    // When string-level diff is present, the remap is derived by the patch tool.
    // Only include explicit remap if string diff is NOT used.
    // For now, always include empty remap (string diff handles everything).
    {
        uint32_t marker = 0xFFFFFFFE, count = 0;
        wval(patch, &marker, 4); wval(patch, &count, 4);
        std::cerr << "  String remap: derived from string diff (0 explicit entries)" << std::endl;
    }

    // --- Section: Per-file merge sequences ---
    // Track fixups per file (applied before merge, included in patch for patch tool)
    std::unordered_map<uint32_t, std::vector<std::pair<uint32_t,uint32_t>>> file_fixups;

    // Stored merge sequences for ID remap derivation
    std::unordered_map<uint32_t, MergeSequence> stored_merges;

    auto write_merge = [&](PatchFileId id, const std::string& name,
                             const std::vector<char>& old_data, const std::vector<char>& new_data,
                             size_t stride) {
        auto seq = build_merge_seq(old_data, new_data, stride);
        stored_merges[static_cast<uint32_t>(id)] = seq; // store for entry rebuild
        uint32_t fid = static_cast<uint32_t>(id);
        uint64_t os = old_data.size(), ns = new_data.size(), ss = seq.data.size();

        // Check for fixups
        auto fit = file_fixups.find(fid);
        uint32_t n_fixups = fit != file_fixups.end() ? static_cast<uint32_t>(fit->second.size()) : 0;

        wval(patch, &fid, 4);
        uint32_t st = static_cast<uint32_t>(stride);
        wval(patch, &st, 4);
        wval(patch, &os, 8); wval(patch, &ns, 8);
        // Fixup count + delta-encoded data before merge sequence
        wval(patch, &n_fixups, 4);
        if (n_fixups > 0) {
            std::vector<char> delta_buf;
            uint32_t prev_idx = 0, prev_val = 0;
            for (auto& [idx, val] : fit->second) {
                write_varint(delta_buf, idx - prev_idx);
                write_varint(delta_buf, val - prev_val);
                prev_idx = idx;
                prev_val = val;
            }
            uint32_t delta_size = static_cast<uint32_t>(delta_buf.size());
            wval(patch, &delta_size, 4);
            patch.insert(patch.end(), delta_buf.begin(), delta_buf.end());
        }
        wval(patch, &ss, 8);
        patch.insert(patch.end(), seq.data.begin(), seq.data.end());
        std::cerr << "  " << name << ": seq=" << ss << " fixups=" << n_fixups
                  << " (" << std::fixed << std::setprecision(2)
                  << (ns > 0 ? ss * 100.0 / ns : 0) << "%)" << std::endl;
    };

    // strings.bin: string-level diff (both pools are sorted alphabetically)
    // Walk both pools, emit new strings to insert and indices of deleted strings.
    // The patch tool merges new strings into old pool alphabetically.
    {
        // Parse both pools into string lists
        std::vector<std::string> old_strs, new_strs;
        {
            size_t p = 0;
            while (p < old_strings.size()) {
                const char* s = old_strings.data() + p;
                old_strs.push_back(s);
                p += strlen(s) + 1;
            }
        }
        {
            size_t p = 0;
            while (p < new_strings.size()) {
                const char* s = new_strings.data() + p;
                new_strs.push_back(s);
                p += strlen(s) + 1;
            }
        }

        // Merge-walk to find insertions and deletions
        std::vector<std::string> added_strings;
        std::vector<uint32_t> deleted_indices;
        size_t oi = 0, ni = 0;
        while (oi < old_strs.size() && ni < new_strs.size()) {
            int c = old_strs[oi].compare(new_strs[ni]);
            if (c == 0) { oi++; ni++; } // same string
            else if (c < 0) { deleted_indices.push_back(oi); oi++; } // old only → deleted
            else { added_strings.push_back(new_strs[ni]); ni++; } // new only → added
        }
        while (oi < old_strs.size()) { deleted_indices.push_back(oi); oi++; }
        while (ni < new_strs.size()) { added_strings.push_back(new_strs[ni]); ni++; }

        // Serialize: marker, n_added, n_deleted, added_string_data, deleted_indices
        uint32_t str_diff_marker = 0xFFFFFFF7;
        uint32_t n_added = static_cast<uint32_t>(added_strings.size());
        uint32_t n_deleted = static_cast<uint32_t>(deleted_indices.size());
        wval(patch, &str_diff_marker, 4);
        wval(patch, &n_added, 4);
        wval(patch, &n_deleted, 4);
        for (auto& s : added_strings) {
            patch.insert(patch.end(), s.begin(), s.end());
            patch.push_back('\0');
        }
        for (auto idx : deleted_indices) wval(patch, &idx, 4);

        size_t str_diff_size = 12 + (n_added > 0 ? new_strings.size() - old_strings.size() + n_deleted * 16 : 0);
        std::cerr << "  strings.bin: +" << n_added << " -" << n_deleted << " strings" << std::endl;
    }

    // addr_points.bin
    {
        auto old_data = read_file(old_dir + "/addr_points.bin");
        remap_addr_points(old_data, str_remap);
        auto new_data = read_file(new_dir + "/addr_points.bin");
        write_merge(PatchFileId::ADDR_POINTS, "addr_points.bin", old_data, new_data, 16);
    }

    // street_ways.bin (string remap + offset fixup for node_offset)
    {
        auto old_w = read_file(old_dir + "/street_ways.bin");
        remap_field(old_w, way_stride, way_stride == 12 ? 8 : 5, str_remap);
        auto new_w = read_file(new_dir + "/street_ways.bin");
        auto old_n = read_file(old_dir + "/street_nodes.bin");
        auto new_n = read_file(new_dir + "/street_nodes.bin");
        // Record fixups: (record_index, new_node_offset)
        auto& fixups = file_fixups[static_cast<uint32_t>(PatchFileId::STREET_WAYS)];
        // Save old offsets, apply fixups, then record changes
        size_t wn = old_w.size() / way_stride;
        std::vector<uint32_t> old_offsets(wn);
        for (size_t i = 0; i < wn; i++) memcpy(&old_offsets[i], old_w.data() + i * way_stride, 4);
        fixup_way_offsets(old_w, old_n, new_w, new_n, way_stride);
        for (size_t i = 0; i < wn; i++) {
            uint32_t new_off; memcpy(&new_off, old_w.data() + i * way_stride, 4);
            if (new_off != old_offsets[i]) fixups.push_back({static_cast<uint32_t>(i), new_off});


   500→        }
   501→        write_merge(PatchFileId::STREET_WAYS, "street_ways.bin", old_w, new_w, way_stride);
   502→    }
   503→
   504→    // street_nodes.bin — derive from street_ways merge (parent-aware)
   505→    // Instead of merging 8-byte records independently (fails at scale),
   506→    // use the way merge to determine which node blocks to COPY vs INSERT.
   507→    {
   508→        auto& way_seq = stored_merges[static_cast<uint32_t>(PatchFileId::STREET_WAYS)];
   509→        // Read ORIGINAL old ways (before fixup) to get correct old node offsets
   510→        auto old_w_orig = read_file(old_dir + "/street_ways.bin");
   511→        auto new_w = read_file(new_dir + "/street_ways.bin");
   512→        auto old_n = read_file(old_dir + "/street_nodes.bin");
   513→        auto new_n = read_file(new_dir + "/street_nodes.bin");
   514→        // For MATCH: use original old node_offsets (not the fixup'd ones)
   515→        // For INSERT: read from new node file using new way's node_offset
   516→        auto& old_w = old_w_orig; // original offsets
   517→
   518→        // Walk the way merge to build a node merge
   519→        MergeSequence node_seq;
   520→        size_t way_oi = 0, way_ni = 0;
   521→        size_t wpos = 0;
   522→        while (wpos < way_seq.data.size()) {
   523→            uint8_t op = static_cast<uint8_t>(way_seq.data[wpos]); wpos++;
   524→            uint32_t count; memcpy(&count, way_seq.data.data() + wpos, 4); wpos += 4;
   525→
   526→            if (op == OP_MATCH_RUN) {
   527→                // Matched ways: verify node blocks are actually identical
   528→                for (uint32_t k = 0; k < count; k++) {
   529→                    uint32_t old_no; memcpy(&old_no, old_w.data() + (way_oi + k) * way_stride, 4);
   530→                    uint8_t nc = static_cast<uint8_t>(old_w[(way_oi + k) * way_stride + 4]);
   531→                    // Get new way's node_offset (from new_w at way_ni + k)
   532→                    uint32_t new_no; memcpy(&new_no, new_w.data() + (way_ni + k) * way_stride, 4);
   533→                    if (nc > 0) {
   534→                        // Check if node data is actually identical
   535→                        bool nodes_match = (old_no + nc) * 8 <= old_n.size() &&
   536→                                          (new_no + nc) * 8 <= new_n.size() &&
   537→                                          memcmp(old_n.data() + old_no * 8,
   538→                                                 new_n.data() + new_no * 8, nc * 8) == 0;
   539→                        if (nodes_match) {
   540→                            node_seq.add_match(nc);
   541→                        } else {
   542→                            // Way matched but nodes differ → INSERT new nodes
   543→                            node_seq.add_delete(nc);
   544→                            node_seq.add_insert(new_n.data() + new_no * 8, nc, 8);
   545→                        }
   546→                    }
   547→                }
   548→                way_oi += count; way_ni += count;
   549→            } else if (op == OP_INSERT_RUN) {
   550→                // New ways: their nodes are new data → INSERT from new
   551→                for (uint32_t k = 0; k < count; k++) {
   552→                    const char* new_way = way_seq.data.data() + wpos + k * way_stride;
   553→                    uint32_t new_node_off; memcpy(&new_node_off, new_way, 4);
   554→                    uint8_t nc = static_cast<uint8_t>(new_way[4]);
   555→                    if (nc > 0 && (new_node_off + nc) * 8 <= new_n.size())
   556→                        node_seq.add_insert(new_n.data() + new_node_off * 8, nc, 8);
   557→                }
   558→                wpos += count * way_stride;
   559→                way_ni += count;
   560→            } else if (op == OP_DELETE_RUN) {
   561→                // Deleted ways: skip their nodes
   562→                for (uint32_t k = 0; k < count; k++) {
   563→                    uint8_t nc = static_cast<uint8_t>(old_w[(way_oi + k) * way_stride + 4]);
   564→                    if (nc > 0) node_seq.add_delete(nc);
   565→                }
   566→                way_oi += count;
   567→            }
   568→        }
   569→
   570→        // Write the derived node merge
   571→        uint32_t fid = static_cast<uint32_t>(PatchFileId::STREET_NODES);
   572→        uint64_t os = old_n.size(), ns = new_n.size(), ss = node_seq.data.size();
   573→        uint32_t st = 8, n_fixups = 0;
   574→        wval(patch, &fid, 4); wval(patch, &st, 4);
   575→        wval(patch, &os, 8); wval(patch, &ns, 8);
   576→        wval(patch, &n_fixups, 4); wval(patch, &ss, 8);
   577→        patch.insert(patch.end(), node_seq.data.begin(), node_seq.data.end());
   578→        std::cerr << "  street_nodes.bin: parent-derived seq=" << ss << " bytes ("
   579→                  << std::fixed << std::setprecision(2) << (ns > 0 ? ss * 100.0 / ns : 0) << "%)" << std::endl;
   580→    }
   581→
   582→    // interp_ways.bin
   583→    {
   584→        auto old_data = read_file(old_dir + "/interp_ways.bin");
   585→        auto new_idata = read_file(new_dir + "/interp_ways.bin");
   586→        // Zero padding bytes for deterministic comparison
   587→        if (interp_stride == 24) {
   588→            for (size_t i = 0; i + interp_stride <= old_data.size(); i += interp_stride) {
   589→                memset(old_data.data() + i + 5, 0, 3);  // after node_count
   590→                memset(old_data.data() + i + 21, 0, 3);  // after interpolation
   591→            }
   592→            for (size_t i = 0; i + interp_stride <= new_idata.size(); i += interp_stride) {
   593→                memset(new_idata.data() + i + 5, 0, 3);
   594→                memset(new_idata.data() + i + 21, 0, 3);
   595→            }
   596→        }
   597→        remap_field(old_data, interp_stride, interp_stride >= 20 ? 8 : 5, str_remap);
   598→        auto old_n = read_file(old_dir + "/interp_nodes.bin");
   599→        auto& new_data = new_idata; // use padding-zeroed version
   600→        auto new_n = read_file(new_dir + "/interp_nodes.bin");
   601→        auto& fixups = file_fixups[static_cast<uint32_t>(PatchFileId::INTERP_WAYS)];
   602→        size_t in_count = old_data.size() / interp_stride;
   603→        std::vector<uint32_t> old_offsets(in_count);
   604→        for (size_t i = 0; i < in_count; i++) memcpy(&old_offsets[i], old_data.data() + i * interp_stride, 4);
   605→        fixup_interp_offsets(old_data, old_n, new_data, new_n, interp_stride);
   606→        for (size_t i = 0; i < in_count; i++) {
   607→            uint32_t new_off; memcpy(&new_off, old_data.data() + i * interp_stride, 4);
   608→            if (new_off != old_offsets[i]) fixups.push_back({static_cast<uint32_t>(i), new_off});
   609→        }
   610→        write_merge(PatchFileId::INTERP_WAYS, "interp_ways.bin", old_data, new_data, interp_stride);
   611→    }
   612→
   613→    // interp_nodes.bin — derive from interp_ways merge (parent-aware)
   614→    {
   615→        auto& iw_seq = stored_merges[static_cast<uint32_t>(PatchFileId::INTERP_WAYS)];
   616→        auto old_iw_data = read_file(old_dir + "/interp_ways.bin");
   617→        auto new_iw_data = read_file(new_dir + "/interp_ways.bin");
   618→        auto& old_iw = old_iw_data;
   619→        auto& new_iw = new_iw_data;
   620→        auto old_in = read_file(old_dir + "/interp_nodes.bin");
   621→        auto new_in = read_file(new_dir + "/interp_nodes.bin");
   622→        MergeSequence in_seq;
   623→        size_t iw_oi = 0, iw_ni = 0;
   624→        size_t iwpos = 0;
   625→        while (iwpos < iw_seq.data.size()) {
   626→            uint8_t op = static_cast<uint8_t>(iw_seq.data[iwpos]); iwpos++;
   627→            uint32_t count; memcpy(&count, iw_seq.data.data() + iwpos, 4); iwpos += 4;
   628→            if (op == OP_MATCH_RUN) {
   629→                for (uint32_t k = 0; k < count; k++) {
   630→                    uint32_t old_no; memcpy(&old_no, old_iw.data() + (iw_oi + k) * interp_stride, 4);
   631→                    uint8_t nc = static_cast<uint8_t>(old_iw[(iw_oi + k) * interp_stride + 4]);
   632→                    uint32_t new_no; memcpy(&new_no, new_iw.data() + (iw_ni + k) * interp_stride, 4);
   633→                    if (nc > 0) {
   634→                        bool match = (old_no + nc) * 8 <= old_in.size() &&
   635→                                    (new_no + nc) * 8 <= new_in.size() &&
   636→                                    memcmp(old_in.data() + old_no * 8, new_in.data() + new_no * 8, nc * 8) == 0;
   637→                        if (match) in_seq.add_match(nc);
   638→                        else { in_seq.add_delete(nc); in_seq.add_insert(new_in.data() + new_no * 8, nc, 8); }
   639→                    }
   640→                }
   641→                iw_oi += count; iw_ni += count;
   642→            } else if (op == OP_INSERT_RUN) {
   643→                for (uint32_t k = 0; k < count; k++) {
   644→                    const char* rec = iw_seq.data.data() + iwpos + k * interp_stride;
   645→                    uint32_t no; memcpy(&no, rec, 4);
   646→                    uint8_t nc = static_cast<uint8_t>(rec[4]);
   647→                    if (nc > 0 && (no + nc) * 8 <= new_in.size())
   648→                        in_seq.add_insert(new_in.data() + no * 8, nc, 8);
   649→                }
   650→                iwpos += count * interp_stride;
   651→                iw_ni += count;
   652→            } else if (op == OP_DELETE_RUN) {
   653→                for (uint32_t k = 0; k < count; k++) {
   654→                    uint8_t nc = static_cast<uint8_t>(old_iw[(iw_oi + k) * interp_stride + 4]);
   655→                    if (nc > 0) in_seq.add_delete(nc);
   656→                }
   657→                iw_oi += count;
   658→            }
   659→        }
   660→        uint32_t fid = static_cast<uint32_t>(PatchFileId::INTERP_NODES);
   661→        uint64_t os = old_in.size(), ns = new_in.size(), ss = in_seq.data.size();
   662→        uint32_t st = 8, n_fixups = 0;
   663→        wval(patch, &fid, 4); wval(patch, &st, 4);
   664→        wval(patch, &os, 8); wval(patch, &ns, 8);
   665→        wval(patch, &n_fixups, 4); wval(patch, &ss, 8);
   666→        patch.insert(patch.end(), in_seq.data.begin(), in_seq.data.end());
   667→        std::cerr << "  interp_nodes.bin: parent-derived seq=" << ss << " bytes" << std::endl;
   668→    }
   669→
   670→    // admin_polygons.bin
   671→    {
   672→        auto old_data = read_file(old_dir + "/admin_polygons.bin");
   673→        auto new_data = read_file(new_dir + "/admin_polygons.bin");
   674→        // Zero padding bytes (offset 13-15 between admin_level and area) for deterministic comparison
   675→        if (admin_stride == 24) {
   676→            for (size_t i = 0; i + admin_stride <= old_data.size(); i += admin_stride)
   677→                memset(old_data.data() + i + 13, 0, 3);
   678→            for (size_t i = 0; i + admin_stride <= new_data.size(); i += admin_stride)
   679→                memset(new_data.data() + i + 13, 0, 3);
   680→        }
   681→        remap_field(old_data, admin_stride, 8, str_remap);
   682→        auto old_v = read_file(old_dir + "/admin_vertices.bin");
   683→        auto new_v = read_file(new_dir + "/admin_vertices.bin");
   684→        auto& fixups = file_fixups[static_cast<uint32_t>(PatchFileId::ADMIN_POLYGONS)];
   685→        size_t an = old_data.size() / admin_stride;
   686→        std::vector<uint32_t> old_offsets(an);
   687→        for (size_t i = 0; i < an; i++) memcpy(&old_offsets[i], old_data.data() + i * admin_stride, 4);
   688→        fixup_admin_offsets(old_data, old_v, new_data, new_v, admin_stride);
   689→        for (size_t i = 0; i < an; i++) {
   690→            uint32_t new_off; memcpy(&new_off, old_data.data() + i * admin_stride, 4);
   691→            if (new_off != old_offsets[i]) fixups.push_back({static_cast<uint32_t>(i), new_off});
   692→        }
   693→        write_merge(PatchFileId::ADMIN_POLYGONS, "admin_polygons.bin", old_data, new_data, admin_stride);
   694→    }
   695→
   696→    // admin_vertices.bin — derive from admin_polygons merge (parent-aware)
   697→    {
   698→        auto& ap_seq = stored_merges[static_cast<uint32_t>(PatchFileId::ADMIN_POLYGONS)];
   699→        auto old_ap = read_file(old_dir + "/admin_polygons.bin");

        }
        write_merge(PatchFileId::STREET_WAYS, "street_ways.bin", old_w, new_w, way_stride);
    }

    // street_nodes.bin — derive from street_ways merge (parent-aware)
    // Instead of merging 8-byte records independently (fails at scale),
    // use the way merge to determine which node blocks to COPY vs INSERT.
    {
        auto& way_seq = stored_merges[static_cast<uint32_t>(PatchFileId::STREET_WAYS)];
        // Read ORIGINAL old ways (before fixup) to get correct old node offsets
        auto old_w_orig = read_file(old_dir + "/street_ways.bin");
        auto new_w = read_file(new_dir + "/street_ways.bin");
        auto old_n = read_file(old_dir + "/street_nodes.bin");
        auto new_n = read_file(new_dir + "/street_nodes.bin");
        // For MATCH: use original old node_offsets (not the fixup'd ones)
        // For INSERT: read from new node file using new way's node_offset
        auto& old_w = old_w_orig; // original offsets

        // Walk the way merge to build a node merge
        MergeSequence node_seq;
        size_t way_oi = 0, way_ni = 0;
        size_t wpos = 0;
        while (wpos < way_seq.data.size()) {
            uint8_t op = static_cast<uint8_t>(way_seq.data[wpos]); wpos++;
            uint32_t count; memcpy(&count, way_seq.data.data() + wpos, 4); wpos += 4;

            if (op == OP_MATCH_RUN) {
                // Matched ways: verify node blocks are actually identical
                for (uint32_t k = 0; k < count; k++) {
                    uint32_t old_no; memcpy(&old_no, old_w.data() + (way_oi + k) * way_stride, 4);
                    uint8_t nc = static_cast<uint8_t>(old_w[(way_oi + k) * way_stride + 4]);
                    // Get new way's node_offset (from new_w at way_ni + k)
                    uint32_t new_no; memcpy(&new_no, new_w.data() + (way_ni + k) * way_stride, 4);
                    if (nc > 0) {
                        // Check if node data is actually identical
                        bool nodes_match = (old_no + nc) * 8 <= old_n.size() &&
                                          (new_no + nc) * 8 <= new_n.size() &&
                                          memcmp(old_n.data() + old_no * 8,
                                                 new_n.data() + new_no * 8, nc * 8) == 0;
                        if (nodes_match) {
                            node_seq.add_match(nc);
                        } else {
                            // Way matched but nodes differ → INSERT new nodes
                            node_seq.add_delete(nc);
                            node_seq.add_insert(new_n.data() + new_no * 8, nc, 8);
                        }
                    }
                }
                way_oi += count; way_ni += count;
            } else if (op == OP_INSERT_RUN) {
                // New ways: their nodes are new data → INSERT from new
                for (uint32_t k = 0; k < count; k++) {
                    const char* new_way = way_seq.data.data() + wpos + k * way_stride;
                    uint32_t new_node_off; memcpy(&new_node_off, new_way, 4);
                    uint8_t nc = static_cast<uint8_t>(new_way[4]);
                    if (nc > 0 && (new_node_off + nc) * 8 <= new_n.size())
                        node_seq.add_insert(new_n.data() + new_node_off * 8, nc, 8);
                }
                wpos += count * way_stride;
                way_ni += count;
            } else if (op == OP_DELETE_RUN) {
                // Deleted ways: skip their nodes
                for (uint32_t k = 0; k < count; k++) {
                    uint8_t nc = static_cast<uint8_t>(old_w[(way_oi + k) * way_stride + 4]);
                    if (nc > 0) node_seq.add_delete(nc);
                }
                way_oi += count;
            }
        }

        // Write the derived node merge
        uint32_t fid = static_cast<uint32_t>(PatchFileId::STREET_NODES);
        uint64_t os = old_n.size(), ns = new_n.size(), ss = node_seq.data.size();
        uint32_t st = 8, n_fixups = 0;
        wval(patch, &fid, 4); wval(patch, &st, 4);
        wval(patch, &os, 8); wval(patch, &ns, 8);
        wval(patch, &n_fixups, 4); wval(patch, &ss, 8);
        patch.insert(patch.end(), node_seq.data.begin(), node_seq.data.end());
        std::cerr << "  street_nodes.bin: parent-derived seq=" << ss << " bytes ("
                  << std::fixed << std::setprecision(2) << (ns > 0 ? ss * 100.0 / ns : 0) << "%)" << std::endl;
    }

    // interp_ways.bin
    {
        auto old_data = read_file(old_dir + "/interp_ways.bin");
        auto new_idata = read_file(new_dir + "/interp_ways.bin");
        // Zero padding bytes for deterministic comparison
        if (interp_stride == 24) {
            for (size_t i = 0; i + interp_stride <= old_data.size(); i += interp_stride) {
                memset(old_data.data() + i + 5, 0, 3);  // after node_count
                memset(old_data.data() + i + 21, 0, 3);  // after interpolation
            }
            for (size_t i = 0; i + interp_stride <= new_idata.size(); i += interp_stride) {
                memset(new_idata.data() + i + 5, 0, 3);
                memset(new_idata.data() + i + 21, 0, 3);
            }
        }
        remap_field(old_data, interp_stride, interp_stride >= 20 ? 8 : 5, str_remap);
        auto old_n = read_file(old_dir + "/interp_nodes.bin");
        auto& new_data = new_idata; // use padding-zeroed version
        auto new_n = read_file(new_dir + "/interp_nodes.bin");
        auto& fixups = file_fixups[static_cast<uint32_t>(PatchFileId::INTERP_WAYS)];
        size_t in_count = old_data.size() / interp_stride;
        std::vector<uint32_t> old_offsets(in_count);
        for (size_t i = 0; i < in_count; i++) memcpy(&old_offsets[i], old_data.data() + i * interp_stride, 4);
        fixup_interp_offsets(old_data, old_n, new_data, new_n, interp_stride);
        for (size_t i = 0; i < in_count; i++) {
            uint32_t new_off; memcpy(&new_off, old_data.data() + i * interp_stride, 4);
            if (new_off != old_offsets[i]) fixups.push_back({static_cast<uint32_t>(i), new_off});
        }
        write_merge(PatchFileId::INTERP_WAYS, "interp_ways.bin", old_data, new_data, interp_stride);
    }

    // interp_nodes.bin — derive from interp_ways merge (parent-aware)
    {
        auto& iw_seq = stored_merges[static_cast<uint32_t>(PatchFileId::INTERP_WAYS)];
        auto old_iw_data = read_file(old_dir + "/interp_ways.bin");
        auto new_iw_data = read_file(new_dir + "/interp_ways.bin");
        auto& old_iw = old_iw_data;
        auto& new_iw = new_iw_data;
        auto old_in = read_file(old_dir + "/interp_nodes.bin");
        auto new_in = read_file(new_dir + "/interp_nodes.bin");
        MergeSequence in_seq;
        size_t iw_oi = 0, iw_ni = 0;
        size_t iwpos = 0;
        while (iwpos < iw_seq.data.size()) {
            uint8_t op = static_cast<uint8_t>(iw_seq.data[iwpos]); iwpos++;
            uint32_t count; memcpy(&count, iw_seq.data.data() + iwpos, 4); iwpos += 4;
            if (op == OP_MATCH_RUN) {
                for (uint32_t k = 0; k < count; k++) {
                    uint32_t old_no; memcpy(&old_no, old_iw.data() + (iw_oi + k) * interp_stride, 4);
                    uint8_t nc = static_cast<uint8_t>(old_iw[(iw_oi + k) * interp_stride + 4]);
                    uint32_t new_no; memcpy(&new_no, new_iw.data() + (iw_ni + k) * interp_stride, 4);
                    if (nc > 0) {
                        bool match = (old_no + nc) * 8 <= old_in.size() &&
                                    (new_no + nc) * 8 <= new_in.size() &&
                                    memcmp(old_in.data() + old_no * 8, new_in.data() + new_no * 8, nc * 8) == 0;
                        if (match) in_seq.add_match(nc);
                        else { in_seq.add_delete(nc); in_seq.add_insert(new_in.data() + new_no * 8, nc, 8); }
                    }
                }
                iw_oi += count; iw_ni += count;
            } else if (op == OP_INSERT_RUN) {
                for (uint32_t k = 0; k < count; k++) {
                    const char* rec = iw_seq.data.data() + iwpos + k * interp_stride;
                    uint32_t no; memcpy(&no, rec, 4);
                    uint8_t nc = static_cast<uint8_t>(rec[4]);
                    if (nc > 0 && (no + nc) * 8 <= new_in.size())
                        in_seq.add_insert(new_in.data() + no * 8, nc, 8);
                }
                iwpos += count * interp_stride;
                iw_ni += count;
            } else if (op == OP_DELETE_RUN) {
                for (uint32_t k = 0; k < count; k++) {
                    uint8_t nc = static_cast<uint8_t>(old_iw[(iw_oi + k) * interp_stride + 4]);
                    if (nc > 0) in_seq.add_delete(nc);
                }
                iw_oi += count;
            }
        }
        uint32_t fid = static_cast<uint32_t>(PatchFileId::INTERP_NODES);
        uint64_t os = old_in.size(), ns = new_in.size(), ss = in_seq.data.size();
        uint32_t st = 8, n_fixups = 0;
        wval(patch, &fid, 4); wval(patch, &st, 4);
        wval(patch, &os, 8); wval(patch, &ns, 8);
        wval(patch, &n_fixups, 4); wval(patch, &ss, 8);
        patch.insert(patch.end(), in_seq.data.begin(), in_seq.data.end());
        std::cerr << "  interp_nodes.bin: parent-derived seq=" << ss << " bytes" << std::endl;
    }

    // admin_polygons.bin
    {
        auto old_data = read_file(old_dir + "/admin_polygons.bin");
        auto new_data = read_file(new_dir + "/admin_polygons.bin");
        // Zero padding bytes (offset 13-15 between admin_level and area) for deterministic comparison
        if (admin_stride == 24) {
            for (size_t i = 0; i + admin_stride <= old_data.size(); i += admin_stride)
                memset(old_data.data() + i + 13, 0, 3);
            for (size_t i = 0; i + admin_stride <= new_data.size(); i += admin_stride)
                memset(new_data.data() + i + 13, 0, 3);
        }
        remap_field(old_data, admin_stride, 8, str_remap);
        auto old_v = read_file(old_dir + "/admin_vertices.bin");
        auto new_v = read_file(new_dir + "/admin_vertices.bin");
        auto& fixups = file_fixups[static_cast<uint32_t>(PatchFileId::ADMIN_POLYGONS)];
        size_t an = old_data.size() / admin_stride;
        std::vector<uint32_t> old_offsets(an);
        for (size_t i = 0; i < an; i++) memcpy(&old_offsets[i], old_data.data() + i * admin_stride, 4);
        fixup_admin_offsets(old_data, old_v, new_data, new_v, admin_stride);
        for (size_t i = 0; i < an; i++) {
            uint32_t new_off; memcpy(&new_off, old_data.data() + i * admin_stride, 4);
            if (new_off != old_offsets[i]) fixups.push_back({static_cast<uint32_t>(i), new_off});
        }
        write_merge(PatchFileId::ADMIN_POLYGONS, "admin_polygons.bin", old_data, new_data, admin_stride);
    }

    // admin_vertices.bin — derive from admin_polygons merge (parent-aware)
    {
        auto& ap_seq = stored_merges[static_cast<uint32_t>(PatchFileId::ADMIN_POLYGONS)];
        auto old_ap = read_file(old_dir + "/admin_polygons.bin");