Write

# Custom Geocoder Patch System

## Context
Binary delta patching (zstd --patch-from) produces patches that are 25-67% of the original file sizes because independent builds have different string offsets, record orderings, and entry layouts. The user wants a custom tool that understands the format and produces small (~5-15 MiB) patches for a 7 GiB Europe dataset with ~0.3% daily changes.

## Architecture

Three tools:
1. **geocoder-canonicalize** — normalizes a fresh build into deterministic byte-identical canonical format
2. **geocoder-diff** — compares two canonical directories, produces `.gcpatch` file (COPY/INSERT byte stream)
3. **geocoder-patch** — applies `.gcpatch` to a canonical directory, producing new canonical directory

### Build Server Pipeline
```
fresh build → geocoder-canonicalize → canonical/
diff(prev_canonical/, canonical/) → patch.gcpatch
```

### Client Pipeline
```
canonical_day1 + patch_1→2 → canonical_day2
canonical_day2 + patch_2→3 → canonical_day3
```

## Canonical Format
- **strings.bin**: strings sorted alphabetically
- **Data files** (addr_points, street_ways, interp_ways, admin_polygons): records sorted by content fingerprint (reusing logic from compare_indexes.cpp)
- **Node/vertex files**: grouped by parent record in canonical order
- **Entries/cells**: derived from canonical data ordering + cell assignments carried from original build (no S2 recomputation)

## Patch Format (.gcpatch)
Binary COPY/INSERT stream per file, zstd compressed. For each of the 14 files:
- COPY(old_offset, length) — copy bytes from old canonical file
- INSERT(length, data) — insert new bytes
Since canonical format is stable, ~99.7% of content is COPYed.

## Files to Create
- `builder/tools/patch_format.h` — shared format definitions
- `builder/tools/canonical_order.h` — fingerprint/sort functions (extracted from compare_indexes.cpp)
- `builder/tools/geocoder_canonicalize.cpp`
- `builder/tools/geocoder_diff.cpp`
- `builder/tools/geocoder_patch.cpp`

## Key Files to Read
- `builder/src/types.h` — binary record structs
- `builder/test/compare_indexes.cpp` — existing fingerprint logic
- `builder/src/cell_index.cpp` — entry/cell write functions
- `builder/src/string_pool.h` — string pool implementation


File created successfully at: /root/.claude/plans/concurrent-humming-sonnet.md

# Custom Geocoder Patch System

## Context
Binary delta patching (zstd --patch-from) produces patches that are 25-67% of the original file sizes because independent builds have different string offsets, record orderings, and entry layouts. The user wants a custom tool that understands the format and produces small (~5-15 MiB) patches for a 7 GiB Europe dataset with ~0.3% daily changes.

## Architecture

Three tools:
1. **geocoder-canonicalize** — normalizes a fresh build into deterministic byte-identical canonical format
2. **geocoder-diff** — compares two canonical directories, produces `.gcpatch` file (COPY/INSERT byte stream)
3. **geocoder-patch** — applies `.gcpatch` to a canonical directory, producing new canonical directory

### Build Server Pipeline
```
fresh build → geocoder-canonicalize → canonical/
diff(prev_canonical/, canonical/) → patch.gcpatch
```

### Client Pipeline
```
canonical_day1 + patch_1→2 → canonical_day2
canonical_day2 + patch_2→3 → canonical_day3
```

## Canonical Format
- **strings.bin**: strings sorted alphabetically
- **Data files** (addr_points, street_ways, interp_ways, admin_polygons): records sorted by content fingerprint (reusing logic from compare_indexes.cpp)
- **Node/vertex files**: grouped by parent record in canonical order
- **Entries/cells**: derived from canonical data ordering + cell assignments carried from original build (no S2 recomputation)

## Patch Format (.gcpatch)
Binary COPY/INSERT stream per file, zstd compressed. For each of the 14 files:
- COPY(old_offset, length) — copy bytes from old canonical file
- INSERT(length, data) — insert new bytes
Since canonical format is stable, ~99.7% of content is COPYed.

## Files to Create
- `builder/tools/patch_format.h` — shared format definitions
- `builder/tools/canonical_order.h` — fingerprint/sort functions (extracted from compare_indexes.cpp)
- `builder/tools/geocoder_canonicalize.cpp`
- `builder/tools/geocoder_diff.cpp`
- `builder/tools/geocoder_patch.cpp`

## Key Files to Read
- `builder/src/types.h` — binary record structs
- `builder/test/compare_indexes.cpp` — existing fingerprint logic
- `builder/src/cell_index.cpp` — entry/cell write functions
- `builder/src/string_pool.h` — string pool implementation


ExitPlanMode

Prompt: build the project with cmake

Prompt: deploy binary to node 3 via scp

Prompt: run geocoder tools on node 3 via ssh

Prompt: list files and check sizes on node 3

# Custom Geocoder Patch System

## Context
Binary delta patching (zstd --patch-from) produces patches that are 25-67% of the original file sizes because independent builds have different string offsets, record orderings, and entry layouts. The user wants a custom tool that understands the format and produces small (~5-15 MiB) patches for a 7 GiB Europe dataset with ~0.3% daily changes.

## Architecture

Three tools:
1. **geocoder-canonicalize** — normalizes a fresh build into deterministic byte-identical canonical format
2. **geocoder-diff** — compares two canonical directories, produces `.gcpatch` file (COPY/INSERT byte stream)
3. **geocoder-patch** — applies `.gcpatch` to a canonical directory, producing new canonical directory

### Build Server Pipeline
```
fresh build → geocoder-canonicalize → canonical/
diff(prev_canonical/, canonical/) → patch.gcpatch
```

### Client Pipeline
```
canonical_day1 + patch_1→2 → canonical_day2
canonical_day2 + patch_2→3 → canonical_day3
```

## Canonical Format
- **strings.bin**: strings sorted alphabetically
- **Data files** (addr_points, street_ways, interp_ways, admin_polygons): records sorted by content fingerprint (reusing logic from compare_indexes.cpp)
- **Node/vertex files**: grouped by parent record in canonical order
- **Entries/cells**: derived from canonical data ordering + cell assignments carried from original build (no S2 recomputation)

## Patch Format (.gcpatch)
Binary COPY/INSERT stream per file, zstd compressed. For each of the 14 files:
- COPY(old_offset, length) — copy bytes from old canonical file
- INSERT(length, data) — insert new bytes
Since canonical format is stable, ~99.7% of content is COPYed.

## Files to Create
- `builder/tools/patch_format.h` — shared format definitions
- `builder/tools/canonical_order.h` — fingerprint/sort functions (extracted from compare_indexes.cpp)
- `builder/tools/geocoder_canonicalize.cpp`
- `builder/tools/geocoder_diff.cpp`
- `builder/tools/geocoder_patch.cpp`

## Key Files to Read
- `builder/src/types.h` — binary record structs
- `builder/test/compare_indexes.cpp` — existing fingerprint logic
- `builder/src/cell_index.cpp` — entry/cell write functions
- `builder/src/string_pool.h` — string pool implementation


<tool_use_error>You are not in plan mode. This tool is only for exiting plan mode after writing a plan. If your plan was already approved, continue with implementation.</tool_use_error>

Error: You are not in plan mode. This tool is only for exiting plan mode after writing a plan. If your plan was already approved, continue with implementation.


The user already took me out of plan mode. Let me start implementing. I'll begin with the foundation — the shared headers and the canonicalize tool, since that's what enables everything else.

First, let me revert the `--previous-build` and stabilization changes from `build_index.cpp` since the user doesn't want to modify the build process. The patching will be entirely post-build.


     1→// Semantic comparison of two geocoder index directories.
     2→// Decodes all binary files, creates canonical fingerprints for each item,
     3→// and compares as multisets — handles any ordering differences between
     4→// parallel and sequential builds.
     5→// Returns 0 if semantically identical, 1 if different.
     6→//
     7→// Usage: compare_indexes <dir-A> <dir-B>
     8→
     9→#include <algorithm>
    10→#include <cmath>
    11→#include <cstdint>
    12→#include <cstring>
    13→#include <fstream>
    14→#include <iostream>
    15→#include <map>
    16→#include <set>
    17→#include <string>
    18→#include <unordered_map>
    19→#include <vector>
    20→
    21→// --- Binary structs (must match build_index.cpp and server main.rs) ---
    22→
    23→struct WayHeader {
    24→    uint32_t node_offset;
    25→    uint8_t node_count;
    26→    uint32_t name_id;
    27→};
    28→
    29→struct AddrPoint {
    30→    float lat;
    31→    float lng;
    32→    uint32_t housenumber_id;
    33→    uint32_t street_id;
    34→};
    35→
    36→struct InterpWay {
    37→    uint32_t node_offset;
    38→    uint8_t node_count;
    39→    uint32_t street_id;
    40→    uint32_t start_number;
    41→    uint32_t end_number;
    42→    uint8_t interpolation;
    43→};
    44→
    45→struct AdminPolygon {
    46→    uint32_t vertex_offset;
    47→    uint16_t vertex_count;
    48→    uint32_t name_id;
    49→    uint8_t admin_level;
    50→    float area;
    51→    uint16_t country_code;
    52→};
    53→
    54→struct NodeCoord {
    55→    float lat;
    56→    float lng;
    57→};
    58→
    59→// --- Helpers ---
    60→
    61→static std::vector<char> read_file(const std::string& path) {
    62→    std::ifstream f(path, std::ios::binary | std::ios::ate);
    63→    if (!f) return {};
    64→    auto size = f.tellg();
    65→    f.seekg(0);
    66→    std::vector<char> data(size);
    67→    f.read(data.data(), size);
    68→    return data;
    69→}
    70→
    71→static const char* get_string(const std::vector<char>& pool, uint32_t offset) {
    72→    if (offset >= pool.size()) return "(invalid)";
    73→    return pool.data() + offset;
    74→}
    75→
    76→template<typename T>
    77→static std::vector<T> read_structs(const std::string& path) {
    78→    auto data = read_file(path);
    79→    size_t count = data.size() / sizeof(T);
    80→    std::vector<T> result(count);
    81→    memcpy(result.data(), data.data(), count * sizeof(T));
    82→    return result;
    83→}
    84→
    85→// Round a float coordinate to an integer grid (1e5 ~ 1m precision)
    86→static int to_grid(float v) {
    87→    return (int)(v * 1e5f + (v >= 0 ? 0.5f : -0.5f));
    88→}
    89→
    90→// --- Canonical fingerprint builders ---
    91→// Each function produces a string that uniquely identifies an item,
    92→// independent of storage order.
    93→
    94→// For a way: "name|sorted_node_coords"
    95→// Nodes are kept in original order (path order matters for ways),
    96→// but we also provide a sorted-set version for order-independent matching.
    97→static std::string way_fingerprint(const std::string& name,
    98→                                    const std::vector<std::pair<float,float>>& nodes) {
    99→    // Use sorted coordinate set as the canonical form.
   100→    // Two ways are "the same" if they have the same name and same set of nodes.
   101→    std::vector<std::pair<int,int>> grid_nodes;
   102→    grid_nodes.reserve(nodes.size());
   103→    for (const auto& n : nodes) {
   104→        grid_nodes.push_back({to_grid(n.first), to_grid(n.second)});
   105→    }
   106→    std::sort(grid_nodes.begin(), grid_nodes.end());
   107→
   108→    std::string fp = name + "|" + std::to_string(nodes.size());
   109→    for (const auto& g : grid_nodes) {
   110→        fp += "|";
   111→        fp += std::to_string(g.first);
   112→        fp += ",";
   113→        fp += std::to_string(g.second);
   114→    }
   115→    return fp;
   116→}
   117→
   118→// For an address: "grid_lat,grid_lng|housenumber|street"
   119→static std::string addr_fingerprint(float lat, float lng,
   120→                                     const std::string& housenumber,

// Semantic comparison of two geocoder index directories.
// Decodes all binary files, creates canonical fingerprints for each item,
// and compares as multisets — handles any ordering differences between
// parallel and sequential builds.
// Returns 0 if semantically identical, 1 if different.
//
// Usage: compare_indexes <dir-A> <dir-B>

#include <algorithm>
#include <cmath>
#include <cstdint>
#include <cstring>
#include <fstream>
#include <iostream>
#include <map>
#include <set>
#include <string>
#include <unordered_map>
#include <vector>

// --- Binary structs (must match build_index.cpp and server main.rs) ---

struct WayHeader {
    uint32_t node_offset;
    uint8_t node_count;
    uint32_t name_id;
};

struct AddrPoint {
    float lat;
    float lng;
    uint32_t housenumber_id;
    uint32_t street_id;
};

struct InterpWay {
    uint32_t node_offset;
    uint8_t node_count;
    uint32_t street_id;
    uint32_t start_number;
    uint32_t end_number;
    uint8_t interpolation;
};

struct AdminPolygon {
    uint32_t vertex_offset;
    uint16_t vertex_count;
    uint32_t name_id;
    uint8_t admin_level;
    float area;
    uint16_t country_code;
};

struct NodeCoord {
    float lat;
    float lng;
};

// --- Helpers ---

static std::vector<char> read_file(const std::string& path) {
    std::ifstream f(path, std::ios::binary | std::ios::ate);
    if (!f) return {};
    auto size = f.tellg();
    f.seekg(0);
    std::vector<char> data(size);
    f.read(data.data(), size);
    return data;
}

static const char* get_string(const std::vector<char>& pool, uint32_t offset) {
    if (offset >= pool.size()) return "(invalid)";
    return pool.data() + offset;
}

template<typename T>
static std::vector<T> read_structs(const std::string& path) {
    auto data = read_file(path);
    size_t count = data.size() / sizeof(T);
    std::vector<T> result(count);
    memcpy(result.data(), data.data(), count * sizeof(T));
    return result;
}

// Round a float coordinate to an integer grid (1e5 ~ 1m precision)
static int to_grid(float v) {
    return (int)(v * 1e5f + (v >= 0 ? 0.5f : -0.5f));
}

// --- Canonical fingerprint builders ---
// Each function produces a string that uniquely identifies an item,
// independent of storage order.

// For a way: "name|sorted_node_coords"
// Nodes are kept in original order (path order matters for ways),
// but we also provide a sorted-set version for order-independent matching.
static std::string way_fingerprint(const std::string& name,
                                    const std::vector<std::pair<float,float>>& nodes) {
    // Use sorted coordinate set as the canonical form.
    // Two ways are "the same" if they have the same name and same set of nodes.
    std::vector<std::pair<int,int>> grid_nodes;
    grid_nodes.reserve(nodes.size());
    for (const auto& n : nodes) {
        grid_nodes.push_back({to_grid(n.first), to_grid(n.second)});
    }
    std::sort(grid_nodes.begin(), grid_nodes.end());

    std::string fp = name + "|" + std::to_string(nodes.size());
    for (const auto& g : grid_nodes) {
        fp += "|";
        fp += std::to_string(g.first);
        fp += ",";
        fp += std::to_string(g.second);
    }
    return fp;
}

// For an address: "grid_lat,grid_lng|housenumber|street"
static std::string addr_fingerprint(float lat, float lng,
                                     const std::string& housenumber,


   120→                                     const std::string& housenumber,
   121→                                     const std::string& street) {
   122→    return std::to_string(to_grid(lat)) + "," + std::to_string(to_grid(lng)) +
   123→           "|" + housenumber + "|" + street;
   124→}
   125→
   126→// For an admin polygon: "name|level|country|sorted_vertices"
   127→static std::string admin_fingerprint(const std::string& name,
   128→                                      uint8_t admin_level,
   129→                                      uint16_t country_code,
   130→                                      const std::vector<std::pair<float,float>>& vertices) {
   131→    std::vector<std::pair<int,int>> grid_verts;
   132→    grid_verts.reserve(vertices.size());
   133→    for (const auto& v : vertices) {
   134→        grid_verts.push_back({to_grid(v.first), to_grid(v.second)});
   135→    }
   136→    std::sort(grid_verts.begin(), grid_verts.end());
   137→
   138→    std::string fp = name + "|" + std::to_string(admin_level) + "|" + std::to_string(country_code);
   139→    for (const auto& g : grid_verts) {
   140→        fp += "|";
   141→        fp += std::to_string(g.first);
   142→        fp += ",";
   143→        fp += std::to_string(g.second);
   144→    }
   145→    return fp;
   146→}
   147→
   148→// For interpolation ways: "street|start|end|interp|sorted_nodes"
   149→static std::string interp_fingerprint(const std::string& street,
   150→                                       uint32_t start_number, uint32_t end_number,
   151→                                       uint8_t interpolation,
   152→                                       const std::vector<std::pair<float,float>>& nodes) {
   153→    std::vector<std::pair<int,int>> grid_nodes;
   154→    grid_nodes.reserve(nodes.size());
   155→    for (const auto& n : nodes) {
   156→        grid_nodes.push_back({to_grid(n.first), to_grid(n.second)});
   157→    }
   158→    std::sort(grid_nodes.begin(), grid_nodes.end());
   159→
   160→    std::string fp = street + "|" + std::to_string(start_number) + "-" +
   161→                     std::to_string(end_number) + "|" + std::to_string(interpolation);
   162→    for (const auto& g : grid_nodes) {
   163→        fp += "|";
   164→        fp += std::to_string(g.first);
   165→        fp += ",";
   166→        fp += std::to_string(g.second);
   167→    }
   168→    return fp;
   169→}
   170→
   171→// --- Multiset comparison ---
   172→// Compare two collections by their fingerprints. Reports missing/extra items.
   173→
   174→static bool compare_multisets(const char* label,
   175→                               const std::vector<std::string>& fps_a,
   176→                               const std::vector<std::string>& fps_b) {
   177→    // Build frequency maps
   178→    std::unordered_map<std::string, int> freq_a, freq_b;
   179→    for (const auto& fp : fps_a) freq_a[fp]++;
   180→    for (const auto& fp : fps_b) freq_b[fp]++;
   181→
   182→    // Find items in A but not in B (missing from B)
   183→    size_t missing = 0;
   184→    for (const auto& [fp, count] : freq_a) {
   185→        int b_count = 0;
   186→        auto it = freq_b.find(fp);
   187→        if (it != freq_b.end()) b_count = it->second;
   188→        if (count > b_count) missing += (count - b_count);
   189→    }
   190→
   191→    // Find items in B but not in A (extra in B)
   192→    size_t extra = 0;
   193→    for (const auto& [fp, count] : freq_b) {
   194→        int a_count = 0;
   195→        auto it = freq_a.find(fp);
   196→        if (it != freq_a.end()) a_count = it->second;
   197→        if (count > a_count) extra += (count - a_count);
   198→    }
   199→
   200→    size_t matched = fps_a.size() - missing;
   201→
   202→    if (missing == 0 && extra == 0) {
   203→        std::cout << "PASS: " << label << " (" << fps_a.size() << " items, all matched)" << std::endl;
   204→        return true;
   205→    } else {
   206→        std::cerr << "FAIL: " << label << " — A=" << fps_a.size() << " B=" << fps_b.size()
   207→                  << " matched=" << matched
   208→                  << " only_in_A=" << missing
   209→                  << " only_in_B=" << extra << std::endl;
   210→        return false;
   211→    }
   212→}
   213→
   214→// --- Decode + fingerprint functions ---
   215→
   216→static std::vector<std::string> fingerprint_admin(const std::string& dir) {
   217→    auto strings = read_file(dir + "/strings.bin");
   218→    auto polygons = read_structs<AdminPolygon>(dir + "/admin_polygons.bin");
   219→    auto vertices = read_structs<NodeCoord>(dir + "/admin_vertices.bin");
   220→
   221→    std::vector<std::string> fps;
   222→    fps.reserve(polygons.size());
   223→    for (const auto& p : polygons) {
   224→        std::string name = get_string(strings, p.name_id);
   225→        std::vector<std::pair<float,float>> verts;
   226→        for (uint16_t i = 0; i < p.vertex_count; i++) {
   227→            uint32_t vi = p.vertex_offset + i;
   228→            if (vi < vertices.size()) {
   229→                verts.push_back({vertices[vi].lat, vertices[vi].lng});
   230→            }
   231→        }
   232→        fps.push_back(admin_fingerprint(name, p.admin_level, p.country_code, verts));
   233→    }
   234→    return fps;
   235→}
   236→
   237→static std::vector<std::string> fingerprint_ways(const std::string& dir) {
   238→    auto strings = read_file(dir + "/strings.bin");
   239→    auto ways = read_structs<WayHeader>(dir + "/street_ways.bin");
   240→    auto nodes = read_structs<NodeCoord>(dir + "/street_nodes.bin");
   241→
   242→    std::vector<std::string> fps;
   243→    fps.reserve(ways.size());
   244→    for (const auto& w : ways) {
   245→        std::string name = get_string(strings, w.name_id);
   246→        std::vector<std::pair<float,float>> way_nodes;
   247→        for (uint8_t i = 0; i < w.node_count; i++) {
   248→            uint32_t ni = w.node_offset + i;
   249→            if (ni < nodes.size()) {
   250→                way_nodes.push_back({nodes[ni].lat, nodes[ni].lng});
   251→            }
   252→        }
   253→        fps.push_back(way_fingerprint(name, way_nodes));
   254→    }
   255→    return fps;
   256→}
   257→
   258→static std::vector<std::string> fingerprint_addrs(const std::string& dir) {
   259→    auto strings = read_file(dir + "/strings.bin");
   260→    auto points = read_structs<AddrPoint>(dir + "/addr_points.bin");
   261→
   262→    std::vector<std::string> fps;
   263→    fps.reserve(points.size());
   264→    for (const auto& p : points) {
   265→        std::string hn = get_string(strings, p.housenumber_id);
   266→        std::string st = get_string(strings, p.street_id);
   267→        fps.push_back(addr_fingerprint(p.lat, p.lng, hn, st));
   268→    }
   269→    return fps;
   270→}
   271→
   272→static std::vector<std::string> fingerprint_interps(const std::string& dir) {
   273→    auto strings = read_file(dir + "/strings.bin");
   274→    auto interps = read_structs<InterpWay>(dir + "/interp_ways.bin");
   275→    auto nodes = read_structs<NodeCoord>(dir + "/interp_nodes.bin");
   276→
   277→    std::vector<std::string> fps;
   278→    fps.reserve(interps.size());
   279→    for (const auto& iw : interps) {
   280→        std::string street = get_string(strings, iw.street_id);
   281→        std::vector<std::pair<float,float>> way_nodes;
   282→        for (uint8_t i = 0; i < iw.node_count; i++) {
   283→            uint32_t ni = iw.node_offset + i;
   284→            if (ni < nodes.size()) {
   285→                way_nodes.push_back({nodes[ni].lat, nodes[ni].lng});
   286→            }
   287→        }
   288→        fps.push_back(interp_fingerprint(street, iw.start_number, iw.end_number,
   289→                                          iw.interpolation, way_nodes));
   290→    }
   291→    return fps;
   292→}
   293→
   294→// --- Semantic (cross-version) fingerprints ---
   295→// Match by name/attributes only, ignoring coordinate precision differences
   296→
   297→// Way: name + node_count only
   298→static std::string way_semantic_fp(const std::string& name, size_t node_count) {
   299→    return name + "|" + std::to_string(node_count);
   300→}
   301→
   302→// Address: housenumber + street + coarse grid coords (~100m precision)
   303→// Centroid may differ slightly across code versions, so use coarse grid
   304→static std::string addr_semantic_fp(float lat, float lng,
   305→                                     const std::string& housenumber, const std::string& street) {
   306→    int glat = (int)(lat * 1e3f + (lat >= 0 ? 0.5f : -0.5f));
   307→    int glng = (int)(lng * 1e3f + (lng >= 0 ? 0.5f : -0.5f));
   308→    return std::to_string(glat) + "," + std::to_string(glng) + "|" + housenumber + "|" + street;
   309→}
   310→
   311→// Admin: name + level
   312→static std::string admin_semantic_fp(const std::string& name, uint8_t admin_level) {
   313→    return name + "|" + std::to_string(admin_level);
   314→}
   315→
   316→static std::vector<std::string> semantic_fp_ways(const std::string& dir) {
   317→    auto strings = read_file(dir + "/strings.bin");
   318→    auto ways = read_structs<WayHeader>(dir + "/street_ways.bin");
   319→

                                     const std::string& housenumber,
                                     const std::string& street) {
    return std::to_string(to_grid(lat)) + "," + std::to_string(to_grid(lng)) +
           "|" + housenumber + "|" + street;
}

// For an admin polygon: "name|level|country|sorted_vertices"
static std::string admin_fingerprint(const std::string& name,
                                      uint8_t admin_level,
                                      uint16_t country_code,
                                      const std::vector<std::pair<float,float>>& vertices) {
    std::vector<std::pair<int,int>> grid_verts;
    grid_verts.reserve(vertices.size());
    for (const auto& v : vertices) {
        grid_verts.push_back({to_grid(v.first), to_grid(v.second)});
    }
    std::sort(grid_verts.begin(), grid_verts.end());

    std::string fp = name + "|" + std::to_string(admin_level) + "|" + std::to_string(country_code);
    for (const auto& g : grid_verts) {
        fp += "|";
        fp += std::to_string(g.first);
        fp += ",";
        fp += std::to_string(g.second);
    }
    return fp;
}

// For interpolation ways: "street|start|end|interp|sorted_nodes"
static std::string interp_fingerprint(const std::string& street,
                                       uint32_t start_number, uint32_t end_number,
                                       uint8_t interpolation,
                                       const std::vector<std::pair<float,float>>& nodes) {
    std::vector<std::pair<int,int>> grid_nodes;
    grid_nodes.reserve(nodes.size());
    for (const auto& n : nodes) {
        grid_nodes.push_back({to_grid(n.first), to_grid(n.second)});
    }
    std::sort(grid_nodes.begin(), grid_nodes.end());

    std::string fp = street + "|" + std::to_string(start_number) + "-" +
                     std::to_string(end_number) + "|" + std::to_string(interpolation);
    for (const auto& g : grid_nodes) {
        fp += "|";
        fp += std::to_string(g.first);
        fp += ",";
        fp += std::to_string(g.second);
    }
    return fp;
}

// --- Multiset comparison ---
// Compare two collections by their fingerprints. Reports missing/extra items.

static bool compare_multisets(const char* label,
                               const std::vector<std::string>& fps_a,
                               const std::vector<std::string>& fps_b) {
    // Build frequency maps
    std::unordered_map<std::string, int> freq_a, freq_b;
    for (const auto& fp : fps_a) freq_a[fp]++;
    for (const auto& fp : fps_b) freq_b[fp]++;

    // Find items in A but not in B (missing from B)
    size_t missing = 0;
    for (const auto& [fp, count] : freq_a) {
        int b_count = 0;
        auto it = freq_b.find(fp);
        if (it != freq_b.end()) b_count = it->second;
        if (count > b_count) missing += (count - b_count);
    }

    // Find items in B but not in A (extra in B)
    size_t extra = 0;
    for (const auto& [fp, count] : freq_b) {
        int a_count = 0;
        auto it = freq_a.find(fp);
        if (it != freq_a.end()) a_count = it->second;
        if (count > a_count) extra += (count - a_count);
    }

    size_t matched = fps_a.size() - missing;

    if (missing == 0 && extra == 0) {
        std::cout << "PASS: " << label << " (" << fps_a.size() << " items, all matched)" << std::endl;
        return true;
    } else {
        std::cerr << "FAIL: " << label << " — A=" << fps_a.size() << " B=" << fps_b.size()
                  << " matched=" << matched
                  << " only_in_A=" << missing
                  << " only_in_B=" << extra << std::endl;
        return false;
    }
}

// --- Decode + fingerprint functions ---

static std::vector<std::string> fingerprint_admin(const std::string& dir) {
    auto strings = read_file(dir + "/strings.bin");
    auto polygons = read_structs<AdminPolygon>(dir + "/admin_polygons.bin");
    auto vertices = read_structs<NodeCoord>(dir + "/admin_vertices.bin");

    std::vector<std::string> fps;
    fps.reserve(polygons.size());
    for (const auto& p : polygons) {
        std::string name = get_string(strings, p.name_id);
        std::vector<std::pair<float,float>> verts;
        for (uint16_t i = 0; i < p.vertex_count; i++) {
            uint32_t vi = p.vertex_offset + i;
            if (vi < vertices.size()) {
                verts.push_back({vertices[vi].lat, vertices[vi].lng});
            }
        }
        fps.push_back(admin_fingerprint(name, p.admin_level, p.country_code, verts));
    }
    return fps;
}

static std::vector<std::string> fingerprint_ways(const std::string& dir) {
    auto strings = read_file(dir + "/strings.bin");
    auto ways = read_structs<WayHeader>(dir + "/street_ways.bin");
    auto nodes = read_structs<NodeCoord>(dir + "/street_nodes.bin");

    std::vector<std::string> fps;
    fps.reserve(ways.size());
    for (const auto& w : ways) {
        std::string name = get_string(strings, w.name_id);
        std::vector<std::pair<float,float>> way_nodes;
        for (uint8_t i = 0; i < w.node_count; i++) {
            uint32_t ni = w.node_offset + i;
            if (ni < nodes.size()) {
                way_nodes.push_back({nodes[ni].lat, nodes[ni].lng});
            }
        }
        fps.push_back(way_fingerprint(name, way_nodes));
    }
    return fps;
}

static std::vector<std::string> fingerprint_addrs(const std::string& dir) {
    auto strings = read_file(dir + "/strings.bin");
    auto points = read_structs<AddrPoint>(dir + "/addr_points.bin");

    std::vector<std::string> fps;
    fps.reserve(points.size());
    for (const auto& p : points) {
        std::string hn = get_string(strings, p.housenumber_id);
        std::string st = get_string(strings, p.street_id);
        fps.push_back(addr_fingerprint(p.lat, p.lng, hn, st));
    }
    return fps;
}

static std::vector<std::string> fingerprint_interps(const std::string& dir) {
    auto strings = read_file(dir + "/strings.bin");
    auto interps = read_structs<InterpWay>(dir + "/interp_ways.bin");
    auto nodes = read_structs<NodeCoord>(dir + "/interp_nodes.bin");

    std::vector<std::string> fps;
    fps.reserve(interps.size());
    for (const auto& iw : interps) {
        std::string street = get_string(strings, iw.street_id);
        std::vector<std::pair<float,float>> way_nodes;
        for (uint8_t i = 0; i < iw.node_count; i++) {
            uint32_t ni = iw.node_offset + i;
            if (ni < nodes.size()) {
                way_nodes.push_back({nodes[ni].lat, nodes[ni].lng});
            }
        }
        fps.push_back(interp_fingerprint(street, iw.start_number, iw.end_number,
                                          iw.interpolation, way_nodes));
    }
    return fps;
}

// --- Semantic (cross-version) fingerprints ---
// Match by name/attributes only, ignoring coordinate precision differences

// Way: name + node_count only
static std::string way_semantic_fp(const std::string& name, size_t node_count) {
    return name + "|" + std::to_string(node_count);
}

// Address: housenumber + street + coarse grid coords (~100m precision)
// Centroid may differ slightly across code versions, so use coarse grid
static std::string addr_semantic_fp(float lat, float lng,
                                     const std::string& housenumber, const std::string& street) {
    int glat = (int)(lat * 1e3f + (lat >= 0 ? 0.5f : -0.5f));
    int glng = (int)(lng * 1e3f + (lng >= 0 ? 0.5f : -0.5f));
    return std::to_string(glat) + "," + std::to_string(glng) + "|" + housenumber + "|" + street;
}

// Admin: name + level
static std::string admin_semantic_fp(const std::string& name, uint8_t admin_level) {
    return name + "|" + std::to_string(admin_level);
}

static std::vector<std::string> semantic_fp_ways(const std::string& dir) {
    auto strings = read_file(dir + "/strings.bin");
    auto ways = read_structs<WayHeader>(dir + "/street_ways.bin");


Good, I have a solid understanding of the fingerprint logic. Now let me read the CMakeLists.txt to understand how to add new tools.


     1→cmake_minimum_required(VERSION 3.16)
     2→project(traccar-geocoder-build LANGUAGES CXX)
     3→
     4→set(CMAKE_CXX_STANDARD 17)
     5→set(CMAKE_CXX_STANDARD_REQUIRED ON)
     6→
     7→
     8→find_package(Threads REQUIRED)
     9→find_package(ZLIB REQUIRED)
    10→find_package(BZip2 REQUIRED)
    11→find_package(EXPAT REQUIRED)
    12→find_package(s2 QUIET)
    13→
    14→add_executable(build-index
    15→    src/build_index.cpp
    16→    src/pbf_reader.cpp
    17→    src/s2_helpers.cpp
    18→    src/ring_assembly.cpp
    19→    src/cache.cpp
    20→    src/continent_filter.cpp
    21→    src/continent_boundaries.cpp
    22→    src/cell_index.cpp
    23→    src/interpolation.cpp
    24→)
    25→
    26→if(EXISTS /opt/homebrew/include)
    27→    target_include_directories(build-index PRIVATE /opt/homebrew/include)
    28→endif()
    29→
    30→if(s2_FOUND)
    31→    target_link_libraries(build-index PRIVATE
    32→        s2::s2
    33→        ZLIB::ZLIB
    34→        BZip2::BZip2
    35→        EXPAT::EXPAT
    36→        Threads::Threads
    37→    )
    38→else()
    39→    target_include_directories(build-index PRIVATE /usr/include/s2)
    40→    target_link_libraries(build-index PRIVATE
    41→        s2
    42→        ZLIB::ZLIB
    43→        BZip2::BZip2
    44→        EXPAT::EXPAT
    45→        Threads::Threads
    46→    )
    47→endif()
    48→

cmake_minimum_required(VERSION 3.16)
project(traccar-geocoder-build LANGUAGES CXX)

set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED ON)

find_package(Threads REQUIRED)
find_package(ZLIB REQUIRED)
find_package(BZip2 REQUIRED)
find_package(EXPAT REQUIRED)
find_package(s2 QUIET)

add_executable(build-index
    src/build_index.cpp
    src/pbf_reader.cpp
    src/s2_helpers.cpp
    src/ring_assembly.cpp
    src/cache.cpp
    src/continent_filter.cpp
    src/continent_boundaries.cpp
    src/cell_index.cpp
    src/interpolation.cpp
)

if(EXISTS /opt/homebrew/include)
    target_include_directories(build-index PRIVATE /opt/homebrew/include)
endif()

if(s2_FOUND)
    target_link_libraries(build-index PRIVATE
        s2::s2
        ZLIB::ZLIB
        BZip2::BZip2
        EXPAT::EXPAT
        Threads::Threads
    )
else()
    target_include_directories(build-index PRIVATE /usr/include/s2)
    target_link_libraries(build-index PRIVATE
        s2
        ZLIB::ZLIB
        BZip2::BZip2
        EXPAT::EXPAT
        Threads::Threads
    )
endif()


Now let me check if zstd is available on the build machine:


Bash

dpkg -l | grep -i zstd | head -5; echo "---"; ls /usr/include/zstd* 2>/dev/null; echo "---"; pkg-config --libs libzstd 2>/dev/null || echo "no pkg-config for zstd"

Check zstd availability


ii  libzstd1:amd64                 1.5.7+dfsg-1                         amd64        fast lossless compression algorithm
---
---
no pkg-config for zstd


apt list --installed 2>/dev/null | grep zstd

Check zstd dev package


libzstd1/stable,now 1.5.7+dfsg-1 amd64 [installed,automatic]


Need zstd dev headers. Let me install them and start building the tools. I'll use zlib (already available) for compression in the patch format to avoid adding a new dependency.

Now let me implement the three tools. I'll start with the shared header, then the canonicalize tool (the foundation), then diff, then patch.


Check existing tools directory


comparison.geojson
simplification_map.html
simplification_viewer.html
simplify_compare.cpp


Write

#pragma once
// Shared definitions for the geocoder patch system.
// Used by geocoder-canonicalize, geocoder-diff, and geocoder-patch.

#include <algorithm>
#include <cstdint>
#include <cstring>
#include <fstream>
#include <iostream>
#include <string>
#include <vector>

// --- Binary record structs (must match types.h and server) ---

#pragma pack(push, 1)
struct PatchWayHeader {
    uint32_t node_offset;
    uint8_t node_count;
    uint32_t name_id;
};
static_assert(sizeof(PatchWayHeader) == 9, "WayHeader must be 9 bytes packed");

struct PatchAddrPoint {
    float lat;
    float lng;
    uint32_t housenumber_id;
    uint32_t street_id;
};
static_assert(sizeof(PatchAddrPoint) == 16, "AddrPoint must be 16 bytes");

struct PatchNodeCoord {
    float lat;
    float lng;
};
static_assert(sizeof(PatchNodeCoord) == 8, "NodeCoord must be 8 bytes");
#pragma pack(pop)

// InterpWay and AdminPolygon have compiler-dependent padding.
// Read them field-by-field instead of struct-casting.

struct PatchInterpWay {
    uint32_t node_offset;
    uint8_t node_count;
    uint32_t street_id;
    uint32_t start_number;
    uint32_t end_number;
    uint8_t interpolation;
};

struct PatchAdminPolygon {
    uint32_t vertex_offset;
    uint32_t vertex_count;
    uint32_t name_id;
    uint8_t admin_level;
    float area;
    uint16_t country_code;
};

// --- File I/O helpers ---

inline std::vector<char> read_file(const std::string& path) {
    std::ifstream f(path, std::ios::binary | std::ios::ate);
    if (!f) return {};
    auto size = f.tellg();
    f.seekg(0);
    std::vector<char> data(size);
    f.read(data.data(), size);
    return data;
}

inline bool write_file(const std::string& path, const char* data, size_t size) {
    std::ofstream f(path, std::ios::binary);
    if (!f) return false;
    f.write(data, size);
    return f.good();
}

inline bool write_file(const std::string& path, const std::vector<char>& data) {
    return write_file(path, data.data(), data.size());
}

inline const char* get_string(const std::vector<char>& pool, uint32_t offset) {
    if (offset >= pool.size()) return "";
    return pool.data() + offset;
}

// Read packed WayHeader array (9 bytes each, no padding)
inline std::vector<PatchWayHeader> read_ways(const std::string& path) {
    auto data = read_file(path);
    size_t count = data.size() / 9;
    std::vector<PatchWayHeader> result(count);
    for (size_t i = 0; i < count; i++) {
        memcpy(&result[i], data.data() + i * 9, 9);
    }
    return result;
}

template<typename T>
inline std::vector<T> read_structs(const std::string& path) {
    auto data = read_file(path);
    size_t count = data.size() / sizeof(T);
    std::vector<T> result(count);
    if (count > 0) memcpy(result.data(), data.data(), count * sizeof(T));
    return result;
}

// Read InterpWay with field-by-field parsing to handle padding
inline std::vector<PatchInterpWay> read_interps(const std::string& path) {
    auto data = read_file(path);
    // Determine record size from file: try sizeof(InterpWay) as compiled by the builder
    // The builder uses the same struct, so sizeof should match.
    size_t rec_size = sizeof(PatchInterpWay);
    // But the builder's InterpWay has padding. Read with the builder's size.
    // From types.h the struct is: u32 + u8 + u32 + u32 + u32 + u8 = 18 bytes raw
    // but compiler pads to 20 bytes typically. Check by trying both.
    size_t count_20 = data.size() / 20;
    size_t count_18 = data.size() / 18;
    // Use whichever divides evenly
    size_t stride = (data.size() % 20 == 0 && count_20 > 0) ? 20 :
                    (data.size() % 18 == 0 && count_18 > 0) ? 18 : 20;
    size_t count = data.size() / stride;
    std::vector<PatchInterpWay> result(count);
    for (size_t i = 0; i < count; i++) {
        const char* p = data.data() + i * stride;
        memcpy(&result[i].node_offset, p, 4); p += 4;
        result[i].node_count = *reinterpret_cast<const uint8_t*>(p); p += 1;
        // Skip padding to align street_id
        if (stride == 20) p += 3; // 3 bytes padding after node_count
        memcpy(&result[i].street_id, p, 4); p += 4;
        memcpy(&result[i].start_number, p, 4); p += 4;
        memcpy(&result[i].end_number, p, 4); p += 4;
        result[i].interpolation = *reinterpret_cast<const uint8_t*>(p);
    }
    return result;
}

// Read AdminPolygon with field-by-field parsing
inline std::vector<PatchAdminPolygon> read_admin_polygons(const std::string& path) {
    auto data = read_file(path);
    // AdminPolygon: u32 + u32 + u32 + u8 + f32 + u16 = 19 raw bytes
    // Compiler likely pads to 20 or 24 bytes. Try common sizes.
    size_t stride = 0;
    for (size_t s : {24, 20, 19}) {
        if (data.size() % s == 0 && data.size() / s > 0) { stride = s; break; }
    }
    if (stride == 0) stride = 24; // fallback
    size_t count = data.size() / stride;
    std::vector<PatchAdminPolygon> result(count);
    for (size_t i = 0; i < count; i++) {
        const char* p = data.data() + i * stride;
        memcpy(&result[i].vertex_offset, p, 4); p += 4;
        memcpy(&result[i].vertex_count, p, 4); p += 4;
        memcpy(&result[i].name_id, p, 4); p += 4;
        result[i].admin_level = *reinterpret_cast<const uint8_t*>(p); p += 1;
        // padding for area alignment
        if (stride >= 20) {
            size_t skip = (stride == 24) ? 3 : (stride == 20 ? 3 : 0);
            p += skip;
        }
        memcpy(&result[i].area, p, 4); p += 4;
        memcpy(&result[i].country_code, p, 2);
    }
    return result;
}

// --- Patch file format ---

static constexpr char GCPATCH_MAGIC[8] = {'G','C','P','A','T','C','H','\0'};
static constexpr uint32_t GCPATCH_VERSION = 1;

enum class PatchFileId : uint32_t {
    STRINGS = 0,
    STREET_WAYS = 1,
    STREET_NODES = 2,
    ADDR_POINTS = 3,
    INTERP_WAYS = 4,
    INTERP_NODES = 5,
    ADMIN_POLYGONS = 6,
    ADMIN_VERTICES = 7,
    GEO_CELLS = 8,
    STREET_ENTRIES = 9,
    ADDR_ENTRIES = 10,
    INTERP_ENTRIES = 11,
    ADMIN_CELLS = 12,
    ADMIN_ENTRIES = 13,
    COUNT = 14
};

static const char* patch_file_names[] = {
    "strings.bin", "street_ways.bin", "street_nodes.bin", "addr_points.bin",
    "interp_ways.bin", "interp_nodes.bin", "admin_polygons.bin", "admin_vertices.bin",
    "geo_cells.bin", "street_entries.bin", "addr_entries.bin", "interp_entries.bin",
    "admin_cells.bin", "admin_entries.bin"
};

// Encoding types for each section
enum class PatchEncoding : uint32_t {
    RAW_REPLACE = 0,  // Full replacement (zlib compressed)
    COPY_INSERT = 1,  // COPY/INSERT byte stream (zlib compressed)
};

// Patch file header
struct PatchHeader {
    char magic[8];
    uint32_t version;
    uint32_t flags;
    // Per-file sections follow
};

// Section header (one per file)
struct PatchSection {
    uint32_t file_id;       // PatchFileId
    uint32_t encoding;      // PatchEncoding
    uint64_t old_size;      // Expected size of old file
    uint64_t new_size;      // Size of reconstructed file
    uint64_t data_size;     // Size of compressed section data
};

// COPY_INSERT opcodes
enum : uint8_t {
    OP_COPY = 0,    // Copy from old file: [u8 tag][u64 offset][u32 length]
    OP_INSERT = 1,  // Insert new data:    [u8 tag][u32 length][data...]
};

// --- Grid coordinate for fingerprinting ---

inline int to_grid(float v) {
    return (int)(v * 1e5f + (v >= 0 ? 0.5f : -0.5f));
}

// --- Directory creation ---
inline void ensure_dir(const std::string& path) {
    std::string cmd = "mkdir -p '" + path + "'";
    system(cmd.c_str());
}


File created successfully at: /root/traccar-geocoder/builder/tools/patch_format.h

#pragma once
// Shared definitions for the geocoder patch system.
// Used by geocoder-canonicalize, geocoder-diff, and geocoder-patch.

#include <algorithm>
#include <cstdint>
#include <cstring>
#include <fstream>
#include <iostream>
#include <string>
#include <vector>

// --- Binary record structs (must match types.h and server) ---

#pragma pack(push, 1)
struct PatchWayHeader {
    uint32_t node_offset;
    uint8_t node_count;
    uint32_t name_id;
};
static_assert(sizeof(PatchWayHeader) == 9, "WayHeader must be 9 bytes packed");

struct PatchAddrPoint {
    float lat;
    float lng;
    uint32_t housenumber_id;
    uint32_t street_id;
};
static_assert(sizeof(PatchAddrPoint) == 16, "AddrPoint must be 16 bytes");

struct PatchNodeCoord {
    float lat;
    float lng;
};
static_assert(sizeof(PatchNodeCoord) == 8, "NodeCoord must be 8 bytes");
#pragma pack(pop)

// InterpWay and AdminPolygon have compiler-dependent padding.
// Read them field-by-field instead of struct-casting.

struct PatchInterpWay {
    uint32_t node_offset;
    uint8_t node_count;
    uint32_t street_id;
    uint32_t start_number;
    uint32_t end_number;
    uint8_t interpolation;
};

struct PatchAdminPolygon {
    uint32_t vertex_offset;
    uint32_t vertex_count;
    uint32_t name_id;
    uint8_t admin_level;
    float area;
    uint16_t country_code;
};

// --- File I/O helpers ---

inline std::vector<char> read_file(const std::string& path) {
    std::ifstream f(path, std::ios::binary | std::ios::ate);
    if (!f) return {};
    auto size = f.tellg();
    f.seekg(0);
    std::vector<char> data(size);
    f.read(data.data(), size);
    return data;
}

inline bool write_file(const std::string& path, const char* data, size_t size) {
    std::ofstream f(path, std::ios::binary);
    if (!f) return false;
    f.write(data, size);
    return f.good();
}

inline bool write_file(const std::string& path, const std::vector<char>& data) {
    return write_file(path, data.data(), data.size());
}

inline const char* get_string(const std::vector<char>& pool, uint32_t offset) {
    if (offset >= pool.size()) return "";
    return pool.data() + offset;
}

// Read packed WayHeader array (9 bytes each, no padding)
inline std::vector<PatchWayHeader> read_ways(const std::string& path) {
    auto data = read_file(path);
    size_t count = data.size() / 9;
    std::vector<PatchWayHeader> result(count);
    for (size_t i = 0; i < count; i++) {
        memcpy(&result[i], data.data() + i * 9, 9);
    }
    return result;
}

template<typename T>
inline std::vector<T> read_structs(const std::string& path) {
    auto data = read_file(path);
    size_t count = data.size() / sizeof(T);
    std::vector<T> result(count);
    if (count > 0) memcpy(result.data(), data.data(), count * sizeof(T));
    return result;
}

// Read InterpWay with field-by-field parsing to handle padding
inline std::vector<PatchInterpWay> read_interps(const std::string& path) {
    auto data = read_file(path);
    // Determine record size from file: try sizeof(InterpWay) as compiled by the builder
    // The builder uses the same struct, so sizeof should match.
    size_t rec_size = sizeof(PatchInterpWay);
    // But the builder's InterpWay has padding. Read with the builder's size.
    // From types.h the struct is: u32 + u8 + u32 + u32 + u32 + u8 = 18 bytes raw
    // but compiler pads to 20 bytes typically. Check by trying both.
    size_t count_20 = data.size() / 20;
    size_t count_18 = data.size() / 18;
    // Use whichever divides evenly
    size_t stride = (data.size() % 20 == 0 && count_20 > 0) ? 20 :
                    (data.size() % 18 == 0 && count_18 > 0) ? 18 : 20;
    size_t count = data.size() / stride;
    std::vector<PatchInterpWay> result(count);
    for (size_t i = 0; i < count; i++) {
        const char* p = data.data() + i * stride;
        memcpy(&result[i].node_offset, p, 4); p += 4;
        result[i].node_count = *reinterpret_cast<const uint8_t*>(p); p += 1;
        // Skip padding to align street_id
        if (stride == 20) p += 3; // 3 bytes padding after node_count
        memcpy(&result[i].street_id, p, 4); p += 4;
        memcpy(&result[i].start_number, p, 4); p += 4;
        memcpy(&result[i].end_number, p, 4); p += 4;
        result[i].interpolation = *reinterpret_cast<const uint8_t*>(p);
    }
    return result;
}

// Read AdminPolygon with field-by-field parsing
inline std::vector<PatchAdminPolygon> read_admin_polygons(const std::string& path) {
    auto data = read_file(path);
    // AdminPolygon: u32 + u32 + u32 + u8 + f32 + u16 = 19 raw bytes
    // Compiler likely pads to 20 or 24 bytes. Try common sizes.
    size_t stride = 0;
    for (size_t s : {24, 20, 19}) {
        if (data.size() % s == 0 && data.size() / s > 0) { stride = s; break; }
    }
    if (stride == 0) stride = 24; // fallback
    size_t count = data.size() / stride;
    std::vector<PatchAdminPolygon> result(count);
    for (size_t i = 0; i < count; i++) {
        const char* p = data.data() + i * stride;
        memcpy(&result[i].vertex_offset, p, 4); p += 4;
        memcpy(&result[i].vertex_count, p, 4); p += 4;
        memcpy(&result[i].name_id, p, 4); p += 4;
        result[i].admin_level = *reinterpret_cast<const uint8_t*>(p); p += 1;
        // padding for area alignment
        if (stride >= 20) {
            size_t skip = (stride == 24) ? 3 : (stride == 20 ? 3 : 0);
            p += skip;
        }
        memcpy(&result[i].area, p, 4); p += 4;
        memcpy(&result[i].country_code, p, 2);
    }
    return result;
}

// --- Patch file format ---

static constexpr char GCPATCH_MAGIC[8] = {'G','C','P','A','T','C','H','\0'};
static constexpr uint32_t GCPATCH_VERSION = 1;

enum class PatchFileId : uint32_t {
    STRINGS = 0,
    STREET_WAYS = 1,
    STREET_NODES = 2,
    ADDR_POINTS = 3,
    INTERP_WAYS = 4,
    INTERP_NODES = 5,
    ADMIN_POLYGONS = 6,
    ADMIN_VERTICES = 7,
    GEO_CELLS = 8,
    STREET_ENTRIES = 9,
    ADDR_ENTRIES = 10,
    INTERP_ENTRIES = 11,
    ADMIN_CELLS = 12,
    ADMIN_ENTRIES = 13,
    COUNT = 14
};

static const char* patch_file_names[] = {
    "strings.bin", "street_ways.bin", "street_nodes.bin", "addr_points.bin",
    "interp_ways.bin", "interp_nodes.bin", "admin_polygons.bin", "admin_vertices.bin",
    "geo_cells.bin", "street_entries.bin", "addr_entries.bin", "interp_entries.bin",
    "admin_cells.bin", "admin_entries.bin"
};

// Encoding types for each section
enum class PatchEncoding : uint32_t {
    RAW_REPLACE = 0,  // Full replacement (zlib compressed)
    COPY_INSERT = 1,  // COPY/INSERT byte stream (zlib compressed)
};

// Patch file header
struct PatchHeader {
    char magic[8];
    uint32_t version;
    uint32_t flags;
    // Per-file sections follow
};

// Section header (one per file)
struct PatchSection {
    uint32_t file_id;       // PatchFileId
    uint32_t encoding;      // PatchEncoding
    uint64_t old_size;      // Expected size of old file
    uint64_t new_size;      // Size of reconstructed file
    uint64_t data_size;     // Size of compressed section data
};

// COPY_INSERT opcodes
enum : uint8_t {
    OP_COPY = 0,    // Copy from old file: [u8 tag][u64 offset][u32 length]
    OP_INSERT = 1,  // Insert new data:    [u8 tag][u32 length][data...]
};

// --- Grid coordinate for fingerprinting ---

inline int to_grid(float v) {
    return (int)(v * 1e5f + (v >= 0 ? 0.5f : -0.5f));
}

// --- Directory creation ---
inline void ensure_dir(const std::string& path) {
    std::string cmd = "mkdir -p '" + path + "'";
    system(cmd.c_str());
}


Now let me implement the canonicalize tool. This is the critical piece — it produces deterministic output from any build.


// geocoder-canonicalize: Converts a geocoder index to canonical (deterministic) format.
//
// Canonical format properties:
// - strings.bin: strings sorted alphabetically
// - Data files: records sorted by content fingerprint
// - Node/vertex files: grouped by parent record in canonical order
// - Entries/cells: derived from canonical data ordering + original cell assignments
//
// Usage: geocoder-canonicalize <input-dir> <output-dir>

#include <algorithm>
#include <cstdint>
#include <cstring>
#include <fstream>
#include <functional>
#include <iostream>
#include <numeric>
#include <string>
#include <unordered_map>
#include <vector>

#include "patch_format.h"

// --- Canonical string pool ---
// Sort all strings alphabetically, build old_offset -> new_offset remap.

struct CanonicalStringPool {
    std::vector<char> data;
    std::unordered_map<uint32_t, uint32_t> remap; // old_offset -> new_offset

    void build(const std::vector<char>& old_pool) {
        // Extract all strings with their offsets
        std::vector<std::pair<uint32_t, const char*>> strings;
        size_t pos = 0;
        while (pos < old_pool.size()) {
            const char* s = old_pool.data() + pos;
            strings.emplace_back(static_cast<uint32_t>(pos), s);
            pos += strlen(s) + 1;
        }

        // Sort alphabetically
        std::sort(strings.begin(), strings.end(), [](const auto& a, const auto& b) {
            return strcmp(a.second, b.second) < 0;
        });

        // Build new pool + remap
        data.clear();
        data.reserve(old_pool.size());
        remap.reserve(strings.size());
        for (auto& [old_off, str] : strings) {
            uint32_t new_off = static_cast<uint32_t>(data.size());
            remap[old_off] = new_off;
            size_t len = strlen(str);
            data.insert(data.end(), str, str + len + 1);
        }
    }

    uint32_t map(uint32_t old_off) const {
        auto it = remap.find(old_off);
        return (it != remap.end()) ? it->second : 0;
    }
};

// --- Cell assignment extraction ---
// Parse geo_cells.bin + entries files to extract cell -> [item_id] mappings.

struct GeoCellEntry {
    uint64_t cell_id;
    uint32_t street_offset;
    uint32_t addr_offset;
    uint32_t interp_offset;
};

struct CellAssignments {
    // For each item type: cell_id -> sorted list of item_ids
    std::unordered_map<uint64_t, std::vector<uint32_t>> street_cells;
    std::unordered_map<uint64_t, std::vector<uint32_t>> addr_cells;
    std::unordered_map<uint64_t, std::vector<uint32_t>> interp_cells;
    std::unordered_map<uint64_t, std::vector<uint32_t>> admin_cells; // with flags
};

static std::vector<uint32_t> parse_entry(const std::vector<char>& entries, uint32_t offset) {
    if (offset == 0xFFFFFFFF || offset + 2 > entries.size()) return {};
    uint16_t count;
    memcpy(&count, entries.data() + offset, 2);
    std::vector<uint32_t> ids(count);
    if (offset + 2 + count * 4 > entries.size()) return {};
    memcpy(ids.data(), entries.data() + offset + 2, count * 4);
    return ids;
}

static CellAssignments extract_cell_assignments(const std::string& dir) {
    CellAssignments ca;

    // Geo cells
    auto geo_data = read_file(dir + "/geo_cells.bin");
    auto street_entries = read_file(dir + "/street_entries.bin");
    auto addr_entries = read_file(dir + "/addr_entries.bin");
    auto interp_entries = read_file(dir + "/interp_entries.bin");

    size_t n_geo = geo_data.size() / 20;
    for (size_t i = 0; i < n_geo; i++) {
        GeoCellEntry e;
        memcpy(&e.cell_id, geo_data.data() + i * 20, 8);
        memcpy(&e.street_offset, geo_data.data() + i * 20 + 8, 4);
        memcpy(&e.addr_offset, geo_data.data() + i * 20 + 12, 4);
        memcpy(&e.interp_offset, geo_data.data() + i * 20 + 16, 4);

        auto s_ids = parse_entry(street_entries, e.street_offset);
        if (!s_ids.empty()) ca.street_cells[e.cell_id] = std::move(s_ids);

        auto a_ids = parse_entry(addr_entries, e.addr_offset);
        if (!a_ids.empty()) ca.addr_cells[e.cell_id] = std::move(a_ids);

        auto i_ids = parse_entry(interp_entries, e.interp_offset);
        if (!i_ids.empty()) ca.interp_cells[e.cell_id] = std::move(i_ids);
    }

    // Admin cells
    auto admin_cell_data = read_file(dir + "/admin_cells.bin");
    auto admin_entry_data = read_file(dir + "/admin_entries.bin");
    size_t n_admin = admin_cell_data.size() / 12;
    for (size_t i = 0; i < n_admin; i++) {
        uint64_t cell_id;
        uint32_t offset;
        memcpy(&cell_id, admin_cell_data.data() + i * 12, 8);
        memcpy(&offset, admin_cell_data.data() + i * 12 + 8, 4);
        auto ids = parse_entry(admin_entry_data, offset);
        if (!ids.empty()) ca.admin_cells[cell_id] = std::move(ids);
    }

    return ca;
}

// --- Remap cell assignments with old->new ID permutation ---
static void remap_cells(std::unordered_map<uint64_t, std::vector<uint32_t>>& cells,
                         const std::vector<uint32_t>& old_to_new) {
    for (auto& [cell_id, ids] : cells) {
        for (auto& id : ids) {
            uint32_t masked = id & 0x7FFFFFFF;
            uint32_t flags = id & 0x80000000;
            if (masked < old_to_new.size()) {
                id = old_to_new[masked] | flags;
            }
        }
        std::sort(ids.begin(), ids.end());
    }
}

// --- Write entries + cells files ---
static void write_entries_and_cells(
    const std::string& cells_path,
    const std::string& entries_path,
    const std::unordered_map<uint64_t, std::vector<uint32_t>>& cell_map,
    size_t cell_entry_size) // 20 for geo, 12 for admin
{
    // Collect and sort cells
    std::vector<uint64_t> sorted_cells;
    sorted_cells.reserve(cell_map.size());
    for (auto& [id, _] : cell_map) sorted_cells.push_back(id);
    std::sort(sorted_cells.begin(), sorted_cells.end());

    // Write entries file
    std::vector<char> entries_buf;
    std::unordered_map<uint64_t, uint32_t> cell_offsets;
    for (uint64_t cell_id : sorted_cells) {
        auto it = cell_map.find(cell_id);
        if (it == cell_map.end() || it->second.empty()) continue;
        cell_offsets[cell_id] = static_cast<uint32_t>(entries_buf.size());
        uint16_t count = static_cast<uint16_t>(it->second.size());
        entries_buf.insert(entries_buf.end(), reinterpret_cast<const char*>(&count),
                          reinterpret_cast<const char*>(&count) + 2);
        entries_buf.insert(entries_buf.end(),
                          reinterpret_cast<const char*>(it->second.data()),
                          reinterpret_cast<const char*>(it->second.data()) + it->second.size() * 4);
    }
    write_file(entries_path, entries_buf);

    // Write cells file
    std::vector<char> cells_buf;
    uint32_t no_data = 0xFFFFFFFF;
    for (uint64_t cell_id : sorted_cells) {
        cells_buf.insert(cells_buf.end(), reinterpret_cast<const char*>(&cell_id),
                        reinterpret_cast<const char*>(&cell_id) + 8);
        auto it = cell_offsets.find(cell_id);
        uint32_t off = (it != cell_offsets.end()) ? it->second : no_data;
        cells_buf.insert(cells_buf.end(), reinterpret_cast<const char*>(&off),
                        reinterpret_cast<const char*>(&off) + 4);
    }
    write_file(cells_path, cells_buf);
}

// Write merged geo_cells with 3 offset columns
static void write_geo_cells_and_entries(
    const std::string& dir,
    const std::unordered_map<uint64_t, std::vector<uint32_t>>& street_cells,
    const std::unordered_map<uint64_t, std::vector<uint32_t>>& addr_cells,
    const std::unordered_map<uint64_t, std::vector<uint32_t>>& interp_cells)
{
    // Collect all unique cell IDs
    std::vector<uint64_t> all_cells;
    auto add = [&](const auto& m) { for (auto& [id, _] : m) all_cells.push_back(id); };
    add(street_cells); add(addr_cells); add(interp_cells);
    std::sort(all_cells.begin(), all_cells.end());
    all_cells.erase(std::unique(all_cells.begin(), all_cells.end()), all_cells.end());

    // Write each entry file and collect offsets
    auto write_entries = [&](const std::string& path,
                              const std::unordered_map<uint64_t, std::vector<uint32_t>>& cells)
        -> std::unordered_map<uint64_t, uint32_t>
    {
        std::vector<char> buf;
        std::unordered_map<uint64_t, uint32_t> offsets;
        for (uint64_t cell_id : all_cells) {
            auto it = cells.find(cell_id);
            if (it == cells.end() || it->second.empty()) continue;
            offsets[cell_id] = static_cast<uint32_t>(buf.size());
            uint16_t count = static_cast<uint16_t>(it->second.size());
            buf.insert(buf.end(), reinterpret_cast<const char*>(&count),
                      reinterpret_cast<const char*>(&count) + 2);
            buf.insert(buf.end(), reinterpret_cast<const char*>(it->second.data()),
                      reinterpret_cast<const char*>(it->second.data()) + it->second.size() * 4);
        }
        write_file(path, buf);
        return offsets;
    };

    auto s_off = write_entries(dir + "/street_entries.bin", street_cells);
    auto a_off = write_entries(dir + "/addr_entries.bin", addr_cells);
    auto i_off = write_entries(dir + "/interp_entries.bin", interp_cells);

    // Write geo_cells.bin
    uint32_t no_data = 0xFFFFFFFF;
    std::vector<char> buf;
    buf.reserve(all_cells.size() * 20);
    for (uint64_t cell_id : all_cells) {
        buf.insert(buf.end(), reinterpret_cast<const char*>(&cell_id),
                  reinterpret_cast<const char*>(&cell_id) + 8);
        auto get = [&](const auto& m) -> uint32_t {
            auto it = m.find(cell_id);
            return (it != m.end()) ? it->second : no_data;
        };
        uint32_t so = get(s_off), ao = get(a_off), io = get(i_off);
        buf.insert(buf.end(), reinterpret_cast<const char*>(&so), reinterpret_cast<const char*>(&so) + 4);
        buf.insert(buf.end(), reinterpret_cast<const char*>(&ao), reinterpret_cast<const char*>(&ao) + 4);
        buf.insert(buf.end(), reinterpret_cast<const char*>(&io), reinterpret_cast<const char*>(&io) + 4);
    }
    write_file(dir + "/geo_cells.bin", buf);
}

// --- Main ---

int main(int argc, char* argv[]) {
    if (argc < 3) {
        std::cerr << "Usage: geocoder-canonicalize <input-dir> <output-dir>" << std::endl;
        return 1;
    }

    std::string input_dir = argv[1];
    std::string output_dir = argv[2];
    ensure_dir(output_dir);

    std::cerr << "Loading index from " << input_dir << "..." << std::endl;

    // 1. Sort string pool
    std::cerr << "  Sorting string pool..." << std::endl;
    auto old_strings = read_file(input_dir + "/strings.bin");
    CanonicalStringPool pool;
    pool.build(old_strings);
    write_file(output_dir + "/strings.bin", pool.data);
    std::cerr << "    " << pool.remap.size() << " strings" << std::endl;

    // 2. Extract cell assignments before reordering
    std::cerr << "  Extracting cell assignments..." << std::endl;
    auto ca = extract_cell_assignments(input_dir);

    // 3. Canonicalize addr_points
    std::cerr << "  Sorting addr_points..." << std::endl;
    {
        auto points = read_structs<PatchAddrPoint>(input_dir + "/addr_points.bin");
        size_t n = points.size();
        std::vector<uint32_t> order(n);
        std::iota(order.begin(), order.end(), 0);

        // Sort by canonical key: (street, housenumber, lat_grid, lng_grid)
        std::sort(order.begin(), order.end(), [&](uint32_t a, uint32_t b) {
            const auto& pa = points[a];
            const auto& pb = points[b];
            // Compare by remapped street name
            uint32_t sa = pool.map(pa.street_id), sb = pool.map(pb.street_id);
            int c = strcmp(pool.data.data() + sa, pool.data.data() + sb);
            if (c != 0) return c < 0;
            // Then housenumber
            uint32_t ha = pool.map(pa.housenumber_id), hb = pool.map(pb.housenumber_id);
            c = strcmp(pool.data.data() + ha, pool.data.data() + hb);
            if (c != 0) return c < 0;
            // Then coordinates
            int la = to_grid(pa.lat), lb = to_grid(pb.lat);
            if (la != lb) return la < lb;
            return to_grid(pa.lng) < to_grid(pb.lng);
        });

        // Build old->new mapping and write
        std::vector<uint32_t> old_to_new(n);
        for (uint32_t i = 0; i < n; i++) old_to_new[order[i]] = i;

        std::vector<PatchAddrPoint> sorted(n);
        for (uint32_t i = 0; i < n; i++) {
            sorted[i] = points[order[i]];
            sorted[i].housenumber_id = pool.map(sorted[i].housenumber_id);
            sorted[i].street_id = pool.map(sorted[i].street_id);
        }
        write_file(output_dir + "/addr_points.bin",
                   reinterpret_cast<const char*>(sorted.data()), n * sizeof(PatchAddrPoint));
        remap_cells(ca.addr_cells, old_to_new);
        std::cerr << "    " << n << " addr points" << std::endl;
    }

    // 4. Canonicalize street_ways + street_nodes
    std::cerr << "  Sorting street_ways..." << std::endl;
    {
        auto ways = read_ways(input_dir + "/street_ways.bin");
        auto nodes = read_structs<PatchNodeCoord>(input_dir + "/street_nodes.bin");
        size_t n = ways.size();
        std::vector<uint32_t> order(n);
        std::iota(order.begin(), order.end(), 0);

        // Sort by (name, node_count, first_node_lat, first_node_lng)
        std::sort(order.begin(), order.end(), [&](uint32_t a, uint32_t b) {
            const auto& wa = ways[a];
            const auto& wb = ways[b];
            uint32_t na = pool.map(wa.name_id), nb = pool.map(wb.name_id);
            int c = strcmp(pool.data.data() + na, pool.data.data() + nb);
            if (c != 0) return c < 0;
            if (wa.node_count != wb.node_count) return wa.node_count < wb.node_count;
            // Compare first node
            if (wa.node_count > 0 && wb.node_count > 0) {
                int la = to_grid(nodes[wa.node_offset].lat);
                int lb = to_grid(nodes[wb.node_offset].lat);
                if (la != lb) return la < lb;
                int lga = to_grid(nodes[wa.node_offset].lng);
                int lgb = to_grid(nodes[wb.node_offset].lng);
                if (lga != lgb) return lga < lgb;
            }
            return false;
        });

        std::vector<uint32_t> old_to_new(n);
        for (uint32_t i = 0; i < n; i++) old_to_new[order[i]] = i;

        // Write sorted ways + nodes
        std::vector<char> ways_buf;
        ways_buf.reserve(n * 9);
        std::vector<PatchNodeCoord> sorted_nodes;
        sorted_nodes.reserve(nodes.size());

        for (uint32_t i = 0; i < n; i++) {
            PatchWayHeader w = ways[order[i]];
            uint32_t old_offset = w.node_offset;
            w.node_offset = static_cast<uint32_t>(sorted_nodes.size());
            w.name_id = pool.map(w.name_id);
            ways_buf.insert(ways_buf.end(), reinterpret_cast<const char*>(&w),
                           reinterpret_cast<const char*>(&w) + 9);
            for (uint8_t j = 0; j < w.node_count; j++) {
                if (old_offset + j < nodes.size())
                    sorted_nodes.push_back(nodes[old_offset + j]);
            }
        }
        write_file(output_dir + "/street_ways.bin", ways_buf);
        write_file(output_dir + "/street_nodes.bin",
                   reinterpret_cast<const char*>(sorted_nodes.data()),
                   sorted_nodes.size() * sizeof(PatchNodeCoord));
        remap_cells(ca.street_cells, old_to_new);
        std::cerr << "    " << n << " ways, " << sorted_nodes.size() << " nodes" << std::endl;
    }

    // 5. Canonicalize interp_ways + interp_nodes
    std::cerr << "  Sorting interp_ways..." << std::endl;
    {
        auto interps = read_interps(input_dir + "/interp_ways.bin");
        auto nodes = read_structs<PatchNodeCoord>(input_dir + "/interp_nodes.bin");
        size_t n = interps.size();
        std::vector<uint32_t> order(n);
        std::iota(order.begin(), order.end(), 0);

        std::sort(order.begin(), order.end(), [&](uint32_t a, uint32_t b) {
            const auto& ia = interps[a];
            const auto& ib = interps[b];
            uint32_t sa = pool.map(ia.street_id), sb = pool.map(ib.street_id);
            int c = strcmp(pool.data.data() + sa, pool.data.data() + sb);
            if (c != 0) return c < 0;
            if (ia.start_number != ib.start_number) return ia.start_number < ib.start_number;
            if (ia.end_number != ib.end_number) return ia.end_number < ib.end_number;
            return ia.interpolation < ib.interpolation;
        });

        std::vector<uint32_t> old_to_new(n);
        for (uint32_t i = 0; i < n; i++) old_to_new[order[i]] = i;

        // Write sorted interps + nodes using builder's native struct size
        auto orig_data = read_file(input_dir + "/interp_ways.bin");
        size_t stride = (orig_data.size() % 20 == 0 && n > 0) ? 20 :
                        (orig_data.size() % 18 == 0 && n > 0) ? 18 : 20;

        std::vector<char> interp_buf;
        interp_buf.reserve(n * stride);
        std::vector<PatchNodeCoord> sorted_nodes;
        sorted_nodes.reserve(nodes.size());

        for (uint32_t i = 0; i < n; i++) {
            // Copy the raw bytes from original file for correct padding
            const char* src = orig_data.data() + order[i] * stride;
            char record[24] = {};
            memcpy(record, src, stride);

            // Update node_offset
            uint32_t new_node_offset = static_cast<uint32_t>(sorted_nodes.size());
            memcpy(record, &new_node_offset, 4);

            // Update street_id (at offset 5 or 8 depending on padding)
            size_t street_off = (stride == 20) ? 8 : 5;
            uint32_t old_street_id;
            memcpy(&old_street_id, record + street_off, 4);
            uint32_t new_street_id = pool.map(old_street_id);
            memcpy(record + street_off, &new_street_id, 4);

            interp_buf.insert(interp_buf.end(), record, record + stride);

            // Copy nodes
            const auto& iw = interps[order[i]];
            for (uint8_t j = 0; j < iw.node_count; j++) {
                if (iw.node_offset + j < nodes.size())
                    sorted_nodes.push_back(nodes[iw.node_offset + j]);
            }
        }
        write_file(output_dir + "/interp_ways.bin", interp_buf);
        write_file(output_dir + "/interp_nodes.bin",
                   reinterpret_cast<const char*>(sorted_nodes.data()),
                   sorted_nodes.size() * sizeof(PatchNodeCoord));
        remap_cells(ca.interp_cells, old_to_new);
        std::cerr << "    " << n << " interps, " << sorted_nodes.size() << " nodes" << std::endl;
    }

    // 6. Canonicalize admin_polygons + admin_vertices
    std::cerr << "  Sorting admin_polygons..." << std::endl;
    {
        auto polygons = read_admin_polygons(input_dir + "/admin_polygons.bin");
        auto vertices = read_structs<PatchNodeCoord>(input_dir + "/admin_vertices.bin");
        size_t n = polygons.size();
        std::vector<uint32_t> order(n);
        std::iota(order.begin(), order.end(), 0);

        std::sort(order.begin(), order.end(), [&](uint32_t a, uint32_t b) {
            const auto& pa = polygons[a];
            const auto& pb = polygons[b];
            uint32_t na = pool.map(pa.name_id), nb = pool.map(pb.name_id);
            int c = strcmp(pool.data.data() + na, pool.data.data() + nb);
            if (c != 0) return c < 0;
            if (pa.admin_level != pb.admin_level) return pa.admin_level < pb.admin_level;
            if (pa.country_code != pb.country_code) return pa.country_code < pb.country_code;
            return pa.vertex_count < pb.vertex_count;
        });

        std::vector<uint32_t> old_to_new(n);
        for (uint32_t i = 0; i < n; i++) old_to_new[order[i]] = i;

        // Write sorted polygons + vertices using builder's native struct size
        auto orig_data = read_file(input_dir + "/admin_polygons.bin");
        size_t stride = 0;
        for (size_t s : {24, 20, 19}) {
            if (orig_data.size() % s == 0 && n > 0) { stride = s; break; }
        }
        if (stride == 0) stride = 24;

        std::vector<char> poly_buf;
        poly_buf.reserve(n * stride);
        std::vector<PatchNodeCoord> sorted_verts;
        sorted_verts.reserve(vertices.size());

        for (uint32_t i = 0; i < n; i++) {
            const char* src = orig_data.data() + order[i] * stride;
            char record[32] = {};
            memcpy(record, src, stride);

            // Update vertex_offset
            uint32_t new_vert_offset = static_cast<uint32_t>(sorted_verts.size());
            memcpy(record, &new_vert_offset, 4);

            // Update name_id (at offset 8)
            uint32_t old_name_id;
            memcpy(&old_name_id, record + 8, 4);
            uint32_t new_name_id = pool.map(old_name_id);
            memcpy(record + 8, &new_name_id, 4);

            // Recompute area for canonical output
            // (area depends on vertex order which hasn't changed, so copy as-is)

            poly_buf.insert(poly_buf.end(), record, record + stride);

            // Copy vertices
            const auto& p = polygons[order[i]];
            for (uint32_t j = 0; j < p.vertex_count; j++) {
                if (p.vertex_offset + j < vertices.size())
                    sorted_verts.push_back(vertices[p.vertex_offset + j]);
            }
        }
        write_file(output_dir + "/admin_polygons.bin", poly_buf);
        write_file(output_dir + "/admin_vertices.bin",
                   reinterpret_cast<const char*>(sorted_verts.data()),
                   sorted_verts.size() * sizeof(PatchNodeCoord));
        remap_cells(ca.admin_cells, old_to_new);
        std::cerr << "    " << n << " polygons, " << sorted_verts.size() << " vertices" << std::endl;
    }

    // 7. Write canonical entries and cell index files
    std::cerr << "  Writing cell indexes..." << std::endl;
    write_geo_cells_and_entries(output_dir, ca.street_cells, ca.addr_cells, ca.interp_cells);
    write_entries_and_cells(output_dir + "/admin_cells.bin", output_dir + "/admin_entries.bin",
                           ca.admin_cells, 12);

    std::cerr << "Done. Canonical output in " << output_dir << std::endl;
    return 0;
}


File created successfully at: /root/traccar-geocoder/builder/tools/geocoder_canonicalize.cpp

// geocoder-canonicalize: Converts a geocoder index to canonical (deterministic) format.
//
// Canonical format properties:
// - strings.bin: strings sorted alphabetically
// - Data files: records sorted by content fingerprint
// - Node/vertex files: grouped by parent record in canonical order
// - Entries/cells: derived from canonical data ordering + original cell assignments
//
// Usage: geocoder-canonicalize <input-dir> <output-dir>

#include <algorithm>
#include <cstdint>
#include <cstring>
#include <fstream>
#include <functional>
#include <iostream>
#include <numeric>
#include <string>
#include <unordered_map>
#include <vector>

#include "patch_format.h"

// --- Canonical string pool ---
// Sort all strings alphabetically, build old_offset -> new_offset remap.

struct CanonicalStringPool {
    std::vector<char> data;
    std::unordered_map<uint32_t, uint32_t> remap; // old_offset -> new_offset

    void build(const std::vector<char>& old_pool) {
        // Extract all strings with their offsets
        std::vector<std::pair<uint32_t, const char*>> strings;
        size_t pos = 0;
        while (pos < old_pool.size()) {
            const char* s = old_pool.data() + pos;
            strings.emplace_back(static_cast<uint32_t>(pos), s);
            pos += strlen(s) + 1;
        }

        // Sort alphabetically
        std::sort(strings.begin(), strings.end(), [](const auto& a, const auto& b) {
            return strcmp(a.second, b.second) < 0;
        });

        // Build new pool + remap
        data.clear();
        data.reserve(old_pool.size());
        remap.reserve(strings.size());
        for (auto& [old_off, str] : strings) {
            uint32_t new_off = static_cast<uint32_t>(data.size());
            remap[old_off] = new_off;
            size_t len = strlen(str);
            data.insert(data.end(), str, str + len + 1);
        }
    }

    uint32_t map(uint32_t old_off) const {
        auto it = remap.find(old_off);
        return (it != remap.end()) ? it->second : 0;
    }
};

// --- Cell assignment extraction ---
// Parse geo_cells.bin + entries files to extract cell -> [item_id] mappings.

struct GeoCellEntry {
    uint64_t cell_id;
    uint32_t street_offset;
    uint32_t addr_offset;
    uint32_t interp_offset;
};

struct CellAssignments {
    // For each item type: cell_id -> sorted list of item_ids
    std::unordered_map<uint64_t, std::vector<uint32_t>> street_cells;
    std::unordered_map<uint64_t, std::vector<uint32_t>> addr_cells;
    std::unordered_map<uint64_t, std::vector<uint32_t>> interp_cells;
    std::unordered_map<uint64_t, std::vector<uint32_t>> admin_cells; // with flags
};

static std::vector<uint32_t> parse_entry(const std::vector<char>& entries, uint32_t offset) {
    if (offset == 0xFFFFFFFF || offset + 2 > entries.size()) return {};
    uint16_t count;
    memcpy(&count, entries.data() + offset, 2);
    std::vector<uint32_t> ids(count);
    if (offset + 2 + count * 4 > entries.size()) return {};
    memcpy(ids.data(), entries.data() + offset + 2, count * 4);
    return ids;
}

static CellAssignments extract_cell_assignments(const std::string& dir) {
    CellAssignments ca;

    // Geo cells
    auto geo_data = read_file(dir + "/geo_cells.bin");
    auto street_entries = read_file(dir + "/street_entries.bin");
    auto addr_entries = read_file(dir + "/addr_entries.bin");
    auto interp_entries = read_file(dir + "/interp_entries.bin");

    size_t n_geo = geo_data.size() / 20;
    for (size_t i = 0; i < n_geo; i++) {
        GeoCellEntry e;
        memcpy(&e.cell_id, geo_data.data() + i * 20, 8);
        memcpy(&e.street_offset, geo_data.data() + i * 20 + 8, 4);
        memcpy(&e.addr_offset, geo_data.data() + i * 20 + 12, 4);
        memcpy(&e.interp_offset, geo_data.data() + i * 20 + 16, 4);

        auto s_ids = parse_entry(street_entries, e.street_offset);
        if (!s_ids.empty()) ca.street_cells[e.cell_id] = std::move(s_ids);

        auto a_ids = parse_entry(addr_entries, e.addr_offset);
        if (!a_ids.empty()) ca.addr_cells[e.cell_id] = std::move(a_ids);

        auto i_ids = parse_entry(interp_entries, e.interp_offset);
        if (!i_ids.empty()) ca.interp_cells[e.cell_id] = std::move(i_ids);
    }

    // Admin cells
    auto admin_cell_data = read_file(dir + "/admin_cells.bin");
    auto admin_entry_data = read_file(dir + "/admin_entries.bin");
    size_t n_admin = admin_cell_data.size() / 12;
    for (size_t i = 0; i < n_admin; i++) {
        uint64_t cell_id;
        uint32_t offset;
        memcpy(&cell_id, admin_cell_data.data() + i * 12, 8);
        memcpy(&offset, admin_cell_data.data() + i * 12 + 8, 4);
        auto ids = parse_entry(admin_entry_data, offset);
        if (!ids.empty()) ca.admin_cells[cell_id] = std::move(ids);
    }

    return ca;
}

// --- Remap cell assignments with old->new ID permutation ---
static void remap_cells(std::unordered_map<uint64_t, std::vector<uint32_t>>& cells,
                         const std::vector<uint32_t>& old_to_new) {
    for (auto& [cell_id, ids] : cells) {
        for (auto& id : ids) {
            uint32_t masked = id & 0x7FFFFFFF;
            uint32_t flags = id & 0x80000000;
            if (masked < old_to_new.size()) {
                id = old_to_new[masked] | flags;
            }
        }
        std::sort(ids.begin(), ids.end());
    }
}

// --- Write entries + cells files ---
static void write_entries_and_cells(
    const std::string& cells_path,
    const std::string& entries_path,
    const std::unordered_map<uint64_t, std::vector<uint32_t>>& cell_map,
    size_t cell_entry_size) // 20 for geo, 12 for admin
{
    // Collect and sort cells
    std::vector<uint64_t> sorted_cells;
    sorted_cells.reserve(cell_map.size());
    for (auto& [id, _] : cell_map) sorted_cells.push_back(id);
    std::sort(sorted_cells.begin(), sorted_cells.end());

    // Write entries file
    std::vector<char> entries_buf;
    std::unordered_map<uint64_t, uint32_t> cell_offsets;
    for (uint64_t cell_id : sorted_cells) {
        auto it = cell_map.find(cell_id);
        if (it == cell_map.end() || it->second.empty()) continue;
        cell_offsets[cell_id] = static_cast<uint32_t>(entries_buf.size());
        uint16_t count = static_cast<uint16_t>(it->second.size());
        entries_buf.insert(entries_buf.end(), reinterpret_cast<const char*>(&count),
                          reinterpret_cast<const char*>(&count) + 2);
        entries_buf.insert(entries_buf.end(),
                          reinterpret_cast<const char*>(it->second.data()),
                          reinterpret_cast<const char*>(it->second.data()) + it->second.size() * 4);
    }
    write_file(entries_path, entries_buf);

    // Write cells file
    std::vector<char> cells_buf;
    uint32_t no_data = 0xFFFFFFFF;
    for (uint64_t cell_id : sorted_cells) {
        cells_buf.insert(cells_buf.end(), reinterpret_cast<const char*>(&cell_id),
                        reinterpret_cast<const char*>(&cell_id) + 8);
        auto it = cell_offsets.find(cell_id);
        uint32_t off = (it != cell_offsets.end()) ? it->second : no_data;
        cells_buf.insert(cells_buf.end(), reinterpret_cast<const char*>(&off),
                        reinterpret_cast<const char*>(&off) + 4);
    }
    write_file(cells_path, cells_buf);
}

// Write merged geo_cells with 3 offset columns
static void write_geo_cells_and_entries(
    const std::string& dir,
    const std::unordered_map<uint64_t, std::vector<uint32_t>>& street_cells,
    const std::unordered_map<uint64_t, std::vector<uint32_t>>& addr_cells,
    const std::unordered_map<uint64_t, std::vector<uint32_t>>& interp_cells)
{
    // Collect all unique cell IDs
    std::vector<uint64_t> all_cells;
    auto add = [&](const auto& m) { for (auto& [id, _] : m) all_cells.push_back(id); };
    add(street_cells); add(addr_cells); add(interp_cells);
    std::sort(all_cells.begin(), all_cells.end());
    all_cells.erase(std::unique(all_cells.begin(), all_cells.end()), all_cells.end());

    // Write each entry file and collect offsets
    auto write_entries = [&](const std::string& path,
                              const std::unordered_map<uint64_t, std::vector<uint32_t>>& cells)
        -> std::unordered_map<uint64_t, uint32_t>
    {
        std::vector<char> buf;
        std::unordered_map<uint64_t, uint32_t> offsets;
        for (uint64_t cell_id : all_cells) {
            auto it = cells.find(cell_id);
            if (it == cells.end() || it->second.empty()) continue;
            offsets[cell_id] = static_cast<uint32_t>(buf.size());
            uint16_t count = static_cast<uint16_t>(it->second.size());
            buf.insert(buf.end(), reinterpret_cast<const char*>(&count),
                      reinterpret_cast<const char*>(&count) + 2);
            buf.insert(buf.end(), reinterpret_cast<const char*>(it->second.data()),
                      reinterpret_cast<const char*>(it->second.data()) + it->second.size() * 4);
        }
        write_file(path, buf);
        return offsets;
    };

    auto s_off = write_entries(dir + "/street_entries.bin", street_cells);
    auto a_off = write_entries(dir + "/addr_entries.bin", addr_cells);
    auto i_off = write_entries(dir + "/interp_entries.bin", interp_cells);

    // Write geo_cells.bin
    uint32_t no_data = 0xFFFFFFFF;
    std::vector<char> buf;
    buf.reserve(all_cells.size() * 20);
    for (uint64_t cell_id : all_cells) {
        buf.insert(buf.end(), reinterpret_cast<const char*>(&cell_id),
                  reinterpret_cast<const char*>(&cell_id) + 8);
        auto get = [&](const auto& m) -> uint32_t {
            auto it = m.find(cell_id);
            return (it != m.end()) ? it->second : no_data;
        };
        uint32_t so = get(s_off), ao = get(a_off), io = get(i_off);
        buf.insert(buf.end(), reinterpret_cast<const char*>(&so), reinterpret_cast<const char*>(&so) + 4);
        buf.insert(buf.end(), reinterpret_cast<const char*>(&ao), reinterpret_cast<const char*>(&ao) + 4);
        buf.insert(buf.end(), reinterpret_cast<const char*>(&io), reinterpret_cast<const char*>(&io) + 4);
    }
    write_file(dir + "/geo_cells.bin", buf);
}

// --- Main ---

int main(int argc, char* argv[]) {
    if (argc < 3) {
        std::cerr << "Usage: geocoder-canonicalize <input-dir> <output-dir>" << std::endl;
        return 1;
    }

    std::string input_dir = argv[1];
    std::string output_dir = argv[2];
    ensure_dir(output_dir);

    std::cerr << "Loading index from " << input_dir << "..." << std::endl;

    // 1. Sort string pool
    std::cerr << "  Sorting string pool..." << std::endl;
    auto old_strings = read_file(input_dir + "/strings.bin");
    CanonicalStringPool pool;
    pool.build(old_strings);
    write_file(output_dir + "/strings.bin", pool.data);
    std::cerr << "    " << pool.remap.size() << " strings" << std::endl;

    // 2. Extract cell assignments before reordering
    std::cerr << "  Extracting cell assignments..." << std::endl;
    auto ca = extract_cell_assignments(input_dir);

    // 3. Canonicalize addr_points
    std::cerr << "  Sorting addr_points..." << std::endl;
    {
        auto points = read_structs<PatchAddrPoint>(input_dir + "/addr_points.bin");
        size_t n = points.size();
        std::vector<uint32_t> order(n);
        std::iota(order.begin(), order.end(), 0);

        // Sort by canonical key: (street, housenumber, lat_grid, lng_grid)
        std::sort(order.begin(), order.end(), [&](uint32_t a, uint32_t b) {
            const auto& pa = points[a];
            const auto& pb = points[b];
            // Compare by remapped street name
            uint32_t sa = pool.map(pa.street_id), sb = pool.map(pb.street_id);
            int c = strcmp(pool.data.data() + sa, pool.data.data() + sb);
            if (c != 0) return c < 0;
            // Then housenumber
            uint32_t ha = pool.map(pa.housenumber_id), hb = pool.map(pb.housenumber_id);
            c = strcmp(pool.data.data() + ha, pool.data.data() + hb);
            if (c != 0) return c < 0;
            // Then coordinates
            int la = to_grid(pa.lat), lb = to_grid(pb.lat);
            if (la != lb) return la < lb;
            return to_grid(pa.lng) < to_grid(pb.lng);
        });

        // Build old->new mapping and write
        std::vector<uint32_t> old_to_new(n);
        for (uint32_t i = 0; i < n; i++) old_to_new[order[i]] = i;

        std::vector<PatchAddrPoint> sorted(n);
        for (uint32_t i = 0; i < n; i++) {
            sorted[i] = points[order[i]];
            sorted[i].housenumber_id = pool.map(sorted[i].housenumber_id);
            sorted[i].street_id = pool.map(sorted[i].street_id);
        }
        write_file(output_dir + "/addr_points.bin",
                   reinterpret_cast<const char*>(sorted.data()), n * sizeof(PatchAddrPoint));
        remap_cells(ca.addr_cells, old_to_new);
        std::cerr << "    " << n << " addr points" << std::endl;
    }

    // 4. Canonicalize street_ways + street_nodes
    std::cerr << "  Sorting street_ways..." << std::endl;
    {
        auto ways = read_ways(input_dir + "/street_ways.bin");
        auto nodes = read_structs<PatchNodeCoord>(input_dir + "/street_nodes.bin");
        size_t n = ways.size();
        std::vector<uint32_t> order(n);
        std::iota(order.begin(), order.end(), 0);

        // Sort by (name, node_count, first_node_lat, first_node_lng)
        std::sort(order.begin(), order.end(), [&](uint32_t a, uint32_t b) {
            const auto& wa = ways[a];
            const auto& wb = ways[b];
            uint32_t na = pool.map(wa.name_id), nb = pool.map(wb.name_id);
            int c = strcmp(pool.data.data() + na, pool.data.data() + nb);
            if (c != 0) return c < 0;
            if (wa.node_count != wb.node_count) return wa.node_count < wb.node_count;
            // Compare first node
            if (wa.node_count > 0 && wb.node_count > 0) {
                int la = to_grid(nodes[wa.node_offset].lat);
                int lb = to_grid(nodes[wb.node_offset].lat);
                if (la != lb) return la < lb;
                int lga = to_grid(nodes[wa.node_offset].lng);
                int lgb = to_grid(nodes[wb.node_offset].lng);
                if (lga != lgb) return lga < lgb;
            }
            return false;
        });

        std::vector<uint32_t> old_to_new(n);
        for (uint32_t i = 0; i < n; i++) old_to_new[order[i]] = i;

        // Write sorted ways + nodes
        std::vector<char> ways_buf;
        ways_buf.reserve(n * 9);
        std::vector<PatchNodeCoord> sorted_nodes;
        sorted_nodes.reserve(nodes.size());

        for (uint32_t i = 0; i < n; i++) {
            PatchWayHeader w = ways[order[i]];
            uint32_t old_offset = w.node_offset;
            w.node_offset = static_cast<uint32_t>(sorted_nodes.size());
            w.name_id = pool.map(w.name_id);
            ways_buf.insert(ways_buf.end(), reinterpret_cast<const char*>(&w),
                           reinterpret_cast<const char*>(&w) + 9);
            for (uint8_t j = 0; j < w.node_count; j++) {
                if (old_offset + j < nodes.size())
                    sorted_nodes.push_back(nodes[old_offset + j]);
            }
        }
        write_file(output_dir + "/street_ways.bin", ways_buf);
        write_file(output_dir + "/street_nodes.bin",
                   reinterpret_cast<const char*>(sorted_nodes.data()),
                   sorted_nodes.size() * sizeof(PatchNodeCoord));
        remap_cells(ca.street_cells, old_to_new);
        std::cerr << "    " << n << " ways, " << sorted_nodes.size() << " nodes" << std::endl;
    }

    // 5. Canonicalize interp_ways + interp_nodes
    std::cerr << "  Sorting interp_ways..." << std::endl;
    {
        auto interps = read_interps(input_dir + "/interp_ways.bin");
        auto nodes = read_structs<PatchNodeCoord>(input_dir + "/interp_nodes.bin");
        size_t n = interps.size();
        std::vector<uint32_t> order(n);
        std::iota(order.begin(), order.end(), 0);

        std::sort(order.begin(), order.end(), [&](uint32_t a, uint32_t b) {
            const auto& ia = interps[a];
            const auto& ib = interps[b];
            uint32_t sa = pool.map(ia.street_id), sb = pool.map(ib.street_id);
            int c = strcmp(pool.data.data() + sa, pool.data.data() + sb);
            if (c != 0) return c < 0;
            if (ia.start_number != ib.start_number) return ia.start_number < ib.start_number;
            if (ia.end_number != ib.end_number) return ia.end_number < ib.end_number;
            return ia.interpolation < ib.interpolation;
        });

        std::vector<uint32_t> old_to_new(n);
        for (uint32_t i = 0; i < n; i++) old_to_new[order[i]] = i;

        // Write sorted interps + nodes using builder's native struct size
        auto orig_data = read_file(input_dir + "/interp_ways.bin");
        size_t stride = (orig_data.size() % 20 == 0 && n > 0) ? 20 :
                        (orig_data.size() % 18 == 0 && n > 0) ? 18 : 20;

        std::vector<char> interp_buf;
        interp_buf.reserve(n * stride);
        std::vector<PatchNodeCoord> sorted_nodes;
        sorted_nodes.reserve(nodes.size());

        for (uint32_t i = 0; i < n; i++) {
            // Copy the raw bytes from original file for correct padding
            const char* src = orig_data.data() + order[i] * stride;
            char record[24] = {};
            memcpy(record, src, stride);

            // Update node_offset
            uint32_t new_node_offset = static_cast<uint32_t>(sorted_nodes.size());
            memcpy(record, &new_node_offset, 4);

            // Update street_id (at offset 5 or 8 depending on padding)
            size_t street_off = (stride == 20) ? 8 : 5;
            uint32_t old_street_id;
            memcpy(&old_street_id, record + street_off, 4);
            uint32_t new_street_id = pool.map(old_street_id);
            memcpy(record + street_off, &new_street_id, 4);

            interp_buf.insert(interp_buf.end(), record, record + stride);

            // Copy nodes
            const auto& iw = interps[order[i]];
            for (uint8_t j = 0; j < iw.node_count; j++) {
                if (iw.node_offset + j < nodes.size())
                    sorted_nodes.push_back(nodes[iw.node_offset + j]);
            }
        }
        write_file(output_dir + "/interp_ways.bin", interp_buf);
        write_file(output_dir + "/interp_nodes.bin",
                   reinterpret_cast<const char*>(sorted_nodes.data()),
                   sorted_nodes.size() * sizeof(PatchNodeCoord));
        remap_cells(ca.interp_cells, old_to_new);
        std::cerr << "    " << n << " interps, " << sorted_nodes.size() << " nodes" << std::endl;
    }

    // 6. Canonicalize admin_polygons + admin_vertices
    std::cerr << "  Sorting admin_polygons..." << std::endl;
    {
        auto polygons = read_admin_polygons(input_dir + "/admin_polygons.bin");
        auto vertices = read_structs<PatchNodeCoord>(input_dir + "/admin_vertices.bin");
        size_t n = polygons.size();
        std::vector<uint32_t> order(n);
        std::iota(order.begin(), order.end(), 0);

        std::sort(order.begin(), order.end(), [&](uint32_t a, uint32_t b) {
            const auto& pa = polygons[a];
            const auto& pb = polygons[b];
            uint32_t na = pool.map(pa.name_id), nb = pool.map(pb.name_id);
            int c = strcmp(pool.data.data() + na, pool.data.data() + nb);
            if (c != 0) return c < 0;
            if (pa.admin_level != pb.admin_level) return pa.admin_level < pb.admin_level;
            if (pa.country_code != pb.country_code) return pa.country_code < pb.country_code;
            return pa.vertex_count < pb.vertex_count;
        });

        std::vector<uint32_t> old_to_new(n);
        for (uint32_t i = 0; i < n; i++) old_to_new[order[i]] = i;

        // Write sorted polygons + vertices using builder's native struct size
        auto orig_data = read_file(input_dir + "/admin_polygons.bin");
        size_t stride = 0;
        for (size_t s : {24, 20, 19}) {
            if (orig_data.size() % s == 0 && n > 0) { stride = s; break; }
        }
        if (stride == 0) stride = 24;

        std::vector<char> poly_buf;
        poly_buf.reserve(n * stride);
        std::vector<PatchNodeCoord> sorted_verts;
        sorted_verts.reserve(vertices.size());

        for (uint32_t i = 0; i < n; i++) {
            const char* src = orig_data.data() + order[i] * stride;
            char record[32] = {};
            memcpy(record, src, stride);

            // Update vertex_offset
            uint32_t new_vert_offset = static_cast<uint32_t>(sorted_verts.size());
            memcpy(record, &new_vert_offset, 4);

            // Update name_id (at offset 8)
            uint32_t old_name_id;
            memcpy(&old_name_id, record + 8, 4);
            uint32_t new_name_id = pool.map(old_name_id);
            memcpy(record + 8, &new_name_id, 4);

            // Recompute area for canonical output
            // (area depends on vertex order which hasn't changed, so copy as-is)

            poly_buf.insert(poly_buf.end(), record, record + stride);

            // Copy vertices
            const auto& p = polygons[order[i]];
            for (uint32_t j = 0; j < p.vertex_count; j++) {
                if (p.vertex_offset + j < vertices.size())
                    sorted_verts.push_back(vertices[p.vertex_offset + j]);
            }
        }
        write_file(output_dir + "/admin_polygons.bin", poly_buf);
        write_file(output_dir + "/admin_vertices.bin",
                   reinterpret_cast<const char*>(sorted_verts.data()),
                   sorted_verts.size() * sizeof(PatchNodeCoord));
        remap_cells(ca.admin_cells, old_to_new);
        std::cerr << "    " << n << " polygons, " << sorted_verts.size() << " vertices" << std::endl;
    }

    // 7. Write canonical entries and cell index files
    std::cerr << "  Writing cell indexes..." << std::endl;
    write_geo_cells_and_entries(output_dir, ca.street_cells, ca.addr_cells, ca.interp_cells);
    write_entries_and_cells(output_dir + "/admin_cells.bin", output_dir + "/admin_entries.bin",
                           ca.admin_cells, 12);

    std::cerr << "Done. Canonical output in " << output_dir << std::endl;
    return 0;
}


Now add it to CMakeLists.txt and build:


Edit

# Patch tools (no S2 dependency)
add_executable(geocoder-canonicalize tools/geocoder_canonicalize.cpp)
target_link_libraries(geocoder-canonicalize PRIVATE Threads::Threads)
target_include_directories(geocoder-canonicalize PRIVATE tools)

if(EXISTS /opt/homebrew/include)


The file /root/traccar-geocoder/builder/CMakeLists.txt has been updated successfully.

# Patch tools (no S2 dependency)
add_executable(geocoder-canonicalize tools/geocoder_canonicalize.cpp)
target_link_libraries(geocoder-canonicalize PRIVATE Threads::Threads)
target_include_directories(geocoder-canonicalize PRIVATE tools)

if(EXISTS /opt/homebrew/include)

cmake_minimum_required(VERSION 3.16)
project(traccar-geocoder-build LANGUAGES CXX)

set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED ON)

find_package(Threads REQUIRED)
find_package(ZLIB REQUIRED)
find_package(BZip2 REQUIRED)
find_package(EXPAT REQUIRED)
find_package(s2 QUIET)

add_executable(build-index
    src/build_index.cpp
    src/pbf_reader.cpp
    src/s2_helpers.cpp
    src/ring_assembly.cpp
    src/cache.cpp
    src/continent_filter.cpp
    src/continent_boundaries.cpp
    src/cell_index.cpp
    src/interpolation.cpp
)

if(EXISTS /opt/homebrew/include)
    target_include_directories(build-index PRIVATE /opt/homebrew/include)
endif()

if(s2_FOUND)
    target_link_libraries(build-index PRIVATE
        s2::s2
        ZLIB::ZLIB
        BZip2::BZip2
        EXPAT::EXPAT
        Threads::Threads
    )
else()
    target_include_directories(build-index PRIVATE /usr/include/s2)
    target_link_libraries(build-index PRIVATE
        s2
        ZLIB::ZLIB
        BZip2::BZip2
        EXPAT::EXPAT
        Threads::Threads
    )
endif()


Bash

cmake --build build -j$(nproc) --target geocoder-canonicalize 2>&1 | tail -15

Build the canonicalize tool


gmake: *** No rule to make target 'geocoder-canonicalize'.  Stop.


cmake -B build . 2>&1 | tail -10 && cmake --build build -j$(nproc) --target geocoder-canonicalize 2>&1 | tail -15

Reconfigure and build canonicalize tool


-- Configuring done (0.1s)
-- Generating done (0.0s)
-- Build files have been written to: /root/traccar-geocoder/builder/build
[ 50%] Building CXX object CMakeFiles/geocoder-canonicalize.dir/tools/geocoder_canonicalize.cpp.o
[100%] Linking CXX executable geocoder-canonicalize
[100%] Built target geocoder-canonicalize


Good, it builds. Now let me also revert the `--previous-build` and stabilization changes from `build_index.cpp` since the user doesn't want the build process modified. Let me first check what was there before.