Let me read the existing settings file and update it.


     1→{}
     2→


Edit

{
  "permissions": {
    "defaultMode": "bypassPermissions"
  }
}


The file /root/.claude/settings.json has been updated successfully.

-{}
+{
+  "permissions": {
+    "defaultMode": "bypassPermissions"
+  }
+}


Done. The default permission mode is now set to `bypassPermissions` in `~/.claude/settings.json`. Restart the session and it'll take effect — no more permission prompts.


Alright, lets go. Continue with the previous task


Picking up where we left off. Tailscale is connected — node 3 (`michtest-ng`) is at `100.113.88.121`. Since we're in userspace networking mode, I need to set up SSH to proxy through Tailscale.


Bash

mkdir -p ~/.ssh && cat >> ~/.ssh/config << 'EOF'

Host node3
    HostName 100.113.88.121
    User michtest
    ProxyCommand tailscale nc %h %p
    StrictHostKeyChecking no
    UserKnownHostsFile /dev/null
EOF
echo "SSH config added"

Configure SSH to proxy through tailscale for node3


SSH config added


ssh node3 "echo connected; hostname; uptime" 2>&1

Test SSH connection to node3 via tailscale


Error: Exit code 255
Warning: Permanently added '100.113.88.121' (ED25519) to the list of known hosts.
Permission denied, please try again.
Permission denied, please try again.
michtest@100.113.88.121: Permission denied (publickey,password).


Need password auth. Let me use `sshpass` with the password from the context (`elon-is-insane`).


which sshpass 2>/dev/null || apt-get install -y sshpass 2>&1 | tail -3

Install sshpass for password auth


Preparing to unpack .../sshpass_1.10-0.1_amd64.deb ...
Unpacking sshpass (1.10-0.1) ...
Setting up sshpass (1.10-0.1) ...


sshpass -p 'elon-is-insane' ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ProxyCommand="tailscale nc %h %p" michtest@100.113.88.121 "echo connected; hostname; uptime" 2>&1

Test SSH to node3 with password


Warning: Permanently added '100.113.88.121' (ED25519) to the list of known hosts.
connected
michtest-ng
 07:31:01 up 12:40,  2 users,  load average: 0.00, 0.00, 0.00


Connected to node 3. Now let me check on the sequential build and the parallel build outputs.


sshpass -p 'elon-is-insane' ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ProxyCommand="tailscale nc %h %p" michtest@100.113.88.121 "echo '=== Running processes ==='; ps aux | grep traccar-geocoder | grep -v grep; echo '=== Parallel output ==='; ls -la /tmp/de-parallel/ 2>/dev/null; echo '=== Sequential output ==='; ls -la /tmp/de-sequential/ 2>/dev/null; echo '=== diff_admin tool ==='; ls -la /tmp/diff_admin 2>/dev/null; echo '=== compare_indexes ==='; ls -la /tmp/compare_indexes* 2>/dev/null" 2>&1

Check build status and output files on node3


Warning: Permanently added '100.113.88.121' (ED25519) to the list of known hosts.
=== Running processes ===
=== Parallel output ===
total 1125784
drwxrwxr-x  2 michtest michtest       320 Mar 21 06:07 .
drwxrwxrwt 16 root     root           440 Mar 21 06:18 ..
-rw-rw-r--  1 michtest michtest  91266798 Mar 21 06:07 addr_entries.bin
-rw-rw-r--  1 michtest michtest 319218496 Mar 21 06:07 addr_points.bin
-rw-rw-r--  1 michtest michtest     73236 Mar 21 06:07 admin_cells.bin
-rw-rw-r--  1 michtest michtest    557718 Mar 21 06:07 admin_entries.bin
-rw-rw-r--  1 michtest michtest   1036272 Mar 21 06:07 admin_polygons.bin
-rw-rw-r--  1 michtest michtest 101362368 Mar 21 06:07 admin_vertices.bin
-rw-rw-r--  1 michtest michtest 195412000 Mar 21 06:07 geo_cells.bin
-rw-rw-r--  1 michtest michtest       914 Mar 21 06:07 interp_entries.bin
-rw-rw-r--  1 michtest michtest      4624 Mar 21 06:07 interp_nodes.bin
-rw-rw-r--  1 michtest michtest      1872 Mar 21 06:07 interp_ways.bin
-rw-rw-r--  1 michtest michtest  71239790 Mar 21 06:06 street_entries.bin
-rw-rw-r--  1 michtest michtest 183794472 Mar 21 06:07 street_nodes.bin
-rw-rw-r--  1 michtest michtest  39752076 Mar 21 06:07 street_ways.bin
-rw-rw-r--  1 michtest michtest 149049831 Mar 21 06:07 strings.bin
=== Sequential output ===
total 1125824
drwxrwxr-x  2 michtest michtest       320 Mar 21 06:21 .
drwxrwxrwt 16 root     root           440 Mar 21 06:18 ..
-rw-rw-r--  1 michtest michtest  91266798 Mar 21 06:20 addr_entries.bin
-rw-rw-r--  1 michtest michtest 319218496 Mar 21 06:21 addr_points.bin
-rw-rw-r--  1 michtest michtest     73236 Mar 21 06:21 admin_cells.bin
-rw-rw-r--  1 michtest michtest    558138 Mar 21 06:21 admin_entries.bin
-rw-rw-r--  1 michtest michtest   1037304 Mar 21 06:21 admin_polygons.bin
-rw-rw-r--  1 michtest michtest 101400464 Mar 21 06:21 admin_vertices.bin
-rw-rw-r--  1 michtest michtest 195412000 Mar 21 06:20 geo_cells.bin
-rw-rw-r--  1 michtest michtest       914 Mar 21 06:20 interp_entries.bin
-rw-rw-r--  1 michtest michtest      4624 Mar 21 06:21 interp_nodes.bin
-rw-rw-r--  1 michtest michtest      1872 Mar 21 06:21 interp_ways.bin
-rw-rw-r--  1 michtest michtest  71239790 Mar 21 06:20 street_entries.bin
-rw-rw-r--  1 michtest michtest 183794472 Mar 21 06:21 street_nodes.bin
-rw-rw-r--  1 michtest michtest  39752076 Mar 21 06:21 street_ways.bin
-rw-rw-r--  1 michtest michtest 149050034 Mar 21 06:21 strings.bin
=== diff_admin tool ===
-rwxrwxr-x 1 michtest michtest 63560 Mar 21 05:34 /tmp/diff_admin
=== compare_indexes ===
-rwxrwxr-x 1 michtest michtest 105680 Mar 21 04:44 /tmp/compare_indexes
-rw-r--r-- 1 michtest michtest  11255 Mar 21 04:44 /tmp/compare_indexes.cpp


Both builds are complete. Let me compare the key admin files and run the diff tool.


sshpass -p 'elon-is-insane' ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ProxyCommand="tailscale nc %h %p" michtest@100.113.88.121 "echo '=== File size comparison ==='; echo 'File | Parallel | Sequential | Diff'; for f in admin_entries.bin admin_polygons.bin admin_vertices.bin admin_cells.bin strings.bin; do p=\$(stat -c%s /tmp/de-parallel/\$f 2>/dev/null || echo 0); s=\$(stat -c%s /tmp/de-sequential/\$f 2>/dev/null || echo 0); d=\$((s - p)); echo \"\$f | \$p | \$s | \$d\"; done; echo; echo '=== Running diff_admin ==='; /tmp/diff_admin /tmp/de-parallel /tmp/de-sequential 2>&1" 2>&1

Compare admin polygon counts between parallel and sequential Germany builds


Warning: Permanently added '100.113.88.121' (ED25519) to the list of known hosts.
=== File size comparison ===
File | Parallel | Sequential | Diff
admin_entries.bin | 557718 | 558138 | 420
admin_polygons.bin | 1036272 | 1037304 | 1032
admin_vertices.bin | 101362368 | 101400464 | 38096
admin_cells.bin | 73236 | 73236 | 0
strings.bin | 149049831 | 149050034 | 203

=== Running diff_admin ===
New code: 43178 polygons
Old code: 43221 polygons
Missing from new: 55
Extra in new: 12

=== Missing by admin level ===
  Level 2: 1 polygons
  Level 6: 2 polygons
  Level 8: 6 polygons
  Level 9: 7 polygons
  Level 10: 37 polygons
  Level 11: 2 polygons

=== Top 20 missing names ===
  2x Großfischbach
  1x Wäschegrund
  1x Werl
  1x Wense-Fallingbostel
  1x Wald-Michelbach
  1x Walchensee
  1x Verr
  1x Umbertshausen
  1x Süd
  1x Steinacker
  1x Solalinden
  1x Siedelsbrunn
  1x Schwaigwall
  1x Sauschütt
  1x Sankt Johannisrain
  1x Putzbrunn
  1x Ottfingen
  1x Osterholz-Scharmbeck
  1x Olpe
  1x Oedenstockach

=== Polygon count by level ===
  Level           New        Old       Diff
  2                10          9          1
  4                49         48          1
  5                56         56          0
  6               663        664         -1
  7              1667       1667          0
  8             12495      12499         -4
  9             10812      10816         -4
  10             8724       8758        -34
  11             8702       8704         -2


Good progress. The gap has narrowed dramatically:

- **Before node-ID fix (planet)**: ~12K fewer polygons (931,963 vs 943,638)
- **After node-ID fix (Germany)**: Only **43 fewer** polygons (43,178 vs 43,221) — 55 missing, 12 extra

That's a huge improvement. The remaining 55 missing polygons are mostly small admin level 10 boundaries. Let me investigate what's causing these remaining failures to see if we can close the gap further.


sshpass -p 'elon-is-insane' ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ProxyCommand="tailscale nc %h %p" michtest@100.113.88.121 "echo '=== Extra in new (12) ==='; /tmp/diff_admin /tmp/de-parallel /tmp/de-sequential 2>&1 | grep -A 100 'Extra in new' | head -50" 2>&1

Get more details about missing/extra polygons


Warning: Permanently added '100.113.88.121' (ED25519) to the list of known hosts.
=== Extra in new (12) ===
Extra in new: 12

=== Missing by admin level ===
  Level 2: 1 polygons
  Level 6: 2 polygons
  Level 8: 6 polygons
  Level 9: 7 polygons
  Level 10: 37 polygons
  Level 11: 2 polygons

=== Top 20 missing names ===
  2x Großfischbach
  1x Wäschegrund
  1x Werl
  1x Wense-Fallingbostel
  1x Wald-Michelbach
  1x Walchensee
  1x Verr
  1x Umbertshausen
  1x Süd
  1x Steinacker
  1x Solalinden
  1x Siedelsbrunn
  1x Schwaigwall
  1x Sauschütt
  1x Sankt Johannisrain
  1x Putzbrunn
  1x Ottfingen
  1x Osterholz-Scharmbeck
  1x Olpe
  1x Oedenstockach

=== Polygon count by level ===
  Level           New        Old       Diff
  2                10          9          1
  4                49         48          1
  5                56         56          0
  6               663        664         -1
  7              1667       1667          0
  8             12495      12499         -4
  9             10812      10816         -4
  10             8724       8758        -34
  11             8702       8704         -2


Let me look at the diff_admin source to understand what details it can give us, and also look at the ring stitcher code to understand what might cause these remaining failures.


sshpass -p 'elon-is-insane' ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ProxyCommand="tailscale nc %h %p" michtest@100.113.88.121 "cat /tmp/compare_indexes.cpp" 2>&1

Read the compare_indexes source


Warning: Permanently added '100.113.88.121' (ED25519) to the list of known hosts.
// Semantic comparison of two geocoder index directories.
// Decodes all binary files, normalizes ordering, and compares content.
// Returns 0 if semantically identical, 1 if different.
//
// Usage: compare_indexes <dir-A> <dir-B>

#include <algorithm>
#include <cmath>
#include <cstdint>
#include <cstring>
#include <fstream>
#include <iostream>
#include <map>
#include <set>
#include <string>
#include <vector>

// --- Binary structs (must match build_index.cpp and server main.rs) ---

struct WayHeader {
    uint32_t node_offset;
    uint8_t node_count;
    uint32_t name_id;
};

struct AddrPoint {
    float lat;
    float lng;
    uint32_t housenumber_id;
    uint32_t street_id;
};

struct InterpWay {
    uint32_t node_offset;
    uint8_t node_count;
    uint32_t street_id;
    uint32_t start_number;
    uint32_t end_number;
    uint8_t interpolation;
};

struct AdminPolygon {
    uint32_t vertex_offset;
    uint16_t vertex_count;
    uint32_t name_id;
    uint8_t admin_level;
    float area;
    uint16_t country_code;
};

struct NodeCoord {
    float lat;
    float lng;
};

// --- Helpers ---

static std::vector<char> read_file(const std::string& path) {
    std::ifstream f(path, std::ios::binary | std::ios::ate);
    if (!f) return {};
    auto size = f.tellg();
    f.seekg(0);
    std::vector<char> data(size);
    f.read(data.data(), size);
    return data;
}

static const char* get_string(const std::vector<char>& pool, uint32_t offset) {
    if (offset >= pool.size()) return "(invalid)";
    return pool.data() + offset;
}

template<typename T>
static std::vector<T> read_structs(const std::string& path) {
    auto data = read_file(path);
    size_t count = data.size() / sizeof(T);
    std::vector<T> result(count);
    memcpy(result.data(), data.data(), count * sizeof(T));
    return result;
}

// --- Decoded representations for comparison ---

struct DecodedAdminPolygon {
    std::string name;
    uint8_t admin_level;
    float area;
    uint16_t country_code;
    std::vector<std::pair<float, float>> vertices;

    // Canonical sort key: name + admin_level + vertex_count
    // This ensures consistent ordering regardless of insertion order
    std::string sort_key() const {
        return name + "|" + std::to_string(admin_level) + "|" + std::to_string(vertices.size());
    }

    bool operator<(const DecodedAdminPolygon& o) const {
        return sort_key() < o.sort_key();
    }

    // Compare vertex rings as sets — same vertices regardless of order/rotation/direction
    static bool rings_equal(const std::vector<std::pair<float,float>>& a,
                            const std::vector<std::pair<float,float>>& b) {
        if (a.size() != b.size()) return false;
        if (a.empty()) return true;

        // Round to grid and compare as sorted sets
        auto to_key = [](const std::pair<float,float>& v) -> std::pair<int,int> {
            return {(int)(v.first * 1e5f + 0.5f), (int)(v.second * 1e5f + 0.5f)};
        };

        std::vector<std::pair<int,int>> sa, sb;
        for (auto& v : a) sa.push_back(to_key(v));
        for (auto& v : b) sb.push_back(to_key(v));
        std::sort(sa.begin(), sa.end());
        std::sort(sb.begin(), sb.end());
        return sa == sb;
    }

    bool operator==(const DecodedAdminPolygon& o) const {
        if (admin_level != o.admin_level) return false;
        if (name != o.name) return false;
        if (country_code != o.country_code) return false;
        if (vertices.size() != o.vertices.size()) return false;
        // Area tolerance: relative comparison for large values, absolute for small
        float area_diff = std::fabs(area - o.area);
        float area_max = std::max(std::fabs(area), std::fabs(o.area));
        if (area_max > 1e-6f && area_diff / area_max > 0.01f) return false;
        if (area_max <= 1e-6f && area_diff > 1e-8f) return false;
        return rings_equal(vertices, o.vertices);
    }
};

struct DecodedWay {
    std::string name;
    std::vector<std::pair<float, float>> nodes;

    std::string sort_key() const {
        std::string key = name + "|" + std::to_string(nodes.size());
        if (!nodes.empty()) {
            key += "|" + std::to_string((int)(nodes[0].first * 1e5)) + "," + std::to_string((int)(nodes[0].second * 1e5));
        }
        return key;
    }

    bool operator<(const DecodedWay& o) const {
        return sort_key() < o.sort_key();
    }

    bool operator==(const DecodedWay& o) const {
        if (name != o.name) return false;
        if (nodes.size() != o.nodes.size()) return false;
        // Compare as sorted vertex sets (order may differ due to parallel processing)
        auto to_key = [](const std::pair<float,float>& v) -> std::pair<int,int> {
            return {(int)(v.first * 1e5f + 0.5f), (int)(v.second * 1e5f + 0.5f)};
        };
        std::vector<std::pair<int,int>> sa, sb;
        for (auto& v : nodes) sa.push_back(to_key(v));
        for (auto& v : o.nodes) sb.push_back(to_key(v));
        std::sort(sa.begin(), sa.end());
        std::sort(sb.begin(), sb.end());
        return sa == sb;
    }
};

struct DecodedAddr {
    float lat, lng;
    std::string housenumber;
    std::string street;

    std::string sort_key() const {
        return std::to_string((int)(lat * 1e5)) + "," + std::to_string((int)(lng * 1e5)) + "|" + housenumber + "|" + street;
    }

    bool operator<(const DecodedAddr& o) const {
        return sort_key() < o.sort_key();
    }

    bool operator==(const DecodedAddr& o) const {
        return std::fabs(lat - o.lat) < 1e-5f &&
               std::fabs(lng - o.lng) < 1e-5f &&
               housenumber == o.housenumber &&
               street == o.street;
    }
};

// --- Decode functions ---

static std::vector<DecodedAdminPolygon> decode_admin(const std::string& dir) {
    auto strings = read_file(dir + "/strings.bin");
    auto polygons = read_structs<AdminPolygon>(dir + "/admin_polygons.bin");
    auto vertices = read_structs<NodeCoord>(dir + "/admin_vertices.bin");

    std::vector<DecodedAdminPolygon> result;
    for (const auto& p : polygons) {
        DecodedAdminPolygon dp;
        dp.name = get_string(strings, p.name_id);
        dp.admin_level = p.admin_level;
        dp.area = p.area;
        dp.country_code = p.country_code;
        for (uint16_t i = 0; i < p.vertex_count; i++) {
            uint32_t vi = p.vertex_offset + i;
            if (vi < vertices.size()) {
                dp.vertices.push_back({vertices[vi].lat, vertices[vi].lng});
            }
        }
        result.push_back(std::move(dp));
    }
    return result;
}

static std::vector<DecodedWay> decode_ways(const std::string& dir) {
    auto strings = read_file(dir + "/strings.bin");
    auto ways = read_structs<WayHeader>(dir + "/street_ways.bin");
    auto nodes = read_structs<NodeCoord>(dir + "/street_nodes.bin");

    std::vector<DecodedWay> result;
    for (const auto& w : ways) {
        DecodedWay dw;
        dw.name = get_string(strings, w.name_id);
        for (uint8_t i = 0; i < w.node_count; i++) {
            uint32_t ni = w.node_offset + i;
            if (ni < nodes.size()) {
                dw.nodes.push_back({nodes[ni].lat, nodes[ni].lng});
            }
        }
        result.push_back(std::move(dw));
    }
    return result;
}

static std::vector<DecodedAddr> decode_addrs(const std::string& dir) {
    auto strings = read_file(dir + "/strings.bin");
    auto points = read_structs<AddrPoint>(dir + "/addr_points.bin");

    std::vector<DecodedAddr> result;
    for (const auto& p : points) {
        DecodedAddr da;
        da.lat = p.lat;
        da.lng = p.lng;
        da.housenumber = get_string(strings, p.housenumber_id);
        da.street = get_string(strings, p.street_id);
        result.push_back(std::move(da));
    }
    return result;
}

// --- Comparison ---

template<typename T>
static bool compare_sorted(const char* label, std::vector<T> a, std::vector<T> b) {
    std::sort(a.begin(), a.end());
    std::sort(b.begin(), b.end());

    if (a.size() != b.size()) {
        std::cerr << "FAIL: " << label << " count differs (A=" << a.size()
                  << " B=" << b.size() << ")" << std::endl;
        return false;
    }

    size_t diffs = 0;
    for (size_t i = 0; i < a.size(); i++) {
        if (!(a[i] == b[i])) {
            if (diffs < 3) {
                std::cerr << "  Mismatch at sorted position " << i << std::endl;
            }
            diffs++;
        }
    }

    if (diffs > 0) {
        std::cerr << "FAIL: " << label << " has " << diffs << " differences out of "
                  << a.size() << " items" << std::endl;
        return false;
    }

    std::cout << "PASS: " << label << " (" << a.size() << " items)" << std::endl;
    return true;
}

int main(int argc, char* argv[]) {
    if (argc != 3) {
        std::cerr << "Usage: compare_indexes <dir-A> <dir-B>" << std::endl;
        return 2;
    }

    std::string dir_a = argv[1];
    std::string dir_b = argv[2];

    std::cout << "Comparing:" << std::endl;
    std::cout << "  A: " << dir_a << std::endl;
    std::cout << "  B: " << dir_b << std::endl;
    std::cout << std::endl;

    bool all_pass = true;

    // Compare admin polygons
    {
        auto a = decode_admin(dir_a);
        auto b = decode_admin(dir_b);
        if (!compare_sorted("admin_polygons", std::move(a), std::move(b)))
            all_pass = false;
    }

    // Compare street ways
    {
        auto a = decode_ways(dir_a);
        auto b = decode_ways(dir_b);
        if (!compare_sorted("street_ways", std::move(a), std::move(b)))
            all_pass = false;
    }

    // Compare address points
    {
        auto a = decode_addrs(dir_a);
        auto b = decode_addrs(dir_b);
        if (!compare_sorted("address_points", std::move(a), std::move(b)))
            all_pass = false;
    }

    // Compare cell index sizes (should be identical regardless of ordering)
    {
        auto ac = read_file(dir_a + "/admin_cells.bin");
        auto bc = read_file(dir_b + "/admin_cells.bin");
        if (ac == bc) {
            std::cout << "PASS: admin_cells.bin byte-identical (" << ac.size() << " bytes)" << std::endl;
        } else if (ac.size() == bc.size()) {
            std::cout << "INFO: admin_cells.bin same size (" << ac.size() << " bytes) but different content" << std::endl;
        } else {
            std::cerr << "FAIL: admin_cells.bin size differs (A=" << ac.size()
                      << " B=" << bc.size() << ")" << std::endl;
            all_pass = false;
        }

        auto gc_a = read_file(dir_a + "/geo_cells.bin");
        auto gc_b = read_file(dir_b + "/geo_cells.bin");
        if (gc_a.size() == gc_b.size()) {
            std::cout << "PASS: geo_cells.bin same size (" << gc_a.size() << " bytes)" << std::endl;
        } else {
            std::cerr << "FAIL: geo_cells.bin size differs (A=" << gc_a.size()
                      << " B=" << gc_b.size() << ")" << std::endl;
            all_pass = false;
        }
    }

    std::cout << std::endl;
    if (all_pass) {
        std::cout << "=== ALL COMPARISONS PASSED ===" << std::endl;
        return 0;
    } else {
        std::cout << "=== SOME COMPARISONS FAILED ===" << std::endl;
        return 1;
    }
}


Now let me run the full semantic comparison, and also look at the ring stitcher code to understand what could cause the remaining 55 missing polygons.


sshpass -p 'elon-is-insane' ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ProxyCommand="tailscale nc %h %p" michtest@100.113.88.121 "/tmp/compare_indexes /tmp/de-parallel /tmp/de-sequential 2>&1" 2>&1

Run full semantic comparison of parallel vs sequential Germany builds


Command running in background with ID: bm64ylbx8. Output is being written to: /tmp/claude-0/-root-traccar-geocoder/af1c4baa-0f9a-4420-a8b3-ce20dc28c8b9/tasks/bm64ylbx8.output


Check semantic comparison output


Warning: Permanently added '100.113.88.121' (ED25519) to the list of known hosts.
Comparing:
  A: /tmp/de-parallel
  B: /tmp/de-sequential

FAIL: admin_polygons count differs (A=43178 B=43221)
  Mismatch at sorted position 0
  Mismatch at sorted position 1
  Mismatch at sorted position 2
FAIL: street_ways has 3312660 differences out of 3312673 items


The semantic comparison is still running (street_ways). But the key finding is clear — the `street_ways` comparison is failing massively because it's comparing by sorted position after sorting by sort key, but parallel processing changes the node ordering within ways. The current comparison is too strict.

Let me look at the ring stitcher source and then update the comparison tool. First, let me understand the current codebase.


Check if comparison finished


Still running. While that completes, let me look at the ring stitcher code to understand the remaining 55 missing polygons, and also plan the comparison tool improvements.


/root/traccar-geocoder/builder/src/build_index.cpp
/root/traccar-geocoder/builder/test/compare_indexes.cpp


No files found


466:// --- Parallel admin: assemble outer rings from collected way geometries ---
468:static std::vector<std::vector<std::pair<double,double>>> assemble_outer_rings(
2290:                                auto rings = assemble_outer_rings(rel.members, data.way_geometries);


   466→// --- Parallel admin: assemble outer rings from collected way geometries ---
   467→
   468→static std::vector<std::vector<std::pair<double,double>>> assemble_outer_rings(
   469→    const std::vector<std::pair<int64_t, std::string>>& members,
   470→    const std::unordered_map<int64_t, ParsedData::WayGeometry>& way_geoms)
   471→{
   472→    // Match way endpoints by OSM node ID (exact) rather than coordinates (lossy)
   473→    struct WayRef {
   474→        int64_t way_id;
   475→        const ParsedData::WayGeometry* geom;
   476→        bool used;
   477→    };
   478→    std::vector<WayRef> outer_ways;
   479→    for (const auto& [way_id, role] : members) {
   480→        if (role == "outer" || role.empty()) {
   481→            auto it = way_geoms.find(way_id);
   482→            if (it != way_geoms.end() && !it->second.coords.empty()) {
   483→                outer_ways.push_back({way_id, &it->second, false});
   484→            }
   485→        }
   486→    }
   487→
   488→    std::vector<std::vector<std::pair<double,double>>> rings;
   489→
   490→    for (size_t start_idx = 0; start_idx < outer_ways.size(); start_idx++) {
   491→        if (outer_ways[start_idx].used) continue;
   492→
   493→        outer_ways[start_idx].used = true;
   494→        std::vector<std::pair<double,double>> ring = outer_ways[start_idx].geom->coords;
   495→
   496→        // Track the current last node ID of the ring
   497→        int64_t ring_first_nid = outer_ways[start_idx].geom->first_node_id;
   498→        int64_t ring_last_nid = outer_ways[start_idx].geom->last_node_id;
   499→
   500→        bool extended = true;
   501→        while (extended) {
   502→            extended = false;
   503→
   504→            // Check if ring is closed (first node ID == last node ID)
   505→            if (ring.size() >= 4 && ring_first_nid == ring_last_nid) {
   506→                break;
   507→            }
   508→
   509→            for (size_t ci = 0; ci < outer_ways.size(); ci++) {
   510→                if (outer_ways[ci].used) continue;
   511→                const auto* cand = outer_ways[ci].geom;
   512→                if (cand->coords.empty()) continue;
   513→
   514→                // Try forward: candidate's first node matches ring's last node
   515→                if (cand->first_node_id == ring_last_nid) {
   516→                    outer_ways[ci].used = true;
   517→                    ring.insert(ring.end(), cand->coords.begin() + 1, cand->coords.end());
   518→                    ring_last_nid = cand->last_node_id;
   519→                    extended = true;
   520→                    break;
   521→                }
   522→
   523→                // Try reversed: candidate's last node matches ring's last node
   524→                if (cand->last_node_id == ring_last_nid) {
   525→                    outer_ways[ci].used = true;
   526→                    for (auto rit = cand->coords.rbegin() + 1; rit != cand->coords.rend(); ++rit) {
   527→                        ring.push_back(*rit);
   528→                    }
   529→                    ring_last_nid = cand->first_node_id;
   530→                    extended = true;
   531→                    break;
   532→                }
   533→            }
   534→        }
   535→
   536→        // Only keep closed rings
   537→        if (ring.size() >= 4 && ring_first_nid == ring_last_nid) {
   538→            rings.push_back(std::move(ring));
   539→        }
   540→    }
   541→
   542→    return rings;
   543→}
   544→
   545→// --- Highway filter ---
   546→
   547→static const std::vector<std::string> kExcludedHighways = {
   548→    "footway", "path", "track", "steps", "cycleway",
   549→    "service", "pedestrian", "bridleway", "construction"
   550→};
   551→
   552→static bool is_included_highway(const char* value) {
   553→    for (const auto& excluded : kExcludedHighways) {
   554→        if (excluded == value) return false;
   555→    }
   556→    return true;
   557→}
   558→
   559→// --- Parse house number (leading digits) ---
   560→
   561→static uint32_t parse_house_number(const char* s) {
   562→    if (!s) return 0;
   563→    uint32_t n = 0;
   564→    while (*s >= '0' && *s <= '9') {
   565→        n = n * 10 + (*s - '0');
   566→        s++;
   567→    }
   568→    return n;
   569→}
   570→
   571→// --- Add an address point ---
   572→
   573→static void add_addr_point(ParsedData& data, double lat, double lng,
   574→                           const char* housenumber, const char* street,
   575→                           uint64_t& addr_count_total) {
   576→    uint32_t addr_id = static_cast<uint32_t>(data.addr_points.size());
   577→    data.addr_points.push_back({
   578→        static_cast<float>(lat),
   579→        static_cast<float>(lng),
   580→        data.string_pool.intern(housenumber),
   581→        data.string_pool.intern(street)
   582→    });
   583→
   584→    S2CellId cell = point_to_cell(lat, lng);
   585→    data.cell_to_addrs[cell.id()].push_back(addr_id);

// --- Parallel admin: assemble outer rings from collected way geometries ---

static std::vector<std::vector<std::pair<double,double>>> assemble_outer_rings(
    const std::vector<std::pair<int64_t, std::string>>& members,
    const std::unordered_map<int64_t, ParsedData::WayGeometry>& way_geoms)
{
    // Match way endpoints by OSM node ID (exact) rather than coordinates (lossy)
    struct WayRef {
        int64_t way_id;
        const ParsedData::WayGeometry* geom;
        bool used;
    };
    std::vector<WayRef> outer_ways;
    for (const auto& [way_id, role] : members) {
        if (role == "outer" || role.empty()) {
            auto it = way_geoms.find(way_id);
            if (it != way_geoms.end() && !it->second.coords.empty()) {
                outer_ways.push_back({way_id, &it->second, false});
            }
        }
    }

    std::vector<std::vector<std::pair<double,double>>> rings;

    for (size_t start_idx = 0; start_idx < outer_ways.size(); start_idx++) {
        if (outer_ways[start_idx].used) continue;

        outer_ways[start_idx].used = true;
        std::vector<std::pair<double,double>> ring = outer_ways[start_idx].geom->coords;

        // Track the current last node ID of the ring
        int64_t ring_first_nid = outer_ways[start_idx].geom->first_node_id;
        int64_t ring_last_nid = outer_ways[start_idx].geom->last_node_id;

        bool extended = true;
        while (extended) {
            extended = false;

            // Check if ring is closed (first node ID == last node ID)
            if (ring.size() >= 4 && ring_first_nid == ring_last_nid) {
                break;
            }

            for (size_t ci = 0; ci < outer_ways.size(); ci++) {
                if (outer_ways[ci].used) continue;
                const auto* cand = outer_ways[ci].geom;
                if (cand->coords.empty()) continue;

                // Try forward: candidate's first node matches ring's last node
                if (cand->first_node_id == ring_last_nid) {
                    outer_ways[ci].used = true;
                    ring.insert(ring.end(), cand->coords.begin() + 1, cand->coords.end());
                    ring_last_nid = cand->last_node_id;
                    extended = true;
                    break;
                }

                // Try reversed: candidate's last node matches ring's last node
                if (cand->last_node_id == ring_last_nid) {
                    outer_ways[ci].used = true;
                    for (auto rit = cand->coords.rbegin() + 1; rit != cand->coords.rend(); ++rit) {
                        ring.push_back(*rit);
                    }
                    ring_last_nid = cand->first_node_id;
                    extended = true;
                    break;
                }
            }
        }

        // Only keep closed rings
        if (ring.size() >= 4 && ring_first_nid == ring_last_nid) {
            rings.push_back(std::move(ring));
        }
    }

    return rings;
}

// --- Highway filter ---

static const std::vector<std::string> kExcludedHighways = {
    "footway", "path", "track", "steps", "cycleway",
    "service", "pedestrian", "bridleway", "construction"
};

static bool is_included_highway(const char* value) {
    for (const auto& excluded : kExcludedHighways) {
        if (excluded == value) return false;
    }
    return true;
}

// --- Parse house number (leading digits) ---

static uint32_t parse_house_number(const char* s) {
    if (!s) return 0;
    uint32_t n = 0;
    while (*s >= '0' && *s <= '9') {
        n = n * 10 + (*s - '0');
        s++;
    }
    return n;
}

// --- Add an address point ---

static void add_addr_point(ParsedData& data, double lat, double lng,
                           const char* housenumber, const char* street,
                           uint64_t& addr_count_total) {
    uint32_t addr_id = static_cast<uint32_t>(data.addr_points.size());
    data.addr_points.push_back({
        static_cast<float>(lat),
        static_cast<float>(lng),
        data.string_pool.intern(housenumber),
        data.string_pool.intern(street)
    });

    S2CellId cell = point_to_cell(lat, lng);
    data.cell_to_addrs[cell.id()].push_back(addr_id);


Let me look at the full comparison tool in the repo and update it to handle order-independent comparison properly.


     1→// Semantic comparison of two geocoder index directories.
     2→// Decodes all binary files, normalizes ordering, and compares content.
     3→// Returns 0 if semantically identical, 1 if different.
     4→//
     5→// Usage: compare_indexes <dir-A> <dir-B>
     6→
     7→#include <algorithm>
     8→#include <cmath>
     9→#include <cstdint>
    10→#include <cstring>
    11→#include <fstream>
    12→#include <iostream>
    13→#include <map>
    14→#include <set>
    15→#include <string>
    16→#include <vector>
    17→
    18→// --- Binary structs (must match build_index.cpp and server main.rs) ---
    19→
    20→struct WayHeader {
    21→    uint32_t node_offset;
    22→    uint8_t node_count;
    23→    uint32_t name_id;
    24→};
    25→
    26→struct AddrPoint {
    27→    float lat;
    28→    float lng;
    29→    uint32_t housenumber_id;
    30→    uint32_t street_id;
    31→};
    32→
    33→struct InterpWay {
    34→    uint32_t node_offset;
    35→    uint8_t node_count;
    36→    uint32_t street_id;
    37→    uint32_t start_number;
    38→    uint32_t end_number;
    39→    uint8_t interpolation;
    40→};
    41→
    42→struct AdminPolygon {
    43→    uint32_t vertex_offset;
    44→    uint16_t vertex_count;
    45→    uint32_t name_id;
    46→    uint8_t admin_level;
    47→    float area;
    48→    uint16_t country_code;
    49→};
    50→
    51→struct NodeCoord {
    52→    float lat;
    53→    float lng;
    54→};
    55→
    56→// --- Helpers ---
    57→
    58→static std::vector<char> read_file(const std::string& path) {
    59→    std::ifstream f(path, std::ios::binary | std::ios::ate);
    60→    if (!f) return {};
    61→    auto size = f.tellg();
    62→    f.seekg(0);
    63→    std::vector<char> data(size);
    64→    f.read(data.data(), size);
    65→    return data;
    66→}
    67→
    68→static const char* get_string(const std::vector<char>& pool, uint32_t offset) {
    69→    if (offset >= pool.size()) return "(invalid)";
    70→    return pool.data() + offset;
    71→}
    72→
    73→template<typename T>
    74→static std::vector<T> read_structs(const std::string& path) {
    75→    auto data = read_file(path);
    76→    size_t count = data.size() / sizeof(T);
    77→    std::vector<T> result(count);
    78→    memcpy(result.data(), data.data(), count * sizeof(T));
    79→    return result;
    80→}
    81→
    82→// --- Decoded representations for comparison ---
    83→
    84→struct DecodedAdminPolygon {
    85→    std::string name;
    86→    uint8_t admin_level;
    87→    float area;
    88→    uint16_t country_code;
    89→    std::vector<std::pair<float, float>> vertices;
    90→
    91→    // Canonical sort key: name + admin_level + vertex_count
    92→    // This ensures consistent ordering regardless of insertion order
    93→    std::string sort_key() const {
    94→        return name + "|" + std::to_string(admin_level) + "|" + std::to_string(vertices.size());
    95→    }
    96→
    97→    bool operator<(const DecodedAdminPolygon& o) const {
    98→        return sort_key() < o.sort_key();
    99→    }
   100→
   101→    // Compare vertex rings as sets — same vertices regardless of order/rotation/direction
   102→    static bool rings_equal(const std::vector<std::pair<float,float>>& a,
   103→                            const std::vector<std::pair<float,float>>& b) {
   104→        if (a.size() != b.size()) return false;
   105→        if (a.empty()) return true;
   106→
   107→        // Round to grid and compare as sorted sets
   108→        auto to_key = [](const std::pair<float,float>& v) -> std::pair<int,int> {
   109→            return {(int)(v.first * 1e5f + 0.5f), (int)(v.second * 1e5f + 0.5f)};
   110→        };
   111→
   112→        std::vector<std::pair<int,int>> sa, sb;
   113→        for (auto& v : a) sa.push_back(to_key(v));
   114→        for (auto& v : b) sb.push_back(to_key(v));
   115→        std::sort(sa.begin(), sa.end());
   116→        std::sort(sb.begin(), sb.end());
   117→        return sa == sb;
   118→    }
   119→
   120→    bool operator==(const DecodedAdminPolygon& o) const {
   121→        if (admin_level != o.admin_level) return false;
   122→        if (name != o.name) return false;
   123→        if (country_code != o.country_code) return false;
   124→        if (vertices.size() != o.vertices.size()) return false;
   125→        // Area tolerance: relative comparison for large values, absolute for small
   126→        float area_diff = std::fabs(area - o.area);
   127→        float area_max = std::max(std::fabs(area), std::fabs(o.area));
   128→        if (area_max > 1e-6f && area_diff / area_max > 0.01f) return false;
   129→        if (area_max <= 1e-6f && area_diff > 1e-8f) return false;
   130→        return rings_equal(vertices, o.vertices);
   131→    }
   132→};
   133→
   134→struct DecodedWay {
   135→    std::string name;
   136→    std::vector<std::pair<float, float>> nodes;
   137→
   138→    std::string sort_key() const {
   139→        std::string key = name + "|" + std::to_string(nodes.size());
   140→        if (!nodes.empty()) {
   141→            key += "|" + std::to_string((int)(nodes[0].first * 1e5)) + "," + std::to_string((int)(nodes[0].second * 1e5));
   142→        }
   143→        return key;
   144→    }
   145→
   146→    bool operator<(const DecodedWay& o) const {
   147→        return sort_key() < o.sort_key();
   148→    }
   149→
   150→    bool operator==(const DecodedWay& o) const {
   151→        if (name != o.name) return false;
   152→        if (nodes.size() != o.nodes.size()) return false;
   153→        // Compare as sorted vertex sets (order may differ due to parallel processing)
   154→        auto to_key = [](const std::pair<float,float>& v) -> std::pair<int,int> {
   155→            return {(int)(v.first * 1e5f + 0.5f), (int)(v.second * 1e5f + 0.5f)};
   156→        };
   157→        std::vector<std::pair<int,int>> sa, sb;
   158→        for (auto& v : nodes) sa.push_back(to_key(v));
   159→        for (auto& v : o.nodes) sb.push_back(to_key(v));
   160→        std::sort(sa.begin(), sa.end());
   161→        std::sort(sb.begin(), sb.end());
   162→        return sa == sb;
   163→    }
   164→};
   165→
   166→struct DecodedAddr {
   167→    float lat, lng;
   168→    std::string housenumber;
   169→    std::string street;
   170→
   171→    std::string sort_key() const {
   172→        return std::to_string((int)(lat * 1e5)) + "," + std::to_string((int)(lng * 1e5)) + "|" + housenumber + "|" + street;
   173→    }
   174→
   175→    bool operator<(const DecodedAddr& o) const {
   176→        return sort_key() < o.sort_key();
   177→    }
   178→
   179→    bool operator==(const DecodedAddr& o) const {
   180→        return std::fabs(lat - o.lat) < 1e-5f &&
   181→               std::fabs(lng - o.lng) < 1e-5f &&
   182→               housenumber == o.housenumber &&
   183→               street == o.street;
   184→    }
   185→};
   186→
   187→// --- Decode functions ---
   188→
   189→static std::vector<DecodedAdminPolygon> decode_admin(const std::string& dir) {
   190→    auto strings = read_file(dir + "/strings.bin");
   191→    auto polygons = read_structs<AdminPolygon>(dir + "/admin_polygons.bin");
   192→    auto vertices = read_structs<NodeCoord>(dir + "/admin_vertices.bin");
   193→
   194→    std::vector<DecodedAdminPolygon> result;
   195→    for (const auto& p : polygons) {
   196→        DecodedAdminPolygon dp;
   197→        dp.name = get_string(strings, p.name_id);
   198→        dp.admin_level = p.admin_level;
   199→        dp.area = p.area;
   200→        dp.country_code = p.country_code;
   201→        for (uint16_t i = 0; i < p.vertex_count; i++) {
   202→            uint32_t vi = p.vertex_offset + i;
   203→            if (vi < vertices.size()) {
   204→                dp.vertices.push_back({vertices[vi].lat, vertices[vi].lng});
   205→            }
   206→        }
   207→        result.push_back(std::move(dp));
   208→    }
   209→    return result;
   210→}
   211→
   212→static std::vector<DecodedWay> decode_ways(const std::string& dir) {
   213→    auto strings = read_file(dir + "/strings.bin");
   214→    auto ways = read_structs<WayHeader>(dir + "/street_ways.bin");
   215→    auto nodes = read_structs<NodeCoord>(dir + "/street_nodes.bin");
   216→
   217→    std::vector<DecodedWay> result;
   218→    for (const auto& w : ways) {
   219→        DecodedWay dw;
   220→        dw.name = get_string(strings, w.name_id);
   221→        for (uint8_t i = 0; i < w.node_count; i++) {
   222→            uint32_t ni = w.node_offset + i;
   223→            if (ni < nodes.size()) {
   224→                dw.nodes.push_back({nodes[ni].lat, nodes[ni].lng});
   225→            }
   226→        }
   227→        result.push_back(std::move(dw));
   228→    }
   229→    return result;
   230→}
   231→
   232→static std::vector<DecodedAddr> decode_addrs(const std::string& dir) {
   233→    auto strings = read_file(dir + "/strings.bin");
   234→    auto points = read_structs<AddrPoint>(dir + "/addr_points.bin");
   235→
   236→    std::vector<DecodedAddr> result;
   237→    for (const auto& p : points) {
   238→        DecodedAddr da;
   239→        da.lat = p.lat;
   240→        da.lng = p.lng;
   241→        da.housenumber = get_string(strings, p.housenumber_id);
   242→        da.street = get_string(strings, p.street_id);
   243→        result.push_back(std::move(da));
   244→    }
   245→    return result;
   246→}
   247→
   248→// --- Comparison ---
   249→
   250→template<typename T>
   251→static bool compare_sorted(const char* label, std::vector<T> a, std::vector<T> b) {
   252→    std::sort(a.begin(), a.end());
   253→    std::sort(b.begin(), b.end());
   254→
   255→    if (a.size() != b.size()) {
   256→        std::cerr << "FAIL: " << label << " count differs (A=" << a.size()
   257→                  << " B=" << b.size() << ")" << std::endl;
   258→        return false;
   259→    }
   260→
   261→    size_t diffs = 0;
   262→    for (size_t i = 0; i < a.size(); i++) {
   263→        if (!(a[i] == b[i])) {
   264→            if (diffs < 3) {
   265→                std::cerr << "  Mismatch at sorted position " << i << std::endl;
   266→            }
   267→            diffs++;
   268→        }
   269→    }
   270→
   271→    if (diffs > 0) {
   272→        std::cerr << "FAIL: " << label << " has " << diffs << " differences out of "
   273→                  << a.size() << " items" << std::endl;
   274→        return false;
   275→    }
   276→
   277→    std::cout << "PASS: " << label << " (" << a.size() << " items)" << std::endl;
   278→    return true;
   279→}
   280→
   281→int main(int argc, char* argv[]) {
   282→    if (argc != 3) {
   283→        std::cerr << "Usage: compare_indexes <dir-A> <dir-B>" << std::endl;
   284→        return 2;
   285→    }
   286→
   287→    std::string dir_a = argv[1];
   288→    std::string dir_b = argv[2];
   289→
   290→    std::cout << "Comparing:" << std::endl;
   291→    std::cout << "  A: " << dir_a << std::endl;
   292→    std::cout << "  B: " << dir_b << std::endl;
   293→    std::cout << std::endl;
   294→
   295→    bool all_pass = true;
   296→
   297→    // Compare admin polygons
   298→    {
   299→        auto a = decode_admin(dir_a);
   300→        auto b = decode_admin(dir_b);
   301→        if (!compare_sorted("admin_polygons", std::move(a), std::move(b)))
   302→            all_pass = false;
   303→    }
   304→
   305→    // Compare street ways
   306→    {
   307→        auto a = decode_ways(dir_a);
   308→        auto b = decode_ways(dir_b);
   309→        if (!compare_sorted("street_ways", std::move(a), std::move(b)))
   310→            all_pass = false;
   311→    }
   312→
   313→    // Compare address points
   314→    {
   315→        auto a = decode_addrs(dir_a);
   316→        auto b = decode_addrs(dir_b);
   317→        if (!compare_sorted("address_points", std::move(a), std::move(b)))
   318→            all_pass = false;
   319→    }
   320→
   321→    // Compare cell index sizes (should be identical regardless of ordering)
   322→    {
   323→        auto ac = read_file(dir_a + "/admin_cells.bin");
   324→        auto bc = read_file(dir_b + "/admin_cells.bin");
   325→        if (ac == bc) {
   326→            std::cout << "PASS: admin_cells.bin byte-identical (" << ac.size() << " bytes)" << std::endl;
   327→        } else if (ac.size() == bc.size()) {
   328→            std::cout << "INFO: admin_cells.bin same size (" << ac.size() << " bytes) but different content" << std::endl;
   329→        } else {
   330→            std::cerr << "FAIL: admin_cells.bin size differs (A=" << ac.size()
   331→                      << " B=" << bc.size() << ")" << std::endl;
   332→            all_pass = false;
   333→        }
   334→
   335→        auto gc_a = read_file(dir_a + "/geo_cells.bin");
   336→        auto gc_b = read_file(dir_b + "/geo_cells.bin");
   337→        if (gc_a.size() == gc_b.size()) {
   338→            std::cout << "PASS: geo_cells.bin same size (" << gc_a.size() << " bytes)" << std::endl;
   339→        } else {
   340→            std::cerr << "FAIL: geo_cells.bin size differs (A=" << gc_a.size()
   341→                      << " B=" << gc_b.size() << ")" << std::endl;
   342→            all_pass = false;
   343→        }
   344→    }
   345→
   346→    std::cout << std::endl;
   347→    if (all_pass) {
   348→        std::cout << "=== ALL COMPARISONS PASSED ===" << std::endl;
   349→        return 0;
   350→    } else {
   351→        std::cout << "=== SOME COMPARISONS FAILED ===" << std::endl;
   352→        return 1;
   353→    }
   354→}
   355→

// Semantic comparison of two geocoder index directories.
// Decodes all binary files, normalizes ordering, and compares content.
// Returns 0 if semantically identical, 1 if different.
//
// Usage: compare_indexes <dir-A> <dir-B>

#include <algorithm>
#include <cmath>
#include <cstdint>
#include <cstring>
#include <fstream>
#include <iostream>
#include <map>
#include <set>
#include <string>
#include <vector>

// --- Binary structs (must match build_index.cpp and server main.rs) ---

struct WayHeader {
    uint32_t node_offset;
    uint8_t node_count;
    uint32_t name_id;
};

struct AddrPoint {
    float lat;
    float lng;
    uint32_t housenumber_id;
    uint32_t street_id;
};

struct InterpWay {
    uint32_t node_offset;
    uint8_t node_count;
    uint32_t street_id;
    uint32_t start_number;
    uint32_t end_number;
    uint8_t interpolation;
};

struct AdminPolygon {
    uint32_t vertex_offset;
    uint16_t vertex_count;
    uint32_t name_id;
    uint8_t admin_level;
    float area;
    uint16_t country_code;
};

struct NodeCoord {
    float lat;
    float lng;
};

// --- Helpers ---

static std::vector<char> read_file(const std::string& path) {
    std::ifstream f(path, std::ios::binary | std::ios::ate);
    if (!f) return {};
    auto size = f.tellg();
    f.seekg(0);
    std::vector<char> data(size);
    f.read(data.data(), size);
    return data;
}

static const char* get_string(const std::vector<char>& pool, uint32_t offset) {
    if (offset >= pool.size()) return "(invalid)";
    return pool.data() + offset;
}

template<typename T>
static std::vector<T> read_structs(const std::string& path) {
    auto data = read_file(path);
    size_t count = data.size() / sizeof(T);
    std::vector<T> result(count);
    memcpy(result.data(), data.data(), count * sizeof(T));
    return result;
}

// --- Decoded representations for comparison ---

struct DecodedAdminPolygon {
    std::string name;
    uint8_t admin_level;
    float area;
    uint16_t country_code;
    std::vector<std::pair<float, float>> vertices;

    // Canonical sort key: name + admin_level + vertex_count
    // This ensures consistent ordering regardless of insertion order
    std::string sort_key() const {
        return name + "|" + std::to_string(admin_level) + "|" + std::to_string(vertices.size());
    }

    bool operator<(const DecodedAdminPolygon& o) const {
        return sort_key() < o.sort_key();
    }

    // Compare vertex rings as sets — same vertices regardless of order/rotation/direction
    static bool rings_equal(const std::vector<std::pair<float,float>>& a,
                            const std::vector<std::pair<float,float>>& b) {
        if (a.size() != b.size()) return false;
        if (a.empty()) return true;

        // Round to grid and compare as sorted sets
        auto to_key = [](const std::pair<float,float>& v) -> std::pair<int,int> {
            return {(int)(v.first * 1e5f + 0.5f), (int)(v.second * 1e5f + 0.5f)};
        };

        std::vector<std::pair<int,int>> sa, sb;
        for (auto& v : a) sa.push_back(to_key(v));
        for (auto& v : b) sb.push_back(to_key(v));
        std::sort(sa.begin(), sa.end());
        std::sort(sb.begin(), sb.end());
        return sa == sb;
    }

    bool operator==(const DecodedAdminPolygon& o) const {
        if (admin_level != o.admin_level) return false;
        if (name != o.name) return false;
        if (country_code != o.country_code) return false;
        if (vertices.size() != o.vertices.size()) return false;
        // Area tolerance: relative comparison for large values, absolute for small
        float area_diff = std::fabs(area - o.area);
        float area_max = std::max(std::fabs(area), std::fabs(o.area));
        if (area_max > 1e-6f && area_diff / area_max > 0.01f) return false;
        if (area_max <= 1e-6f && area_diff > 1e-8f) return false;
        return rings_equal(vertices, o.vertices);
    }
};

struct DecodedWay {
    std::string name;
    std::vector<std::pair<float, float>> nodes;

    std::string sort_key() const {
        std::string key = name + "|" + std::to_string(nodes.size());
        if (!nodes.empty()) {
            key += "|" + std::to_string((int)(nodes[0].first * 1e5)) + "," + std::to_string((int)(nodes[0].second * 1e5));
        }
        return key;
    }

    bool operator<(const DecodedWay& o) const {
        return sort_key() < o.sort_key();
    }

    bool operator==(const DecodedWay& o) const {
        if (name != o.name) return false;
        if (nodes.size() != o.nodes.size()) return false;
        // Compare as sorted vertex sets (order may differ due to parallel processing)
        auto to_key = [](const std::pair<float,float>& v) -> std::pair<int,int> {
            return {(int)(v.first * 1e5f + 0.5f), (int)(v.second * 1e5f + 0.5f)};
        };
        std::vector<std::pair<int,int>> sa, sb;
        for (auto& v : nodes) sa.push_back(to_key(v));
        for (auto& v : o.nodes) sb.push_back(to_key(v));
        std::sort(sa.begin(), sa.end());
        std::sort(sb.begin(), sb.end());
        return sa == sb;
    }
};

struct DecodedAddr {
    float lat, lng;
    std::string housenumber;
    std::string street;

    std::string sort_key() const {
        return std::to_string((int)(lat * 1e5)) + "," + std::to_string((int)(lng * 1e5)) + "|" + housenumber + "|" + street;
    }

    bool operator<(const DecodedAddr& o) const {
        return sort_key() < o.sort_key();
    }

    bool operator==(const DecodedAddr& o) const {
        return std::fabs(lat - o.lat) < 1e-5f &&
               std::fabs(lng - o.lng) < 1e-5f &&
               housenumber == o.housenumber &&
               street == o.street;
    }
};

// --- Decode functions ---

static std::vector<DecodedAdminPolygon> decode_admin(const std::string& dir) {
    auto strings = read_file(dir + "/strings.bin");
    auto polygons = read_structs<AdminPolygon>(dir + "/admin_polygons.bin");
    auto vertices = read_structs<NodeCoord>(dir + "/admin_vertices.bin");

    std::vector<DecodedAdminPolygon> result;
    for (const auto& p : polygons) {
        DecodedAdminPolygon dp;
        dp.name = get_string(strings, p.name_id);
        dp.admin_level = p.admin_level;
        dp.area = p.area;
        dp.country_code = p.country_code;
        for (uint16_t i = 0; i < p.vertex_count; i++) {
            uint32_t vi = p.vertex_offset + i;
            if (vi < vertices.size()) {
                dp.vertices.push_back({vertices[vi].lat, vertices[vi].lng});
            }
        }
        result.push_back(std::move(dp));
    }
    return result;
}

static std::vector<DecodedWay> decode_ways(const std::string& dir) {
    auto strings = read_file(dir + "/strings.bin");
    auto ways = read_structs<WayHeader>(dir + "/street_ways.bin");
    auto nodes = read_structs<NodeCoord>(dir + "/street_nodes.bin");

    std::vector<DecodedWay> result;
    for (const auto& w : ways) {
        DecodedWay dw;
        dw.name = get_string(strings, w.name_id);
        for (uint8_t i = 0; i < w.node_count; i++) {
            uint32_t ni = w.node_offset + i;
            if (ni < nodes.size()) {
                dw.nodes.push_back({nodes[ni].lat, nodes[ni].lng});
            }
        }
        result.push_back(std::move(dw));
    }
    return result;
}

static std::vector<DecodedAddr> decode_addrs(const std::string& dir) {
    auto strings = read_file(dir + "/strings.bin");
    auto points = read_structs<AddrPoint>(dir + "/addr_points.bin");

    std::vector<DecodedAddr> result;
    for (const auto& p : points) {
        DecodedAddr da;
        da.lat = p.lat;
        da.lng = p.lng;
        da.housenumber = get_string(strings, p.housenumber_id);
        da.street = get_string(strings, p.street_id);
        result.push_back(std::move(da));
    }
    return result;
}

// --- Comparison ---

template<typename T>
static bool compare_sorted(const char* label, std::vector<T> a, std::vector<T> b) {
    std::sort(a.begin(), a.end());
    std::sort(b.begin(), b.end());

    if (a.size() != b.size()) {
        std::cerr << "FAIL: " << label << " count differs (A=" << a.size()
                  << " B=" << b.size() << ")" << std::endl;
        return false;
    }

    size_t diffs = 0;
    for (size_t i = 0; i < a.size(); i++) {
        if (!(a[i] == b[i])) {
            if (diffs < 3) {
                std::cerr << "  Mismatch at sorted position " << i << std::endl;
            }
            diffs++;
        }
    }

    if (diffs > 0) {
        std::cerr << "FAIL: " << label << " has " << diffs << " differences out of "
                  << a.size() << " items" << std::endl;
        return false;
    }

    std::cout << "PASS: " << label << " (" << a.size() << " items)" << std::endl;
    return true;
}

int main(int argc, char* argv[]) {
    if (argc != 3) {
        std::cerr << "Usage: compare_indexes <dir-A> <dir-B>" << std::endl;
        return 2;
    }

    std::string dir_a = argv[1];
    std::string dir_b = argv[2];

    std::cout << "Comparing:" << std::endl;
    std::cout << "  A: " << dir_a << std::endl;
    std::cout << "  B: " << dir_b << std::endl;
    std::cout << std::endl;

    bool all_pass = true;

    // Compare admin polygons
    {
        auto a = decode_admin(dir_a);
        auto b = decode_admin(dir_b);
        if (!compare_sorted("admin_polygons", std::move(a), std::move(b)))
            all_pass = false;
    }

    // Compare street ways
    {
        auto a = decode_ways(dir_a);
        auto b = decode_ways(dir_b);
        if (!compare_sorted("street_ways", std::move(a), std::move(b)))
            all_pass = false;
    }

    // Compare address points
    {
        auto a = decode_addrs(dir_a);
        auto b = decode_addrs(dir_b);
        if (!compare_sorted("address_points", std::move(a), std::move(b)))
            all_pass = false;
    }

    // Compare cell index sizes (should be identical regardless of ordering)
    {
        auto ac = read_file(dir_a + "/admin_cells.bin");
        auto bc = read_file(dir_b + "/admin_cells.bin");
        if (ac == bc) {
            std::cout << "PASS: admin_cells.bin byte-identical (" << ac.size() << " bytes)" << std::endl;
        } else if (ac.size() == bc.size()) {
            std::cout << "INFO: admin_cells.bin same size (" << ac.size() << " bytes) but different content" << std::endl;
        } else {
            std::cerr << "FAIL: admin_cells.bin size differs (A=" << ac.size()
                      << " B=" << bc.size() << ")" << std::endl;
            all_pass = false;
        }

        auto gc_a = read_file(dir_a + "/geo_cells.bin");
        auto gc_b = read_file(dir_b + "/geo_cells.bin");
        if (gc_a.size() == gc_b.size()) {
            std::cout << "PASS: geo_cells.bin same size (" << gc_a.size() << " bytes)" << std::endl;
        } else {
            std::cerr << "FAIL: geo_cells.bin size differs (A=" << gc_a.size()
                      << " B=" << gc_b.size() << ")" << std::endl;
            all_pass = false;
        }
    }

    std::cout << std::endl;
    if (all_pass) {
        std::cout << "=== ALL COMPARISONS PASSED ===" << std::endl;
        return 0;
    } else {
        std::cout << "=== SOME COMPARISONS FAILED ===" << std::endl;
        return 1;
    }
}


Now I understand the issues. The current comparison sorts by `sort_key()` then compares position-by-position, which fails badly when there are sort key collisions (millions of ways with the same name + node count). The fix is to use a canonical fingerprint for each item and compare as multisets.


Write

// Semantic comparison of two geocoder index directories.
// Decodes all binary files, creates canonical fingerprints for each item,
// and compares as multisets — handles any ordering differences between
// parallel and sequential builds.
// Returns 0 if semantically identical, 1 if different.
//
// Usage: compare_indexes <dir-A> <dir-B>

#include <algorithm>
#include <cmath>
#include <cstdint>
#include <cstring>
#include <fstream>
#include <iostream>
#include <map>
#include <set>
#include <string>
#include <unordered_map>
#include <vector>

// --- Binary structs (must match build_index.cpp and server main.rs) ---

struct WayHeader {
    uint32_t node_offset;
    uint8_t node_count;
    uint32_t name_id;
};

struct AddrPoint {
    float lat;
    float lng;
    uint32_t housenumber_id;
    uint32_t street_id;
};

struct InterpWay {
    uint32_t node_offset;
    uint8_t node_count;
    uint32_t street_id;
    uint32_t start_number;
    uint32_t end_number;
    uint8_t interpolation;
};

struct AdminPolygon {
    uint32_t vertex_offset;
    uint16_t vertex_count;
    uint32_t name_id;
    uint8_t admin_level;
    float area;
    uint16_t country_code;
};

struct NodeCoord {
    float lat;
    float lng;
};

// --- Helpers ---

static std::vector<char> read_file(const std::string& path) {
    std::ifstream f(path, std::ios::binary | std::ios::ate);
    if (!f) return {};
    auto size = f.tellg();
    f.seekg(0);
    std::vector<char> data(size);
    f.read(data.data(), size);
    return data;
}

static const char* get_string(const std::vector<char>& pool, uint32_t offset) {
    if (offset >= pool.size()) return "(invalid)";
    return pool.data() + offset;
}

template<typename T>
static std::vector<T> read_structs(const std::string& path) {
    auto data = read_file(path);
    size_t count = data.size() / sizeof(T);
    std::vector<T> result(count);
    memcpy(result.data(), data.data(), count * sizeof(T));
    return result;
}

// Round a float coordinate to an integer grid (1e5 ~ 1m precision)
static int to_grid(float v) {
    return (int)(v * 1e5f + (v >= 0 ? 0.5f : -0.5f));
}

// --- Canonical fingerprint builders ---
// Each function produces a string that uniquely identifies an item,
// independent of storage order.

// For a way: "name|sorted_node_coords"
// Nodes are kept in original order (path order matters for ways),
// but we also provide a sorted-set version for order-independent matching.
static std::string way_fingerprint(const std::string& name,
                                    const std::vector<std::pair<float,float>>& nodes) {
    // Use sorted coordinate set as the canonical form.
    // Two ways are "the same" if they have the same name and same set of nodes.
    std::vector<std::pair<int,int>> grid_nodes;
    grid_nodes.reserve(nodes.size());
    for (const auto& n : nodes) {
        grid_nodes.push_back({to_grid(n.first), to_grid(n.second)});
    }
    std::sort(grid_nodes.begin(), grid_nodes.end());

    std::string fp = name + "|" + std::to_string(nodes.size());
    for (const auto& g : grid_nodes) {
        fp += "|";
        fp += std::to_string(g.first);
        fp += ",";
        fp += std::to_string(g.second);
    }
    return fp;
}

// For an address: "grid_lat,grid_lng|housenumber|street"
static std::string addr_fingerprint(float lat, float lng,
                                     const std::string& housenumber,
                                     const std::string& street) {
    return std::to_string(to_grid(lat)) + "," + std::to_string(to_grid(lng)) +
           "|" + housenumber + "|" + street;
}

// For an admin polygon: "name|level|country|sorted_vertices"
static std::string admin_fingerprint(const std::string& name,
                                      uint8_t admin_level,
                                      uint16_t country_code,
                                      const std::vector<std::pair<float,float>>& vertices) {
    std::vector<std::pair<int,int>> grid_verts;
    grid_verts.reserve(vertices.size());
    for (const auto& v : vertices) {
        grid_verts.push_back({to_grid(v.first), to_grid(v.second)});
    }
    std::sort(grid_verts.begin(), grid_verts.end());

    std::string fp = name + "|" + std::to_string(admin_level) + "|" + std::to_string(country_code);
    for (const auto& g : grid_verts) {
        fp += "|";
        fp += std::to_string(g.first);
        fp += ",";
        fp += std::to_string(g.second);
    }
    return fp;
}

// For interpolation ways: "street|start|end|interp|sorted_nodes"
static std::string interp_fingerprint(const std::string& street,
                                       uint32_t start_number, uint32_t end_number,
                                       uint8_t interpolation,
                                       const std::vector<std::pair<float,float>>& nodes) {
    std::vector<std::pair<int,int>> grid_nodes;
    grid_nodes.reserve(nodes.size());
    for (const auto& n : nodes) {
        grid_nodes.push_back({to_grid(n.first), to_grid(n.second)});
    }
    std::sort(grid_nodes.begin(), grid_nodes.end());

    std::string fp = street + "|" + std::to_string(start_number) + "-" +
                     std::to_string(end_number) + "|" + std::to_string(interpolation);
    for (const auto& g : grid_nodes) {
        fp += "|";
        fp += std::to_string(g.first);
        fp += ",";
        fp += std::to_string(g.second);
    }
    return fp;
}

// --- Multiset comparison ---
// Compare two collections by their fingerprints. Reports missing/extra items.

static bool compare_multisets(const char* label,
                               const std::vector<std::string>& fps_a,
                               const std::vector<std::string>& fps_b) {
    // Build frequency maps
    std::unordered_map<std::string, int> freq_a, freq_b;
    for (const auto& fp : fps_a) freq_a[fp]++;
    for (const auto& fp : fps_b) freq_b[fp]++;

    // Find items in A but not in B (missing from B)
    size_t missing = 0;
    for (const auto& [fp, count] : freq_a) {
        int b_count = 0;
        auto it = freq_b.find(fp);
        if (it != freq_b.end()) b_count = it->second;
        if (count > b_count) missing += (count - b_count);
    }

    // Find items in B but not in A (extra in B)
    size_t extra = 0;
    for (const auto& [fp, count] : freq_b) {
        int a_count = 0;
        auto it = freq_a.find(fp);
        if (it != freq_a.end()) a_count = it->second;
        if (count > a_count) extra += (count - a_count);
    }

    size_t matched = fps_a.size() - missing;

    if (missing == 0 && extra == 0) {
        std::cout << "PASS: " << label << " (" << fps_a.size() << " items, all matched)" << std::endl;
        return true;
    } else {
        std::cerr << "FAIL: " << label << " — A=" << fps_a.size() << " B=" << fps_b.size()
                  << " matched=" << matched
                  << " only_in_A=" << missing
                  << " only_in_B=" << extra << std::endl;
        return false;
    }
}

// --- Decode + fingerprint functions ---

static std::vector<std::string> fingerprint_admin(const std::string& dir) {
    auto strings = read_file(dir + "/strings.bin");
    auto polygons = read_structs<AdminPolygon>(dir + "/admin_polygons.bin");
    auto vertices = read_structs<NodeCoord>(dir + "/admin_vertices.bin");

    std::vector<std::string> fps;
    fps.reserve(polygons.size());
    for (const auto& p : polygons) {
        std::string name = get_string(strings, p.name_id);
        std::vector<std::pair<float,float>> verts;
        for (uint16_t i = 0; i < p.vertex_count; i++) {
            uint32_t vi = p.vertex_offset + i;
            if (vi < vertices.size()) {
                verts.push_back({vertices[vi].lat, vertices[vi].lng});
            }
        }
        fps.push_back(admin_fingerprint(name, p.admin_level, p.country_code, verts));
    }
    return fps;
}

static std::vector<std::string> fingerprint_ways(const std::string& dir) {
    auto strings = read_file(dir + "/strings.bin");
    auto ways = read_structs<WayHeader>(dir + "/street_ways.bin");
    auto nodes = read_structs<NodeCoord>(dir + "/street_nodes.bin");

    std::vector<std::string> fps;
    fps.reserve(ways.size());
    for (const auto& w : ways) {
        std::string name = get_string(strings, w.name_id);
        std::vector<std::pair<float,float>> way_nodes;
        for (uint8_t i = 0; i < w.node_count; i++) {
            uint32_t ni = w.node_offset + i;
            if (ni < nodes.size()) {
                way_nodes.push_back({nodes[ni].lat, nodes[ni].lng});
            }
        }
        fps.push_back(way_fingerprint(name, way_nodes));
    }
    return fps;
}

static std::vector<std::string> fingerprint_addrs(const std::string& dir) {
    auto strings = read_file(dir + "/strings.bin");
    auto points = read_structs<AddrPoint>(dir + "/addr_points.bin");

    std::vector<std::string> fps;
    fps.reserve(points.size());
    for (const auto& p : points) {
        std::string hn = get_string(strings, p.housenumber_id);
        std::string st = get_string(strings, p.street_id);
        fps.push_back(addr_fingerprint(p.lat, p.lng, hn, st));
    }
    return fps;
}

static std::vector<std::string> fingerprint_interps(const std::string& dir) {
    auto strings = read_file(dir + "/strings.bin");
    auto interps = read_structs<InterpWay>(dir + "/interp_ways.bin");
    auto nodes = read_structs<NodeCoord>(dir + "/interp_nodes.bin");

    std::vector<std::string> fps;
    fps.reserve(interps.size());
    for (const auto& iw : interps) {
        std::string street = get_string(strings, iw.street_id);
        std::vector<std::pair<float,float>> way_nodes;
        for (uint8_t i = 0; i < iw.node_count; i++) {
            uint32_t ni = iw.node_offset + i;
            if (ni < nodes.size()) {
                way_nodes.push_back({nodes[ni].lat, nodes[ni].lng});
            }
        }
        fps.push_back(interp_fingerprint(street, iw.start_number, iw.end_number,
                                          iw.interpolation, way_nodes));
    }
    return fps;
}

int main(int argc, char* argv[]) {
    if (argc != 3) {
        std::cerr << "Usage: compare_indexes <dir-A> <dir-B>" << std::endl;
        return 2;
    }

    std::string dir_a = argv[1];
    std::string dir_b = argv[2];

    std::cout << "Comparing:" << std::endl;
    std::cout << "  A: " << dir_a << std::endl;
    std::cout << "  B: " << dir_b << std::endl;
    std::cout << std::endl;

    bool all_pass = true;

    // Compare admin polygons
    {
        std::cout << "Loading admin polygons..." << std::flush;
        auto fps_a = fingerprint_admin(dir_a);
        auto fps_b = fingerprint_admin(dir_b);
        std::cout << " done." << std::endl;
        if (!compare_multisets("admin_polygons", fps_a, fps_b))
            all_pass = false;
    }

    // Compare street ways
    {
        std::cout << "Loading street ways..." << std::flush;
        auto fps_a = fingerprint_ways(dir_a);
        auto fps_b = fingerprint_ways(dir_b);
        std::cout << " done." << std::endl;
        if (!compare_multisets("street_ways", fps_a, fps_b))
            all_pass = false;
    }

    // Compare address points
    {
        std::cout << "Loading address points..." << std::flush;
        auto fps_a = fingerprint_addrs(dir_a);
        auto fps_b = fingerprint_addrs(dir_b);
        std::cout << " done." << std::endl;
        if (!compare_multisets("address_points", fps_a, fps_b))
            all_pass = false;
    }

    // Compare interpolation ways
    {
        std::cout << "Loading interpolation ways..." << std::flush;
        auto fps_a = fingerprint_interps(dir_a);
        auto fps_b = fingerprint_interps(dir_b);
        std::cout << " done." << std::endl;
        if (!compare_multisets("interp_ways", fps_a, fps_b))
            all_pass = false;
    }

    // Compare cell index sizes (should be identical regardless of ordering)
    {
        auto ac = read_file(dir_a + "/admin_cells.bin");
        auto bc = read_file(dir_b + "/admin_cells.bin");
        if (ac == bc) {
            std::cout << "PASS: admin_cells.bin byte-identical (" << ac.size() << " bytes)" << std::endl;
        } else if (ac.size() == bc.size()) {
            std::cout << "INFO: admin_cells.bin same size (" << ac.size() << " bytes) but different content" << std::endl;
        } else {
            std::cerr << "FAIL: admin_cells.bin size differs (A=" << ac.size()
                      << " B=" << bc.size() << ")" << std::endl;
            all_pass = false;
        }

        auto gc_a = read_file(dir_a + "/geo_cells.bin");
        auto gc_b = read_file(dir_b + "/geo_cells.bin");
        if (gc_a.size() == gc_b.size()) {
            std::cout << "PASS: geo_cells.bin same size (" << gc_a.size() << " bytes)" << std::endl;
        } else {
            std::cerr << "FAIL: geo_cells.bin size differs (A=" << gc_a.size()
                      << " B=" << gc_b.size() << ")" << std::endl;
            all_pass = false;
        }
    }

    std::cout << std::endl;
    if (all_pass) {
        std::cout << "=== ALL COMPARISONS PASSED ===" << std::endl;
        return 0;
    } else {
        std::cout << "=== SOME COMPARISONS FAILED ===" << std::endl;
        return 1;
    }
}