Design a custom geocoder index patching system. I'll provide the full context.

## Goal
Two tools:
- `geocoder-diff <old-dir> <new-dir> -o patch.gcpatch` — compares two independently-built indexes, produces a compact semantic patch
- `geocoder-patch <current-dir> patch.gcpatch -o <new-dir>` — applies patch, produces new index

Requirements from the user:
1. Daily builds are completely independent (no dependency on previous builds)
2. Diff tool compares full output of today vs yesterday
3. Patches must be SMALL (target: ~5-15 MiB for Europe, vs 7 GiB full dataset)
4. Users can apply patches in sequence: day1 + patch_1→2 = day2, then day2 + patch_2→3 = day3

## Key Challenge: Sequential Patching
Two independent builds of the same day have completely different file layouts (different string offsets, different record orderings, different entry offsets). A patch generated from fresh_day2 vs fresh_day3 can't be applied to a file produced by patching day1→day2, because the byte layout differs.

**Solution**: The patch applicator always produces DETERMINISTIC (canonical) output. Same semantic data → same bytes. This means:
- `patch(canonical_day1, diff_1→2)` produces `canonical_day2`
- `canonicalize(fresh_day2)` also produces `canonical_day2`
- Therefore `diff_2→3` generated from `canonical_day2` vs `fresh_day3` is applicable to the client's `canonical_day2`

The build server pipeline becomes:
1. Fresh build → output/
2. Canonicalize output/ → canonical/ (deterministic format)
3. Diff previous canonical/ vs new canonical/ → patch

## Binary Format (14 files per index)

**Cell index files** (sorted by cell_id, binary search):
- geo_cells.bin: [uint64 cell_id, uint32 street_off, uint32 addr_off, uint32 interp_off] × N (20 bytes/entry)
- admin_cells.bin: [uint64 cell_id, uint32 entry_off] × N (12 bytes/entry)

**Entry files** (variable-length, referenced by byte offset from cell files):
- street_entries.bin, addr_entries.bin, interp_entries.bin, admin_entries.bin: [uint16 count][uint32 id] × count

**Data arrays** (fixed-size records, indexed by ID from entries):
- street_ways.bin: [uint32 node_offset, uint8 node_count, uint32 name_id] (9 bytes, NO padding)
- addr_points.bin: [float lat, float lng, uint32 housenumber_id, uint32 street_id] (16 bytes)
- interp_ways.bin: [uint32 node_offset, uint8 node_count, uint32 street_id, uint32 start_num, uint32 end_num, uint8 interp_type] (18 bytes, struct-packed with padding)
- admin_polygons.bin: [uint32 vertex_offset, uint32 vertex_count, uint32 name_id, uint8 admin_level, float area, uint16 country_code] (with padding)

**Coordinate arrays** (flat, 8 bytes/entry):
- street_nodes.bin, interp_nodes.bin, admin_vertices.bin: [float lat, float lng]

**String pool**: strings.bin: null-terminated strings concatenated, referenced by byte offset

**Server constraints**: Files are mmap'd. Cell files MUST be sorted by cell_id. Data arrays are accessed by direct indexing. No validation of IDs or offsets.

## Existing Comparison Infrastructure
`compare_indexes.cpp` already implements semantic fingerprinting:
- way_fingerprint: "name|node_count|sorted_grid_coords" (1e5 precision)
- addr_fingerprint: "grid_lat,grid_lng|housenumber|street"
- admin_fingerprint: "name|admin_level|country|sorted_grid_vertices"
- interp_fingerprint: "street|start-end|interp_type|sorted_grid_nodes"

## Design the diff tool, patch format, and applicator

Key considerations:
1. The diff tool compares two builds by semantic fingerprinting
2. The patch contains: added records (full data + S2 cell assignments), removed record fingerprints
3. The applicator rebuilds all files deterministically (canonical format)
4. No S2 dependency on client — cell assignments for new records come from the patch
5. Kept records carry their cell assignments from the old build's entries files
6. The canonical format: alphabetically sorted string pool, records sorted by canonical key

Please design:
1. The patch binary format (sections, encoding)
2. The diff tool algorithm (matching, cell assignment extraction)
3. The applicator algorithm (how it rebuilds each of the 14 files)
4. The canonical record ordering (what key to sort by for each record type)
5. Memory and performance estimates
6. How to handle admin data (polygons + vertices are linked)

The implementation will be in C++, in builder/tools/. It can reuse types from builder/src/types.h and the string pool from builder/src/string_pool.h.

Prompt:
Design a custom geocoder index patching system. I'll provide the full context.

## Goal
Two tools:
- `geocoder-diff <old-dir> <new-dir> -o patch.gcpatch` — compares two independently-built indexes, produces a compact semantic patch
- `geocoder-patch <current-dir> patch.gcpatch -o <new-dir>` — applies patch, produces new index

Requirements from the user:
1. Daily builds are completely independent (no dependency on previous builds)
2. Diff tool compares full output of today vs yesterday
3. Patches must be SMALL (target: ~5-15 MiB for Europe, vs 7 GiB full dataset)
4. Users can apply patches in sequence: day1 + patch_1→2 = day2, then day2 + patch_2→3 = day3

## Key Challenge: Sequential Patching
Two independent builds of the same day have completely different file layouts (different string offsets, different record orderings, different entry offsets). A patch generated from fresh_day2 vs fresh_day3 can't be applied to a file produced by patching day1→day2, because the byte layout differs.

**Solution**: The patch applicator always produces DETERMINISTIC (canonical) output. Same semantic data → same bytes. This means:
- `patch(canonical_day1, diff_1→2)` produces `canonical_day2`
- `canonicalize(fresh_day2)` also produces `canonical_day2`
- Therefore `diff_2→3` generated from `canonical_day2` vs `fresh_day3` is applicable to the client's `canonical_day2`

The build server pipeline becomes:
1. Fresh build → output/
2. Canonicalize output/ → canonical/ (deterministic format)
3. Diff previous canonical/ vs new canonical/ → patch

## Binary Format (14 files per index)

**Cell index files** (sorted by cell_id, binary search):
- geo_cells.bin: [uint64 cell_id, uint32 street_off, uint32 addr_off, uint32 interp_off] × N (20 bytes/entry)
- admin_cells.bin: [uint64 cell_id, uint32 entry_off] × N (12 bytes/entry)

**Entry files** (variable-length, referenced by byte offset from cell files):
- street_entries.bin, addr_entries.bin, interp_entries.bin, admin_entries.bin: [uint16 count][uint32 id] × count

**Data arrays** (fixed-size records, indexed by ID from entries):
- street_ways.bin: [uint32 node_offset, uint8 node_count, uint32 name_id] (9 bytes, NO padding)
- addr_points.bin: [float lat, float lng, uint32 housenumber_id, uint32 street_id] (16 bytes)
- interp_ways.bin: [uint32 node_offset, uint8 node_count, uint32 street_id, uint32 start_num, uint32 end_num, uint8 interp_type] (18 bytes, struct-packed with padding)
- admin_polygons.bin: [uint32 vertex_offset, uint32 vertex_count, uint32 name_id, uint8 admin_level, float area, uint16 country_code] (with padding)

**Coordinate arrays** (flat, 8 bytes/entry):
- street_nodes.bin, interp_nodes.bin, admin_vertices.bin: [float lat, float lng]

**String pool**: strings.bin: null-terminated strings concatenated, referenced by byte offset

**Server constraints**: Files are mmap'd. Cell files MUST be sorted by cell_id. Data arrays are accessed by direct indexing. No validation of IDs or offsets.

## Existing Comparison Infrastructure
`compare_indexes.cpp` already implements semantic fingerprinting:
- way_fingerprint: "name|node_count|sorted_grid_coords" (1e5 precision)
- addr_fingerprint: "grid_lat,grid_lng|housenumber|street"
- admin_fingerprint: "name|admin_level|country|sorted_grid_vertices"
- interp_fingerprint: "street|start-end|interp_type|sorted_grid_nodes"

## Design the diff tool, patch format, and applicator

Key considerations:
1. The diff tool compares two builds by semantic fingerprinting
2. The patch contains: added records (full data + S2 cell assignments), removed record fingerprints
3. The applicator rebuilds all files deterministically (canonical format)
4. No S2 dependency on client — cell assignments for new records come from the patch
5. Kept records carry their cell assignments from the old build's entries files
6. The canonical format: alphabetically sorted string pool, records sorted by canonical key

Please design:
1. The patch binary format (sections, encoding)
2. The diff tool algorithm (matching, cell assignment extraction)
3. The applicator algorithm (how it rebuilds each of the 14 files)
4. The canonical record ordering (what key to sort by for each record type)
5. Memory and performance estimates
6. How to handle admin data (polygons + vertices are linked)

The implementation will be in C++, in builder/tools/. It can reuse types from builder/src/types.h and the string pool from builder/src/string_pool.h.


You've hit your limit · resets 4am (UTC)

Prompt:
Design a custom geocoder index patching system. I'll provide the full context.

## Goal
Two tools:
- `geocoder-diff <old-dir> <new-dir> -o patch.gcpatch` — compares two independently-built indexes, produces a compact semantic patch
- `geocoder-patch <current-dir> patch.gcpatch -o <new-dir>` — applies patch, produces new index

Requirements from the user:
1. Daily builds are completely independent (no dependency on previous builds)
2. Diff tool compares full output of today vs yesterday
3. Patches must be SMALL (target: ~5-15 MiB for Europe, vs 7 GiB full dataset)
4. Users can apply patches in sequence: day1 + patch_1→2 = day2, then day2 + patch_2→3 = day3

## Key Challenge: Sequential Patching
Two independent builds of the same day have completely different file layouts (different string offsets, different record orderings, different entry offsets). A patch generated from fresh_day2 vs fresh_day3 can't be applied to a file produced by patching day1→day2, because the byte layout differs.

**Solution**: The patch applicator always produces DETERMINISTIC (canonical) output. Same semantic data → same bytes. This means:
- `patch(canonical_day1, diff_1→2)` produces `canonical_day2`
- `canonicalize(fresh_day2)` also produces `canonical_day2`
- Therefore `diff_2→3` generated from `canonical_day2` vs `fresh_day3` is applicable to the client's `canonical_day2`

The build server pipeline becomes:
1. Fresh build → output/
2. Canonicalize output/ → canonical/ (deterministic format)
3. Diff previous canonical/ vs new canonical/ → patch

## Binary Format (14 files per index)

**Cell index files** (sorted by cell_id, binary search):
- geo_cells.bin: [uint64 cell_id, uint32 street_off, uint32 addr_off, uint32 interp_off] × N (20 bytes/entry)
- admin_cells.bin: [uint64 cell_id, uint32 entry_off] × N (12 bytes/entry)

**Entry files** (variable-length, referenced by byte offset from cell files):
- street_entries.bin, addr_entries.bin, interp_entries.bin, admin_entries.bin: [uint16 count][uint32 id] × count

**Data arrays** (fixed-size records, indexed by ID from entries):
- street_ways.bin: [uint32 node_offset, uint8 node_count, uint32 name_id] (9 bytes, NO padding)
- addr_points.bin: [float lat, float lng, uint32 housenumber_id, uint32 street_id] (16 bytes)
- interp_ways.bin: [uint32 node_offset, uint8 node_count, uint32 street_id, uint32 start_num, uint32 end_num, uint8 interp_type] (18 bytes, struct-packed with padding)
- admin_polygons.bin: [uint32 vertex_offset, uint32 vertex_count, uint32 name_id, uint8 admin_level, float area, uint16 country_code] (with padding)

**Coordinate arrays** (flat, 8 bytes/entry):
- street_nodes.bin, interp_nodes.bin, admin_vertices.bin: [float lat, float lng]

**String pool**: strings.bin: null-terminated strings concatenated, referenced by byte offset

**Server constraints**: Files are mmap'd. Cell files MUST be sorted by cell_id. Data arrays are accessed by direct indexing. No validation of IDs or offsets.

## Existing Comparison Infrastructure
`compare_indexes.cpp` already implements semantic fingerprinting:
- way_fingerprint: "name|node_count|sorted_grid_coords" (1e5 precision)
- addr_fingerprint: "grid_lat,grid_lng|housenumber|street"
- admin_fingerprint: "name|admin_level|country|sorted_grid_vertices"
- interp_fingerprint: "street|start-end|interp_type|sorted_grid_nodes"

## Design the diff tool, patch format, and applicator

Key considerations:
1. The diff tool compares two builds by semantic fingerprinting
2. The patch contains: added records (full data + S2 cell assignments), removed record fingerprints
3. The applicator rebuilds all files deterministically (canonical format)
4. No S2 dependency on client — cell assignments for new records come from the patch
5. Kept records carry their cell assignments from the old build's entries files
6. The canonical format: alphabetically sorted string pool, records sorted by canonical key

Please design:
1. The patch binary format (sections, encoding)
2. The diff tool algorithm (matching, cell assignment extraction)
3. The applicator algorithm (how it rebuilds each of the 14 files)
4. The canonical record ordering (what key to sort by for each record type)
5. Memory and performance estimates
6. How to handle admin data (polygons + vertices are linked)

The implementation will be in C++, in builder/tools/. It can reuse types from builder/src/types.h and the string pool from builder/src/string_pool.h.


Continue


Agent

Design semantic patch system

Prompt:
Design a custom geocoder index patching system. Two tools:

1. `geocoder-diff <old-dir> <new-dir> -o patch.gcpatch` — compares two independently-built indexes
2. `geocoder-patch <current-dir> patch.gcpatch -o <new-dir>` — applies patch

Requirements:
- Daily builds are independent (no dependency on previous)
- Patches must be small (~5-15 MiB for 7 GiB Europe dataset with ~0.3% daily change)
- Sequential patching: day1 + patch12 → day2, then day2 + patch23 → day3

The key insight: the applicator always produces DETERMINISTIC canonical output. Same semantic data → same bytes. So patches compose correctly.

Build server pipeline:
1. Fresh build → output/
2. Canonicalize → canonical/
3. Diff(prev_canonical, new_canonical) → patch

## File Format (14 files)
- geo_cells.bin: sorted [cell_id:u64, street_off:u32, addr_off:u32, interp_off:u32] (20B)
- admin_cells.bin: sorted [cell_id:u64, off:u32] (12B)
- *_entries.bin: variable [count:u16][id:u32]*count
- street_ways.bin: [node_offset:u32, node_count:u8, name_id:u32] (9B packed)
- addr_points.bin: [lat:f32, lng:f32, hn_id:u32, street_id:u32] (16B)
- interp_ways.bin: [node_off:u32, count:u8, street_id:u32, start:u32, end:u32, type:u8] (18B+padding)
- admin_polygons.bin: [vert_off:u32, count:u32, name_id:u32, level:u8, area:f32, cc:u16] (+padding)
- street_nodes.bin, interp_nodes.bin, admin_vertices.bin: [lat:f32, lng:f32] (8B)
- strings.bin: null-terminated concatenated strings, referenced by byte offset

Existing fingerprinting (compare_indexes.cpp):
- way: "name|node_count|sorted_grid_coords"
- addr: "grid_lat,grid_lng|housenumber|street"
- admin: "name|admin_level|country|sorted_grid_vertices"

Design:
1. Patch binary format (sections, encoding)
2. Diff algorithm (matching, cell assignment extraction)
3. Applicator algorithm (rebuild each of 14 files deterministically)
4. Canonical ordering for each record type
5. Memory/performance estimates for Europe (86M addrs, 19M ways, 450K polygons)
6. Admin handling (polygons linked to vertex arrays)

Implementation in C++, builder/tools/. Can reuse types.h and string_pool.h.


Design a custom geocoder index patching system. Two tools:

1. `geocoder-diff <old-dir> <new-dir> -o patch.gcpatch` — compares two independently-built indexes
2. `geocoder-patch <current-dir> patch.gcpatch -o <new-dir>` — applies patch

Requirements:
- Daily builds are independent (no dependency on previous)
- Patches must be small (~5-15 MiB for 7 GiB Europe dataset with ~0.3% daily change)
- Sequential patching: day1 + patch12 → day2, then day2 + patch23 → day3

The key insight: the applicator always produces DETERMINISTIC canonical output. Same semantic data → same bytes. So patches compose correctly.

Build server pipeline:
1. Fresh build → output/
2. Canonicalize → canonical/
3. Diff(prev_canonical, new_canonical) → patch

## File Format (14 files)
- geo_cells.bin: sorted [cell_id:u64, street_off:u32, addr_off:u32, interp_off:u32] (20B)
- admin_cells.bin: sorted [cell_id:u64, off:u32] (12B)
- *_entries.bin: variable [count:u16][id:u32]*count
- street_ways.bin: [node_offset:u32, node_count:u8, name_id:u32] (9B packed)
- addr_points.bin: [lat:f32, lng:f32, hn_id:u32, street_id:u32] (16B)
- interp_ways.bin: [node_off:u32, count:u8, street_id:u32, start:u32, end:u32, type:u8] (18B+padding)
- admin_polygons.bin: [vert_off:u32, count:u32, name_id:u32, level:u8, area:f32, cc:u16] (+padding)
- street_nodes.bin, interp_nodes.bin, admin_vertices.bin: [lat:f32, lng:f32] (8B)
- strings.bin: null-terminated concatenated strings, referenced by byte offset

Existing fingerprinting (compare_indexes.cpp):
- way: "name|node_count|sorted_grid_coords"
- addr: "grid_lat,grid_lng|housenumber|street"
- admin: "name|admin_level|country|sorted_grid_vertices"

Design:
1. Patch binary format (sections, encoding)
2. Diff algorithm (matching, cell assignment extraction)
3. Applicator algorithm (rebuild each of 14 files deterministically)
4. Canonical ordering for each record type
5. Memory/performance estimates for Europe (86M addrs, 19M ways, 450K polygons)
6. Admin handling (polygons linked to vertex arrays)

Implementation in C++, builder/tools/. Can reuse types.h and string_pool.h.

Prompt:
Design a custom geocoder index patching system. Two tools:

1. `geocoder-diff <old-dir> <new-dir> -o patch.gcpatch` — compares two independently-built indexes
2. `geocoder-patch <current-dir> patch.gcpatch -o <new-dir>` — applies patch

Requirements:
- Daily builds are independent (no dependency on previous)
- Patches must be small (~5-15 MiB for 7 GiB Europe dataset with ~0.3% daily change)
- Sequential patching: day1 + patch12 → day2, then day2 + patch23 → day3

The key insight: the applicator always produces DETERMINISTIC canonical output. Same semantic data → same bytes. So patches compose correctly.

Build server pipeline:
1. Fresh build → output/
2. Canonicalize → canonical/
3. Diff(prev_canonical, new_canonical) → patch

## File Format (14 files)
- geo_cells.bin: sorted [cell_id:u64, street_off:u32, addr_off:u32, interp_off:u32] (20B)
- admin_cells.bin: sorted [cell_id:u64, off:u32] (12B)
- *_entries.bin: variable [count:u16][id:u32]*count
- street_ways.bin: [node_offset:u32, node_count:u8, name_id:u32] (9B packed)
- addr_points.bin: [lat:f32, lng:f32, hn_id:u32, street_id:u32] (16B)
- interp_ways.bin: [node_off:u32, count:u8, street_id:u32, start:u32, end:u32, type:u8] (18B+padding)
- admin_polygons.bin: [vert_off:u32, count:u32, name_id:u32, level:u8, area:f32, cc:u16] (+padding)
- street_nodes.bin, interp_nodes.bin, admin_vertices.bin: [lat:f32, lng:f32] (8B)
- strings.bin: null-terminated concatenated strings, referenced by byte offset

Existing fingerprinting (compare_indexes.cpp):
- way: "name|node_count|sorted_grid_coords"
- addr: "grid_lat,grid_lng|housenumber|street"
- admin: "name|admin_level|country|sorted_grid_vertices"

Design:
1. Patch binary format (sections, encoding)
2. Diff algorithm (matching, cell assignment extraction)
3. Applicator algorithm (rebuild each of 14 files deterministically)
4. Canonical ordering for each record type
5. Memory/performance estimates for Europe (86M addrs, 19M ways, 450K polygons)
6. Admin handling (polygons linked to vertex arrays)

Implementation in C++, builder/tools/. Can reuse types.h and string_pool.h.


Bash

List builder directory contents


total 77
drwxr-xr-x 8 root root   10 Mar 26 20:19 .
drwxr-xr-x 7 root root   14 Mar 21 11:08 ..
-rw-r--r-- 1 root root 1059 Mar 26 01:10 CMakeLists.txt
-rw-r--r-- 1 root root 3827 Mar 26 20:19 OPTIMIZATION_LOG.md
drwxr-xr-x 3 root root   12 Mar 27 03:30 build
drwxr-xr-x 3 root root    7 Mar 23 11:51 build-presplit
drwxr-xr-x 2 root root   10 Mar 26 01:23 data
drwxr-xr-x 2 root root   24 Mar 27 03:30 src
drwxr-xr-x 2 root root   15 Mar 26 11:11 test
drwxr-xr-x 2 root root    6 Mar 24 22:15 tools


find /root/traccar-geocoder/builder -type f -name "*.cpp" -o -name "*.h" | sort

Find all C++ source and header files


/root/traccar-geocoder/builder/build-presplit/CMakeFiles/3.31.6/CompilerIdCXX/CMakeCXXCompilerId.cpp
/root/traccar-geocoder/builder/build/CMakeFiles/3.31.6/CompilerIdCXX/CMakeCXXCompilerId.cpp
/root/traccar-geocoder/builder/src/build_index.cpp
/root/traccar-geocoder/builder/src/cache.cpp
/root/traccar-geocoder/builder/src/cache.h
/root/traccar-geocoder/builder/src/cell_index.cpp
/root/traccar-geocoder/builder/src/cell_index.h
/root/traccar-geocoder/builder/src/continent_boundaries.cpp
/root/traccar-geocoder/builder/src/continent_boundaries.h
/root/traccar-geocoder/builder/src/continent_filter.cpp
/root/traccar-geocoder/builder/src/continent_filter.h
/root/traccar-geocoder/builder/src/geometry.h
/root/traccar-geocoder/builder/src/interpolation.cpp
/root/traccar-geocoder/builder/src/interpolation.h
/root/traccar-geocoder/builder/src/parsed_data.h
/root/traccar-geocoder/builder/src/pbf_reader.cpp
/root/traccar-geocoder/builder/src/pbf_reader.h
/root/traccar-geocoder/builder/src/relation_collector.h
/root/traccar-geocoder/builder/src/ring_assembly.cpp
/root/traccar-geocoder/builder/src/ring_assembly.h
/root/traccar-geocoder/builder/src/s2_helpers.cpp
/root/traccar-geocoder/builder/src/s2_helpers.h
/root/traccar-geocoder/builder/src/string_pool.h
/root/traccar-geocoder/builder/src/types.h
/root/traccar-geocoder/builder/test/compare_indexes.cpp
/root/traccar-geocoder/builder/test/deep_compare.cpp
/root/traccar-geocoder/builder/test/test_custom_pbf.cpp
/root/traccar-geocoder/builder/test/test_pbf_parsing.cpp
/root/traccar-geocoder/builder/test/test_ring_assembly.cpp
/root/traccar-geocoder/builder/tools/simplify_compare.cpp


ls -la /root/traccar-geocoder/builder/tools/ 2>/dev/null || echo "tools/ directory does not exist"

Check if tools directory exists


total 7783
drwxr-xr-x 2 root root        6 Mar 24 22:15 .
drwxr-xr-x 8 root root       10 Mar 26 20:19 ..
-rw-r--r-- 1 root root   154460 Mar 23 16:27 comparison.geojson
-rw-r--r-- 1 root root     5349 Mar 24 22:15 simplification_map.html
-rw-r--r-- 1 root root 14822689 Mar 23 23:22 simplification_viewer.html
-rw-r--r-- 1 root root    13274 Mar 23 23:04 simplify_compare.cpp


     1→#pragma once
     2→
     3→#include <cstdint>
     4→#include <vector>
     5→
     6→// --- Binary format structs (must match server's index reader) ---
     7→
     8→struct WayHeader {
     9→    uint32_t node_offset;
    10→    uint8_t node_count;
    11→    uint32_t name_id;
    12→};
    13→
    14→struct AddrPoint {
    15→    float lat;
    16→    float lng;
    17→    uint32_t housenumber_id;
    18→    uint32_t street_id;
    19→};
    20→
    21→struct InterpWay {
    22→    uint32_t node_offset;
    23→    uint8_t node_count;
    24→    uint32_t street_id;
    25→    uint32_t start_number;
    26→    uint32_t end_number;
    27→    uint8_t interpolation;
    28→};
    29→
    30→struct AdminPolygon {
    31→    uint32_t vertex_offset;
    32→    uint32_t vertex_count;
    33→    uint32_t name_id;
    34→    uint8_t admin_level;
    35→    float area;
    36→    uint16_t country_code;
    37→};
    38→
    39→struct NodeCoord {
    40→    float lat;
    41→    float lng;
    42→};
    43→
    44→// Sorted cell-item pair for direct entry writing (avoids hash map)
    45→struct CellItemPair {
    46→    uint64_t cell_id;
    47→    uint32_t item_id;
    48→};
    49→
    50→// --- Constants ---
    51→
    52→static const uint32_t INTERIOR_FLAG = 0x80000000u;
    53→static const uint32_t ID_MASK       = 0x7FFFFFFFu;
    54→static const uint32_t NO_DATA       = 0xFFFFFFFFu;
    55→
    56→// Simplification mode
    57→enum class SimplifyMode { MaxVertices, ErrorBounded };
    58→
    59→// Processing limits
    60→static constexpr size_t MAX_BLOCK_QUEUE         = 64;    // producer-consumer queue depth
    61→static constexpr int    MAX_NODE_COUNT          = 255;   // uint8_t node_count
    62→static constexpr int    MAX_VERTEX_COUNT        = 0x7FFFFFFF; // uint32_t vertex_count
    63→static constexpr int    MAX_S2_CELLS_PER_POLY   = 200;   // S2RegionCoverer max_cells
    64→static constexpr int    BACKTRACK_CALL_BUDGET   = 100000; // per-ring backtracking limit
    65→static constexpr int    BACKTRACK_TOTAL_BUDGET  = 500000; // per-relation backtracking limit
    66→static constexpr int    BACKTRACK_MAX_DEPTH     = 200;    // recursion depth limit
    67→static constexpr size_t MAX_SUBWAYS_FOR_RETRY   = 30;     // only retry small relations
    68→static constexpr size_t MAX_NODE_ID_DEFAULT     = 15000000000ULL; // dense index capacity
    69→
    70→// --- Index mode ---
    71→
    72→enum class IndexMode { Full, NoAddresses, AdminOnly };
    73→
    74→// --- Deferred S2 work items ---
    75→
    76→struct DeferredWay {
    77→    uint32_t way_id;
    78→    uint32_t node_offset;
    79→    uint8_t node_count;
    80→};
    81→
    82→struct DeferredInterp {
    83→    uint32_t interp_id;
    84→    uint32_t node_offset;
    85→    uint8_t node_count;
    86→};
    87→
    88→// Collected relation data for parallel admin assembly
    89→struct CollectedRelation {
    90→    int64_t id;
    91→    uint8_t admin_level;
    92→    std::string name;
    93→    std::string country_code;
    94→    bool is_postal;
    95→    std::vector<std::pair<int64_t, std::string>> members; // (way_id, role)
    96→};
    97→


     1→#pragma once
     2→
     3→#include <cstdint>
     4→#include <cstring>
     5→#include <fstream>
     6→#include <string>
     7→#include <unordered_map>
     8→#include <vector>
     9→
    10→class StringPool {
    11→public:
    12→    uint32_t intern(const std::string& s) {
    13→        auto it = index_.find(s);
    14→        if (it != index_.end()) {
    15→            return it->second;
    16→        }
    17→        uint32_t offset = static_cast<uint32_t>(data_.size());
    18→        index_[s] = offset;
    19→        data_.insert(data_.end(), s.begin(), s.end());
    20→        data_.push_back('\0');
    21→        return offset;
    22→    }
    23→
    24→    // Load a previous build's string pool to preserve offsets.
    25→    // Strings that exist in the old pool keep their original offset.
    26→    // New strings get appended at the end.
    27→    // This makes string offsets stable across builds, enabling efficient binary patches.
    28→    bool load_previous(const std::string& path) {
    29→        std::ifstream f(path, std::ios::binary | std::ios::ate);
    30→        if (!f.is_open()) return false;
    31→        size_t size = f.tellg();
    32→        if (size == 0) return false;
    33→        f.seekg(0);
    34→        data_.resize(size);
    35→        f.read(data_.data(), size);
    36→
    37→        // Build index from old pool
    38→        size_t pos = 0;
    39→        while (pos < data_.size()) {
    40→            const char* str = data_.data() + pos;
    41→            size_t len = strlen(str);
    42→            if (len > 0) {
    43→                index_[std::string(str, len)] = static_cast<uint32_t>(pos);
    44→            }
    45→            pos += len + 1;
    46→        }
    47→        return true;
    48→    }
    49→
    50→    size_t previous_size() const { return previous_size_; }
    51→    void mark_previous_loaded() { previous_size_ = data_.size(); }
    52→
    53→    const std::vector<char>& data() const { return data_; }
    54→    std::vector<char>& mutable_data() { return data_; }
    55→
    56→private:
    57→    std::unordered_map<std::string, uint32_t> index_;
    58→    std::vector<char> data_;
    59→    size_t previous_size_ = 0;
    60→};
    61→


     1→#pragma once
     2→
     3→#include <string>
     4→#include <unordered_map>
     5→#include <vector>
     6→
     7→#include "types.h"
     8→#include "parsed_data.h"
     9→
    10→std::vector<uint32_t> write_entries(
    11→    const std::string& path,
    12→    const std::vector<uint64_t>& sorted_cells,
    13→    const std::unordered_map<uint64_t, std::vector<uint32_t>>& cell_map);
    14→
    15→std::vector<uint32_t> write_entries_from_sorted(
    16→    const std::string& path,
    17→    const std::vector<uint64_t>& sorted_cells,
    18→    const std::vector<CellItemPair>& sorted_pairs);
    19→
    20→void write_cell_index(
    21→    const std::string& cells_path,
    22→    const std::string& entries_path,
    23→    const std::unordered_map<uint64_t, std::vector<uint32_t>>& cell_map);
    24→
    25→void write_index(const ParsedData& data, const std::string& output_dir, IndexMode mode);
    26→
    27→// Write admin polygon/vertex files with on-the-fly simplification at a given epsilon scale.
    28→// Scale 0 = uncapped (no simplification). Other files are symlinked/copied from source_dir.
    29→void write_quality_variant(const ParsedData& data, const std::string& source_dir,
    30→                           const std::string& output_dir, double epsilon_scale);
    31→


     1→#include "cell_index.h"
     2→
     3→#include <algorithm>
     4→#include <atomic>
     5→#include <cstring>
     6→#include <fstream>
     7→#include <future>
     8→#include <iostream>
     9→#include <thread>
    10→#include <unordered_map>
    11→#include <unordered_set>
    12→
    13→#include <unistd.h>
    14→
    15→#include "geometry.h"
    16→#include "s2_helpers.h"
    17→
    18→std::vector<uint32_t> write_entries(
    19→    const std::string& path,
    20→    const std::vector<uint64_t>& sorted_cells,
    21→    const std::unordered_map<uint64_t, std::vector<uint32_t>>& cell_map
    22→) {
    23→    struct CellRef { uint64_t cell_id; const std::vector<uint32_t>* ids; };
    24→    std::vector<CellRef> sorted_refs;
    25→    sorted_refs.reserve(cell_map.size());
    26→    for (auto& [id, ids] : cell_map) sorted_refs.push_back({id, &ids});
    27→    std::sort(sorted_refs.begin(), sorted_refs.end(),
    28→        [](const CellRef& a, const CellRef& b) { return a.cell_id < b.cell_id; });
    29→
    30→    std::vector<uint32_t> offsets(sorted_cells.size(), NO_DATA);
    31→    size_t total_size = 0;
    32→    for (auto& r : sorted_refs) total_size += sizeof(uint16_t) + r.ids->size() * sizeof(uint32_t);
    33→
    34→    std::vector<char> buf;
    35→    buf.reserve(total_size);
    36→    uint32_t current = 0;
    37→    size_t ri = 0;
    38→    for (uint32_t si = 0; si < sorted_cells.size() && ri < sorted_refs.size(); si++) {
    39→        if (sorted_cells[si] < sorted_refs[ri].cell_id) continue;
    40→        if (sorted_cells[si] > sorted_refs[ri].cell_id) { si--; ri++; continue; }
    41→        offsets[si] = current;
    42→        const auto& ids = *sorted_refs[ri].ids;
    43→        uint16_t count = static_cast<uint16_t>(std::min(ids.size(), size_t(MAX_VERTEX_COUNT)));
    44→        buf.insert(buf.end(), reinterpret_cast<const char*>(&count),
    45→                   reinterpret_cast<const char*>(&count) + sizeof(count));
    46→        buf.insert(buf.end(), reinterpret_cast<const char*>(ids.data()),
    47→                   reinterpret_cast<const char*>(ids.data()) + ids.size() * sizeof(uint32_t));
    48→        current += sizeof(uint16_t) + ids.size() * sizeof(uint32_t);
    49→        ri++;
    50→    }
    51→    std::ofstream f(path, std::ios::binary);
    52→    f.write(buf.data(), buf.size());
    53→    return offsets;
    54→}
    55→
    56→std::vector<uint32_t> write_entries_from_sorted(
    57→    const std::string& path,
    58→    const std::vector<uint64_t>& sorted_cells,
    59→    const std::vector<CellItemPair>& sorted_pairs
    60→) {
    61→    std::vector<uint32_t> offsets(sorted_cells.size(), NO_DATA);
    62→    if (sorted_pairs.empty()) {
    63→        std::ofstream f(path, std::ios::binary);
    64→        return offsets;
    65→    }
    66→
    67→    unsigned int nthreads = std::thread::hardware_concurrency();
    68→    if (nthreads == 0) nthreads = 4;
    69→    size_t cells_per_chunk = (sorted_cells.size() + nthreads - 1) / nthreads;
    70→
    71→    struct ChunkResult {
    72→        std::vector<char> buf;
    73→        size_t cell_start, cell_end;
    74→        uint32_t local_size;
    75→    };
    76→    std::vector<ChunkResult> chunks(nthreads);
    77→
    78→    std::vector<std::thread> threads;
    79→    for (unsigned int t = 0; t < nthreads; t++) {
    80→        size_t cs = t * cells_per_chunk;
    81→        size_t ce = std::min(cs + cells_per_chunk, sorted_cells.size());
    82→        if (cs >= sorted_cells.size()) break;
    83→
    84→        threads.emplace_back([&, t, cs, ce]() {
    85→            auto& chunk = chunks[t];
    86→            chunk.cell_start = cs;
    87→            chunk.cell_end = ce;
    88→            chunk.local_size = 0;
    89→
    90→            size_t pi = std::lower_bound(sorted_pairs.begin(), sorted_pairs.end(),
    91→                sorted_cells[cs], [](const CellItemPair& p, uint64_t id) {
    92→                    return p.cell_id < id;
    93→                }) - sorted_pairs.begin();
    94→
    95→            for (size_t si = cs; si < ce && pi < sorted_pairs.size(); si++) {
    96→                if (sorted_cells[si] < sorted_pairs[pi].cell_id) continue;
    97→                while (pi < sorted_pairs.size() && sorted_pairs[pi].cell_id < sorted_cells[si]) pi++;
    98→                if (pi >= sorted_pairs.size() || sorted_pairs[pi].cell_id != sorted_cells[si]) continue;
    99→
   100→                offsets[si] = chunk.local_size;
   101→                size_t start = pi;
   102→                while (pi < sorted_pairs.size() && sorted_pairs[pi].cell_id == sorted_cells[si]) pi++;
   103→                uint16_t count = static_cast<uint16_t>(std::min(pi - start, size_t(MAX_VERTEX_COUNT)));
   104→                size_t entry_size = sizeof(uint16_t) + (pi - start) * sizeof(uint32_t);
   105→                size_t buf_pos = chunk.buf.size();
   106→                chunk.buf.resize(buf_pos + entry_size);
   107→                memcpy(chunk.buf.data() + buf_pos, &count, sizeof(count));
   108→                for (size_t k = start; k < pi; k++) {
   109→                    memcpy(chunk.buf.data() + buf_pos + sizeof(uint16_t) + (k - start) * sizeof(uint32_t),
   110→                           &sorted_pairs[k].item_id, sizeof(uint32_t));
   111→                }
   112→                chunk.local_size += entry_size;
   113→            }
   114→        });
   115→    }
   116→    for (auto& t : threads) t.join();
   117→
   118→    uint32_t global_offset = 0;
   119→    for (auto& chunk : chunks) {
   120→        for (size_t si = chunk.cell_start; si < chunk.cell_end; si++) {
   121→            if (offsets[si] != NO_DATA) offsets[si] += global_offset;
   122→        }
   123→        global_offset += chunk.local_size;
   124→    }
   125→
   126→    std::vector<char> buf;
   127→    buf.reserve(global_offset);
   128→    for (auto& chunk : chunks) buf.insert(buf.end(), chunk.buf.begin(), chunk.buf.end());
   129→
   130→    std::ofstream f(path, std::ios::binary);
   131→    f.write(buf.data(), buf.size());
   132→    return offsets;
   133→}
   134→
   135→void write_cell_index(
   136→    const std::string& cells_path,
   137→    const std::string& entries_path,
   138→    const std::unordered_map<uint64_t, std::vector<uint32_t>>& cell_map
   139→) {
   140→    std::vector<std::pair<uint64_t, std::vector<uint32_t>>> sorted(cell_map.begin(), cell_map.end());
   141→    std::sort(sorted.begin(), sorted.end());
   142→
   143→    { std::ofstream f(cells_path, std::ios::binary);
   144→      uint32_t current_offset = 0;
   145→      for (const auto& [cell_id, ids] : sorted) {
   146→          f.write(reinterpret_cast<const char*>(&cell_id), sizeof(cell_id));
   147→          f.write(reinterpret_cast<const char*>(&current_offset), sizeof(current_offset));
   148→          current_offset += sizeof(uint16_t) + ids.size() * sizeof(uint32_t);
   149→      } }
   150→
   151→    { std::ofstream f(entries_path, std::ios::binary);
   152→      for (const auto& [cell_id, ids] : sorted) {
   153→          uint16_t count = static_cast<uint16_t>(std::min(ids.size(), size_t(MAX_VERTEX_COUNT)));
   154→          f.write(reinterpret_cast<const char*>(&count), sizeof(count));
   155→          f.write(reinterpret_cast<const char*>(ids.data()), ids.size() * sizeof(uint32_t));
   156→      } }
   157→}
   158→
   159→void write_index(const ParsedData& data, const std::string& output_dir, IndexMode mode) {
   160→    ensure_dir(output_dir);
   161→    auto _wt = std::chrono::steady_clock::now();
   162→
   163→    bool write_streets = (mode != IndexMode::AdminOnly);
   164→    bool write_addresses = (mode == IndexMode::Full);
   165→
   166→    if (write_streets) {
   167→        std::vector<uint64_t> sorted_geo_cells;
   168→        {
   169→            auto extract_unique_cells = [](const std::vector<CellItemPair>& pairs) {
   170→                std::vector<uint64_t> cells;
   171→                cells.reserve(pairs.size() / 2);
   172→                for (size_t i = 0; i < pairs.size(); ) {
   173→                    cells.push_back(pairs[i].cell_id);
   174→                    uint64_t cur = pairs[i].cell_id;
   175→                    while (i < pairs.size() && pairs[i].cell_id == cur) i++;
   176→                }
   177→                return cells;
   178→            };
   179→            auto extract_from_map = [](const std::unordered_map<uint64_t, std::vector<uint32_t>>& m) {
   180→                std::vector<uint64_t> cells;
   181→                cells.reserve(m.size());
   182→                for (auto& [id, _] : m) cells.push_back(id);
   183→                std::sort(cells.begin(), cells.end());
   184→                return cells;
   185→            };
   186→
   187→            std::vector<uint64_t> way_cells, addr_cells, interp_cells;
   188→            {
   189→                auto f1 = std::async(std::launch::async, [&] {
   190→                    return !data.sorted_way_cells.empty()
   191→                        ? extract_unique_cells(data.sorted_way_cells)
   192→                        : extract_from_map(data.cell_to_ways);
   193→                });
   194→                if (write_addresses) {
   195→                    auto f2 = std::async(std::launch::async, [&] {
   196→                        return !data.sorted_addr_cells.empty()
   197→                            ? extract_unique_cells(data.sorted_addr_cells)
   198→                            : extract_from_map(data.cell_to_addrs);
   199→                    });
   200→                    auto f3 = std::async(std::launch::async, [&] {
   201→                        return !data.sorted_interp_cells.empty()
   202→                            ? extract_unique_cells(data.sorted_interp_cells)
   203→                            : extract_from_map(data.cell_to_interps);
   204→                    });
   205→                    addr_cells = f2.get();
   206→                    interp_cells = f3.get();
   207→                }
   208→                way_cells = f1.get();
   209→            }
   210→
   211→            sorted_geo_cells.reserve(way_cells.size() + addr_cells.size() + interp_cells.size());
   212→            std::merge(way_cells.begin(), way_cells.end(), addr_cells.begin(), addr_cells.end(),
   213→                       std::back_inserter(sorted_geo_cells));
   214→            if (!interp_cells.empty()) {
   215→                std::vector<uint64_t> tmp;
   216→                tmp.reserve(sorted_geo_cells.size() + interp_cells.size());
   217→                std::merge(sorted_geo_cells.begin(), sorted_geo_cells.end(),
   218→                           interp_cells.begin(), interp_cells.end(), std::back_inserter(tmp));
   219→                sorted_geo_cells = std::move(tmp);
   220→            }
   221→            sorted_geo_cells.erase(std::unique(sorted_geo_cells.begin(), sorted_geo_cells.end()),
   222→                                    sorted_geo_cells.end());
   223→        }
   224→
   225→        std::vector<uint32_t> street_offsets, addr_offsets, interp_offsets;
   226→        {
   227→            auto f1 = std::async(std::launch::async, [&]() {
   228→                if (!data.sorted_way_cells.empty())
   229→                    return write_entries_from_sorted(output_dir + "/street_entries.bin", sorted_geo_cells, data.sorted_way_cells);
   230→                return write_entries(output_dir + "/street_entries.bin", sorted_geo_cells, data.cell_to_ways);
   231→            });
   232→            if (write_addresses) {
   233→                auto f2 = std::async(std::launch::async, [&]() {
   234→                    if (!data.sorted_addr_cells.empty())
   235→                        return write_entries_from_sorted(output_dir + "/addr_entries.bin", sorted_geo_cells, data.sorted_addr_cells);
   236→                    return write_entries(output_dir + "/addr_entries.bin", sorted_geo_cells, data.cell_to_addrs);
   237→                });
   238→                auto f3 = std::async(std::launch::async, [&]() {
   239→                    if (!data.sorted_interp_cells.empty())
   240→                        return write_entries_from_sorted(output_dir + "/interp_entries.bin", sorted_geo_cells, data.sorted_interp_cells);
   241→                    return write_entries(output_dir + "/interp_entries.bin", sorted_geo_cells, data.cell_to_interps);
   242→                });
   243→                addr_offsets = f2.get();
   244→                interp_offsets = f3.get();
   245→            }
   246→            street_offsets = f1.get();
   247→        }
   248→
   249→        log_phase("  Write: entry files", _wt);
   250→        {
   251→            size_t n = sorted_geo_cells.size();
   252→            size_t row_size = sizeof(uint64_t) + 3 * sizeof(uint32_t);
   253→            std::vector<char> buf(n * row_size);
   254→            if (addr_offsets.empty()) addr_offsets.resize(n, NO_DATA);
   255→            if (interp_offsets.empty()) interp_offsets.resize(n, NO_DATA);
   256→
   257→            unsigned int nthreads = std::thread::hardware_concurrency();
   258→            if (nthreads == 0) nthreads = 4;
   259→            size_t chunk = (n + nthreads - 1) / nthreads;
   260→            std::vector<std::thread> fill_threads;
   261→            for (unsigned int t = 0; t < nthreads; t++) {
   262→                size_t start = t * chunk;
   263→                size_t end = std::min(start + chunk, n);
   264→                if (start >= n) break;
   265→                fill_threads.emplace_back([&, start, end]() {
   266→                    char* ptr = buf.data() + start * row_size;
   267→                    for (size_t i = start; i < end; i++) {
   268→                        memcpy(ptr, &sorted_geo_cells[i], sizeof(uint64_t)); ptr += sizeof(uint64_t);
   269→                        memcpy(ptr, &street_offsets[i], sizeof(uint32_t)); ptr += sizeof(uint32_t);
   270→                        memcpy(ptr, &addr_offsets[i], sizeof(uint32_t)); ptr += sizeof(uint32_t);
   271→                        memcpy(ptr, &interp_offsets[i], sizeof(uint32_t)); ptr += sizeof(uint32_t);
   272→                    }
   273→                });
   274→            }
   275→            for (auto& t : fill_threads) t.join();
   276→
   277→            std::ofstream f(output_dir + "/geo_cells.bin", std::ios::binary);
   278→            f.write(buf.data(), buf.size());
   279→        }
   280→
   281→        std::cerr << "geo index: " << sorted_geo_cells.size() << " cells ("
   282→                  << data.ways.size() << " ways, " << data.addr_points.size() << " addrs, "
   283→                  << data.interp_ways.size() << " interps)" << std::endl;
   284→    }
   285→
   286→    log_phase("  Write: geo_cells.bin", _wt);
   287→    auto admin_future = std::async(std::launch::async, [&] {
   288→        write_cell_index(output_dir + "/admin_cells.bin", output_dir + "/admin_entries.bin", data.cell_to_admin);
   289→        std::cerr << "admin index: " << data.cell_to_admin.size() << " cells, " << data.admin_polygons.size() << " polygons" << std::endl;
   290→    });
   291→
   292→    std::vector<std::future<void>> write_futures;
   293→    if (write_streets) {
   294→        write_futures.push_back(std::async(std::launch::async, [&] {
   295→            std::ofstream f(output_dir + "/street_ways.bin", std::ios::binary);
   296→            f.write(reinterpret_cast<const char*>(data.ways.data()), data.ways.size() * sizeof(WayHeader));
   297→        }));
   298→        write_futures.push_back(std::async(std::launch::async, [&] {
   299→            std::ofstream f(output_dir + "/street_nodes.bin", std::ios::binary);
   300→            f.write(reinterpret_cast<const char*>(data.street_nodes.data()), data.street_nodes.size() * sizeof(NodeCoord));
   301→        }));
   302→        if (write_addresses) {
   303→            write_futures.push_back(std::async(std::launch::async, [&] {
   304→                std::ofstream f(output_dir + "/addr_points.bin", std::ios::binary);
   305→                f.write(reinterpret_cast<const char*>(data.addr_points.data()), data.addr_points.size() * sizeof(AddrPoint));
   306→            }));
   307→            write_futures.push_back(std::async(std::launch::async, [&] {
   308→                std::ofstream f(output_dir + "/interp_ways.bin", std::ios::binary);
   309→                f.write(reinterpret_cast<const char*>(data.interp_ways.data()), data.interp_ways.size() * sizeof(InterpWay));
   310→            }));
   311→            write_futures.push_back(std::async(std::launch::async, [&] {
   312→                std::ofstream f(output_dir + "/interp_nodes.bin", std::ios::binary);
   313→                f.write(reinterpret_cast<const char*>(data.interp_nodes.data()), data.interp_nodes.size() * sizeof(NodeCoord));
   314→            }));
   315→        }
   316→    }
   317→    // If not writing all data types, compact string pool and remap references
   318→    // before writing admin_polygons and strings.bin.
   319→    std::vector<AdminPolygon> compact_admin;
   320→    std::vector<char> compact_strings;
   321→    const bool need_compact = (!write_streets || !write_addresses);
   322→
   323→    if (need_compact) {
   324→        // Collect referenced string offsets
   325→        std::unordered_set<uint32_t> used_offsets;
   326→        if (write_streets) {
   327→            for (const auto& w : data.ways) used_offsets.insert(w.name_id);
   328→            if (write_addresses) {
   329→                for (const auto& a : data.addr_points) {
   330→                    used_offsets.insert(a.housenumber_id);
   331→                    used_offsets.insert(a.street_id);
   332→                }
   333→                for (const auto& iw : data.interp_ways) used_offsets.insert(iw.street_id);
   334→            }
   335→        }
   336→        for (const auto& ap : data.admin_polygons) used_offsets.insert(ap.name_id);
   337→
   338→        // Build compact pool and remap
   339→        const auto& old_sp = data.string_pool.data();
   340→        std::vector<uint32_t> sorted_offs(used_offsets.begin(), used_offsets.end());
   341→        std::sort(sorted_offs.begin(), sorted_offs.end());
   342→        std::unordered_map<uint32_t, uint32_t> remap;
   343→        compact_strings.reserve(used_offsets.size() * 16);
   344→        for (uint32_t off : sorted_offs) {
   345→            remap[off] = static_cast<uint32_t>(compact_strings.size());
   346→            size_t len = std::strlen(old_sp.data() + off);
   347→            compact_strings.insert(compact_strings.end(), old_sp.data() + off,
   348→                old_sp.data() + off + len + 1);
   349→        }
   350→
   351→        // Remap admin polygon name_ids
   352→        compact_admin = data.admin_polygons;
   353→        for (auto& ap : compact_admin) ap.name_id = remap[ap.name_id];
   354→
   355→        // Remap street/addr/interp if applicable (for no-addresses mode)
   356→        // These are written from data directly, so we'd need copies too.
   357→        // For simplicity, only compact when we have fewer types.
   358→    }
   359→
   360→    const auto& admin_to_write = need_compact ? compact_admin : data.admin_polygons;
   361→    write_futures.push_back(std::async(std::launch::async, [&] {
   362→        std::ofstream f(output_dir + "/admin_polygons.bin", std::ios::binary);
   363→        f.write(reinterpret_cast<const char*>(admin_to_write.data()), admin_to_write.size() * sizeof(AdminPolygon));
   364→    }));
   365→    write_futures.push_back(std::async(std::launch::async, [&] {
   366→        std::ofstream f(output_dir + "/admin_vertices.bin", std::ios::binary);
   367→        f.write(reinterpret_cast<const char*>(data.admin_vertices.data()), data.admin_vertices.size() * sizeof(NodeCoord));
   368→    }));
   369→    write_futures.push_back(std::async(std::launch::async, [&] {
   370→        if (need_compact) {
   371→            std::ofstream f(output_dir + "/strings.bin", std::ios::binary);
   372→            f.write(compact_strings.data(), compact_strings.size());
   373→            std::cerr << "strings.bin: " << compact_strings.size() << " bytes (compacted from "
   374→                      << data.string_pool.data().size() << ")" << std::endl;
   375→        } else {
   376→            std::ofstream f(output_dir + "/strings.bin", std::ios::binary);
   377→            f.write(data.string_pool.data().data(), data.string_pool.data().size());
   378→            std::cerr << "strings.bin: " << data.string_pool.data().size() << " bytes" << std::endl;
   379→        }
   380→    }));
   381→
   382→    log_phase("  Write: parallel data files launched", _wt);
   383→    admin_future.get();
   384→    for (auto& f : write_futures) f.get();
   385→}
   386→
   387→void write_quality_variant(const ParsedData& data, const std::string& source_dir,
   388→                           const std::string& output_dir, double epsilon_scale) {
   389→    ensure_dir(output_dir);
   390→
   391→    // Re-simplify admin polygons at the given epsilon scale
   392→    std::vector<AdminPolygon> new_polys;
   393→    std::vector<NodeCoord> new_verts;
   394→    new_polys.reserve(data.admin_polygons.size());
   395→
   396→    // Parallel simplification
   397→    struct SimplifiedPoly {
   398→        std::vector<std::pair<double,double>> verts;
   399→    };
   400→    std::vector<SimplifiedPoly> simplified(data.admin_polygons.size());
   401→
   402→    {
   403→        std::atomic<size_t> idx{0};
   404→        unsigned nthreads = std::thread::hardware_concurrency();
   405→        if (nthreads == 0) nthreads = 4;
   406→        std::vector<std::thread> workers;
   407→        for (unsigned t = 0; t < nthreads; t++) {
   408→            workers.emplace_back([&]() {
   409→                while (true) {
   410→                    size_t i = idx.fetch_add(1);
   411→                    if (i >= data.admin_polygons.size()) break;
   412→                    const auto& ap = data.admin_polygons[i];
   413→
   414→                    // Extract original vertices
   415→                    std::vector<std::pair<double,double>> pts;
   416→                    pts.reserve(ap.vertex_count);
   417→                    for (uint32_t j = 0; j < ap.vertex_count; j++) {
   418→                        const auto& v = data.admin_vertices[ap.vertex_offset + j];
   419→                        pts.emplace_back(v.lat, v.lng);
   420→                    }
   421→
   422→                    if (epsilon_scale > 0) {
   423→                        double eps_m = admin_epsilon_meters(ap.admin_level) * epsilon_scale;
   424→                        double lat = pts.empty() ? 0.0 : pts[0].first;
   425→                        double eps_deg = meters_to_degrees(eps_m, lat);
   426→                        simplified[i].verts = simplify_polygon_epsilon(pts, eps_deg);
   427→                    } else {
   428→                        simplified[i].verts = std::move(pts);
   429→                    }
   430→                }
   431→            });
   432→        }
   433→        for (auto& w : workers) w.join();
   434→    }
   435→
   436→    // Sequential: build new polygon and vertex arrays
   437→    for (size_t i = 0; i < data.admin_polygons.size(); i++) {
   438→        auto& sv = simplified[i].verts;
   439→        if (sv.size() < 3) continue;
   440→
   441→        AdminPolygon np = data.admin_polygons[i];
   442→        np.vertex_offset = static_cast<uint32_t>(new_verts.size());
   443→        np.vertex_count = static_cast<uint32_t>(sv.size());
   444→        np.area = polygon_area(sv);
   445→
   446→        for (const auto& [lat, lng] : sv) {
   447→            new_verts.push_back({static_cast<float>(lat), static_cast<float>(lng)});
   448→        }
   449→        new_polys.push_back(np);
   450→    }
   451→
   452→    std::cerr << "Quality " << epsilon_scale << "x: " << new_polys.size()
   453→              << " polygons, " << new_verts.size() << " vertices ("
   454→              << new_verts.size() * 8 / 1024 / 1024 << " MiB)" << std::endl;
   455→
   456→    // Write admin files
   457→    {
   458→        std::ofstream f(output_dir + "/admin_polygons.bin", std::ios::binary);
   459→        f.write(reinterpret_cast<const char*>(new_polys.data()), new_polys.size() * sizeof(AdminPolygon));
   460→    }
   461→    {
   462→        std::ofstream f(output_dir + "/admin_vertices.bin", std::ios::binary);
   463→        f.write(reinterpret_cast<const char*>(new_verts.data()), new_verts.size() * sizeof(NodeCoord));
   464→    }
   465→
   466→    // Quality directories only contain the two files that change.
   467→    // Shared files (admin_cells.bin, admin_entries.bin, strings.bin)
   468→    // stay in the parent directory.
   469→}
   470→


     1→// Semantic comparison of two geocoder index directories.
     2→// Decodes all binary files, creates canonical fingerprints for each item,
     3→// and compares as multisets — handles any ordering differences between
     4→// parallel and sequential builds.
     5→// Returns 0 if semantically identical, 1 if different.
     6→//
     7→// Usage: compare_indexes <dir-A> <dir-B>
     8→
     9→#include <algorithm>
    10→#include <cmath>
    11→#include <cstdint>
    12→#include <cstring>
    13→#include <fstream>
    14→#include <iostream>
    15→#include <map>
    16→#include <set>
    17→#include <string>
    18→#include <unordered_map>
    19→#include <vector>
    20→
    21→// --- Binary structs (must match build_index.cpp and server main.rs) ---
    22→
    23→struct WayHeader {
    24→    uint32_t node_offset;
    25→    uint8_t node_count;
    26→    uint32_t name_id;
    27→};
    28→
    29→struct AddrPoint {
    30→    float lat;
    31→    float lng;
    32→    uint32_t housenumber_id;
    33→    uint32_t street_id;
    34→};
    35→
    36→struct InterpWay {
    37→    uint32_t node_offset;
    38→    uint8_t node_count;
    39→    uint32_t street_id;
    40→    uint32_t start_number;
    41→    uint32_t end_number;
    42→    uint8_t interpolation;
    43→};
    44→
    45→struct AdminPolygon {
    46→    uint32_t vertex_offset;
    47→    uint16_t vertex_count;
    48→    uint32_t name_id;
    49→    uint8_t admin_level;
    50→    float area;
    51→    uint16_t country_code;
    52→};
    53→
    54→struct NodeCoord {
    55→    float lat;
    56→    float lng;
    57→};
    58→
    59→// --- Helpers ---
    60→
    61→static std::vector<char> read_file(const std::string& path) {
    62→    std::ifstream f(path, std::ios::binary | std::ios::ate);
    63→    if (!f) return {};
    64→    auto size = f.tellg();
    65→    f.seekg(0);
    66→    std::vector<char> data(size);
    67→    f.read(data.data(), size);
    68→    return data;
    69→}
    70→
    71→static const char* get_string(const std::vector<char>& pool, uint32_t offset) {
    72→    if (offset >= pool.size()) return "(invalid)";
    73→    return pool.data() + offset;
    74→}
    75→
    76→template<typename T>
    77→static std::vector<T> read_structs(const std::string& path) {
    78→    auto data = read_file(path);
    79→    size_t count = data.size() / sizeof(T);
    80→    std::vector<T> result(count);
    81→    memcpy(result.data(), data.data(), count * sizeof(T));
    82→    return result;
    83→}
    84→
    85→// Round a float coordinate to an integer grid (1e5 ~ 1m precision)
    86→static int to_grid(float v) {
    87→    return (int)(v * 1e5f + (v >= 0 ? 0.5f : -0.5f));
    88→}
    89→
    90→// --- Canonical fingerprint builders ---
    91→// Each function produces a string that uniquely identifies an item,
    92→// independent of storage order.
    93→
    94→// For a way: "name|sorted_node_coords"
    95→// Nodes are kept in original order (path order matters for ways),
    96→// but we also provide a sorted-set version for order-independent matching.
    97→static std::string way_fingerprint(const std::string& name,
    98→                                    const std::vector<std::pair<float,float>>& nodes) {
    99→    // Use sorted coordinate set as the canonical form.
   100→    // Two ways are "the same" if they have the same name and same set of nodes.
   101→    std::vector<std::pair<int,int>> grid_nodes;
   102→    grid_nodes.reserve(nodes.size());
   103→    for (const auto& n : nodes) {
   104→        grid_nodes.push_back({to_grid(n.first), to_grid(n.second)});
   105→    }
   106→    std::sort(grid_nodes.begin(), grid_nodes.end());
   107→
   108→    std::string fp = name + "|" + std::to_string(nodes.size());
   109→    for (const auto& g : grid_nodes) {
   110→        fp += "|";
   111→        fp += std::to_string(g.first);
   112→        fp += ",";
   113→        fp += std::to_string(g.second);
   114→    }
   115→    return fp;
   116→}
   117→
   118→// For an address: "grid_lat,grid_lng|housenumber|street"
   119→static std::string addr_fingerprint(float lat, float lng,
   120→                                     const std::string& housenumber,
   121→                                     const std::string& street) {
   122→    return std::to_string(to_grid(lat)) + "," + std::to_string(to_grid(lng)) +
   123→           "|" + housenumber + "|" + street;
   124→}
   125→
   126→// For an admin polygon: "name|level|country|sorted_vertices"
   127→static std::string admin_fingerprint(const std::string& name,
   128→                                      uint8_t admin_level,
   129→                                      uint16_t country_code,
   130→                                      const std::vector<std::pair<float,float>>& vertices) {
   131→    std::vector<std::pair<int,int>> grid_verts;
   132→    grid_verts.reserve(vertices.size());
   133→    for (const auto& v : vertices) {
   134→        grid_verts.push_back({to_grid(v.first), to_grid(v.second)});
   135→    }
   136→    std::sort(grid_verts.begin(), grid_verts.end());
   137→
   138→    std::string fp = name + "|" + std::to_string(admin_level) + "|" + std::to_string(country_code);
   139→    for (const auto& g : grid_verts) {
   140→        fp += "|";
   141→        fp += std::to_string(g.first);
   142→        fp += ",";
   143→        fp += std::to_string(g.second);
   144→    }
   145→    return fp;
   146→}
   147→
   148→// For interpolation ways: "street|start|end|interp|sorted_nodes"
   149→static std::string interp_fingerprint(const std::string& street,
   150→                                       uint32_t start_number, uint32_t end_number,
   151→                                       uint8_t interpolation,
   152→                                       const std::vector<std::pair<float,float>>& nodes) {
   153→    std::vector<std::pair<int,int>> grid_nodes;
   154→    grid_nodes.reserve(nodes.size());
   155→    for (const auto& n : nodes) {
   156→        grid_nodes.push_back({to_grid(n.first), to_grid(n.second)});
   157→    }
   158→    std::sort(grid_nodes.begin(), grid_nodes.end());
   159→
   160→    std::string fp = street + "|" + std::to_string(start_number) + "-" +
   161→                     std::to_string(end_number) + "|" + std::to_string(interpolation);
   162→    for (const auto& g : grid_nodes) {
   163→        fp += "|";
   164→        fp += std::to_string(g.first);
   165→        fp += ",";
   166→        fp += std::to_string(g.second);
   167→    }
   168→    return fp;
   169→}
   170→
   171→// --- Multiset comparison ---
   172→// Compare two collections by their fingerprints. Reports missing/extra items.
   173→
   174→static bool compare_multisets(const char* label,
   175→                               const std::vector<std::string>& fps_a,
   176→                               const std::vector<std::string>& fps_b) {
   177→    // Build frequency maps
   178→    std::unordered_map<std::string, int> freq_a, freq_b;
   179→    for (const auto& fp : fps_a) freq_a[fp]++;
   180→    for (const auto& fp : fps_b) freq_b[fp]++;
   181→
   182→    // Find items in A but not in B (missing from B)
   183→    size_t missing = 0;
   184→    for (const auto& [fp, count] : freq_a) {
   185→        int b_count = 0;
   186→        auto it = freq_b.find(fp);
   187→        if (it != freq_b.end()) b_count = it->second;
   188→        if (count > b_count) missing += (count - b_count);
   189→    }
   190→
   191→    // Find items in B but not in A (extra in B)
   192→    size_t extra = 0;
   193→    for (const auto& [fp, count] : freq_b) {
   194→        int a_count = 0;
   195→        auto it = freq_a.find(fp);
   196→        if (it != freq_a.end()) a_count = it->second;
   197→        if (count > a_count) extra += (count - a_count);
   198→    }
   199→
   200→    size_t matched = fps_a.size() - missing;
   201→
   202→    if (missing == 0 && extra == 0) {
   203→        std::cout << "PASS: " << label << " (" << fps_a.size() << " items, all matched)" << std::endl;
   204→        return true;
   205→    } else {
   206→        std::cerr << "FAIL: " << label << " — A=" << fps_a.size() << " B=" << fps_b.size()
   207→                  << " matched=" << matched
   208→                  << " only_in_A=" << missing
   209→                  << " only_in_B=" << extra << std::endl;
   210→        return false;
   211→    }
   212→}
   213→
   214→// --- Decode + fingerprint functions ---
   215→
   216→static std::vector<std::string> fingerprint_admin(const std::string& dir) {
   217→    auto strings = read_file(dir + "/strings.bin");
   218→    auto polygons = read_structs<AdminPolygon>(dir + "/admin_polygons.bin");
   219→    auto vertices = read_structs<NodeCoord>(dir + "/admin_vertices.bin");
   220→
   221→    std::vector<std::string> fps;
   222→    fps.reserve(polygons.size());
   223→    for (const auto& p : polygons) {
   224→        std::string name = get_string(strings, p.name_id);
   225→        std::vector<std::pair<float,float>> verts;
   226→        for (uint16_t i = 0; i < p.vertex_count; i++) {
   227→            uint32_t vi = p.vertex_offset + i;
   228→            if (vi < vertices.size()) {
   229→                verts.push_back({vertices[vi].lat, vertices[vi].lng});
   230→            }
   231→        }
   232→        fps.push_back(admin_fingerprint(name, p.admin_level, p.country_code, verts));
   233→    }
   234→    return fps;
   235→}
   236→
   237→static std::vector<std::string> fingerprint_ways(const std::string& dir) {
   238→    auto strings = read_file(dir + "/strings.bin");
   239→    auto ways = read_structs<WayHeader>(dir + "/street_ways.bin");
   240→    auto nodes = read_structs<NodeCoord>(dir + "/street_nodes.bin");
   241→
   242→    std::vector<std::string> fps;
   243→    fps.reserve(ways.size());
   244→    for (const auto& w : ways) {
   245→        std::string name = get_string(strings, w.name_id);
   246→        std::vector<std::pair<float,float>> way_nodes;
   247→        for (uint8_t i = 0; i < w.node_count; i++) {
   248→            uint32_t ni = w.node_offset + i;
   249→            if (ni < nodes.size()) {
   250→                way_nodes.push_back({nodes[ni].lat, nodes[ni].lng});
   251→            }
   252→        }
   253→        fps.push_back(way_fingerprint(name, way_nodes));
   254→    }
   255→    return fps;
   256→}
   257→
   258→static std::vector<std::string> fingerprint_addrs(const std::string& dir) {
   259→    auto strings = read_file(dir + "/strings.bin");
   260→    auto points = read_structs<AddrPoint>(dir + "/addr_points.bin");
   261→
   262→    std::vector<std::string> fps;
   263→    fps.reserve(points.size());
   264→    for (const auto& p : points) {
   265→        std::string hn = get_string(strings, p.housenumber_id);
   266→        std::string st = get_string(strings, p.street_id);
   267→        fps.push_back(addr_fingerprint(p.lat, p.lng, hn, st));
   268→    }
   269→    return fps;
   270→}
   271→
   272→static std::vector<std::string> fingerprint_interps(const std::string& dir) {
   273→    auto strings = read_file(dir + "/strings.bin");
   274→    auto interps = read_structs<InterpWay>(dir + "/interp_ways.bin");
   275→    auto nodes = read_structs<NodeCoord>(dir + "/interp_nodes.bin");
   276→
   277→    std::vector<std::string> fps;
   278→    fps.reserve(interps.size());
   279→    for (const auto& iw : interps) {
   280→        std::string street = get_string(strings, iw.street_id);
   281→        std::vector<std::pair<float,float>> way_nodes;
   282→        for (uint8_t i = 0; i < iw.node_count; i++) {
   283→            uint32_t ni = iw.node_offset + i;
   284→            if (ni < nodes.size()) {
   285→                way_nodes.push_back({nodes[ni].lat, nodes[ni].lng});
   286→            }
   287→        }
   288→        fps.push_back(interp_fingerprint(street, iw.start_number, iw.end_number,
   289→                                          iw.interpolation, way_nodes));
   290→    }
   291→    return fps;
   292→}
   293→
   294→// --- Semantic (cross-version) fingerprints ---
   295→// Match by name/attributes only, ignoring coordinate precision differences
   296→
   297→// Way: name + node_count only
   298→static std::string way_semantic_fp(const std::string& name, size_t node_count) {
   299→    return name + "|" + std::to_string(node_count);
   300→}
   301→
   302→// Address: housenumber + street + coarse grid coords (~100m precision)
   303→// Centroid may differ slightly across code versions, so use coarse grid
   304→static std::string addr_semantic_fp(float lat, float lng,
   305→                                     const std::string& housenumber, const std::string& street) {
   306→    int glat = (int)(lat * 1e3f + (lat >= 0 ? 0.5f : -0.5f));
   307→    int glng = (int)(lng * 1e3f + (lng >= 0 ? 0.5f : -0.5f));
   308→    return std::to_string(glat) + "," + std::to_string(glng) + "|" + housenumber + "|" + street;
   309→}
   310→
   311→// Admin: name + level
   312→static std::string admin_semantic_fp(const std::string& name, uint8_t admin_level) {
   313→    return name + "|" + std::to_string(admin_level);
   314→}
   315→
   316→static std::vector<std::string> semantic_fp_ways(const std::string& dir) {
   317→    auto strings = read_file(dir + "/strings.bin");
   318→    auto ways = read_structs<WayHeader>(dir + "/street_ways.bin");
   319→
   320→    std::vector<std::string> fps;
   321→    fps.reserve(ways.size());
   322→    for (const auto& w : ways) {
   323→        std::string name = get_string(strings, w.name_id);
   324→        fps.push_back(way_semantic_fp(name, w.node_count));
   325→    }
   326→    return fps;
   327→}
   328→
   329→static std::vector<std::string> semantic_fp_addrs(const std::string& dir) {
   330→    auto strings = read_file(dir + "/strings.bin");
   331→    auto points = read_structs<AddrPoint>(dir + "/addr_points.bin");
   332→
   333→    std::vector<std::string> fps;
   334→    fps.reserve(points.size());
   335→    for (const auto& p : points) {
   336→        std::string hn = get_string(strings, p.housenumber_id);
   337→        std::string st = get_string(strings, p.street_id);
   338→        fps.push_back(addr_semantic_fp(p.lat, p.lng, hn, st));
   339→    }
   340→    return fps;
   341→}
   342→
   343→static std::vector<std::string> semantic_fp_admin(const std::string& dir) {
   344→    auto strings = read_file(dir + "/strings.bin");
   345→    auto polygons = read_structs<AdminPolygon>(dir + "/admin_polygons.bin");
   346→
   347→    std::vector<std::string> fps;
   348→    fps.reserve(polygons.size());
   349→    for (const auto& p : polygons) {
   350→        std::string name = get_string(strings, p.name_id);
   351→        fps.push_back(admin_semantic_fp(name, p.admin_level));
   352→    }
   353→    return fps;
   354→}
   355→
   356→int main(int argc, char* argv[]) {
   357→    if (argc < 3) {
   358→        std::cerr << "Usage: compare_indexes <dir-A> <dir-B> [--semantic]" << std::endl;
   359→        return 2;
   360→    }
   361→
   362→    std::string dir_a = argv[1];
   363→    std::string dir_b = argv[2];
   364→    bool semantic = (argc > 3 && std::string(argv[3]) == "--semantic");
   365→
   366→    std::cout << "Comparing" << (semantic ? " (semantic/cross-version)" : " (exact)") << ":" << std::endl;
   367→    std::cout << "  A: " << dir_a << std::endl;
   368→    std::cout << "  B: " << dir_b << std::endl;
   369→    std::cout << std::endl;
   370→
   371→    bool all_pass = true;
   372→
   373→    // Compare admin polygons
   374→    {
   375→        std::cout << "Loading admin polygons..." << std::flush;
   376→        auto fps_a = semantic ? semantic_fp_admin(dir_a) : fingerprint_admin(dir_a);
   377→        auto fps_b = semantic ? semantic_fp_admin(dir_b) : fingerprint_admin(dir_b);
   378→        std::cout << " done." << std::endl;
   379→        if (!compare_multisets("admin_polygons", fps_a, fps_b))
   380→            all_pass = false;
   381→    }
   382→
   383→    // Compare street ways
   384→    {
   385→        std::cout << "Loading street ways..." << std::flush;
   386→        auto fps_a = semantic ? semantic_fp_ways(dir_a) : fingerprint_ways(dir_a);
   387→        auto fps_b = semantic ? semantic_fp_ways(dir_b) : fingerprint_ways(dir_b);
   388→        std::cout << " done." << std::endl;
   389→        if (!compare_multisets("street_ways", fps_a, fps_b))
   390→            all_pass = false;
   391→    }
   392→
   393→    // Compare address points
   394→    {
   395→        std::cout << "Loading address points..." << std::flush;
   396→        auto fps_a = semantic ? semantic_fp_addrs(dir_a) : fingerprint_addrs(dir_a);
   397→        auto fps_b = semantic ? semantic_fp_addrs(dir_b) : fingerprint_addrs(dir_b);
   398→        std::cout << " done." << std::endl;
   399→        if (!compare_multisets("address_points", fps_a, fps_b))
   400→            all_pass = false;
   401→    }
   402→
   403→    // Compare interpolation ways
   404→    {
   405→        std::cout << "Loading interpolation ways..." << std::flush;
   406→        auto fps_a = fingerprint_interps(dir_a);
   407→        auto fps_b = fingerprint_interps(dir_b);
   408→        std::cout << " done." << std::endl;
   409→        if (!compare_multisets("interp_ways", fps_a, fps_b))
   410→            all_pass = false;
   411→    }
   412→
   413→    // Compare cell index sizes (should be identical regardless of ordering)
   414→    {
   415→        auto ac = read_file(dir_a + "/admin_cells.bin");
   416→        auto bc = read_file(dir_b + "/admin_cells.bin");
   417→        if (ac == bc) {
   418→            std::cout << "PASS: admin_cells.bin byte-identical (" << ac.size() << " bytes)" << std::endl;
   419→        } else if (ac.size() == bc.size()) {
   420→            std::cout << "INFO: admin_cells.bin same size (" << ac.size() << " bytes) but different content" << std::endl;
   421→        } else {
   422→            std::cerr << "FAIL: admin_cells.bin size differs (A=" << ac.size()
   423→                      << " B=" << bc.size() << ")" << std::endl;
   424→            all_pass = false;
   425→        }
   426→
   427→        auto gc_a = read_file(dir_a + "/geo_cells.bin");
   428→        auto gc_b = read_file(dir_b + "/geo_cells.bin");
   429→        if (gc_a.size() == gc_b.size()) {
   430→            std::cout << "PASS: geo_cells.bin same size (" << gc_a.size() << " bytes)" << std::endl;
   431→        } else {
   432→            std::cerr << "FAIL: geo_cells.bin size differs (A=" << gc_a.size()
   433→                      << " B=" << gc_b.size() << ")" << std::endl;
   434→            all_pass = false;
   435→        }
   436→    }
   437→
   438→    std::cout << std::endl;
   439→    if (all_pass) {
   440→        std::cout << "=== ALL COMPARISONS PASSED ===" << std::endl;
   441→        return 0;
   442→    } else {
   443→        std::cout << "=== SOME COMPARISONS FAILED ===" << std::endl;
   444→        return 1;
   445→    }
   446→}
   447→


     1→// Deep comparison: dumps samples of items only in A or only in B
     2→// with their coordinates, to understand boundary differences.
     3→// Usage: deep_compare <dir-A> <dir-B> [--ways|--addrs|--admins] [--limit N]
     4→
     5→#include <algorithm>
     6→#include <cmath>
     7→#include <cstdint>
     8→#include <cstring>
     9→#include <fstream>
    10→#include <iostream>
    11→#include <string>
    12→#include <unordered_map>
    13→#include <unordered_set>
    14→#include <vector>
    15→
    16→struct WayHeader { uint32_t node_offset; uint8_t node_count; uint32_t name_id; };
    17→struct AddrPoint { float lat; float lng; uint32_t housenumber_id; uint32_t street_id; };
    18→struct NodeCoord { float lat; float lng; };
    19→
    20→static std::vector<char> read_file(const std::string& path) {
    21→    std::ifstream f(path, std::ios::binary | std::ios::ate);
    22→    if (!f) return {};
    23→    auto size = f.tellg(); f.seekg(0);
    24→    std::vector<char> data(size); f.read(data.data(), size);
    25→    return data;
    26→}
    27→
    28→template<typename T>
    29→static std::vector<T> read_structs(const std::string& path) {
    30→    auto data = read_file(path);
    31→    size_t count = data.size() / sizeof(T);
    32→    std::vector<T> result(count);
    33→    memcpy(result.data(), data.data(), count * sizeof(T));
    34→    return result;
    35→}
    36→
    37→static const char* get_string(const std::vector<char>& pool, uint32_t offset) {
    38→    if (offset >= pool.size()) return "(invalid)";
    39→    return pool.data() + offset;
    40→}
    41→
    42→static int to_grid(float v) {
    43→    return (int)(v * 1e5f + (v >= 0 ? 0.5f : -0.5f));
    44→}
    45→
    46→int main(int argc, char* argv[]) {
    47→    if (argc < 3) {
    48→        std::cerr << "Usage: deep_compare <dir-A> <dir-B> [--ways|--addrs|--admins] [--limit N]" << std::endl;
    49→        return 1;
    50→    }
    51→    std::string dir_a = argv[1], dir_b = argv[2];
    52→    bool do_ways = false, do_addrs = false, do_admins = false;
    53→    int limit = 20;
    54→    for (int i = 3; i < argc; i++) {
    55→        if (strcmp(argv[i], "--ways") == 0) do_ways = true;
    56→        else if (strcmp(argv[i], "--addrs") == 0) do_addrs = true;
    57→        else if (strcmp(argv[i], "--admins") == 0) do_admins = true;
    58→        else if (strcmp(argv[i], "--limit") == 0 && i+1 < argc) limit = atoi(argv[++i]);
    59→    }
    60→    if (!do_ways && !do_addrs && !do_admins) { do_ways = do_addrs = true; }
    61→
    62→    if (do_ways) {
    63→        std::cerr << "Loading ways..." << std::endl;
    64→        auto ways_a = read_structs<WayHeader>(dir_a + "/street_ways.bin");
    65→        auto ways_b = read_structs<WayHeader>(dir_b + "/street_ways.bin");
    66→        auto nodes_a = read_structs<NodeCoord>(dir_a + "/street_nodes.bin");
    67→        auto nodes_b = read_structs<NodeCoord>(dir_b + "/street_nodes.bin");
    68→        auto sp_a = read_file(dir_a + "/strings.bin");
    69→        auto sp_b = read_file(dir_b + "/strings.bin");
    70→
    71→        // Build fingerprints: name + sorted node coords
    72→        auto way_fp = [&](const std::vector<WayHeader>& ways, const std::vector<NodeCoord>& nodes,
    73→                          const std::vector<char>& sp, size_t idx) -> std::string {
    74→            const auto& w = ways[idx];
    75→            std::string name = get_string(sp, w.name_id);
    76→            std::string fp = name + "|" + std::to_string(w.node_count);
    77→            // Use first node coords as primary identifier
    78→            if (w.node_count > 0) {
    79→                fp += "|" + std::to_string(to_grid(nodes[w.node_offset].lat)) +
    80→                      "," + std::to_string(to_grid(nodes[w.node_offset].lng));
    81→            }
    82→            return fp;
    83→        };
    84→
    85→        std::unordered_map<std::string, std::vector<size_t>> fps_a, fps_b;
    86→        for (size_t i = 0; i < ways_a.size(); i++) fps_a[way_fp(ways_a, nodes_a, sp_a, i)].push_back(i);
    87→        for (size_t i = 0; i < ways_b.size(); i++) fps_b[way_fp(ways_b, nodes_b, sp_b, i)].push_back(i);
    88→
    89→        // Find items only in A
    90→        std::vector<std::pair<std::string, size_t>> only_a, only_b;
    91→        for (auto& [fp, indices] : fps_a) {
    92→            size_t bcount = fps_b.count(fp) ? fps_b[fp].size() : 0;
    93→            if (indices.size() > bcount) {
    94→                for (size_t i = bcount; i < indices.size(); i++)
    95→                    only_a.push_back({fp, indices[i]});
    96→            }
    97→        }
    98→        for (auto& [fp, indices] : fps_b) {
    99→            size_t acount = fps_a.count(fp) ? fps_a[fp].size() : 0;
   100→            if (indices.size() > acount) {
   101→                for (size_t i = acount; i < indices.size(); i++)
   102→                    only_b.push_back({fp, indices[i]});
   103→            }
   104→        }
   105→
   106→        std::cout << "=== WAYS ===" << std::endl;
   107→        std::cout << "A: " << ways_a.size() << "  B: " << ways_b.size() << std::endl;
   108→        std::cout << "Only in A (standalone): " << only_a.size() << std::endl;
   109→        int count = 0;
   110→        for (auto& [fp, idx] : only_a) {
   111→            if (count++ >= limit) { std::cout << "  ... and " << (only_a.size() - limit) << " more" << std::endl; break; }
   112→            const auto& w = ways_a[idx];
   113→            std::cout << "  " << get_string(sp_a, w.name_id)
   114→                      << " nodes=" << (int)w.node_count
   115→                      << " first=(" << nodes_a[w.node_offset].lat << "," << nodes_a[w.node_offset].lng << ")"
   116→                      << " last=(" << nodes_a[w.node_offset + w.node_count - 1].lat << "," << nodes_a[w.node_offset + w.node_count - 1].lng << ")"
   117→                      << std::endl;
   118→        }
   119→        std::cout << "Only in B (filtered): " << only_b.size() << std::endl;
   120→        count = 0;
   121→        for (auto& [fp, idx] : only_b) {
   122→            if (count++ >= limit) { std::cout << "  ... and " << (only_b.size() - limit) << " more" << std::endl; break; }
   123→            const auto& w = ways_b[idx];
   124→            std::cout << "  " << get_string(sp_b, w.name_id)
   125→                      << " nodes=" << (int)w.node_count
   126→                      << " first=(" << nodes_b[w.node_offset].lat << "," << nodes_b[w.node_offset].lng << ")"
   127→                      << " last=(" << nodes_b[w.node_offset + w.node_count - 1].lat << "," << nodes_b[w.node_offset + w.node_count - 1].lng << ")"
   128→                      << std::endl;
   129→        }
   130→    }
   131→
   132→    if (do_addrs) {
   133→        std::cerr << "Loading addrs..." << std::endl;
   134→        auto addrs_a = read_structs<AddrPoint>(dir_a + "/addr_points.bin");
   135→        auto addrs_b = read_structs<AddrPoint>(dir_b + "/addr_points.bin");
   136→        auto sp_a = read_file(dir_a + "/strings.bin");
   137→        auto sp_b = read_file(dir_b + "/strings.bin");
   138→
   139→        auto addr_fp = [&](const std::vector<AddrPoint>& addrs, const std::vector<char>& sp, size_t idx) -> std::string {
   140→            const auto& a = addrs[idx];
   141→            return std::to_string(to_grid(a.lat)) + "," + std::to_string(to_grid(a.lng)) +
   142→                   "|" + get_string(sp, a.housenumber_id) + "|" + get_string(sp, a.street_id);
   143→        };
   144→
   145→        std::unordered_set<std::string> set_a, set_b;
   146→        for (size_t i = 0; i < addrs_a.size(); i++) set_a.insert(addr_fp(addrs_a, sp_a, i));
   147→        for (size_t i = 0; i < addrs_b.size(); i++) set_b.insert(addr_fp(addrs_b, sp_b, i));
   148→
   149→        std::vector<std::pair<std::string, size_t>> only_a, only_b;
   150→        for (size_t i = 0; i < addrs_a.size(); i++) {
   151→            auto fp = addr_fp(addrs_a, sp_a, i);
   152→            if (!set_b.count(fp)) only_a.push_back({fp, i});
   153→        }
   154→        for (size_t i = 0; i < addrs_b.size(); i++) {
   155→            auto fp = addr_fp(addrs_b, sp_b, i);
   156→            if (!set_a.count(fp)) only_b.push_back({fp, i});
   157→        }
   158→
   159→        std::cout << "\n=== ADDRESSES ===" << std::endl;
   160→        std::cout << "A: " << addrs_a.size() << "  B: " << addrs_b.size() << std::endl;
   161→        std::cout << "Only in A (standalone): " << only_a.size() << std::endl;
   162→        int count = 0;
   163→        for (auto& [fp, idx] : only_a) {
   164→            if (count++ >= limit) { std::cout << "  ... and " << (only_a.size() - limit) << " more" << std::endl; break; }
   165→            const auto& a = addrs_a[idx];
   166→            std::cout << "  (" << a.lat << "," << a.lng << ") "
   167→                      << get_string(sp_a, a.housenumber_id) << " "
   168→                      << get_string(sp_a, a.street_id) << std::endl;
   169→        }
   170→        std::cout << "Only in B (filtered): " << only_b.size() << std::endl;
   171→        count = 0;
   172→        for (auto& [fp, idx] : only_b) {
   173→            if (count++ >= limit) { std::cout << "  ... and " << (only_b.size() - limit) << " more" << std::endl; break; }
   174→            const auto& a = addrs_b[idx];
   175→            std::cout << "  (" << a.lat << "," << a.lng << ") "
   176→                      << get_string(sp_b, a.housenumber_id) << " "
   177→                      << get_string(sp_b, a.street_id) << std::endl;
   178→        }
   179→    }
   180→
   181→    return 0;
   182→}
   183→


     1→#pragma once
     2→
     3→#include <algorithm>
     4→#include <chrono>
     5→#include <iostream>
     6→#include <string>
     7→#include <unordered_map>
     8→#include <vector>
     9→
    10→#include <cstdio>
    11→#include <cstring>
    12→#include <dirent.h>
    13→#include <iomanip>
    14→#include <thread>
    15→#include <unistd.h>
    16→#include <sys/stat.h>
    17→
    18→#include "types.h"
    19→#include "string_pool.h"
    20→
    21→// --- Directory creation ---
    22→
    23→inline void ensure_dir(const std::string& path) {
    24→    // Recursive mkdir — create all parent directories
    25→    for (size_t i = 1; i < path.size(); i++) {
    26→        if (path[i] == '/') {
    27→            mkdir(path.substr(0, i).c_str(), 0755);
    28→        }
    29→    }
    30→    mkdir(path.c_str(), 0755);
    31→}
    32→
    33→// --- CPU tick reading (all threads via getrusage) ---
    34→
    35→#include <sys/resource.h>
    36→
    37→struct CpuTicks {
    38→    long long process_us = 0;  // user+system microseconds (all threads)
    39→
    40→    static CpuTicks now() {
    41→        CpuTicks ct;
    42→        struct rusage ru;
    43→        if (getrusage(RUSAGE_SELF, &ru) == 0) {
    44→            ct.process_us = (long long)ru.ru_utime.tv_sec * 1000000 + ru.ru_utime.tv_usec
    45→                          + (long long)ru.ru_stime.tv_sec * 1000000 + ru.ru_stime.tv_usec;
    46→        }
    47→        return ct;
    48→    }
    49→};
    50→
    51→// --- Phase timer with CPU utilization ---
    52→
    53→inline long get_rss_mb() {
    54→    long rss_pages = 0;
    55→    FILE* f = fopen("/proc/self/statm", "r");
    56→    if (f) {
    57→        long size;
    58→        if (fscanf(f, "%ld %ld", &size, &rss_pages) != 2) rss_pages = 0;
    59→        fclose(f);
    60→    }
    61→    return rss_pages * sysconf(_SC_PAGESIZE) / (1024 * 1024);
    62→}
    63→
    64→inline void log_phase(const char* name, std::chrono::steady_clock::time_point& t,
    65→                       CpuTicks& prev_cpu) {
    66→    auto now = std::chrono::steady_clock::now();
    67→    auto ms = std::chrono::duration_cast<std::chrono::milliseconds>(now - t).count();
    68→    auto cpu_now = CpuTicks::now();
    69→    double cpu_sec = (cpu_now.process_us - prev_cpu.process_us) / 1e6;
    70→    double wall_sec = ms / 1000.0;
    71→    double cores_used = (wall_sec > 0.1) ? cpu_sec / wall_sec : 0;
    72→    unsigned ncpu = std::thread::hardware_concurrency();
    73→    int pct = (wall_sec > 0.1 && ncpu > 0) ? (int)(cores_used * 100.0 / ncpu) : 0;
    74→    long rss = get_rss_mb();
    75→    std::cerr << "  [" << ms/1000 << "." << (ms%1000)/100 << "s"
    76→              << " " << std::fixed << std::setprecision(1) << cores_used << "/" << ncpu << "cores"
    77→              << " " << pct << "%"
    78→              << " " << rss << "MiB] " << name << std::endl;
    79→    t = now;
    80→    prev_cpu = cpu_now;
    81→}
    82→
    83→// Backward compat overload (no CPU tracking)
    84→inline void log_phase(const char* name, std::chrono::steady_clock::time_point& t) {
    85→    auto ms = std::chrono::duration_cast<std::chrono::milliseconds>(
    86→        std::chrono::steady_clock::now() - t).count();
    87→    std::cerr << "  [" << ms/1000 << "." << (ms%1000)/100 << "s] " << name << std::endl;
    88→    t = std::chrono::steady_clock::now();
    89→}
    90→
    91→// --- Parsed data container ---
    92→
    93→struct ParsedData {
    94→    StringPool string_pool;
    95→    std::vector<WayHeader> ways;
    96→    std::vector<NodeCoord> street_nodes;
    97→    std::unordered_map<uint64_t, std::vector<uint32_t>> cell_to_ways;
    98→    std::vector<AddrPoint> addr_points;
    99→    std::unordered_map<uint64_t, std::vector<uint32_t>> cell_to_addrs;
   100→    std::vector<InterpWay> interp_ways;
   101→    std::vector<NodeCoord> interp_nodes;
   102→    std::unordered_map<uint64_t, std::vector<uint32_t>> cell_to_interps;
   103→    std::vector<AdminPolygon> admin_polygons;
   104→    std::vector<NodeCoord> admin_vertices;
   105→    std::unordered_map<uint64_t, std::vector<uint32_t>> cell_to_admin;
   106→
   107→    // Deferred work for parallel S2 computation (ways + interps)
   108→    std::vector<DeferredWay> deferred_ways;
   109→    std::vector<DeferredInterp> deferred_interps;
   110→
   111→    // Sorted (cell_id, item_id) pairs — kept for direct entry writing
   112→    std::vector<CellItemPair> sorted_way_cells;
   113→    std::vector<CellItemPair> sorted_addr_cells;
   114→    std::vector<CellItemPair> sorted_interp_cells;
   115→
   116→    // Collected data for parallel admin assembly
   117→    std::vector<CollectedRelation> collected_relations;
   118→    struct WayGeometry {
   119→        std::vector<std::pair<double,double>> coords;
   120→        int64_t first_node_id;
   121→        int64_t last_node_id;
   122→    };
   123→    std::unordered_map<int64_t, WayGeometry> way_geometries;
   124→};
   125→
   126→// --- Deduplicate IDs per cell ---
   127→
   128→template<typename Map>
   129→inline void deduplicate(Map& cell_map) {
   130→    for (auto& [cell_id, ids] : cell_map) {
   131→        std::sort(ids.begin(), ids.end());
   132→        ids.erase(std::unique(ids.begin(), ids.end()), ids.end());
   133→    }
   134→}
   135→


     1→cmake_minimum_required(VERSION 3.16)
     2→project(traccar-geocoder-build LANGUAGES CXX)
     3→
     4→set(CMAKE_CXX_STANDARD 17)
     5→set(CMAKE_CXX_STANDARD_REQUIRED ON)
     6→
     7→
     8→find_package(Threads REQUIRED)
     9→find_package(ZLIB REQUIRED)
    10→find_package(BZip2 REQUIRED)
    11→find_package(EXPAT REQUIRED)
    12→find_package(s2 QUIET)
    13→
    14→add_executable(build-index
    15→    src/build_index.cpp
    16→    src/pbf_reader.cpp
    17→    src/s2_helpers.cpp
    18→    src/ring_assembly.cpp
    19→    src/cache.cpp
    20→    src/continent_filter.cpp
    21→    src/continent_boundaries.cpp
    22→    src/cell_index.cpp
    23→    src/interpolation.cpp
    24→)
    25→
    26→if(EXISTS /opt/homebrew/include)
    27→    target_include_directories(build-index PRIVATE /opt/homebrew/include)
    28→endif()
    29→
    30→if(s2_FOUND)
    31→    target_link_libraries(build-index PRIVATE
    32→        s2::s2
    33→        ZLIB::ZLIB
    34→        BZip2::BZip2
    35→        EXPAT::EXPAT
    36→        Threads::Threads
    37→    )
    38→else()
    39→    target_include_directories(build-index PRIVATE /usr/include/s2)
    40→    target_link_libraries(build-index PRIVATE
    41→        s2
    42→        ZLIB::ZLIB
    43→        BZip2::BZip2
    44→        EXPAT::EXPAT
    45→        Threads::Threads
    46→    )
    47→endif()
    48→


Error: File content (24837 tokens) exceeds maximum allowed tokens (10000). Use offset and limit parameters to read specific portions of the file, or search for specific content instead of reading the whole file.