<task-notification>
<task-id>be0ddek7m</task-id>
<tool-use-id>toolu_01NzaqqSsxRopgQffjvqXA9K</tool-use-id>
<output-file>/tmp/claude-0/-root-traccar-geocoder/af1c4baa-0f9a-4420-a8b3-ce20dc28c8b9/tasks/be0ddek7m.output</output-file>
<status>completed</status>
<summary>Background command "Generate zstd patches for all files" completed (exit code 0)</summary>
</task-notification>
Read the output file to retrieve the result: /tmp/claude-0/-root-traccar-geocoder/af1c4baa-0f9a-4420-a8b3-ce20dc28c8b9/tasks/be0ddek7m.output


     1→Warning: Permanently added '100.113.88.121' (ED25519) to the list of known hosts.
     2→Simplification: error-bounded, fixed 0.1m epsilon
     3→Using 31 worker threads.
     4→Allocated dense node index: 111GB virtual address space
     5→

Warning: Permanently added '100.113.88.121' (ED25519) to the list of known hosts.
Simplification: error-bounded, fixed 0.1m epsilon
Using 31 worker threads.
Allocated dense node index: 111GB virtual address space


     1→Warning: Permanently added '100.113.88.121' (ED25519) to the list of known hosts.
     2→=== zstd patch-from results ===
     3→  addr_entries.bin: 377M -> 186M patch (49.3%) 401680ms
     4→Terminated
     5→stat: cannot statx '/home/michtest/patches/full/addr_points.bin.patch': No such file or directory
     6→  addr_points.bin: 1.3G ->  patch (%) 350967ms
     7→  admin_cells.bin: 1.7M -> 209K patch (12.6%) 1992ms
     8→  admin_entries.bin: 6.2M -> 2.5M patch (39.8%) 2736ms
     9→  admin_polygons.bin: 11M -> 4.3M patch (41.4%) 12407ms
    10→  admin_vertices.bin: 1.3G -> 3.3M patch (.2%) 225423ms
    11→  geo_cells.bin: 1.6G -> 260M patch (16.2%) 516554ms
    12→  interp_entries.bin: 156K -> 50K patch (31.8%) 28ms
    13→  interp_nodes.bin: 710K -> 12K patch (1.6%) 293ms
    14→  interp_ways.bin: 328K -> 68K patch (20.5%) 96ms
    15→  street_entries.bin: 560M -> 134M patch (23.9%) 404637ms
    16→  street_nodes.bin: 1.4G -> 3.1M patch (.2%) 286545ms
    17→  street_ways.bin: 223M -> 92M patch (41.2%) 329765ms
    18→  strings.bin: 79M -> 13M patch (16.4%) 164261ms
    19→

Warning: Permanently added '100.113.88.121' (ED25519) to the list of known hosts.
=== zstd patch-from results ===
  addr_entries.bin: 377M -> 186M patch (49.3%) 401680ms
Terminated
stat: cannot statx '/home/michtest/patches/full/addr_points.bin.patch': No such file or directory
  addr_points.bin: 1.3G ->  patch (%) 350967ms
  admin_cells.bin: 1.7M -> 209K patch (12.6%) 1992ms
  admin_entries.bin: 6.2M -> 2.5M patch (39.8%) 2736ms
  admin_polygons.bin: 11M -> 4.3M patch (41.4%) 12407ms
  admin_vertices.bin: 1.3G -> 3.3M patch (.2%) 225423ms
  geo_cells.bin: 1.6G -> 260M patch (16.2%) 516554ms
  interp_entries.bin: 156K -> 50K patch (31.8%) 28ms
  interp_nodes.bin: 710K -> 12K patch (1.6%) 293ms
  interp_ways.bin: 328K -> 68K patch (20.5%) 96ms
  street_entries.bin: 560M -> 134M patch (23.9%) 404637ms
  street_nodes.bin: 1.4G -> 3.1M patch (.2%) 286545ms
  street_ways.bin: 223M -> 92M patch (41.2%) 329765ms
  strings.bin: 79M -> 13M patch (16.4%) 164261ms


These are stale results from a previous session (different epsilon settings, addr_points.bin was terminated). The current v2 results I just showed are the relevant ones.

To recap where we are:

**Current v2 patch results (--previous-build + geographic reorder, no node shuffle):**
- Total: **~1.19 GiB** patches for 7 GiB Europe data (**17% ratio**)
- Build overhead: ~50s additional (~7% of 12min build)
- Biggest wins: addr_points 67% → 22%, strings 41% → 0.05%

**Remaining opportunity:** Extending `--previous-build` to preserve data record IDs (not just strings) could further reduce patches to ~100-200 MiB. The current bottlenecks are geo_cells (378 MiB), addr_entries (215 MiB), and street_entries (141 MiB) — all caused by shifting record IDs when new data is inserted.


These patches are far too big, and changing the build process like this really isn't ideal either. Really I would want to be able to run the process each day, comparing the full output of today to the full output of yesterday, producing the patch files, then people could catch up multiple days by just applying patches in sequence. With this approach that wouldn't work. We could come up with a custom tool that produces small patches because it understands the format and can apply them sensibly, rather than using zstd patch


You're right. The `--previous-build` approach couples the build to the previous output, and sequential patching (day 1 → day 2 → day 3) wouldn't work since each build's string offsets depend on the prior. And the patches are still too large.

A custom tool that understands the format is the right call. It can diff at the semantic level — which cells changed, which records changed — instead of trying to binary-diff files where insertions cascade everywhere.

Let me design this. The key insight from the format analysis:

**The data is organized around S2 cells.** A semantic diff tracks:
- Which cells have different entry lists (streets/addrs/interps)
- Which data records (ways, addr_points, interps) are new/changed/removed
- Which admin polygons changed
- Which strings are new

The patch applicator reconstructs the new files from old + changes.

Let me think through the design before writing code.


ToolSearch

Question: select:EnterPlanMode


Question: select:EnterPlanMode


Entered plan mode. You should now focus on exploring the codebase and designing an implementation approach.

In plan mode, you should:
1. Thoroughly explore the codebase to understand existing patterns
2. Identify similar features and architectural approaches
3. Consider multiple approaches and their trade-offs
4. Use AskUserQuestion if you need to clarify the approach
5. Design a concrete implementation strategy
6. When ready, use ExitPlanMode to present your plan for approval

Remember: DO NOT write or edit any files yet. This is a read-only exploration and planning phase.


Agent

Explore server index reading code

Prompt:
I need to understand how the geocoder server reads and uses the binary index files. This is critical for designing a custom patching tool.

Look in /root/traccar-geocoder/server/ for the Rust server code. Find:
1. How each binary file is loaded (mmap, full read, etc.)
2. How lookups work for each file type:
   - geo_cells.bin: binary search for cell_id → offsets
   - street_entries.bin / addr_entries.bin / interp_entries.bin: read entry lists by offset
   - street_ways.bin / addr_points.bin / interp_ways.bin: index by record ID
   - street_nodes.bin / interp_nodes.bin: index by offset
   - admin_cells.bin / admin_entries.bin / admin_polygons.bin / admin_vertices.bin
   - strings.bin: read by offset to null terminator
3. Whether the server requires contiguous, sorted files or could handle alternative layouts
4. Whether there's any manifest or version metadata

Be thorough - I need to know exactly what format constraints the server imposes.


I need to understand how the geocoder server reads and uses the binary index files. This is critical for designing a custom patching tool.

Look in /root/traccar-geocoder/server/ for the Rust server code. Find:
1. How each binary file is loaded (mmap, full read, etc.)
2. How lookups work for each file type:
   - geo_cells.bin: binary search for cell_id → offsets
   - street_entries.bin / addr_entries.bin / interp_entries.bin: read entry lists by offset
   - street_ways.bin / addr_points.bin / interp_ways.bin: index by record ID
   - street_nodes.bin / interp_nodes.bin: index by offset
   - admin_cells.bin / admin_entries.bin / admin_polygons.bin / admin_vertices.bin
   - strings.bin: read by offset to null terminator
3. Whether the server requires contiguous, sorted files or could handle alternative layouts
4. Whether there's any manifest or version metadata

Be thorough - I need to know exactly what format constraints the server imposes.

Prompt:
I need to understand how the geocoder server reads and uses the binary index files. This is critical for designing a custom patching tool.

Look in /root/traccar-geocoder/server/ for the Rust server code. Find:
1. How each binary file is loaded (mmap, full read, etc.)
2. How lookups work for each file type:
   - geo_cells.bin: binary search for cell_id → offsets
   - street_entries.bin / addr_entries.bin / interp_entries.bin: read entry lists by offset
   - street_ways.bin / addr_points.bin / interp_ways.bin: index by record ID
   - street_nodes.bin / interp_nodes.bin: index by offset
   - admin_cells.bin / admin_entries.bin / admin_polygons.bin / admin_vertices.bin
   - strings.bin: read by offset to null terminator
3. Whether the server requires contiguous, sorted files or could handle alternative layouts
4. Whether there's any manifest or version metadata

Be thorough - I need to know exactly what format constraints the server imposes.


Bash

find /root/traccar-geocoder/server -type f -name "*.rs" | head -20


/root/traccar-geocoder/server/src/auth.rs
/root/traccar-geocoder/server/src/main.rs


Error: EISDIR: illegal operation on a directory, read '/root/traccar-geocoder/server'


find /root/traccar-geocoder/server -type f | head -30


/root/traccar-geocoder/server/Cargo.toml
/root/traccar-geocoder/server/src/auth.rs
/root/traccar-geocoder/server/src/web/dashboard.html
/root/traccar-geocoder/server/src/web/login.html
/root/traccar-geocoder/server/src/main.rs
/root/traccar-geocoder/server/Cargo.lock


Error: File content (10030 tokens) exceeds maximum allowed tokens (10000). Use offset and limit parameters to read specific portions of the file, or search for specific content instead of reading the whole file.


Agent

Explore existing patch/diff tooling

Prompt:
Search the codebase at /root/traccar-geocoder/ for:
1. Any existing patch generation or delta tools (builder/tools/, scripts, etc.)
2. The manifest.json that gets written - what's in it?
3. The compare_indexes.cpp and deep_compare.cpp tools - how do they compare files semantically?
4. How the continent filter produces filtered data (filter_by_bbox_masked, filter_and_remap_sorted) - this shows how data is extracted per-region and might inform the patch design
5. Any download/distribution mechanism for the index files

Also look at builder/src/cell_index.cpp for write_entries_from_sorted and write_index to understand exactly how each file is produced from the in-memory data structures.


Search the codebase at /root/traccar-geocoder/ for:
1. Any existing patch generation or delta tools (builder/tools/, scripts, etc.)
2. The manifest.json that gets written - what's in it?
3. The compare_indexes.cpp and deep_compare.cpp tools - how do they compare files semantically?
4. How the continent filter produces filtered data (filter_by_bbox_masked, filter_and_remap_sorted) - this shows how data is extracted per-region and might inform the patch design
5. Any download/distribution mechanism for the index files

Also look at builder/src/cell_index.cpp for write_entries_from_sorted and write_index to understand exactly how each file is produced from the in-memory data structures.

Prompt:
Search the codebase at /root/traccar-geocoder/ for:
1. Any existing patch generation or delta tools (builder/tools/, scripts, etc.)
2. The manifest.json that gets written - what's in it?
3. The compare_indexes.cpp and deep_compare.cpp tools - how do they compare files semantically?
4. How the continent filter produces filtered data (filter_by_bbox_masked, filter_and_remap_sorted) - this shows how data is extracted per-region and might inform the patch design
5. Any download/distribution mechanism for the index files

Also look at builder/src/cell_index.cpp for write_entries_from_sorted and write_index to understand exactly how each file is produced from the in-memory data structures.


Bash

find /root/traccar-geocoder -type f -name "*.json" | head -20


     1→mod auth;
     2→
     3→use axum::extract::Query;
     4→use axum::http::StatusCode;
     5→use axum::response::{IntoResponse, Response};
     6→use axum::routing::get;
     7→use axum::Router;
     8→use memmap2::Mmap;
     9→use s2::cellid::CellID;
    10→use s2::latlng::LatLng;
    11→use serde::{Deserialize, Serialize};
    12→use std::borrow::Cow;
    13→use std::fs::File;
    14→use std::sync::{Arc, RwLock};
    15→
    16→// --- S2 helpers ---
    17→
    18→const DEFAULT_STREET_CELL_LEVEL: u64 = 17;
    19→const DEFAULT_ADMIN_CELL_LEVEL: u64 = 10;
    20→const DEFAULT_SEARCH_DISTANCE: f64 = 75.0;
    21→
    22→fn cell_id_at_level(lat: f64, lng: f64, level: u64) -> u64 {
    23→    let ll = LatLng::from_degrees(lat, lng);
    24→    CellID::from(ll).parent(level).0
    25→}
    26→
    27→fn cell_neighbors_at_level(cell_id: u64, level: u64) -> Vec<u64> {
    28→    let cell = CellID(cell_id);
    29→    cell.all_neighbors(level).into_iter().map(|c| c.0).collect()
    30→}
    31→
    32→// --- Binary format structs (must match C++ build pipeline) ---
    33→
    34→#[repr(C)]
    35→#[derive(Clone, Copy)]
    36→struct WayHeader {
    37→    node_offset: u32,
    38→    node_count: u8,
    39→    name_id: u32,
    40→}
    41→
    42→#[repr(C)]
    43→#[derive(Clone, Copy)]
    44→struct AddrPoint {
    45→    lat: f32,
    46→    lng: f32,
    47→    housenumber_id: u32,
    48→    street_id: u32,
    49→}
    50→
    51→#[repr(C)]
    52→#[derive(Clone, Copy)]
    53→struct InterpWay {
    54→    node_offset: u32,
    55→    node_count: u8,
    56→    street_id: u32,
    57→    start_number: u32,
    58→    end_number: u32,
    59→    interpolation: u8,
    60→}
    61→
    62→#[repr(C)]
    63→#[derive(Clone, Copy)]
    64→struct AdminPolygon {
    65→    vertex_offset: u32,
    66→    vertex_count: u32,
    67→    name_id: u32,
    68→    admin_level: u8,
    69→    area: f32,
    70→    country_code: u16,
    71→}
    72→
    73→#[repr(C)]
    74→#[derive(Clone, Copy)]
    75→struct NodeCoord {
    76→    lat: f32,
    77→    lng: f32,
    78→}
    79→
    80→// --- Index data ---
    81→
    82→struct Index {
    83→    geo_cells: Option<Mmap>,
    84→    street_entries: Option<Mmap>,
    85→    street_ways: Option<Mmap>,
    86→    street_nodes: Option<Mmap>,
    87→    addr_entries: Option<Mmap>,
    88→    addr_points: Option<Mmap>,
    89→    interp_entries: Option<Mmap>,
    90→    interp_ways: Option<Mmap>,
    91→    interp_nodes: Option<Mmap>,
    92→    admin_cells: Mmap,
    93→    admin_entries: Mmap,
    94→    admin_polygons: Mmap,
    95→    admin_vertices: Mmap,
    96→    strings: Mmap,
    97→    street_cell_level: u64,
    98→    admin_cell_level: u64,
    99→    max_distance_sq: f64,
   100→}


(Bash completed with no output)


   100→}
   101→
   102→const NO_DATA: u32 = 0xFFFFFFFF;
   103→
   104→struct GeoCellOffsets {
   105→    street: u32,
   106→    addr: u32,
   107→    interp: u32,
   108→}
   109→
   110→fn mmap_file(path: &str) -> Result<Mmap, String> {
   111→    let file = File::open(path).map_err(|e| format!("Failed to open {}: {}", path, e))?;
   112→    unsafe { Mmap::map(&file).map_err(|e| format!("Failed to mmap {}: {}", path, e)) }
   113→}
   114→
   115→fn mmap_file_optional(path: &str) -> Option<Mmap> {
   116→    let file = File::open(path).ok()?;
   117→    unsafe { Mmap::map(&file).ok() }
   118→}
   119→
   120→impl Index {
   121→    fn load(dir: &str, street_cell_level: u64, admin_cell_level: u64, search_distance: f64) -> Result<Self, String> {
   122→        let meters_to_rad = search_distance / 111_320.0;
   123→        let max_distance_sq = meters_to_rad * meters_to_rad;
   124→
   125→        let geo_cells = mmap_file_optional(&format!("{}/geo_cells.bin", dir));
   126→        let has_geo = geo_cells.is_some();
   127→
   128→        let required_geo = |name: &str| -> Result<Option<Mmap>, String> {
   129→            let path = format!("{}/{}", dir, name);
   130→            let mmap = mmap_file_optional(&path);
   131→            if has_geo && mmap.is_none() {
   132→                return Err(format!("Partial geo index: {} is missing", name));
   133→            }
   134→            Ok(mmap)
   135→        };
   136→
   137→        Ok(Index {
   138→            geo_cells,
   139→            street_entries: required_geo("street_entries.bin")?,
   140→            street_ways: required_geo("street_ways.bin")?,
   141→            street_nodes: required_geo("street_nodes.bin")?,
   142→            addr_entries: mmap_file_optional(&format!("{}/addr_entries.bin", dir)),
   143→            addr_points: mmap_file_optional(&format!("{}/addr_points.bin", dir)),
   144→            interp_entries: mmap_file_optional(&format!("{}/interp_entries.bin", dir)),
   145→            interp_ways: mmap_file_optional(&format!("{}/interp_ways.bin", dir)),
   146→            interp_nodes: mmap_file_optional(&format!("{}/interp_nodes.bin", dir)),
   147→            admin_cells: mmap_file(&format!("{}/admin_cells.bin", dir))?,
   148→            admin_entries: mmap_file(&format!("{}/admin_entries.bin", dir))?,
   149→            admin_polygons: mmap_file(&format!("{}/admin_polygons.bin", dir))?,
   150→            admin_vertices: mmap_file(&format!("{}/admin_vertices.bin", dir))?,
   151→            strings: mmap_file(&format!("{}/strings.bin", dir))?,
   152→            street_cell_level,
   153→            admin_cell_level,
   154→            max_distance_sq,
   155→        })
   156→    }
   157→
   158→    fn get_string(&self, offset: u32) -> &str {
   159→        let bytes = &self.strings[offset as usize..];
   160→        let end = bytes.iter().position(|&b| b == 0).unwrap_or(bytes.len());
   161→        std::str::from_utf8(&bytes[..end]).unwrap_or("")
   162→    }
   163→
   164→    fn read_u16(data: &[u8], offset: usize) -> u16 {
   165→        u16::from_le_bytes([data[offset], data[offset + 1]])
   166→    }
   167→
   168→    fn read_u32(data: &[u8], offset: usize) -> u32 {
   169→        u32::from_le_bytes(data[offset..offset + 4].try_into().unwrap())
   170→    }
   171→
   172→    fn read_u64(data: &[u8], offset: usize) -> u64 {
   173→        u64::from_le_bytes(data[offset..offset + 8].try_into().unwrap())
   174→    }
   175→
   176→    // Iterate entry IDs inline from entries file at given offset
   177→    fn for_each_entry(entries: &[u8], offset: u32, mut f: impl FnMut(u32)) {
   178→        if offset == NO_DATA { return; }
   179→        let offset = offset as usize;
   180→        if offset + 2 > entries.len() { return; }
   181→
   182→        let id_count = Self::read_u16(entries, offset) as usize;
   183→        let data_start = offset + 2;
   184→        if data_start + id_count * 4 > entries.len() { return; }
   185→
   186→        for i in 0..id_count {
   187→            f(Self::read_u32(entries, data_start + i * 4));
   188→        }
   189→    }
   190→
   191→    // Binary search geo cell index: 20 bytes per entry (u64 cell_id + u32 street + u32 addr + u32 interp)
   192→    fn lookup_geo_cell(cells: &[u8], cell_id: u64) -> GeoCellOffsets {
   193→        let entry_size: usize = 20;
   194→        let count = cells.len() / entry_size;
   195→        let empty = GeoCellOffsets { street: NO_DATA, addr: NO_DATA, interp: NO_DATA };
   196→        if count == 0 { return empty; }
   197→
   198→        let mut lo = 0usize;
   199→        let mut hi = count;
   200→        while lo < hi {
   201→            let mid = lo + (hi - lo) / 2;
   202→            let mid_id = Self::read_u64(cells, mid * entry_size);
   203→            if mid_id == cell_id {
   204→                return GeoCellOffsets {
   205→                    street: Self::read_u32(cells, mid * entry_size + 8),
   206→                    addr: Self::read_u32(cells, mid * entry_size + 12),
   207→                    interp: Self::read_u32(cells, mid * entry_size + 16),
   208→                };
   209→            } else if mid_id < cell_id {
   210→                lo = mid + 1;
   211→            } else {
   212→                hi = mid;
   213→            }
   214→        }
   215→        empty
   216→    }
   217→
   218→    // Binary search admin cell index: 12 bytes per entry (u64 cell_id + u32 offset)
   219→    fn lookup_admin_cell(cells: &[u8], cell_id: u64) -> u32 {
   220→        let entry_size: usize = 12;
   221→        let count = cells.len() / entry_size;
   222→        if count == 0 { return NO_DATA; }
   223→
   224→        let mut lo = 0usize;
   225→        let mut hi = count;
   226→        while lo < hi {
   227→            let mid = lo + (hi - lo) / 2;
   228→            let mid_id = Self::read_u64(cells, mid * entry_size);
   229→            if mid_id == cell_id {
   230→                return Self::read_u32(cells, mid * entry_size + 8);
   231→            } else if mid_id < cell_id {
   232→                lo = mid + 1;
   233→            } else {
   234→                hi = mid;
   235→            }
   236→        }
   237→        NO_DATA
   238→    }
   239→
   240→    // --- Geo lookup (streets, addresses, interpolation from merged index) ---
   241→
   242→    fn query_geo(&self, lat: f64, lng: f64) -> (Option<(f64, &AddrPoint)>, Option<(f64, &str, u32)>, Option<(f64, &WayHeader)>) {
   243→        let geo_cells = match &self.geo_cells {
   244→            Some(gc) => gc,
   245→            None => return (None, None, None),
   246→        };
   247→        let street_entries = self.street_entries.as_ref().unwrap();
   248→        let street_ways_mmap = self.street_ways.as_ref().unwrap();
   249→        let street_nodes_mmap = self.street_nodes.as_ref().unwrap();


No files found


/root/traccar-geocoder/.git/hooks/pre-applypatch.sample
/root/traccar-geocoder/.git/hooks/applypatch-msg.sample


   250→        let cell = cell_id_at_level(lat, lng, self.street_cell_level);
   251→        let neighbors = cell_neighbors_at_level(cell, self.street_cell_level);
   252→
   253→        let empty_addr: &[AddrPoint] = &[];
   254→        let all_points: &[AddrPoint] = if let Some(ref m) = self.addr_points {
   255→            unsafe {
   256→                std::slice::from_raw_parts(
   257→                    m.as_ptr() as *const AddrPoint,
   258→                    m.len() / std::mem::size_of::<AddrPoint>(),
   259→                )
   260→            }
   261→        } else {
   262→            empty_addr
   263→        };
   264→        let all_ways: &[WayHeader] = unsafe {
   265→            std::slice::from_raw_parts(
   266→                street_ways_mmap.as_ptr() as *const WayHeader,
   267→                street_ways_mmap.len() / std::mem::size_of::<WayHeader>(),
   268→            )
   269→        };
   270→        let all_street_nodes: &[NodeCoord] = unsafe {
   271→            std::slice::from_raw_parts(
   272→                street_nodes_mmap.as_ptr() as *const NodeCoord,
   273→                street_nodes_mmap.len() / std::mem::size_of::<NodeCoord>(),
   274→            )
   275→        };
   276→        let empty_interp: &[InterpWay] = &[];
   277→        let all_interps: &[InterpWay] = if let Some(ref m) = self.interp_ways {
   278→            unsafe {
   279→                std::slice::from_raw_parts(
   280→                    m.as_ptr() as *const InterpWay,
   281→                    m.len() / std::mem::size_of::<InterpWay>(),
   282→                )
   283→            }
   284→        } else {
   285→            empty_interp
   286→        };
   287→        let empty_interp_nodes: &[NodeCoord] = &[];
   288→        let all_interp_nodes: &[NodeCoord] = if let Some(ref m) = self.interp_nodes {
   289→            unsafe {
   290→                std::slice::from_raw_parts(
   291→                    m.as_ptr() as *const NodeCoord,
   292→                    m.len() / std::mem::size_of::<NodeCoord>(),
   293→                )
   294→            }
   295→        } else {
   296→            empty_interp_nodes
   297→        };
   298→
   299→        let cos_lat = lat.to_radians().cos();
   300→
   301→        let mut best_addr_dist = f64::MAX;
   302→        let mut best_addr: Option<&AddrPoint> = None;
   303→        let mut best_street_dist = f64::MAX;
   304→        let mut best_street: Option<&WayHeader> = None;
   305→        let mut best_interp_dist = f64::MAX;
   306→        let mut best_interp: Option<&InterpWay> = None;
   307→        let mut best_interp_t: f64 = 0.0;
   308→
   309→        // Fixed-size hash set on stack to skip duplicate street IDs across cells
   310→        let mut seen_streets: [u32; 64] = [u32::MAX; 64];
   311→
   312→        for c in std::iter::once(cell).chain(neighbors.into_iter()) {
   313→            let offsets = Self::lookup_geo_cell(geo_cells, c);
   314→
   315→            // Addresses
   316→            if let Some(ref addr_entries) = self.addr_entries {
   317→                Self::for_each_entry(addr_entries, offsets.addr, |id| {
   318→                    let point = &all_points[id as usize];
   319→                    let dlat = (point.lat as f64 - lat).to_radians();
   320→                    let dlng = (point.lng as f64 - lng).to_radians();
   321→                    let dist = dist_sq(dlat, dlng, cos_lat);
   322→                    if dist < best_addr_dist {
   323→                        best_addr_dist = dist;
   324→                        best_addr = Some(point);
   325→                    }
   326→                });
   327→            }
   328→
   329→            // Streets
   330→            Self::for_each_entry(street_entries, offsets.street, |id| {
   331→                let slot = (id as usize) & 0x3F;
   332→                if seen_streets[slot] == id { return; }
   333→                seen_streets[slot] = id;
   334→
   335→                let way = &all_ways[id as usize];
   336→                let offset = way.node_offset as usize;
   337→                let count = way.node_count as usize;
   338→                let nodes = &all_street_nodes[offset..offset + count];
   339→
   340→                for i in 0..nodes.len() - 1 {
   341→                    let dist = point_to_segment_distance(
   342→                        lat, lng,
   343→                        nodes[i].lat as f64, nodes[i].lng as f64,
   344→                        nodes[i + 1].lat as f64, nodes[i + 1].lng as f64,
   345→                        cos_lat,
   346→                    );
   347→                    if dist < best_street_dist {
   348→                        best_street_dist = dist;
   349→                        best_street = Some(way);
   350→                    }
   351→                }
   352→            });
   353→
   354→            // Interpolation
   355→            if let Some(ref interp_entries) = self.interp_entries {
   356→                Self::for_each_entry(interp_entries, offsets.interp, |id| {
   357→                    let iw = &all_interps[id as usize];
   358→                    if iw.start_number == 0 || iw.end_number == 0 { return; }
   359→
   360→                    let offset = iw.node_offset as usize;
   361→                    let count = iw.node_count as usize;
   362→                    let nodes = &all_interp_nodes[offset..offset + count];
   363→
   364→                    let mut total_len: f64 = 0.0;
   365→                    for i in 0..nodes.len() - 1 {
   366→                        let dlat = (nodes[i + 1].lat as f64 - nodes[i].lat as f64).to_radians();
   367→                        let dlng = (nodes[i + 1].lng as f64 - nodes[i].lng as f64).to_radians();
   368→                        total_len += dist_sq(dlat, dlng, cos_lat);
   369→                    }
   370→                    if total_len == 0.0 { return; }
   371→
   372→                    let mut best_seg_dist = f64::MAX;
   373→                    let mut best_seg_t: f64 = 0.0;
   374→                    let mut prev_accumulated: f64 = 0.0;
   375→
   376→                    for i in 0..nodes.len() - 1 {
   377→                        let dlat = (nodes[i + 1].lat as f64 - nodes[i].lat as f64).to_radians();
   378→                        let dlng = (nodes[i + 1].lng as f64 - nodes[i].lng as f64).to_radians();
   379→                        let seg_len = dist_sq(dlat, dlng, cos_lat);
   380→                        let (dist, seg_t) = point_to_segment_with_t(
   381→                            lat, lng,
   382→                            nodes[i].lat as f64, nodes[i].lng as f64,
   383→                            nodes[i + 1].lat as f64, nodes[i + 1].lng as f64,
   384→                            cos_lat,
   385→                        );
   386→                        if dist < best_seg_dist {
   387→                            best_seg_dist = dist;
   388→                            best_seg_t = (prev_accumulated + seg_t * seg_len) / total_len;
   389→                        }
   390→                        prev_accumulated += seg_len;
   391→                    }
   392→
   393→                    if best_seg_dist < best_interp_dist {
   394→                        best_interp_dist = best_seg_dist;
   395→                        best_interp = Some(iw);
   396→                        best_interp_t = best_seg_t;
   397→                    }
   398→                });
   399→            }
   400→        }
   401→
   402→        let addr_result = best_addr.map(|p| (best_addr_dist, p));
   403→        let street_result = best_street.map(|w| (best_street_dist, w));
   404→        let interp_result = best_interp.map(|iw| {
   405→            let start = iw.start_number as f64;
   406→            let end = iw.end_number as f64;
   407→            let raw = start + best_interp_t * (end - start);
   408→
   409→            let step: u32 = match iw.interpolation {
   410→                1 | 2 => 2,
   411→                _ => 1,
   412→            };
   413→
   414→            let number = if step == 2 {
   415→                let base = iw.start_number;
   416→                let offset = ((raw - base as f64) / step as f64).round() as u32 * step;
   417→                base + offset
   418→            } else {
   419→                raw.round() as u32
   420→            };
   421→
   422→            (best_interp_dist, self.get_string(iw.street_id), number)
   423→        });
   424→
   425→        (addr_result, interp_result, street_result)
   426→    }
   427→
   428→    // --- Admin boundary lookup (point-in-polygon) ---
   429→
   430→    fn find_admin(&self, lat: f64, lng: f64) -> AdminResult<'_> {
   431→        let cell = cell_id_at_level(lat, lng, self.admin_cell_level);
   432→        let neighbors = cell_neighbors_at_level(cell, self.admin_cell_level);
   433→
   434→        let all_polygons: &[AdminPolygon] = unsafe {
   435→            std::slice::from_raw_parts(
   436→                self.admin_polygons.as_ptr() as *const AdminPolygon,
   437→                self.admin_polygons.len() / std::mem::size_of::<AdminPolygon>(),
   438→            )
   439→        };
   440→        let all_vertices: &[NodeCoord] = unsafe {
   441→            std::slice::from_raw_parts(
   442→                self.admin_vertices.as_ptr() as *const NodeCoord,
   443→                self.admin_vertices.len() / std::mem::size_of::<NodeCoord>(),
   444→            )
   445→        };
   446→
   447→        // For each admin level, find the smallest-area polygon containing the point
   448→        let mut best_by_level: [Option<(f32, &AdminPolygon)>; 12] = [None; 12];
   449→


tools/comparison.geojson
tools/simplify_compare.cpp
tools/simplification_viewer.html
tools/simplification_map.html


Found 3 files
src/build_index.cpp
src/continent_filter.cpp
src/continent_filter.h


   450→        const INTERIOR_FLAG: u32 = 0x80000000;
   451→        const ID_MASK: u32 = 0x7FFFFFFF;
   452→
   453→        for c in std::iter::once(cell).chain(neighbors.into_iter()) {
   454→            Self::for_each_entry(&self.admin_entries, Self::lookup_admin_cell(&self.admin_cells, c), |id| {
   455→                let is_interior = (id & INTERIOR_FLAG) != 0;
   456→                let poly_id = (id & ID_MASK) as usize;
   457→                let poly = &all_polygons[poly_id];
   458→                let level = poly.admin_level as usize;
   459→                if level >= 12 { return; }
   460→
   461→                // Skip if we already have a smaller polygon at this level
   462→                if let Some((best_area, _)) = best_by_level[level] {
   463→                    if poly.area >= best_area { return; }
   464→                }
   465→
   466→                // Interior cells skip point-in-polygon test
   467→                if is_interior || point_in_polygon(lat as f32, lng as f32, {
   468→                    let offset = poly.vertex_offset as usize;
   469→                    let count = poly.vertex_count as usize;
   470→                    &all_vertices[offset..offset + count]
   471→                }) {
   472→                    best_by_level[level] = Some((poly.area, poly));
   473→                }
   474→            });
   475→        }
   476→
   477→        let mut result = AdminResult::default();
   478→
   479→        for level in 0..12 {
   480→            if let Some((_, poly)) = best_by_level[level] {
   481→                let name = self.get_string(poly.name_id);
   482→                match poly.admin_level {
   483→                    2 => {
   484→                        result.country = Some(name);
   485→                        if poly.country_code != 0 {
   486→                            result.country_code = Some([
   487→                                (poly.country_code >> 8) as u8,
   488→                                (poly.country_code & 0xFF) as u8,
   489→                            ]);
   490→                        }
   491→                    }
   492→                    4 => result.state = Some(name),
   493→                    6 => result.county = Some(name),
   494→                    8 => result.city = Some(name),
   495→                    11 => result.postcode = Some(name),
   496→                    _ => {}
   497→                }
   498→            }
   499→        }
   500→
   501→        result
   502→    }
   503→
   504→    // --- Combined query ---
   505→
   506→    fn query(&self, lat: f64, lng: f64) -> Address<'_> {
   507→        let max_dist = self.max_distance_sq;
   508→
   509→        let admin = self.find_admin(lat, lng);
   510→        let (addr, interp, street) = self.query_geo(lat, lng);
   511→
   512→        // Determine house_number and road from best geo match (priority: address > interpolation > street)
   513→        let mut house_number: Option<Cow<'_, str>> = None;
   514→        let mut road: Option<&str> = None;
   515→
   516→        if let Some((dist, point)) = addr {
   517→            if dist < max_dist {
   518→                house_number = Some(Cow::Borrowed(self.get_string(point.housenumber_id)));
   519→                road = Some(self.get_string(point.street_id));
   520→            }
   521→        }
   522→        if road.is_none() {
   523→            if let Some((dist, street_name, number)) = interp {
   524→                if dist < max_dist {
   525→                    house_number = Some(Cow::Owned(number.to_string()));
   526→                    road = Some(street_name);
   527→                }
   528→            }
   529→        }
   530→        if road.is_none() {
   531→            if let Some((dist, way)) = street {
   532→                if dist < max_dist {
   533→                    road = Some(self.get_string(way.name_id));
   534→                }
   535→            }
   536→        }
   537→
   538→        if road.is_none() && admin.country.is_none() && admin.city.is_none() && admin.state.is_none() {
   539→            return Address::default();
   540→        }
   541→
   542→        let address = AddressDetails {
   543→            house_number,
   544→            road,
   545→            city: admin.city,
   546→            state: admin.state,
   547→            county: admin.county,
   548→            postcode: admin.postcode,
   549→            country: admin.country,
   550→            country_code: admin.country_code.map(|c| String::from_utf8_lossy(&c).into_owned()),
   551→        };
   552→        let display_name = format_address(&address);
   553→        Address { display_name, address }
   554→    }
   555→}
   556→
   557→// --- Geometry helpers ---
   558→
   559→fn dist_sq(dlat: f64, dlng: f64, cos_lat: f64) -> f64 {
   560→    dlat * dlat + dlng * dlng * cos_lat * cos_lat
   561→}
   562→
   563→fn point_to_segment_with_t(
   564→    px: f64, py: f64,
   565→    ax: f64, ay: f64,
   566→    bx: f64, by: f64,
   567→    cos_lat: f64,
   568→) -> (f64, f64) {
   569→    let dx = bx - ax;
   570→    let dy = by - ay;
   571→    let len_sq = dx * dx + dy * dy;
   572→
   573→    let t = if len_sq == 0.0 {
   574→        0.0
   575→    } else {
   576→        (((px - ax) * dx + (py - ay) * dy) / len_sq).clamp(0.0, 1.0)
   577→    };
   578→
   579→    let proj_x = ax + t * dx;
   580→    let proj_y = ay + t * dy;
   581→    let dlat = (px - proj_x).to_radians();
   582→    let dlng = (py - proj_y).to_radians();
   583→    (dist_sq(dlat, dlng, cos_lat), t)
   584→}
   585→
   586→fn point_to_segment_distance(
   587→    px: f64, py: f64,
   588→    ax: f64, ay: f64,
   589→    bx: f64, by: f64,
   590→    cos_lat: f64,
   591→) -> f64 {
   592→    point_to_segment_with_t(px, py, ax, ay, bx, by, cos_lat).0
   593→}
   594→
   595→// Ray casting point-in-polygon test
   596→fn point_in_polygon(lat: f32, lng: f32, vertices: &[NodeCoord]) -> bool {
   597→    let mut inside = false;
   598→    let n = vertices.len();
   599→    let mut j = n - 1;


Found 1 file
src/build_index.cpp


Error: File does not exist. Note: your current working directory is /root/traccar-geocoder/builder. Did you mean /root/traccar-geocoder/builder/src/cell_index.cpp?


Error: File does not exist. Note: your current working directory is /root/traccar-geocoder/builder. Did you mean /root/traccar-geocoder/builder/test/compare_indexes.cpp?


777:    ([(axum::http::header::CONTENT_TYPE, "application/json")], json).into_response()


find /root/traccar-geocoder -type f \( -name "*.cpp" -o -name "*.h" \) | sort


/root/traccar-geocoder/builder/build-presplit/CMakeFiles/3.31.6/CompilerIdCXX/CMakeCXXCompilerId.cpp
/root/traccar-geocoder/builder/build/CMakeFiles/3.31.6/CompilerIdCXX/CMakeCXXCompilerId.cpp
/root/traccar-geocoder/builder/src/build_index.cpp
/root/traccar-geocoder/builder/src/cache.cpp
/root/traccar-geocoder/builder/src/cache.h
/root/traccar-geocoder/builder/src/cell_index.cpp
/root/traccar-geocoder/builder/src/cell_index.h
/root/traccar-geocoder/builder/src/continent_boundaries.cpp
/root/traccar-geocoder/builder/src/continent_boundaries.h
/root/traccar-geocoder/builder/src/continent_filter.cpp
/root/traccar-geocoder/builder/src/continent_filter.h
/root/traccar-geocoder/builder/src/geometry.h
/root/traccar-geocoder/builder/src/interpolation.cpp
/root/traccar-geocoder/builder/src/interpolation.h
/root/traccar-geocoder/builder/src/parsed_data.h
/root/traccar-geocoder/builder/src/pbf_reader.cpp
/root/traccar-geocoder/builder/src/pbf_reader.h
/root/traccar-geocoder/builder/src/relation_collector.h
/root/traccar-geocoder/builder/src/ring_assembly.cpp
/root/traccar-geocoder/builder/src/ring_assembly.h
/root/traccar-geocoder/builder/src/s2_helpers.cpp
/root/traccar-geocoder/builder/src/s2_helpers.h
/root/traccar-geocoder/builder/src/string_pool.h
/root/traccar-geocoder/builder/src/types.h
/root/traccar-geocoder/builder/test/compare_indexes.cpp
/root/traccar-geocoder/builder/test/deep_compare.cpp
/root/traccar-geocoder/builder/test/test_custom_pbf.cpp
/root/traccar-geocoder/builder/test/test_pbf_parsing.cpp
/root/traccar-geocoder/builder/test/test_ring_assembly.cpp
/root/traccar-geocoder/builder/tools/simplify_compare.cpp


find /root/traccar-geocoder -name "*.rs" -o -name "*.cpp" -o -name "*.h" | grep -E "(builder|build)" | head -20


     1→#include "cell_index.h"
     2→
     3→#include <algorithm>
     4→#include <atomic>
     5→#include <cstring>
     6→#include <fstream>
     7→#include <future>
     8→#include <iostream>
     9→#include <thread>
    10→#include <unordered_map>
    11→#include <unordered_set>
    12→
    13→#include <unistd.h>
    14→
    15→#include "geometry.h"
    16→#include "s2_helpers.h"
    17→
    18→std::vector<uint32_t> write_entries(
    19→    const std::string& path,
    20→    const std::vector<uint64_t>& sorted_cells,
    21→    const std::unordered_map<uint64_t, std::vector<uint32_t>>& cell_map
    22→) {
    23→    struct CellRef { uint64_t cell_id; const std::vector<uint32_t>* ids; };
    24→    std::vector<CellRef> sorted_refs;
    25→    sorted_refs.reserve(cell_map.size());
    26→    for (auto& [id, ids] : cell_map) sorted_refs.push_back({id, &ids});
    27→    std::sort(sorted_refs.begin(), sorted_refs.end(),
    28→        [](const CellRef& a, const CellRef& b) { return a.cell_id < b.cell_id; });
    29→
    30→    std::vector<uint32_t> offsets(sorted_cells.size(), NO_DATA);
    31→    size_t total_size = 0;
    32→    for (auto& r : sorted_refs) total_size += sizeof(uint16_t) + r.ids->size() * sizeof(uint32_t);
    33→
    34→    std::vector<char> buf;
    35→    buf.reserve(total_size);
    36→    uint32_t current = 0;
    37→    size_t ri = 0;
    38→    for (uint32_t si = 0; si < sorted_cells.size() && ri < sorted_refs.size(); si++) {
    39→        if (sorted_cells[si] < sorted_refs[ri].cell_id) continue;
    40→        if (sorted_cells[si] > sorted_refs[ri].cell_id) { si--; ri++; continue; }
    41→        offsets[si] = current;
    42→        const auto& ids = *sorted_refs[ri].ids;
    43→        uint16_t count = static_cast<uint16_t>(std::min(ids.size(), size_t(MAX_VERTEX_COUNT)));
    44→        buf.insert(buf.end(), reinterpret_cast<const char*>(&count),
    45→                   reinterpret_cast<const char*>(&count) + sizeof(count));
    46→        buf.insert(buf.end(), reinterpret_cast<const char*>(ids.data()),
    47→                   reinterpret_cast<const char*>(ids.data()) + ids.size() * sizeof(uint32_t));
    48→        current += sizeof(uint16_t) + ids.size() * sizeof(uint32_t);
    49→        ri++;
    50→    }
    51→    std::ofstream f(path, std::ios::binary);
    52→    f.write(buf.data(), buf.size());
    53→    return offsets;
    54→}
    55→
    56→std::vector<uint32_t> write_entries_from_sorted(
    57→    const std::string& path,
    58→    const std::vector<uint64_t>& sorted_cells,
    59→    const std::vector<CellItemPair>& sorted_pairs
    60→) {
    61→    std::vector<uint32_t> offsets(sorted_cells.size(), NO_DATA);
    62→    if (sorted_pairs.empty()) {
    63→        std::ofstream f(path, std::ios::binary);
    64→        return offsets;
    65→    }
    66→
    67→    unsigned int nthreads = std::thread::hardware_concurrency();
    68→    if (nthreads == 0) nthreads = 4;
    69→    size_t cells_per_chunk = (sorted_cells.size() + nthreads - 1) / nthreads;
    70→
    71→    struct ChunkResult {
    72→        std::vector<char> buf;
    73→        size_t cell_start, cell_end;
    74→        uint32_t local_size;
    75→    };
    76→    std::vector<ChunkResult> chunks(nthreads);
    77→
    78→    std::vector<std::thread> threads;
    79→    for (unsigned int t = 0; t < nthreads; t++) {
    80→        size_t cs = t * cells_per_chunk;
    81→        size_t ce = std::min(cs + cells_per_chunk, sorted_cells.size());
    82→        if (cs >= sorted_cells.size()) break;
    83→
    84→        threads.emplace_back([&, t, cs, ce]() {
    85→            auto& chunk = chunks[t];
    86→            chunk.cell_start = cs;
    87→            chunk.cell_end = ce;
    88→            chunk.local_size = 0;
    89→
    90→            size_t pi = std::lower_bound(sorted_pairs.begin(), sorted_pairs.end(),
    91→                sorted_cells[cs], [](const CellItemPair& p, uint64_t id) {
    92→                    return p.cell_id < id;
    93→                }) - sorted_pairs.begin();
    94→
    95→            for (size_t si = cs; si < ce && pi < sorted_pairs.size(); si++) {
    96→                if (sorted_cells[si] < sorted_pairs[pi].cell_id) continue;
    97→                while (pi < sorted_pairs.size() && sorted_pairs[pi].cell_id < sorted_cells[si]) pi++;
    98→                if (pi >= sorted_pairs.size() || sorted_pairs[pi].cell_id != sorted_cells[si]) continue;
    99→
   100→                offsets[si] = chunk.local_size;
   101→                size_t start = pi;
   102→                while (pi < sorted_pairs.size() && sorted_pairs[pi].cell_id == sorted_cells[si]) pi++;
   103→                uint16_t count = static_cast<uint16_t>(std::min(pi - start, size_t(MAX_VERTEX_COUNT)));
   104→                size_t entry_size = sizeof(uint16_t) + (pi - start) * sizeof(uint32_t);
   105→                size_t buf_pos = chunk.buf.size();
   106→                chunk.buf.resize(buf_pos + entry_size);
   107→                memcpy(chunk.buf.data() + buf_pos, &count, sizeof(count));
   108→                for (size_t k = start; k < pi; k++) {
   109→                    memcpy(chunk.buf.data() + buf_pos + sizeof(uint16_t) + (k - start) * sizeof(uint32_t),
   110→                           &sorted_pairs[k].item_id, sizeof(uint32_t));
   111→                }
   112→                chunk.local_size += entry_size;
   113→            }
   114→        });
   115→    }
   116→    for (auto& t : threads) t.join();
   117→
   118→    uint32_t global_offset = 0;
   119→    for (auto& chunk : chunks) {
   120→        for (size_t si = chunk.cell_start; si < chunk.cell_end; si++) {
   121→            if (offsets[si] != NO_DATA) offsets[si] += global_offset;
   122→        }
   123→        global_offset += chunk.local_size;
   124→    }
   125→
   126→    std::vector<char> buf;
   127→    buf.reserve(global_offset);
   128→    for (auto& chunk : chunks) buf.insert(buf.end(), chunk.buf.begin(), chunk.buf.end());
   129→
   130→    std::ofstream f(path, std::ios::binary);
   131→    f.write(buf.data(), buf.size());
   132→    return offsets;
   133→}
   134→
   135→void write_cell_index(
   136→    const std::string& cells_path,
   137→    const std::string& entries_path,
   138→    const std::unordered_map<uint64_t, std::vector<uint32_t>>& cell_map
   139→) {
   140→    std::vector<std::pair<uint64_t, std::vector<uint32_t>>> sorted(cell_map.begin(), cell_map.end());
   141→    std::sort(sorted.begin(), sorted.end());
   142→
   143→    { std::ofstream f(cells_path, std::ios::binary);
   144→      uint32_t current_offset = 0;
   145→      for (const auto& [cell_id, ids] : sorted) {
   146→          f.write(reinterpret_cast<const char*>(&cell_id), sizeof(cell_id));
   147→          f.write(reinterpret_cast<const char*>(&current_offset), sizeof(current_offset));
   148→          current_offset += sizeof(uint16_t) + ids.size() * sizeof(uint32_t);
   149→      } }
   150→
   151→    { std::ofstream f(entries_path, std::ios::binary);
   152→      for (const auto& [cell_id, ids] : sorted) {
   153→          uint16_t count = static_cast<uint16_t>(std::min(ids.size(), size_t(MAX_VERTEX_COUNT)));
   154→          f.write(reinterpret_cast<const char*>(&count), sizeof(count));
   155→          f.write(reinterpret_cast<const char*>(ids.data()), ids.size() * sizeof(uint32_t));
   156→      } }
   157→}
   158→
   159→void write_index(const ParsedData& data, const std::string& output_dir, IndexMode mode) {
   160→    ensure_dir(output_dir);
   161→    auto _wt = std::chrono::steady_clock::now();
   162→
   163→    bool write_streets = (mode != IndexMode::AdminOnly);
   164→    bool write_addresses = (mode == IndexMode::Full);
   165→
   166→    if (write_streets) {
   167→        std::vector<uint64_t> sorted_geo_cells;
   168→        {
   169→            auto extract_unique_cells = [](const std::vector<CellItemPair>& pairs) {
   170→                std::vector<uint64_t> cells;
   171→                cells.reserve(pairs.size() / 2);
   172→                for (size_t i = 0; i < pairs.size(); ) {
   173→                    cells.push_back(pairs[i].cell_id);
   174→                    uint64_t cur = pairs[i].cell_id;
   175→                    while (i < pairs.size() && pairs[i].cell_id == cur) i++;
   176→                }
   177→                return cells;
   178→            };
   179→            auto extract_from_map = [](const std::unordered_map<uint64_t, std::vector<uint32_t>>& m) {
   180→                std::vector<uint64_t> cells;
   181→                cells.reserve(m.size());
   182→                for (auto& [id, _] : m) cells.push_back(id);
   183→                std::sort(cells.begin(), cells.end());
   184→                return cells;
   185→            };
   186→
   187→            std::vector<uint64_t> way_cells, addr_cells, interp_cells;
   188→            {
   189→                auto f1 = std::async(std::launch::async, [&] {
   190→                    return !data.sorted_way_cells.empty()
   191→                        ? extract_unique_cells(data.sorted_way_cells)
   192→                        : extract_from_map(data.cell_to_ways);
   193→                });
   194→                if (write_addresses) {
   195→                    auto f2 = std::async(std::launch::async, [&] {
   196→                        return !data.sorted_addr_cells.empty()
   197→                            ? extract_unique_cells(data.sorted_addr_cells)
   198→                            : extract_from_map(data.cell_to_addrs);
   199→                    });
   200→                    auto f3 = std::async(std::launch::async, [&] {
   201→                        return !data.sorted_interp_cells.empty()
   202→                            ? extract_unique_cells(data.sorted_interp_cells)
   203→                            : extract_from_map(data.cell_to_interps);
   204→                    });
   205→                    addr_cells = f2.get();
   206→                    interp_cells = f3.get();
   207→                }
   208→                way_cells = f1.get();
   209→            }
   210→
   211→            sorted_geo_cells.reserve(way_cells.size() + addr_cells.size() + interp_cells.size());
   212→            std::merge(way_cells.begin(), way_cells.end(), addr_cells.begin(), addr_cells.end(),
   213→                       std::back_inserter(sorted_geo_cells));
   214→            if (!interp_cells.empty()) {
   215→                std::vector<uint64_t> tmp;
   216→                tmp.reserve(sorted_geo_cells.size() + interp_cells.size());
   217→                std::merge(sorted_geo_cells.begin(), sorted_geo_cells.end(),
   218→                           interp_cells.begin(), interp_cells.end(), std::back_inserter(tmp));
   219→                sorted_geo_cells = std::move(tmp);
   220→            }
   221→            sorted_geo_cells.erase(std::unique(sorted_geo_cells.begin(), sorted_geo_cells.end()),
   222→                                    sorted_geo_cells.end());
   223→        }
   224→
   225→        std::vector<uint32_t> street_offsets, addr_offsets, interp_offsets;
   226→        {
   227→            auto f1 = std::async(std::launch::async, [&]() {
   228→                if (!data.sorted_way_cells.empty())
   229→                    return write_entries_from_sorted(output_dir + "/street_entries.bin", sorted_geo_cells, data.sorted_way_cells);
   230→                return write_entries(output_dir + "/street_entries.bin", sorted_geo_cells, data.cell_to_ways);
   231→            });
   232→            if (write_addresses) {
   233→                auto f2 = std::async(std::launch::async, [&]() {
   234→                    if (!data.sorted_addr_cells.empty())
   235→                        return write_entries_from_sorted(output_dir + "/addr_entries.bin", sorted_geo_cells, data.sorted_addr_cells);
   236→                    return write_entries(output_dir + "/addr_entries.bin", sorted_geo_cells, data.cell_to_addrs);
   237→                });
   238→                auto f3 = std::async(std::launch::async, [&]() {
   239→                    if (!data.sorted_interp_cells.empty())
   240→                        return write_entries_from_sorted(output_dir + "/interp_entries.bin", sorted_geo_cells, data.sorted_interp_cells);
   241→                    return write_entries(output_dir + "/interp_entries.bin", sorted_geo_cells, data.cell_to_interps);
   242→                });
   243→                addr_offsets = f2.get();
   244→                interp_offsets = f3.get();
   245→            }
   246→            street_offsets = f1.get();
   247→        }
   248→
   249→        log_phase("  Write: entry files", _wt);
   250→        {
   251→            size_t n = sorted_geo_cells.size();
   252→            size_t row_size = sizeof(uint64_t) + 3 * sizeof(uint32_t);
   253→            std::vector<char> buf(n * row_size);
   254→            if (addr_offsets.empty()) addr_offsets.resize(n, NO_DATA);
   255→            if (interp_offsets.empty()) interp_offsets.resize(n, NO_DATA);
   256→
   257→            unsigned int nthreads = std::thread::hardware_concurrency();
   258→            if (nthreads == 0) nthreads = 4;
   259→            size_t chunk = (n + nthreads - 1) / nthreads;
   260→            std::vector<std::thread> fill_threads;
   261→            for (unsigned int t = 0; t < nthreads; t++) {
   262→                size_t start = t * chunk;
   263→                size_t end = std::min(start + chunk, n);
   264→                if (start >= n) break;
   265→                fill_threads.emplace_back([&, start, end]() {
   266→                    char* ptr = buf.data() + start * row_size;
   267→                    for (size_t i = start; i < end; i++) {
   268→                        memcpy(ptr, &sorted_geo_cells[i], sizeof(uint64_t)); ptr += sizeof(uint64_t);
   269→                        memcpy(ptr, &street_offsets[i], sizeof(uint32_t)); ptr += sizeof(uint32_t);
   270→                        memcpy(ptr, &addr_offsets[i], sizeof(uint32_t)); ptr += sizeof(uint32_t);
   271→                        memcpy(ptr, &interp_offsets[i], sizeof(uint32_t)); ptr += sizeof(uint32_t);
   272→                    }
   273→                });
   274→            }
   275→            for (auto& t : fill_threads) t.join();
   276→
   277→            std::ofstream f(output_dir + "/geo_cells.bin", std::ios::binary);
   278→            f.write(buf.data(), buf.size());
   279→        }
   280→
   281→        std::cerr << "geo index: " << sorted_geo_cells.size() << " cells ("
   282→                  << data.ways.size() << " ways, " << data.addr_points.size() << " addrs, "
   283→                  << data.interp_ways.size() << " interps)" << std::endl;
   284→    }
   285→
   286→    log_phase("  Write: geo_cells.bin", _wt);
   287→    auto admin_future = std::async(std::launch::async, [&] {
   288→        write_cell_index(output_dir + "/admin_cells.bin", output_dir + "/admin_entries.bin", data.cell_to_admin);
   289→        std::cerr << "admin index: " << data.cell_to_admin.size() << " cells, " << data.admin_polygons.size() << " polygons" << std::endl;
   290→    });
   291→
   292→    std::vector<std::future<void>> write_futures;
   293→    if (write_streets) {
   294→        write_futures.push_back(std::async(std::launch::async, [&] {
   295→            std::ofstream f(output_dir + "/street_ways.bin", std::ios::binary);
   296→            f.write(reinterpret_cast<const char*>(data.ways.data()), data.ways.size() * sizeof(WayHeader));
   297→        }));
   298→        write_futures.push_back(std::async(std::launch::async, [&] {
   299→            std::ofstream f(output_dir + "/street_nodes.bin", std::ios::binary);
   300→            f.write(reinterpret_cast<const char*>(data.street_nodes.data()), data.street_nodes.size() * sizeof(NodeCoord));
   301→        }));
   302→        if (write_addresses) {
   303→            write_futures.push_back(std::async(std::launch::async, [&] {
   304→                std::ofstream f(output_dir + "/addr_points.bin", std::ios::binary);
   305→                f.write(reinterpret_cast<const char*>(data.addr_points.data()), data.addr_points.size() * sizeof(AddrPoint));
   306→            }));
   307→            write_futures.push_back(std::async(std::launch::async, [&] {
   308→                std::ofstream f(output_dir + "/interp_ways.bin", std::ios::binary);
   309→                f.write(reinterpret_cast<const char*>(data.interp_ways.data()), data.interp_ways.size() * sizeof(InterpWay));
   310→            }));
   311→            write_futures.push_back(std::async(std::launch::async, [&] {
   312→                std::ofstream f(output_dir + "/interp_nodes.bin", std::ios::binary);
   313→                f.write(reinterpret_cast<const char*>(data.interp_nodes.data()), data.interp_nodes.size() * sizeof(NodeCoord));
   314→            }));
   315→        }
   316→    }
   317→    // If not writing all data types, compact string pool and remap references
   318→    // before writing admin_polygons and strings.bin.
   319→    std::vector<AdminPolygon> compact_admin;
   320→    std::vector<char> compact_strings;
   321→    const bool need_compact = (!write_streets || !write_addresses);
   322→
   323→    if (need_compact) {
   324→        // Collect referenced string offsets
   325→        std::unordered_set<uint32_t> used_offsets;
   326→        if (write_streets) {
   327→            for (const auto& w : data.ways) used_offsets.insert(w.name_id);
   328→            if (write_addresses) {
   329→                for (const auto& a : data.addr_points) {
   330→                    used_offsets.insert(a.housenumber_id);
   331→                    used_offsets.insert(a.street_id);
   332→                }
   333→                for (const auto& iw : data.interp_ways) used_offsets.insert(iw.street_id);
   334→            }
   335→        }
   336→        for (const auto& ap : data.admin_polygons) used_offsets.insert(ap.name_id);
   337→
   338→        // Build compact pool and remap
   339→        const auto& old_sp = data.string_pool.data();
   340→        std::vector<uint32_t> sorted_offs(used_offsets.begin(), used_offsets.end());
   341→        std::sort(sorted_offs.begin(), sorted_offs.end());
   342→        std::unordered_map<uint32_t, uint32_t> remap;
   343→        compact_strings.reserve(used_offsets.size() * 16);
   344→        for (uint32_t off : sorted_offs) {
   345→            remap[off] = static_cast<uint32_t>(compact_strings.size());
   346→            size_t len = std::strlen(old_sp.data() + off);
   347→            compact_strings.insert(compact_strings.end(), old_sp.data() + off,
   348→                old_sp.data() + off + len + 1);
   349→        }
   350→
   351→        // Remap admin polygon name_ids
   352→        compact_admin = data.admin_polygons;
   353→        for (auto& ap : compact_admin) ap.name_id = remap[ap.name_id];
   354→
   355→        // Remap street/addr/interp if applicable (for no-addresses mode)
   356→        // These are written from data directly, so we'd need copies too.
   357→        // For simplicity, only compact when we have fewer types.
   358→    }
   359→
   360→    const auto& admin_to_write = need_compact ? compact_admin : data.admin_polygons;
   361→    write_futures.push_back(std::async(std::launch::async, [&] {
   362→        std::ofstream f(output_dir + "/admin_polygons.bin", std::ios::binary);
   363→        f.write(reinterpret_cast<const char*>(admin_to_write.data()), admin_to_write.size() * sizeof(AdminPolygon));
   364→    }));
   365→    write_futures.push_back(std::async(std::launch::async, [&] {
   366→        std::ofstream f(output_dir + "/admin_vertices.bin", std::ios::binary);
   367→        f.write(reinterpret_cast<const char*>(data.admin_vertices.data()), data.admin_vertices.size() * sizeof(NodeCoord));
   368→    }));
   369→    write_futures.push_back(std::async(std::launch::async, [&] {
   370→        if (need_compact) {
   371→            std::ofstream f(output_dir + "/strings.bin", std::ios::binary);
   372→            f.write(compact_strings.data(), compact_strings.size());
   373→            std::cerr << "strings.bin: " << compact_strings.size() << " bytes (compacted from "
   374→                      << data.string_pool.data().size() << ")" << std::endl;
   375→        } else {
   376→            std::ofstream f(output_dir + "/strings.bin", std::ios::binary);
   377→            f.write(data.string_pool.data().data(), data.string_pool.data().size());
   378→            std::cerr << "strings.bin: " << data.string_pool.data().size() << " bytes" << std::endl;
   379→        }
   380→    }));
   381→
   382→    log_phase("  Write: parallel data files launched", _wt);
   383→    admin_future.get();
   384→    for (auto& f : write_futures) f.get();
   385→}
   386→
   387→void write_quality_variant(const ParsedData& data, const std::string& source_dir,
   388→                           const std::string& output_dir, double epsilon_scale) {
   389→    ensure_dir(output_dir);
   390→
   391→    // Re-simplify admin polygons at the given epsilon scale
   392→    std::vector<AdminPolygon> new_polys;
   393→    std::vector<NodeCoord> new_verts;
   394→    new_polys.reserve(data.admin_polygons.size());
   395→
   396→    // Parallel simplification
   397→    struct SimplifiedPoly {
   398→        std::vector<std::pair<double,double>> verts;
   399→    };
   400→    std::vector<SimplifiedPoly> simplified(data.admin_polygons.size());
   401→
   402→    {
   403→        std::atomic<size_t> idx{0};
   404→        unsigned nthreads = std::thread::hardware_concurrency();
   405→        if (nthreads == 0) nthreads = 4;
   406→        std::vector<std::thread> workers;
   407→        for (unsigned t = 0; t < nthreads; t++) {
   408→            workers.emplace_back([&]() {
   409→                while (true) {
   410→                    size_t i = idx.fetch_add(1);
   411→                    if (i >= data.admin_polygons.size()) break;
   412→                    const auto& ap = data.admin_polygons[i];
   413→
   414→                    // Extract original vertices
   415→                    std::vector<std::pair<double,double>> pts;
   416→                    pts.reserve(ap.vertex_count);
   417→                    for (uint32_t j = 0; j < ap.vertex_count; j++) {
   418→                        const auto& v = data.admin_vertices[ap.vertex_offset + j];
   419→                        pts.emplace_back(v.lat, v.lng);
   420→                    }
   421→
   422→                    if (epsilon_scale > 0) {
   423→                        double eps_m = admin_epsilon_meters(ap.admin_level) * epsilon_scale;
   424→                        double lat = pts.empty() ? 0.0 : pts[0].first;
   425→                        double eps_deg = meters_to_degrees(eps_m, lat);
   426→                        simplified[i].verts = simplify_polygon_epsilon(pts, eps_deg);
   427→                    } else {
   428→                        simplified[i].verts = std::move(pts);
   429→                    }
   430→                }
   431→            });
   432→        }
   433→        for (auto& w : workers) w.join();
   434→    }
   435→
   436→    // Sequential: build new polygon and vertex arrays
   437→    for (size_t i = 0; i < data.admin_polygons.size(); i++) {
   438→        auto& sv = simplified[i].verts;
   439→        if (sv.size() < 3) continue;
   440→
   441→        AdminPolygon np = data.admin_polygons[i];
   442→        np.vertex_offset = static_cast<uint32_t>(new_verts.size());
   443→        np.vertex_count = static_cast<uint32_t>(sv.size());
   444→        np.area = polygon_area(sv);
   445→
   446→        for (const auto& [lat, lng] : sv) {
   447→            new_verts.push_back({static_cast<float>(lat), static_cast<float>(lng)});
   448→        }
   449→        new_polys.push_back(np);
   450→    }
   451→
   452→    std::cerr << "Quality " << epsilon_scale << "x: " << new_polys.size()
   453→              << " polygons, " << new_verts.size() << " vertices ("
   454→              << new_verts.size() * 8 / 1024 / 1024 << " MiB)" << std::endl;
   455→
   456→    // Write admin files
   457→    {
   458→        std::ofstream f(output_dir + "/admin_polygons.bin", std::ios::binary);
   459→        f.write(reinterpret_cast<const char*>(new_polys.data()), new_polys.size() * sizeof(AdminPolygon));
   460→    }
   461→    {
   462→        std::ofstream f(output_dir + "/admin_vertices.bin", std::ios::binary);
   463→        f.write(reinterpret_cast<const char*>(new_verts.data()), new_verts.size() * sizeof(NodeCoord));
   464→    }
   465→
   466→    // Quality directories only contain the two files that change.
   467→    // Shared files (admin_cells.bin, admin_entries.bin, strings.bin)
   468→    // stay in the parent directory.
   469→}
   470→


/root/traccar-geocoder/builder/build-presplit/CMakeFiles/3.31.6/CompilerIdCXX/CMakeCXXCompilerId.cpp
/root/traccar-geocoder/builder/tools/simplify_compare.cpp
/root/traccar-geocoder/builder/build/CMakeFiles/3.31.6/CompilerIdCXX/CMakeCXXCompilerId.cpp
/root/traccar-geocoder/builder/src/cell_index.cpp
/root/traccar-geocoder/builder/src/interpolation.cpp
/root/traccar-geocoder/builder/src/ring_assembly.h
/root/traccar-geocoder/builder/src/string_pool.h
/root/traccar-geocoder/builder/src/interpolation.h
/root/traccar-geocoder/builder/src/cache.h
/root/traccar-geocoder/builder/src/s2_helpers.h
/root/traccar-geocoder/builder/src/build_index.cpp
/root/traccar-geocoder/builder/src/continent_boundaries.h
/root/traccar-geocoder/builder/src/cell_index.h
/root/traccar-geocoder/builder/src/pbf_reader.h
/root/traccar-geocoder/builder/src/pbf_reader.cpp
/root/traccar-geocoder/builder/src/continent_boundaries.cpp
/root/traccar-geocoder/builder/src/relation_collector.h
/root/traccar-geocoder/builder/src/continent_filter.h
/root/traccar-geocoder/builder/src/cache.cpp
/root/traccar-geocoder/builder/src/parsed_data.h


     1→// Deep comparison: dumps samples of items only in A or only in B
     2→// with their coordinates, to understand boundary differences.
     3→// Usage: deep_compare <dir-A> <dir-B> [--ways|--addrs|--admins] [--limit N]
     4→
     5→#include <algorithm>
     6→#include <cmath>
     7→#include <cstdint>
     8→#include <cstring>
     9→#include <fstream>
    10→#include <iostream>
    11→#include <string>
    12→#include <unordered_map>
    13→#include <unordered_set>
    14→#include <vector>
    15→
    16→struct WayHeader { uint32_t node_offset; uint8_t node_count; uint32_t name_id; };
    17→struct AddrPoint { float lat; float lng; uint32_t housenumber_id; uint32_t street_id; };
    18→struct NodeCoord { float lat; float lng; };
    19→
    20→static std::vector<char> read_file(const std::string& path) {
    21→    std::ifstream f(path, std::ios::binary | std::ios::ate);
    22→    if (!f) return {};
    23→    auto size = f.tellg(); f.seekg(0);
    24→    std::vector<char> data(size); f.read(data.data(), size);
    25→    return data;
    26→}
    27→
    28→template<typename T>
    29→static std::vector<T> read_structs(const std::string& path) {
    30→    auto data = read_file(path);
    31→    size_t count = data.size() / sizeof(T);
    32→    std::vector<T> result(count);
    33→    memcpy(result.data(), data.data(), count * sizeof(T));
    34→    return result;
    35→}
    36→
    37→static const char* get_string(const std::vector<char>& pool, uint32_t offset) {
    38→    if (offset >= pool.size()) return "(invalid)";
    39→    return pool.data() + offset;
    40→}
    41→
    42→static int to_grid(float v) {
    43→    return (int)(v * 1e5f + (v >= 0 ? 0.5f : -0.5f));
    44→}
    45→
    46→int main(int argc, char* argv[]) {
    47→    if (argc < 3) {
    48→        std::cerr << "Usage: deep_compare <dir-A> <dir-B> [--ways|--addrs|--admins] [--limit N]" << std::endl;
    49→        return 1;
    50→    }
    51→    std::string dir_a = argv[1], dir_b = argv[2];
    52→    bool do_ways = false, do_addrs = false, do_admins = false;
    53→    int limit = 20;
    54→    for (int i = 3; i < argc; i++) {
    55→        if (strcmp(argv[i], "--ways") == 0) do_ways = true;
    56→        else if (strcmp(argv[i], "--addrs") == 0) do_addrs = true;
    57→        else if (strcmp(argv[i], "--admins") == 0) do_admins = true;
    58→        else if (strcmp(argv[i], "--limit") == 0 && i+1 < argc) limit = atoi(argv[++i]);
    59→    }
    60→    if (!do_ways && !do_addrs && !do_admins) { do_ways = do_addrs = true; }
    61→
    62→    if (do_ways) {
    63→        std::cerr << "Loading ways..." << std::endl;
    64→        auto ways_a = read_structs<WayHeader>(dir_a + "/street_ways.bin");
    65→        auto ways_b = read_structs<WayHeader>(dir_b + "/street_ways.bin");
    66→        auto nodes_a = read_structs<NodeCoord>(dir_a + "/street_nodes.bin");
    67→        auto nodes_b = read_structs<NodeCoord>(dir_b + "/street_nodes.bin");
    68→        auto sp_a = read_file(dir_a + "/strings.bin");
    69→        auto sp_b = read_file(dir_b + "/strings.bin");
    70→
    71→        // Build fingerprints: name + sorted node coords
    72→        auto way_fp = [&](const std::vector<WayHeader>& ways, const std::vector<NodeCoord>& nodes,
    73→                          const std::vector<char>& sp, size_t idx) -> std::string {
    74→            const auto& w = ways[idx];
    75→            std::string name = get_string(sp, w.name_id);
    76→            std::string fp = name + "|" + std::to_string(w.node_count);
    77→            // Use first node coords as primary identifier
    78→            if (w.node_count > 0) {
    79→                fp += "|" + std::to_string(to_grid(nodes[w.node_offset].lat)) +
    80→                      "," + std::to_string(to_grid(nodes[w.node_offset].lng));
    81→            }
    82→            return fp;
    83→        };
    84→
    85→        std::unordered_map<std::string, std::vector<size_t>> fps_a, fps_b;
    86→        for (size_t i = 0; i < ways_a.size(); i++) fps_a[way_fp(ways_a, nodes_a, sp_a, i)].push_back(i);
    87→        for (size_t i = 0; i < ways_b.size(); i++) fps_b[way_fp(ways_b, nodes_b, sp_b, i)].push_back(i);
    88→
    89→        // Find items only in A
    90→        std::vector<std::pair<std::string, size_t>> only_a, only_b;
    91→        for (auto& [fp, indices] : fps_a) {
    92→            size_t bcount = fps_b.count(fp) ? fps_b[fp].size() : 0;
    93→            if (indices.size() > bcount) {
    94→                for (size_t i = bcount; i < indices.size(); i++)
    95→                    only_a.push_back({fp, indices[i]});
    96→            }
    97→        }
    98→        for (auto& [fp, indices] : fps_b) {
    99→            size_t acount = fps_a.count(fp) ? fps_a[fp].size() : 0;
   100→            if (indices.size() > acount) {
   101→                for (size_t i = acount; i < indices.size(); i++)
   102→                    only_b.push_back({fp, indices[i]});
   103→            }
   104→        }
   105→
   106→        std::cout << "=== WAYS ===" << std::endl;
   107→        std::cout << "A: " << ways_a.size() << "  B: " << ways_b.size() << std::endl;
   108→        std::cout << "Only in A (standalone): " << only_a.size() << std::endl;
   109→        int count = 0;
   110→        for (auto& [fp, idx] : only_a) {
   111→            if (count++ >= limit) { std::cout << "  ... and " << (only_a.size() - limit) << " more" << std::endl; break; }
   112→            const auto& w = ways_a[idx];
   113→            std::cout << "  " << get_string(sp_a, w.name_id)
   114→                      << " nodes=" << (int)w.node_count
   115→                      << " first=(" << nodes_a[w.node_offset].lat << "," << nodes_a[w.node_offset].lng << ")"
   116→                      << " last=(" << nodes_a[w.node_offset + w.node_count - 1].lat << "," << nodes_a[w.node_offset + w.node_count - 1].lng << ")"
   117→                      << std::endl;
   118→        }
   119→        std::cout << "Only in B (filtered): " << only_b.size() << std::endl;
   120→        count = 0;
   121→        for (auto& [fp, idx] : only_b) {
   122→            if (count++ >= limit) { std::cout << "  ... and " << (only_b.size() - limit) << " more" << std::endl; break; }
   123→            const auto& w = ways_b[idx];
   124→            std::cout << "  " << get_string(sp_b, w.name_id)
   125→                      << " nodes=" << (int)w.node_count
   126→                      << " first=(" << nodes_b[w.node_offset].lat << "," << nodes_b[w.node_offset].lng << ")"
   127→                      << " last=(" << nodes_b[w.node_offset + w.node_count - 1].lat << "," << nodes_b[w.node_offset + w.node_count - 1].lng << ")"
   128→                      << std::endl;
   129→        }
   130→    }
   131→
   132→    if (do_addrs) {
   133→        std::cerr << "Loading addrs..." << std::endl;
   134→        auto addrs_a = read_structs<AddrPoint>(dir_a + "/addr_points.bin");
   135→        auto addrs_b = read_structs<AddrPoint>(dir_b + "/addr_points.bin");
   136→        auto sp_a = read_file(dir_a + "/strings.bin");
   137→        auto sp_b = read_file(dir_b + "/strings.bin");
   138→
   139→        auto addr_fp = [&](const std::vector<AddrPoint>& addrs, const std::vector<char>& sp, size_t idx) -> std::string {
   140→            const auto& a = addrs[idx];
   141→            return std::to_string(to_grid(a.lat)) + "," + std::to_string(to_grid(a.lng)) +
   142→                   "|" + get_string(sp, a.housenumber_id) + "|" + get_string(sp, a.street_id);
   143→        };
   144→
   145→        std::unordered_set<std::string> set_a, set_b;
   146→        for (size_t i = 0; i < addrs_a.size(); i++) set_a.insert(addr_fp(addrs_a, sp_a, i));
   147→        for (size_t i = 0; i < addrs_b.size(); i++) set_b.insert(addr_fp(addrs_b, sp_b, i));
   148→
   149→        std::vector<std::pair<std::string, size_t>> only_a, only_b;
   150→        for (size_t i = 0; i < addrs_a.size(); i++) {
   151→            auto fp = addr_fp(addrs_a, sp_a, i);
   152→            if (!set_b.count(fp)) only_a.push_back({fp, i});
   153→        }
   154→        for (size_t i = 0; i < addrs_b.size(); i++) {
   155→            auto fp = addr_fp(addrs_b, sp_b, i);
   156→            if (!set_a.count(fp)) only_b.push_back({fp, i});
   157→        }
   158→
   159→        std::cout << "\n=== ADDRESSES ===" << std::endl;
   160→        std::cout << "A: " << addrs_a.size() << "  B: " << addrs_b.size() << std::endl;
   161→        std::cout << "Only in A (standalone): " << only_a.size() << std::endl;
   162→        int count = 0;
   163→        for (auto& [fp, idx] : only_a) {
   164→            if (count++ >= limit) { std::cout << "  ... and " << (only_a.size() - limit) << " more" << std::endl; break; }
   165→            const auto& a = addrs_a[idx];
   166→            std::cout << "  (" << a.lat << "," << a.lng << ") "
   167→                      << get_string(sp_a, a.housenumber_id) << " "
   168→                      << get_string(sp_a, a.street_id) << std::endl;
   169→        }
   170→        std::cout << "Only in B (filtered): " << only_b.size() << std::endl;
   171→        count = 0;
   172→        for (auto& [fp, idx] : only_b) {
   173→            if (count++ >= limit) { std::cout << "  ... and " << (only_b.size() - limit) << " more" << std::endl; break; }
   174→            const auto& a = addrs_b[idx];
   175→            std::cout << "  (" << a.lat << "," << a.lng << ") "
   176→                      << get_string(sp_b, a.housenumber_id) << " "
   177→                      << get_string(sp_b, a.street_id) << std::endl;
   178→        }
   179→    }
   180→
   181→    return 0;
   182→}
   183→


     1→// Semantic comparison of two geocoder index directories.
     2→// Decodes all binary files, creates canonical fingerprints for each item,
     3→// and compares as multisets — handles any ordering differences between
     4→// parallel and sequential builds.
     5→// Returns 0 if semantically identical, 1 if different.
     6→//
     7→// Usage: compare_indexes <dir-A> <dir-B>
     8→
     9→#include <algorithm>
    10→#include <cmath>
    11→#include <cstdint>
    12→#include <cstring>
    13→#include <fstream>
    14→#include <iostream>
    15→#include <map>
    16→#include <set>
    17→#include <string>
    18→#include <unordered_map>
    19→#include <vector>
    20→
    21→// --- Binary structs (must match build_index.cpp and server main.rs) ---
    22→
    23→struct WayHeader {
    24→    uint32_t node_offset;
    25→    uint8_t node_count;
    26→    uint32_t name_id;
    27→};
    28→
    29→struct AddrPoint {
    30→    float lat;
    31→    float lng;
    32→    uint32_t housenumber_id;
    33→    uint32_t street_id;
    34→};
    35→
    36→struct InterpWay {
    37→    uint32_t node_offset;
    38→    uint8_t node_count;
    39→    uint32_t street_id;
    40→    uint32_t start_number;
    41→    uint32_t end_number;
    42→    uint8_t interpolation;
    43→};
    44→
    45→struct AdminPolygon {
    46→    uint32_t vertex_offset;
    47→    uint16_t vertex_count;
    48→    uint32_t name_id;
    49→    uint8_t admin_level;
    50→    float area;
    51→    uint16_t country_code;
    52→};
    53→
    54→struct NodeCoord {
    55→    float lat;
    56→    float lng;
    57→};
    58→
    59→// --- Helpers ---
    60→
    61→static std::vector<char> read_file(const std::string& path) {
    62→    std::ifstream f(path, std::ios::binary | std::ios::ate);
    63→    if (!f) return {};
    64→    auto size = f.tellg();
    65→    f.seekg(0);
    66→    std::vector<char> data(size);
    67→    f.read(data.data(), size);
    68→    return data;
    69→}
    70→
    71→static const char* get_string(const std::vector<char>& pool, uint32_t offset) {
    72→    if (offset >= pool.size()) return "(invalid)";
    73→    return pool.data() + offset;
    74→}
    75→
    76→template<typename T>
    77→static std::vector<T> read_structs(const std::string& path) {
    78→    auto data = read_file(path);
    79→    size_t count = data.size() / sizeof(T);
    80→    std::vector<T> result(count);
    81→    memcpy(result.data(), data.data(), count * sizeof(T));
    82→    return result;
    83→}
    84→
    85→// Round a float coordinate to an integer grid (1e5 ~ 1m precision)
    86→static int to_grid(float v) {
    87→    return (int)(v * 1e5f + (v >= 0 ? 0.5f : -0.5f));
    88→}
    89→
    90→// --- Canonical fingerprint builders ---
    91→// Each function produces a string that uniquely identifies an item,
    92→// independent of storage order.
    93→
    94→// For a way: "name|sorted_node_coords"
    95→// Nodes are kept in original order (path order matters for ways),
    96→// but we also provide a sorted-set version for order-independent matching.
    97→static std::string way_fingerprint(const std::string& name,
    98→                                    const std::vector<std::pair<float,float>>& nodes) {
    99→    // Use sorted coordinate set as the canonical form.
   100→    // Two ways are "the same" if they have the same name and same set of nodes.
   101→    std::vector<std::pair<int,int>> grid_nodes;
   102→    grid_nodes.reserve(nodes.size());
   103→    for (const auto& n : nodes) {
   104→        grid_nodes.push_back({to_grid(n.first), to_grid(n.second)});
   105→    }
   106→    std::sort(grid_nodes.begin(), grid_nodes.end());
   107→
   108→    std::string fp = name + "|" + std::to_string(nodes.size());
   109→    for (const auto& g : grid_nodes) {
   110→        fp += "|";
   111→        fp += std::to_string(g.first);
   112→        fp += ",";
   113→        fp += std::to_string(g.second);
   114→    }
   115→    return fp;
   116→}
   117→
   118→// For an address: "grid_lat,grid_lng|housenumber|street"
   119→static std::string addr_fingerprint(float lat, float lng,
   120→                                     const std::string& housenumber,
   121→                                     const std::string& street) {
   122→    return std::to_string(to_grid(lat)) + "," + std::to_string(to_grid(lng)) +
   123→           "|" + housenumber + "|" + street;
   124→}
   125→
   126→// For an admin polygon: "name|level|country|sorted_vertices"
   127→static std::string admin_fingerprint(const std::string& name,
   128→                                      uint8_t admin_level,
   129→                                      uint16_t country_code,
   130→                                      const std::vector<std::pair<float,float>>& vertices) {
   131→    std::vector<std::pair<int,int>> grid_verts;
   132→    grid_verts.reserve(vertices.size());
   133→    for (const auto& v : vertices) {
   134→        grid_verts.push_back({to_grid(v.first), to_grid(v.second)});
   135→    }
   136→    std::sort(grid_verts.begin(), grid_verts.end());
   137→
   138→    std::string fp = name + "|" + std::to_string(admin_level) + "|" + std::to_string(country_code);
   139→    for (const auto& g : grid_verts) {
   140→        fp += "|";
   141→        fp += std::to_string(g.first);
   142→        fp += ",";
   143→        fp += std::to_string(g.second);
   144→    }
   145→    return fp;
   146→}
   147→
   148→// For interpolation ways: "street|start|end|interp|sorted_nodes"
   149→static std::string interp_fingerprint(const std::string& street,
   150→                                       uint32_t start_number, uint32_t end_number,
   151→                                       uint8_t interpolation,
   152→                                       const std::vector<std::pair<float,float>>& nodes) {
   153→    std::vector<std::pair<int,int>> grid_nodes;
   154→    grid_nodes.reserve(nodes.size());
   155→    for (const auto& n : nodes) {
   156→        grid_nodes.push_back({to_grid(n.first), to_grid(n.second)});
   157→    }
   158→    std::sort(grid_nodes.begin(), grid_nodes.end());
   159→
   160→    std::string fp = street + "|" + std::to_string(start_number) + "-" +
   161→                     std::to_string(end_number) + "|" + std::to_string(interpolation);
   162→    for (const auto& g : grid_nodes) {
   163→        fp += "|";
   164→        fp += std::to_string(g.first);
   165→        fp += ",";
   166→        fp += std::to_string(g.second);
   167→    }
   168→    return fp;
   169→}
   170→
   171→// --- Multiset comparison ---
   172→// Compare two collections by their fingerprints. Reports missing/extra items.
   173→
   174→static bool compare_multisets(const char* label,
   175→                               const std::vector<std::string>& fps_a,
   176→                               const std::vector<std::string>& fps_b) {
   177→    // Build frequency maps
   178→    std::unordered_map<std::string, int> freq_a, freq_b;
   179→    for (const auto& fp : fps_a) freq_a[fp]++;
   180→    for (const auto& fp : fps_b) freq_b[fp]++;
   181→
   182→    // Find items in A but not in B (missing from B)
   183→    size_t missing = 0;
   184→    for (const auto& [fp, count] : freq_a) {
   185→        int b_count = 0;
   186→        auto it = freq_b.find(fp);
   187→        if (it != freq_b.end()) b_count = it->second;
   188→        if (count > b_count) missing += (count - b_count);
   189→    }
   190→
   191→    // Find items in B but not in A (extra in B)
   192→    size_t extra = 0;
   193→    for (const auto& [fp, count] : freq_b) {
   194→        int a_count = 0;
   195→        auto it = freq_a.find(fp);
   196→        if (it != freq_a.end()) a_count = it->second;
   197→        if (count > a_count) extra += (count - a_count);
   198→    }
   199→
   200→    size_t matched = fps_a.size() - missing;
   201→
   202→    if (missing == 0 && extra == 0) {
   203→        std::cout << "PASS: " << label << " (" << fps_a.size() << " items, all matched)" << std::endl;
   204→        return true;
   205→    } else {
   206→        std::cerr << "FAIL: " << label << " — A=" << fps_a.size() << " B=" << fps_b.size()
   207→                  << " matched=" << matched
   208→                  << " only_in_A=" << missing
   209→                  << " only_in_B=" << extra << std::endl;
   210→        return false;
   211→    }
   212→}
   213→
   214→// --- Decode + fingerprint functions ---
   215→
   216→static std::vector<std::string> fingerprint_admin(const std::string& dir) {
   217→    auto strings = read_file(dir + "/strings.bin");
   218→    auto polygons = read_structs<AdminPolygon>(dir + "/admin_polygons.bin");
   219→    auto vertices = read_structs<NodeCoord>(dir + "/admin_vertices.bin");
   220→
   221→    std::vector<std::string> fps;
   222→    fps.reserve(polygons.size());
   223→    for (const auto& p : polygons) {
   224→        std::string name = get_string(strings, p.name_id);
   225→        std::vector<std::pair<float,float>> verts;
   226→        for (uint16_t i = 0; i < p.vertex_count; i++) {
   227→            uint32_t vi = p.vertex_offset + i;
   228→            if (vi < vertices.size()) {
   229→                verts.push_back({vertices[vi].lat, vertices[vi].lng});
   230→            }
   231→        }
   232→        fps.push_back(admin_fingerprint(name, p.admin_level, p.country_code, verts));
   233→    }
   234→    return fps;
   235→}
   236→
   237→static std::vector<std::string> fingerprint_ways(const std::string& dir) {
   238→    auto strings = read_file(dir + "/strings.bin");
   239→    auto ways = read_structs<WayHeader>(dir + "/street_ways.bin");
   240→    auto nodes = read_structs<NodeCoord>(dir + "/street_nodes.bin");
   241→
   242→    std::vector<std::string> fps;
   243→    fps.reserve(ways.size());
   244→    for (const auto& w : ways) {
   245→        std::string name = get_string(strings, w.name_id);
   246→        std::vector<std::pair<float,float>> way_nodes;
   247→        for (uint8_t i = 0; i < w.node_count; i++) {
   248→            uint32_t ni = w.node_offset + i;
   249→            if (ni < nodes.size()) {
   250→                way_nodes.push_back({nodes[ni].lat, nodes[ni].lng});
   251→            }
   252→        }
   253→        fps.push_back(way_fingerprint(name, way_nodes));
   254→    }
   255→    return fps;
   256→}
   257→
   258→static std::vector<std::string> fingerprint_addrs(const std::string& dir) {
   259→    auto strings = read_file(dir + "/strings.bin");
   260→    auto points = read_structs<AddrPoint>(dir + "/addr_points.bin");
   261→
   262→    std::vector<std::string> fps;
   263→    fps.reserve(points.size());
   264→    for (const auto& p : points) {
   265→        std::string hn = get_string(strings, p.housenumber_id);
   266→        std::string st = get_string(strings, p.street_id);
   267→        fps.push_back(addr_fingerprint(p.lat, p.lng, hn, st));
   268→    }
   269→    return fps;
   270→}
   271→
   272→static std::vector<std::string> fingerprint_interps(const std::string& dir) {
   273→    auto strings = read_file(dir + "/strings.bin");
   274→    auto interps = read_structs<InterpWay>(dir + "/interp_ways.bin");
   275→    auto nodes = read_structs<NodeCoord>(dir + "/interp_nodes.bin");
   276→
   277→    std::vector<std::string> fps;
   278→    fps.reserve(interps.size());
   279→    for (const auto& iw : interps) {
   280→        std::string street = get_string(strings, iw.street_id);
   281→        std::vector<std::pair<float,float>> way_nodes;
   282→        for (uint8_t i = 0; i < iw.node_count; i++) {
   283→            uint32_t ni = iw.node_offset + i;
   284→            if (ni < nodes.size()) {
   285→                way_nodes.push_back({nodes[ni].lat, nodes[ni].lng});
   286→            }
   287→        }
   288→        fps.push_back(interp_fingerprint(street, iw.start_number, iw.end_number,
   289→                                          iw.interpolation, way_nodes));
   290→    }
   291→    return fps;
   292→}
   293→
   294→// --- Semantic (cross-version) fingerprints ---
   295→// Match by name/attributes only, ignoring coordinate precision differences
   296→
   297→// Way: name + node_count only
   298→static std::string way_semantic_fp(const std::string& name, size_t node_count) {
   299→    return name + "|" + std::to_string(node_count);
   300→}
   301→
   302→// Address: housenumber + street + coarse grid coords (~100m precision)
   303→// Centroid may differ slightly across code versions, so use coarse grid
   304→static std::string addr_semantic_fp(float lat, float lng,
   305→                                     const std::string& housenumber, const std::string& street) {
   306→    int glat = (int)(lat * 1e3f + (lat >= 0 ? 0.5f : -0.5f));
   307→    int glng = (int)(lng * 1e3f + (lng >= 0 ? 0.5f : -0.5f));
   308→    return std::to_string(glat) + "," + std::to_string(glng) + "|" + housenumber + "|" + street;
   309→}
   310→
   311→// Admin: name + level
   312→static std::string admin_semantic_fp(const std::string& name, uint8_t admin_level) {
   313→    return name + "|" + std::to_string(admin_level);
   314→}
   315→
   316→static std::vector<std::string> semantic_fp_ways(const std::string& dir) {
   317→    auto strings = read_file(dir + "/strings.bin");
   318→    auto ways = read_structs<WayHeader>(dir + "/street_ways.bin");
   319→
   320→    std::vector<std::string> fps;
   321→    fps.reserve(ways.size());
   322→    for (const auto& w : ways) {
   323→        std::string name = get_string(strings, w.name_id);
   324→        fps.push_back(way_semantic_fp(name, w.node_count));
   325→    }
   326→    return fps;
   327→}
   328→
   329→static std::vector<std::string> semantic_fp_addrs(const std::string& dir) {
   330→    auto strings = read_file(dir + "/strings.bin");
   331→    auto points = read_structs<AddrPoint>(dir + "/addr_points.bin");
   332→
   333→    std::vector<std::string> fps;
   334→    fps.reserve(points.size());
   335→    for (const auto& p : points) {
   336→        std::string hn = get_string(strings, p.housenumber_id);
   337→        std::string st = get_string(strings, p.street_id);
   338→        fps.push_back(addr_semantic_fp(p.lat, p.lng, hn, st));
   339→    }
   340→    return fps;
   341→}
   342→
   343→static std::vector<std::string> semantic_fp_admin(const std::string& dir) {
   344→    auto strings = read_file(dir + "/strings.bin");
   345→    auto polygons = read_structs<AdminPolygon>(dir + "/admin_polygons.bin");
   346→
   347→    std::vector<std::string> fps;
   348→    fps.reserve(polygons.size());
   349→    for (const auto& p : polygons) {
   350→        std::string name = get_string(strings, p.name_id);
   351→        fps.push_back(admin_semantic_fp(name, p.admin_level));
   352→    }
   353→    return fps;
   354→}
   355→
   356→int main(int argc, char* argv[]) {
   357→    if (argc < 3) {
   358→        std::cerr << "Usage: compare_indexes <dir-A> <dir-B> [--semantic]" << std::endl;
   359→        return 2;
   360→    }
   361→
   362→    std::string dir_a = argv[1];
   363→    std::string dir_b = argv[2];
   364→    bool semantic = (argc > 3 && std::string(argv[3]) == "--semantic");
   365→
   366→    std::cout << "Comparing" << (semantic ? " (semantic/cross-version)" : " (exact)") << ":" << std::endl;
   367→    std::cout << "  A: " << dir_a << std::endl;
   368→    std::cout << "  B: " << dir_b << std::endl;
   369→    std::cout << std::endl;
   370→
   371→    bool all_pass = true;
   372→
   373→    // Compare admin polygons
   374→    {
   375→        std::cout << "Loading admin polygons..." << std::flush;
   376→        auto fps_a = semantic ? semantic_fp_admin(dir_a) : fingerprint_admin(dir_a);
   377→        auto fps_b = semantic ? semantic_fp_admin(dir_b) : fingerprint_admin(dir_b);
   378→        std::cout << " done." << std::endl;
   379→        if (!compare_multisets("admin_polygons", fps_a, fps_b))
   380→            all_pass = false;
   381→    }
   382→
   383→    // Compare street ways
   384→    {
   385→        std::cout << "Loading street ways..." << std::flush;
   386→        auto fps_a = semantic ? semantic_fp_ways(dir_a) : fingerprint_ways(dir_a);
   387→        auto fps_b = semantic ? semantic_fp_ways(dir_b) : fingerprint_ways(dir_b);
   388→        std::cout << " done." << std::endl;
   389→        if (!compare_multisets("street_ways", fps_a, fps_b))
   390→            all_pass = false;
   391→    }
   392→
   393→    // Compare address points
   394→    {
   395→        std::cout << "Loading address points..." << std::flush;
   396→        auto fps_a = semantic ? semantic_fp_addrs(dir_a) : fingerprint_addrs(dir_a);
   397→        auto fps_b = semantic ? semantic_fp_addrs(dir_b) : fingerprint_addrs(dir_b);
   398→        std::cout << " done." << std::endl;
   399→        if (!compare_multisets("address_points", fps_a, fps_b))
   400→            all_pass = false;
   401→    }
   402→
   403→    // Compare interpolation ways
   404→    {
   405→        std::cout << "Loading interpolation ways..." << std::flush;
   406→        auto fps_a = fingerprint_interps(dir_a);
   407→        auto fps_b = fingerprint_interps(dir_b);
   408→        std::cout << " done." << std::endl;
   409→        if (!compare_multisets("interp_ways", fps_a, fps_b))
   410→            all_pass = false;
   411→    }
   412→
   413→    // Compare cell index sizes (should be identical regardless of ordering)
   414→    {
   415→        auto ac = read_file(dir_a + "/admin_cells.bin");
   416→        auto bc = read_file(dir_b + "/admin_cells.bin");
   417→        if (ac == bc) {
   418→            std::cout << "PASS: admin_cells.bin byte-identical (" << ac.size() << " bytes)" << std::endl;
   419→        } else if (ac.size() == bc.size()) {
   420→            std::cout << "INFO: admin_cells.bin same size (" << ac.size() << " bytes) but different content" << std::endl;
   421→        } else {
   422→            std::cerr << "FAIL: admin_cells.bin size differs (A=" << ac.size()
   423→                      << " B=" << bc.size() << ")" << std::endl;
   424→            all_pass = false;
   425→        }
   426→
   427→        auto gc_a = read_file(dir_a + "/geo_cells.bin");
   428→        auto gc_b = read_file(dir_b + "/geo_cells.bin");
   429→        if (gc_a.size() == gc_b.size()) {
   430→            std::cout << "PASS: geo_cells.bin same size (" << gc_a.size() << " bytes)" << std::endl;
   431→        } else {
   432→            std::cerr << "FAIL: geo_cells.bin size differs (A=" << gc_a.size()
   433→                      << " B=" << gc_b.size() << ")" << std::endl;
   434→            all_pass = false;
   435→        }
   436→    }
   437→
   438→    std::cout << std::endl;
   439→    if (all_pass) {
   440→        std::cout << "=== ALL COMPARISONS PASSED ===" << std::endl;
   441→        return 0;
   442→    } else {
   443→        std::cout << "=== SOME COMPARISONS FAILED ===" << std::endl;
   444→        return 1;
   445→    }
   446→}
   447→


Error: File content (11263 tokens) exceeds maximum allowed tokens (10000). Use offset and limit parameters to read specific portions of the file, or search for specific content instead of reading the whole file.