9.7 GiB → 10.0 GiB in 5 seconds = 62 MiB/s write rate. For 500K cells/sec at ~8 bytes/cell = 4 MiB/s. The writes are 15x faster than expected. Something IS writing too much.

The `write_ids` uses `ftell()` which returns the actual position. If the ACTUAL position is correct but the file is growing faster than the progress suggests... the only explanation is that the `fwrite` calls are writing MORE data than intended.

Let me check the `write_ids` function one more time. It writes `uint16_t count` (2 bytes) then `uint32_t[] ids` (n×4 bytes). For `n` average of ~50, that's ~202 bytes per entry. For 500K cells/sec with 95% having streets, that's 475K writes/sec × 202 bytes = 96 MiB/s. The actual rate is 62 MiB/s. That's actually reasonable!

Wait — if 95% of cells have streets and the average is ~50 IDs per street cell, then for 322M cells: 306M × (2 + 50×4) = 306M × 202 = 61.8 GiB. But the reference is only 2.1 GiB!

The average can't be 50 IDs per cell — it must be much less. The reference street_entries is 2.1 GiB for 307M cells that have streets. That's 6.8 bytes/cell average = 1.2 IDs per cell average (2 bytes header + 1.2 × 4 bytes IDs = 6.8).

So the average is ~1.2 IDs per cell. But I'm getting output at 62 MiB/s × 600s = 37 GiB projected. For 307M street cells that's 120 bytes/cell = ~30 IDs per cell. Something is 25x the expected.

The bug must be in what `parse_ids` reads. The old entry files are mmap'd from `cur_dir/` (the old build). If the offsets in old_geo are correct, `parse_ids` should read the correct entry data.

But wait — what if the REMAPPED IDs are correct but the offsets into old entry files are WRONG after the ID remap in the merge replay phase? No — the offsets come directly from old_geo (mmap'd, unchanged).

OK, I think I need to trace the exact bytes being written for one cell. Let me add a diagnostic:


Edit

                if (id_buf.size() > 1000 && cell_count < 100000000) {
                    std::cerr << "    LARGE entry cell=" << cell_count << " cid=" << cid << " oi=" << oi
                              << " geo_off=" << geo_off << " off=" << off << " n=" << id_buf.size() << std::endl;
                }

                if (cell_count <= 5) {
                    std::cerr << "    TRACE cell=" << cell_count << " geo_off=" << geo_off << " off=" << off << " n=" << id_buf.size() << std::endl;
                }


The file /root/traccar-geocoder/builder/tools/geocoder_patch.cpp has been updated successfully.

                if (cell_count <= 5) {
                    std::cerr << "    TRACE cell=" << cell_count << " geo_off=" << geo_off << " off=" << off << " n=" << id_buf.size() << std::endl;
                }

// geocoder-patch v4: Low-memory streaming patch application.
//
// Key design: mmap old files (zero copy), stream output via fwrite (never accumulate),
// process entry pipeline cell-by-cell. Target: <1 GiB peak RSS for planet.
//
// Usage: geocoder-patch <current-dir> <patch-file> -o <output-dir>

#include <algorithm>
#include <cstdint>
#include <cstdio>
#include <cstdlib>
#include <cstring>
#include <ctime>
#include <fcntl.h>
#include <iomanip>
#include <iostream>
#include <malloc.h>
#include <string>
#include <sys/mman.h>
#include <sys/stat.h>
#include <unordered_map>
#include <unordered_set>
#include <unistd.h>
#include <vector>

#include "patch_format.h"

enum MergeOp : uint8_t { OP_MATCH_RUN = 0, OP_INSERT_RUN = 1, OP_DELETE_RUN = 2 };

// --- Helpers ---
static double now_ms() {
    struct timespec ts; clock_gettime(CLOCK_MONOTONIC, &ts);
    return ts.tv_sec * 1000.0 + ts.tv_nsec / 1e6;
}
static size_t get_rss_mb() {
    FILE* f = fopen("/proc/self/statm", "r");
    if (!f) return 0;
    size_t dummy, rss; fscanf(f, "%zu %zu", &dummy, &rss); fclose(f);
    return rss * 4096 / (1024*1024);
}
static void log_phase(const char* label, double start) {
    std::cerr << "  [" << std::fixed << std::setprecision(1) << (now_ms() - start) / 1000.0
              << "s, " << get_rss_mb() << " MiB] " << label << std::endl;
}

// mmap a file read-only. Returns {pointer, size}. Caller must munmap.
struct MappedFile { const char* data; size_t size; };
static MappedFile mmap_file(const std::string& path) {
    int fd = open(path.c_str(), O_RDONLY);
    if (fd < 0) return {nullptr, 0};
    struct stat st; fstat(fd, &st);
    size_t sz = st.st_size;
    if (sz == 0) { close(fd); return {nullptr, 0}; }
    void* p = mmap(nullptr, sz, PROT_READ, MAP_PRIVATE, fd, 0);
    close(fd);
    if (p == MAP_FAILED) return {nullptr, 0};
    return {static_cast<const char*>(p), sz};
}
static void unmap_file(MappedFile& f) {
    if (f.data) { munmap(const_cast<char*>(f.data), f.size); f.data = nullptr; f.size = 0; }
}

// Detect stride from file size
static size_t detect_stride(const std::string& path, std::initializer_list<size_t> candidates) {
    struct stat st; if (stat(path.c_str(), &st) != 0) return *candidates.begin();
    for (size_t s : candidates) if (st.st_size % s == 0 && st.st_size > 0) return s;
    return *candidates.begin();
}

int main(int argc, char* argv[]) {
    if (argc < 5 || std::string(argv[3]) != "-o") {
        std::cerr << "Usage: geocoder-patch <current-dir> <patch-file> -o <output-dir>" << std::endl;
        return 1;
    }
    std::string cur_dir = argv[1], patch_path = argv[2], out_dir = argv[4];
    ensure_dir(out_dir);
    std::string tmpdir = "/tmp/geocoder-patch-" + std::to_string(getpid());
    ensure_dir(tmpdir);
    double t_start = now_ms();

    // --- Phase 1: Decompress + mmap patch ---
    std::string raw_path = tmpdir + "/patch.raw";
    {
        std::string cmd = "zstd -d '" + patch_path + "' -o '" + raw_path + "' -f --quiet 2>/dev/null";
        if (system(cmd.c_str()) != 0) { std::cerr << "Failed to decompress" << std::endl; return 1; }
    }
    MappedFile patch_map = mmap_file(raw_path);
    if (!patch_map.data) { std::cerr << "Failed to mmap patch" << std::endl; return 1; }
    remove(raw_path.c_str());
    const char* P = patch_map.data;
    size_t patch_size = patch_map.size;
    std::cerr << "Patch: " << patch_size << " bytes" << std::endl;
    log_phase("Decompress", t_start);

    size_t pos = 0;
    auto ru32 = [&]() -> uint32_t { uint32_t v; memcpy(&v, P+pos, 4); pos += 4; return v; };
    auto ru64 = [&]() -> uint64_t { uint64_t v; memcpy(&v, P+pos, 8); pos += 8; return v; };

    // Header
    if (memcmp(P, GCPATCH_MAGIC, 8) != 0) { std::cerr << "Bad magic" << std::endl; return 1; }
    pos = 8;
    uint32_t ver = ru32(); if (ver != 2) { std::cerr << "Bad version" << std::endl; return 1; }
    ru32(); // flags

    // --- Phase 2: String rebuild ---
    std::unordered_map<uint32_t, uint32_t> str_remap;
    {
        uint32_t marker = ru32();
        if (marker == 0xFFFFFFF7) {
            uint32_t n_added = ru32(), n_deleted = ru32();
            std::vector<std::string> added;
            for (uint32_t i = 0; i < n_added; i++) { const char* s = P+pos; added.push_back(s); pos += strlen(s)+1; }
            std::vector<uint32_t> del_idx(n_deleted);
            for (uint32_t i = 0; i < n_deleted; i++) del_idx[i] = ru32();
            std::unordered_set<uint32_t> del_set(del_idx.begin(), del_idx.end());

            MappedFile old_pool = mmap_file(cur_dir + "/strings.bin");
            std::vector<std::pair<uint32_t, std::string>> old_strs;
            size_t sp = 0;
            while (sp < old_pool.size) {
                old_strs.push_back({(uint32_t)sp, std::string(old_pool.data + sp)});
                sp += strlen(old_pool.data + sp) + 1;
            }
            std::vector<std::string> merged;
            for (size_t i = 0; i < old_strs.size(); i++)
                if (!del_set.count(i)) merged.push_back(old_strs[i].second);
            for (auto& s : added) merged.push_back(s);
            std::sort(merged.begin(), merged.end());

            // Write new pool and build remap
            FILE* fp = fopen((out_dir + "/strings.bin").c_str(), "wb");
            std::unordered_map<std::string, uint32_t> new_offs;
            uint32_t wpos = 0;
            for (auto& s : merged) {
                new_offs[s] = wpos;
                fwrite(s.c_str(), 1, s.size() + 1, fp);
                wpos += s.size() + 1;
            }
            fclose(fp);
            for (auto& [old_off, s] : old_strs) {
                auto it = new_offs.find(s);
                if (it != new_offs.end() && it->second != old_off) str_remap[old_off] = it->second;
            }
            unmap_file(old_pool);
            std::cerr << "  Strings: +" << n_added << " -" << n_deleted << " → " << merged.size()
                      << ", " << str_remap.size() << " remapped" << std::endl;
            marker = ru32();
        }
        if (marker == 0xFFFFFFFE) { uint32_t c = ru32(); for (uint32_t i = 0; i < c; i++) { uint32_t a = ru32(), b = ru32(); str_remap[a] = b; } }
        else pos -= 4;
    }
    log_phase("Strings", t_start);

    // Detect strides
    size_t way_stride = detect_stride(cur_dir + "/street_ways.bin", {12, 9});
    size_t interp_stride = detect_stride(cur_dir + "/interp_ways.bin", {24, 20, 18});
    size_t admin_stride = detect_stride(cur_dir + "/admin_polygons.bin", {24, 20, 19});

    // String remap applier (works on mutable copy)
    auto apply_str_remap = [&](std::vector<char>& data, uint32_t fid, size_t stride) {
        if (str_remap.empty()) return;
        std::vector<size_t> offs;
        if (fid == (uint32_t)PatchFileId::ADDR_POINTS) offs = {8, 12};
        else if (fid == (uint32_t)PatchFileId::STREET_WAYS) offs = {(stride == 12) ? 8ul : 5ul};
        else if (fid == (uint32_t)PatchFileId::INTERP_WAYS) offs = {(stride >= 20) ? 8ul : 5ul};
        else if (fid == (uint32_t)PatchFileId::ADMIN_POLYGONS) offs = {8};
        else return;
        for (size_t i = 0; i + stride <= data.size(); i += stride)
            for (size_t o : offs) { uint32_t v; memcpy(&v, data.data()+i+o, 4); auto it = str_remap.find(v); if (it != str_remap.end()) memcpy(data.data()+i+o, &it->second, 4); }
    };

    // --- Phase 3: Merge replays (streaming output) ---
    // ID remaps: file_id → vector<uint32_t> where remap[old_idx] = new_idx
    std::unordered_map<uint32_t, std::vector<uint32_t>> id_remaps;
    std::vector<uint64_t> geo_added, geo_removed, admin_added, admin_removed;
    std::unordered_map<uint64_t, uint8_t> flag_corrections;
    struct CellCorr { uint64_t cell_id; std::vector<uint32_t> ids; };
    std::unordered_map<uint32_t, std::vector<CellCorr>> entry_corrections;

    while (pos < patch_size) {
        uint32_t file_id = ru32();
        if (file_id == 0xFFFFFFFF) break;

        // --- Metadata sections ---
        if (file_id == 0xFFFFFFF7) {
            // String diff in main loop — process it (may be the only occurrence)
            uint32_t n_added = ru32(), n_deleted = ru32();
            std::vector<std::string> added;
            for (uint32_t i = 0; i < n_added; i++) { const char* s = P+pos; added.push_back(s); pos += strlen(s)+1; }
            std::vector<uint32_t> del_idx(n_deleted);
            for (uint32_t i = 0; i < n_deleted; i++) del_idx[i] = ru32();
            std::unordered_set<uint32_t> del_set(del_idx.begin(), del_idx.end());
            MappedFile old_pool = mmap_file(cur_dir + "/strings.bin");
            std::vector<std::pair<uint32_t, std::string>> old_strs;
            size_t sp = 0;
            while (sp < old_pool.size) { old_strs.push_back({(uint32_t)sp, std::string(old_pool.data+sp)}); sp += strlen(old_pool.data+sp)+1; }
            std::vector<std::string> merged;
            for (size_t i = 0; i < old_strs.size(); i++) if (!del_set.count(i)) merged.push_back(old_strs[i].second);
            for (auto& s : added) merged.push_back(s);
            std::sort(merged.begin(), merged.end());
            FILE* fp = fopen((out_dir + "/strings.bin").c_str(), "wb");
            std::unordered_map<std::string, uint32_t> new_offs;
            uint32_t wpos = 0;
            for (auto& s : merged) { new_offs[s] = wpos; fwrite(s.c_str(), 1, s.size()+1, fp); wpos += s.size()+1; }
            fclose(fp);
            for (auto& [old_off, s] : old_strs) { auto it = new_offs.find(s); if (it != new_offs.end() && it->second != old_off) str_remap[old_off] = it->second; }
            unmap_file(old_pool);
            std::cerr << "  Strings (loop): +" << n_added << " -" << n_deleted << " → " << merged.size() << std::endl;
            continue;
        }
        if (file_id == CELL_CHANGES_GEO_MARKER) {
            uint32_t na = ru32(), nr = ru32();
            geo_added.resize(na); geo_removed.resize(nr);
            for (uint32_t i = 0; i < na; i++) { memcpy(&geo_added[i], P+pos, 8); pos += 8; }
            for (uint32_t i = 0; i < nr; i++) { memcpy(&geo_removed[i], P+pos, 8); pos += 8; }
            std::cerr << "  Geo cells: +" << na << " -" << nr << std::endl;
            continue;
        }
        if (file_id == CELL_CHANGES_ADMIN_MARKER) {
            uint32_t na = ru32(), nr = ru32();
            admin_added.resize(na); admin_removed.resize(nr);
            for (uint32_t i = 0; i < na; i++) { memcpy(&admin_added[i], P+pos, 8); pos += 8; }
            for (uint32_t i = 0; i < nr; i++) { memcpy(&admin_removed[i], P+pos, 8); pos += 8; }
            std::cerr << "  Admin cells: +" << na << " -" << nr << std::endl;
            continue;
        }
        if (file_id == CELL_FLAGS_MARKER) {
            uint32_t c = ru32();
            for (uint32_t i = 0; i < c; i++) { uint64_t cid; memcpy(&cid, P+pos, 8); pos += 8; flag_corrections[cid] = P[pos++]; }
            std::cerr << "  Flag corrections: " << c << std::endl;
            continue;
        }
        if (file_id == SECONDARY_REMAP_MARKER) {
            uint32_t nf = ru32();
            for (uint32_t f = 0; f < nf; f++) {
                uint32_t fid = ru32(), np = ru32();
                auto& rm = id_remaps[fid];
                for (uint32_t i = 0; i < np; i++) { uint32_t o = ru32(), n = ru32(); if (o < rm.size()) rm[o] = n; }
                std::cerr << "  Secondary remap " << fid << ": " << np << " pairs" << std::endl;
            }
            continue;
        }
        if (file_id == ENTRY_CORRECTION_MARKER) {
            uint32_t fid = ru32(), c = ru32();
            auto& list = entry_corrections[fid];
            for (uint32_t i = 0; i < c; i++) {
                uint64_t cid; memcpy(&cid, P+pos, 8); pos += 8;
                uint16_t ec; memcpy(&ec, P+pos, 2); pos += 2;
                std::vector<uint32_t> ids(ec);
                if (ec > 0) { memcpy(ids.data(), P+pos, ec*4); pos += ec*4; }
                list.push_back({cid, std::move(ids)});
            }
            std::cerr << "  Entry corrections " << fid << ": " << c << std::endl;
            continue;
        }

        // --- Merge sequence replay (streaming) ---
        uint32_t stride = ru32();
        uint64_t old_size = ru64(), new_size = ru64();
        if (file_id >= (uint32_t)PatchFileId::COUNT) { std::cerr << "Unknown file " << file_id << std::endl; break; }
        const char* fname = patch_file_names[file_id];

        if (stride == 0) {
            // Full replacement — write directly from patch mmap
            uint32_t nf = ru32(); (void)nf; uint64_t ds = ru64();
            FILE* fp = fopen((out_dir + "/" + fname).c_str(), "wb");
            fwrite(P+pos, 1, ds, fp); fclose(fp);
            pos += ds;
            std::cerr << "  " << fname << ": full replace " << ds << " bytes" << std::endl;
            continue;
        }
        if (stride == 0xFE) { uint32_t nf = ru32(); (void)nf; uint64_t ds = ru64(); pos += ds; continue; }

        size_t actual_stride = stride;
        if (file_id == (uint32_t)PatchFileId::STREET_WAYS) actual_stride = way_stride;
        else if (file_id == (uint32_t)PatchFileId::INTERP_WAYS) actual_stride = interp_stride;
        else if (file_id == (uint32_t)PatchFileId::ADMIN_POLYGONS) actual_stride = admin_stride;

        // Load old file into mutable buffer (need to apply remap + fixups)
        // For large files (nodes/vertices), this is the peak memory per file.
        // Files are processed sequentially, so only one is in memory at a time.
        auto old_data = read_file(cur_dir + "/" + std::string(fname));

        // Zero padding
        if (file_id == (uint32_t)PatchFileId::ADMIN_POLYGONS && actual_stride == 24)
            for (size_t i = 0; i + actual_stride <= old_data.size(); i += actual_stride) memset(old_data.data()+i+13, 0, 3);
        if (file_id == (uint32_t)PatchFileId::INTERP_WAYS && actual_stride == 24)
            for (size_t i = 0; i + actual_stride <= old_data.size(); i += actual_stride) { memset(old_data.data()+i+5, 0, 3); memset(old_data.data()+i+21, 0, 3); }

        apply_str_remap(old_data, file_id, actual_stride);

        // Apply fixups
        uint32_t n_fixups = ru32();
        if (n_fixups > 0) {
            uint32_t delta_size = ru32();
            uint32_t pi = 0, pv = 0; size_t fp = pos;
            for (uint32_t i = 0; i < n_fixups; i++) {
                uint32_t idx = pi + read_varint(P, fp);
                uint32_t val = pv + read_varint(P, fp);
                if ((size_t)idx * actual_stride + 4 <= old_data.size())
                    memcpy(old_data.data() + (size_t)idx * actual_stride, &val, 4);
                pi = idx; pv = val;
            }
            pos = fp;
        }

        // Replay merge sequence — stream output directly to file
        uint64_t seq_size = ru64();
        size_t seq_end = pos + seq_size;
        FILE* outf = fopen((out_dir + "/" + fname).c_str(), "wb");

        // Track ID remap (only for data files that need entry reconstruction)
        bool track = (file_id == (uint32_t)PatchFileId::ADDR_POINTS ||
                      file_id == (uint32_t)PatchFileId::STREET_WAYS ||
                      file_id == (uint32_t)PatchFileId::INTERP_WAYS ||
                      file_id == (uint32_t)PatchFileId::ADMIN_POLYGONS);
        std::vector<uint32_t> id_map;
        if (track) id_map.assign(old_data.size() / actual_stride, 0xFFFFFFFF);

        size_t old_rec = 0, new_rec = 0, old_bytes = 0, written = 0;
        while (pos < seq_end) {
            uint8_t op = P[pos++];
            uint32_t count; memcpy(&count, P+pos, 4); pos += 4;
            if (op == OP_MATCH_RUN) {
                size_t bytes = count * actual_stride;
                fwrite(old_data.data() + old_bytes, 1, bytes, outf);
                written += bytes;
                if (track) for (uint32_t k = 0; k < count; k++) if (old_rec+k < id_map.size()) id_map[old_rec+k] = new_rec+k;
                old_rec += count; new_rec += count; old_bytes += bytes;
            } else if (op == OP_INSERT_RUN) {
                size_t bytes = count * actual_stride;
                fwrite(P+pos, 1, bytes, outf);
                written += bytes; pos += bytes; new_rec += count;
            } else if (op == OP_DELETE_RUN) {
                old_rec += count; old_bytes += count * actual_stride;
            }
        }
        fclose(outf);

        if (track) {
            id_remaps[file_id] = std::move(id_map);
            size_t matched = 0;
            for (auto v : id_remaps[file_id]) if (v != 0xFFFFFFFF) matched++;
            std::cerr << "  " << fname << ": " << written << " bytes (mapped " << matched << "/" << id_remaps[file_id].size() << ")" << std::endl;
        } else {
            std::cerr << "  " << fname << ": " << written << " bytes" << std::endl;
        }

        // Free old data immediately — don't let glibc hold the pages
        { std::vector<char>().swap(old_data); }
        malloc_trim(0);
    }
    log_phase("Merge replays", t_start);

    // Free string remap (no longer needed)
    { std::unordered_map<uint32_t,uint32_t>().swap(str_remap); }
    malloc_trim(0);

    // --- Phase 4: Streaming entry pipeline ---
    // Walk old geo_cells, remap IDs, apply corrections, write output directly to files.
    {
        double t_entry = now_ms();
        std::cerr << "Entry pipeline..." << std::endl;

        auto get_rm = [&](PatchFileId fid) -> const std::vector<uint32_t>& {
            static std::vector<uint32_t> empty;
            auto it = id_remaps.find((uint32_t)fid); return it != id_remaps.end() ? it->second : empty;
        };
        auto& w_rm = get_rm(PatchFileId::STREET_WAYS);
        auto& a_rm = get_rm(PatchFileId::ADDR_POINTS);
        auto& i_rm = get_rm(PatchFileId::INTERP_WAYS);

        // Build correction lookup maps
        std::unordered_map<uint64_t, const std::vector<uint32_t>*> cs, ca, ci;
        for (auto& [fid, list] : entry_corrections) {
            auto* m = (fid == (uint32_t)PatchFileId::STREET_ENTRIES) ? &cs :
                      (fid == (uint32_t)PatchFileId::ADDR_ENTRIES) ? &ca :
                      (fid == (uint32_t)PatchFileId::INTERP_ENTRIES) ? &ci : nullptr;
            if (m) for (auto& c : list) (*m)[c.cell_id] = &c.ids;
        }

        // mmap old files (zero-copy read)
        MappedFile old_geo = mmap_file(cur_dir + "/geo_cells.bin");
        MappedFile old_se = mmap_file(cur_dir + "/street_entries.bin");
        MappedFile old_ae = mmap_file(cur_dir + "/addr_entries.bin");
        MappedFile old_ie = mmap_file(cur_dir + "/interp_entries.bin");

        // Build derived cell list
        std::unordered_set<uint64_t> rm_set(geo_removed.begin(), geo_removed.end());
        struct CR { uint64_t cid; int32_t idx; };
        std::vector<CR> cells;
        cells.reserve(old_geo.size / 20 + geo_added.size());
        for (size_t i = 0; i < old_geo.size / 20; i++) {
            uint64_t cid; memcpy(&cid, old_geo.data + i*20, 8);
            if (!rm_set.count(cid)) cells.push_back({cid, (int32_t)i});
        }
        for (uint64_t cid : geo_added) cells.push_back({cid, -1});
        std::sort(cells.begin(), cells.end(), [](const CR& a, const CR& b) { return a.cid < b.cid; });
        log_phase("  Cell list", t_entry);

        // Open output files for streaming write
        FILE* f_geo = fopen((out_dir + "/geo_cells.bin").c_str(), "wb");
        FILE* f_se = fopen((out_dir + "/street_entries.bin").c_str(), "wb");
        FILE* f_ae = fopen((out_dir + "/addr_entries.bin").c_str(), "wb");
        FILE* f_ie = fopen((out_dir + "/interp_entries.bin").c_str(), "wb");
        uint32_t no_data = 0xFFFFFFFF;

        // Reusable buffer
        std::vector<uint32_t> id_buf;
        id_buf.reserve(256);

        auto parse_ids = [](const MappedFile& f, uint32_t off, std::vector<uint32_t>& buf) {
            buf.clear();
            if (off == 0xFFFFFFFF || off + 2 > f.size) return;
            uint16_t c; memcpy(&c, f.data+off, 2);
            if (c > 10000 || off+2+(size_t)c*4 > f.size) return; // sanity: no cell has >10K entries
            buf.resize(c); memcpy(buf.data(), f.data+off+2, c*4);
        };
        auto remap_buf = [](std::vector<uint32_t>& buf, const std::vector<uint32_t>& rm) {
            for (auto& id : buf) if (id < rm.size() && rm[id] != 0xFFFFFFFF) id = rm[id];
            std::sort(buf.begin(), buf.end());
        };
        auto write_ids = [](FILE* f, const uint32_t* ids, size_t n) -> uint32_t {
            if (n == 0) return 0xFFFFFFFF;
            uint32_t off = (uint32_t)ftell(f);
            uint16_t c = (uint16_t)n;
            fwrite(&c, 2, 1, f); fwrite(ids, 4, n, f);
            return off;
        };

        size_t sc = 0, ac = 0, ic = 0;
        size_t cell_count = 0;
        for (auto& [cid, oi] : cells) {
            if (++cell_count % 5000000 == 0) {
                std::cerr << "    " << cell_count << "/" << cells.size()
                          << " se=" << ftell(f_se)/1024/1024 << "M ae=" << ftell(f_ae)/1024/1024
                          << "M ie=" << ftell(f_ie)/1024/1024 << "M rss=" << get_rss_mb() << "M" << std::endl;
            }
            // Process one entry type at a time using shared id_buf
            auto process = [&](const MappedFile& old_e, size_t geo_off,
                               const std::vector<uint32_t>& rm,
                               const std::unordered_map<uint64_t, const std::vector<uint32_t>*>& corr,
                               FILE* outf, size_t& cc) -> uint32_t {
                auto cit = corr.find(cid);
                if (cit != corr.end()) { cc++; return write_ids(outf, cit->second->data(), cit->second->size()); }
                if (oi < 0) return no_data;
                uint32_t off; memcpy(&off, old_geo.data + oi*20 + geo_off, 4);
                parse_ids(old_e, off, id_buf);
                if (id_buf.empty()) return no_data;
                if (id_buf.size() > 1000 && cell_count < 100000000) {
                    std::cerr << "    LARGE entry cell=" << cell_count << " cid=" << cid << " oi=" << oi
                              << " geo_off=" << geo_off << " off=" << off << " n=" << id_buf.size() << std::endl;
                }
                remap_buf(id_buf, rm);
                return write_ids(outf, id_buf.data(), id_buf.size());
            };

            // Determine flags
            bool hs = false, ha = false, hi = false;
            if (oi >= 0) {
                uint32_t s, a, ip; memcpy(&s, old_geo.data+oi*20+8, 4); memcpy(&a, old_geo.data+oi*20+12, 4); memcpy(&ip, old_geo.data+oi*20+16, 4);
                hs = (s != no_data); ha = (a != no_data); hi = (ip != no_data);
            }
            auto fc = flag_corrections.find(cid);
            if (fc != flag_corrections.end()) { hs = fc->second & 1; ha = fc->second & 2; hi = fc->second & 4; }
            if (cs.count(cid)) hs = true;
            if (ca.count(cid)) ha = true;
            if (ci.count(cid)) hi = true;

            uint32_t so = hs ? process(old_se, 8, w_rm, cs, f_se, sc) : no_data;
            uint32_t ao = ha ? process(old_ae, 12, a_rm, ca, f_ae, ac) : no_data;
            uint32_t io = hi ? process(old_ie, 16, i_rm, ci, f_ie, ic) : no_data;
            if (cell_count == 1) {
                // Count how many old cells have each flag
                size_t os = 0, oa = 0, oi_c = 0;
                for (size_t ci = 0; ci < old_geo.size / 20; ci++) {
                    uint32_t sv, av, iv;
                    memcpy(&sv, old_geo.data + ci*20+8, 4); memcpy(&av, old_geo.data + ci*20+12, 4); memcpy(&iv, old_geo.data + ci*20+16, 4);
                    if (sv != no_data) os++; if (av != no_data) oa++; if (iv != no_data) oi_c++;
                }
                std::cerr << "    DEBUG: old cells=" << old_geo.size/20 << " has_street=" << os << " has_addr=" << oa << " has_interp=" << oi_c << std::endl;
            }

            fwrite(&cid, 8, 1, f_geo); fwrite(&so, 4, 1, f_geo); fwrite(&ao, 4, 1, f_geo); fwrite(&io, 4, 1, f_geo);
        }
        fclose(f_geo); fclose(f_se); fclose(f_ae); fclose(f_ie);
        unmap_file(old_geo); unmap_file(old_se); unmap_file(old_ae); unmap_file(old_ie);
        std::cerr << "  Geo: " << cells.size() << " cells, corrections: " << sc << " st, " << ac << " ad, " << ic << " in" << std::endl;
        log_phase("  Geo entries", t_entry);

        // Admin: small, use existing rebuild + corrections
        {
            std::unordered_map<uint32_t,uint32_t> ad_rm;
            auto ait = id_remaps.find((uint32_t)PatchFileId::ADMIN_POLYGONS);
            if (ait != id_remaps.end()) {
                for (uint32_t i = 0; i < ait->second.size(); i++)
                    if (ait->second[i] != 0xFFFFFFFF) ad_rm[i] = ait->second[i];
            }
            auto old_ac = read_file(cur_dir + "/admin_cells.bin");
            auto old_adme = read_file(cur_dir + "/admin_entries.bin");
            auto admin = rebuild_admin_from_remap(old_ac, old_adme, ad_rm, admin_added, admin_removed);

            auto ecit = entry_corrections.find((uint32_t)PatchFileId::ADMIN_ENTRIES);
            if (ecit != entry_corrections.end()) {
                std::unordered_map<uint64_t, const std::vector<uint32_t>*> ac_corr;
                for (auto& c : ecit->second) ac_corr[c.cell_id] = &c.ids;
                size_t n = admin.admin_cells_data.size() / 12;
                FILE* fac = fopen((out_dir + "/admin_cells.bin").c_str(), "wb");
                FILE* fae = fopen((out_dir + "/admin_entries.bin").c_str(), "wb");
                uint32_t ae_wpos = 0;
                for (size_t i = 0; i < n; i++) {
                    uint64_t cid; memcpy(&cid, admin.admin_cells_data.data()+i*12, 8);
                    fwrite(&cid, 8, 1, fac);
                    auto cit = ac_corr.find(cid);
                    if (cit != ac_corr.end()) {
                        uint32_t off = cit->second->empty() ? no_data : ae_wpos;
                        fwrite(&off, 4, 1, fac);
                        if (!cit->second->empty()) {
                            uint16_t c = cit->second->size();
                            fwrite(&c, 2, 1, fae); fwrite(cit->second->data(), 4, c, fae);
                            ae_wpos += 2 + c*4;
                        }
                    } else {
                        uint32_t old_off; memcpy(&old_off, admin.admin_cells_data.data()+i*12+8, 4);
                        if (old_off != no_data && old_off+2 <= admin.admin_entries_data.size()) {
                            uint32_t new_off = ae_wpos; fwrite(&new_off, 4, 1, fac);
                            uint16_t c; memcpy(&c, admin.admin_entries_data.data()+old_off, 2);
                            fwrite(admin.admin_entries_data.data()+old_off, 1, 2+c*4, fae);
                            ae_wpos += 2+c*4;
                        } else {
                            fwrite(&no_data, 4, 1, fac);
                        }
                    }
                }
                fclose(fac); fclose(fae);
                std::cerr << "  Admin: " << n << " cells, " << ac_corr.size() << " corrections" << std::endl;
            } else {
                write_file(out_dir + "/admin_cells.bin", admin.admin_cells_data);
                write_file(out_dir + "/admin_entries.bin", admin.admin_entries_data);
            }
            log_phase("  Admin", t_entry);
        }
    }

    // Cleanup
    unmap_file(patch_map);
    { std::string cmd = "rm -rf '" + tmpdir + "'"; system(cmd.c_str()); }
    log_phase("Total", t_start);
    std::cerr << "Patch applied. Output in " << out_dir << std::endl;
    return 0;
}

-                    std::cerr << "    LARGE entry cell=" << cell_count << " cid=" << cid << " oi=" << oi

-                              << " geo_off=" << geo_off << " off=" << off << " n=" << id_buf.size() << std::endl;

+                    std::cerr << "    TRACE cell=" << cell_count << " geo_off=" << geo_off << " off=" << off << " n=" << id_buf.size() << std::endl;


Bash

make -j$(nproc) geocoder-patch 2>&1 | tail -3 && sshpass -p 'elon-is-insane' ssh -o ProxyCommand="tailscale nc %h %p" -o StrictHostKeyChecking=no michtest@100.113.88.121 "kill -9 \$(ps aux | grep 'geocoder-patch' | grep -v grep | awk '{print \$2}') 2>/dev/null; rm -f /home/michtest/geocoder-patch; rm -rf /home/michtest/patched-planet-stream/" 2>&1 && sshpass -p 'elon-is-insane' scp -o ProxyCommand="tailscale nc %h %p" -o StrictHostKeyChecking=no /root/traccar-geocoder/builder/build/geocoder-patch michtest@100.113.88.121:/home/michtest/ 2>&1 && sshpass -p 'elon-is-insane' ssh -o ProxyCommand="tailscale nc %h %p" -o StrictHostKeyChecking=no michtest@100.113.88.121 'cd /home/michtest && rm -rf patched-planet-stream/ && ./geocoder-patch planet-B/ validated-patches/planet-mar16-to-mar23-v2.gcpatch -o patched-planet-stream/ 2>&1 | head -40 && kill -9 $(ps aux | grep "geocoder-patch planet" | grep -v grep | awk "{print \$2}") 2>/dev/null; rm -rf patched-planet-stream/' 2>&1

Trace first cells


[ 50%] Building CXX object CMakeFiles/geocoder-patch.dir/tools/geocoder_patch.cpp.o
[100%] Linking CXX executable geocoder-patch
[100%] Built target geocoder-patch
Patch: 423813763 bytes
  [0.3s, 3 MiB] Decompress
  [0.3s, 3 MiB] Strings
  Strings (loop): +10703 -4486 → 11894530
  addr_points.bin: 2497261184 bytes (mapped 155813360/155902060)
  street_ways.bin: 574853820 bytes (mapped 47686691/47864210)
  street_nodes.bin: 3966274352 bytes
  interp_ways.bin: 1749240 bytes (mapped 72656/72839)
  interp_nodes.bin: 3484432 bytes
  admin_polygons.bin: 22763904 bytes (mapped 942932/947905)
  admin_vertices.bin: 717223152 bytes
  Geo cells: +183728 -45783
  Admin cells: +4 -0
  Secondary remap 1: 55091 pairs
  Secondary remap 3: 58838 pairs
  Secondary remap 4: 88 pairs
  Secondary remap 6: 4603 pairs
  Entry corrections 9: 1588748
  Entry corrections 10: 95288
  Entry corrections 11: 256
  Entry corrections 13: 4556
  Flag corrections: 224324
  [26.1s, 1816 MiB] Merge replays
Entry pipeline...
  [16.0s, 12469 MiB]   Cell list
    TRACE cell=1 geo_off=8 off=0 n=1
    DEBUG: old cells=322588816 has_street=306918478 has_addr=48969395 has_interp=146493
    TRACE cell=2 geo_off=8 off=6 n=1
    TRACE cell=3 geo_off=8 off=12 n=1
    TRACE cell=4 geo_off=8 off=18 n=1
    TRACE cell=5 geo_off=8 off=24 n=1
    5000000/322726761 se=39M ae=4M ie=0M rss=12512M
    10000000/322726761 se=78M ae=12M ie=0M rss=12559M
    15000000/322726761 se=111M ae=17M ie=0M rss=12598M
    20000000/322726761 se=147M ae=27M ie=0M rss=12644M
    25000000/322726761 se=186M ae=39M ie=0M rss=12695M
    30000000/322726761 se=221M ae=40M ie=0M rss=12731M
    35000000/322726761 se=254M ae=41M ie=0M rss=12765M
    40000000/322726761 se=292M ae=43M ie=0M rss=12804M
    45000000/322726761 se=329M ae=79M ie=0M rss=12877M


The first cells: `off=0, n=1` then `off=6, n=1` then `off=12, n=1`. Each cell has 1 street entry ID. `off` advances by 6 (2 bytes count + 4 bytes ID). Normal.

And the progress shows se=329M at 45M cells (ftell, not fpos). That's 7.3 bytes/cell — correct!

So the `head -50` piped run produces CORRECT output. But the nohup background run produces files 10x too large with the SAME binary.

The only difference: `head -50` kills the process after 50 lines. The nohup run continues. Could there be a race condition or buffering issue?

Actually — I wonder if the `head -50` piped version is producing CORRECT data because it STOPS early. And the full nohup run diverges later. Let me check: at 45M cells, the ftell shows 329M for streets. If this rate continues: 322M cells × (329M/45M) = 2.35 GiB. Correct!

But the ls -lh showed 6.8 GiB for streets at ~3 min into a nohup run. If 3 min = ~50M cells processed, that's 6.8 GiB / 50M = 136 bytes/cell. That's WAY off from the 7.3 bytes/cell from ftell.

Unless... `ls -lh` is returning stale sizes. The file might have been preallocated or the filesystem reports a different size than what's actually written. Let me check with `du` instead of `ls`:

For now, the nohup runs keep showing large files but the piped run shows correct ftell values. Let me just let the nohup run complete and check if the FINAL file sizes are correct. The intermediate `ls -lh` might be misleading:


sshpass -p 'elon-is-insane' ssh -o ProxyCommand="tailscale nc %h %p" -o StrictHostKeyChecking=no michtest@100.113.88.121 "killall -9 geocoder-patch 2>/dev/null; rm -rf /home/michtest/patched-planet-stream/" 2>&1 && sshpass -p 'elon-is-insane' ssh -o ProxyCommand="tailscale nc %h %p" -o StrictHostKeyChecking=no michtest@100.113.88.121 'cd /home/michtest && nohup bash -c "time ./geocoder-patch planet-B/ validated-patches/planet-mar16-to-mar23-v2.gcpatch -o patched-planet-stream/ 2>&1 && echo ---VERIFY--- && ls -lh patched-planet-stream/ && for f in planet-C/*; do b=\$(basename \$f); cmp -s patched-planet-stream/\$b planet-C/\$b || echo \"MISMATCH: \$b\"; done && echo All_checked" > planet-patch-final.log 2>&1 &' 2>&1 && echo "Started — monitoring"

Start and monitor


Command running in background with ID: b5saodjw9. Output is being written to: /tmp/claude-0/-root-traccar-geocoder/af1c4baa-0f9a-4420-a8b3-ce20dc28c8b9/tasks/b5saodjw9.output


sleep 60 && sshpass -p 'elon-is-insane' ssh -o ProxyCommand="tailscale nc %h %p" -o StrictHostKeyChecking=no michtest@100.113.88.121 "du -h /home/michtest/patched-planet-stream/street_entries.bin 2>/dev/null && ls -lh /home/michtest/patched-planet-stream/street_entries.bin 2>/dev/null && stat /home/michtest/patched-planet-stream/street_entries.bin 2>/dev/null | grep Size" 2>&1

Compare du vs ls vs stat for street_entries


7.0G	/home/michtest/patched-planet-stream/street_entries.bin
-rw-rw-r-- 1 michtest michtest 7.0G Mar 30 06:19 /home/michtest/patched-planet-stream/street_entries.bin
  Size: 7457308672	Blocks: 14565064   IO Block: 4096   regular file


7.0 GiB actual disk usage matches 7.0 GiB file size. Not a filesystem caching issue. The file IS 7 GiB at 1 min = ~50M cells.

From the `head -50` trace, at 45M cells: ftell=329M. At ~50M cells: should be ~365M. NOT 7 GiB.

This means the nohup run is producing DIFFERENT output from the `head -50` run, even though they use the same binary. The only difference is the process lifecycle (head kills after 50 lines vs full run).

OH WAIT — I just realized: the `head -50` run ALSO deletes the output at the end (`rm -rf patched-planet-stream/` at the end of the command). So I never saw its final file sizes. The ftell values in the log were correct at that point, but maybe something goes wrong LATER.

The 7 GiB at 50M cells means ~140 bytes per cell. But ftell said ~7.3 bytes per cell at 45M cells. The discrepancy is HUGE. Unless ftell is lying (buffered output not yet flushed to disk), the numbers can't be reconciled.

Actually — `ftell` returns the BUFFER position (what's been written to the FILE* buffer), not what's on disk. The `fwrite` buffers internally. So `ftell` should be ahead of the actual disk file size (more written than flushed). But the disk file is LARGER than ftell — that means something else is writing to the file.

Could the OLD entries from a merge replay be hitting the same filename? No — the merge replay writes to `addr_points.bin`, `street_ways.bin`, etc. The entry files are ONLY written in the streaming pipeline. Unless...

**OH!** I bet the issue is: the merge replay's `read_file()` for old data uses `cur_dir/street_entries.bin`. But the MERGE SECTION in the patch might contain a section for `street_entries.bin`! If the patch has a full-replacement or merge section for entry files, the merge replay would write them to `out_dir/street_entries.bin`, and then the streaming pipeline ALSO writes to the same file (opening with `"wb"` which truncates).

But wait — the diff tool doesn't include merge sections for entry files (they're reconstructed by the patch tool). The entry files should never appear as merge sections. Let me verify:

The diff tool's merge sections include: ADDR_POINTS, STREET_WAYS, STREET_NODES, INTERP_WAYS, INTERP_NODES, ADMIN_POLYGONS, ADMIN_VERTICES. Entry files (STREET_ENTRIES, ADDR_ENTRIES, etc.) are NOT included as merge sections.

So the streaming pipeline IS the only writer for entry files. The 7 GiB must be correct... unless `fwrite` is somehow writing more than requested. That would be a system-level bug which is extremely unlikely.

Let me just run it to completion and check the FINAL sizes:


sleep 600 && sshpass -p 'elon-is-insane' ssh -o ProxyCommand="tailscale nc %h %p" -o StrictHostKeyChecking=no michtest@100.113.88.121 "grep -E 'Geo:|Admin pip|Total|All_checked|MISMATCH|real' /home/michtest/planet-patch-final.log && ps aux | grep 'geocoder-patch planet' | grep -v grep | wc -l" 2>&1

Check 10 min


Command running in background with ID: btuw5jrqt. Output is being written to: /tmp/claude-0/-root-traccar-geocoder/af1c4baa-0f9a-4420-a8b3-ce20dc28c8b9/tasks/btuw5jrqt.output


sleep 600 && cat /tmp/claude-0/-root-traccar-geocoder/af1c4baa-0f9a-4420-a8b3-ce20dc28c8b9/tasks/btuw5jrqt.output 2>/dev/null

Check 20 min


(Bash completed with no output)


sshpass -p 'elon-is-insane' ssh -o ProxyCommand="tailscale nc %h %p" -o StrictHostKeyChecking=no michtest@100.113.88.121 "tail -20 /home/michtest/planet-patch-final.log" 2>&1

Check directly


    100000000/322726761 se=690M ae=189M ie=0M rss=13350M
    105000000/322726761 se=720M ae=224M ie=0M rss=13415M
    110000000/322726761 se=2342M ae=2653M ie=3M rss=13441M
    115000000/322726761 se=5243M ae=7217M ie=10M rss=13441M
    120000000/322726761 se=8282M ae=12958M ie=17M rss=13449M
    125000000/322726761 se=12219M ae=18443M ie=25M rss=13455M
    130000000/322726761 se=17253M ae=22727M ie=32M rss=13461M
    135000000/322726761 se=21874M ae=26812M ie=39M rss=13472M
    140000000/322726761 se=24655M ae=33218M ie=46M rss=13480M
    145000000/322726761 se=27163M ae=40080M ie=53M rss=13486M
    150000000/322726761 se=30010M ae=44785M ie=60M rss=13492M
    155000000/322726761 se=34872M ae=49110M ie=68M rss=13497M
    160000000/322726761 se=38456M ae=53340M ie=75M rss=13503M
    165000000/322726761 se=41575M ae=57678M ie=82M rss=13509M
    170000000/322726761 se=44233M ae=62146M ie=89M rss=13515M
    175000000/322726761 se=46855M ae=66347M ie=96M rss=13522M
    180000000/322726761 se=51983M ae=73548M ie=410M rss=15326M
    185000000/322726761 se=61077M ae=83447M ie=1128M rss=15330M
    190000000/322726761 se=68786M ae=95698M ie=2111M rss=15330M
    195000000/322726761 se=76271M ae=107550M ie=2973M rss=15330M


**THERE'S THE BUG!** Look at the jump between 105M and 110M cells:

```
105M: se=720M ae=224M ie=0M
110M: se=2342M ae=2653M ie=3M
```

In just 5M cells, street_entries jumped from 720M to 2342M (+1622M = 324 bytes/cell!) and addr_entries jumped from 224M to 2653M (+2429M = 486 bytes/cell!). Normal rate is ~8 bytes/cell.

Something catastrophic happens around cell 107M-110M that causes entries to explode. This must be a corrupt entry in the old entry file at that position, or the offset from old_geo is wrong at that point.

The geo_cells are sorted by cell_id. Around cell 107M, we hit a region where either the old entry offsets are corrupt or the mmap'd entry files have garbage data.

Actually — this might be where the remap vector entries go wrong. The remap vectors have `0xFFFFFFFF` for unmapped entries. If a remapped ID is `0xFFFFFFFF` and happens to equal the no_data sentinel, the sorted output would have `0xFFFFFFFF` values. But the bug is in entry SIZES, not values.

The `parse_ids` reads `uint16_t count` from the old entry file. If the count is large (e.g., 10000), it would write 40 KB per cell. For 5M cells with 10000 IDs each, that's 200 GiB. But the sanity check limits to 10000.

Wait — I added a sanity check of `c > 10000` but maybe 10000 is still too high. Let me check what the normal max is:

Looking at the jump: 486 bytes/cell for addr for 5M cells = average count of ~120 IDs per cell. That's high but plausible for dense areas. But the REFERENCE has only 14 bytes/cell for addr. Something is reading the wrong data.

I think the issue is that the offset values in `old_geo` for those cells point to the WRONG locations in the old entry files. This could happen if the `oi` (old index) is wrong for some cells. Let me check: the cells are sorted by `cid`. The `oi` comes from the cell list which maps `cid → old_index`. If two cells have the same `cid` (duplicate), the index could be wrong.

Or — the `oi` index is correct but after removing/adding cells, the sorted order puts cells in a different order than the old_geo file. The offsets in old_geo are correct for each cell, but if the pipeline reads them out of order... wait, that should be fine since we look up by `oi * 20`.

I need to dig deeper but the context is getting long. Let me save the current state and continue in the next message cycle. The key finding: entry explosion happens around cell 107-110M in the planet processing, likely due to corrupt offsets or ID remap issues in that region.


sshpass -p 'elon-is-insane' ssh -o ProxyCommand="tailscale nc %h %p" -o StrictHostKeyChecking=no michtest@100.113.88.121 "kill -9 \$(ps aux | grep 'geocoder-patch' | grep -v grep | awk '{print \$2}') 2>/dev/null; rm -rf /home/michtest/patched-planet-stream/" 2>&1

Kill and clean for next iteration


The entry explosion at cell ~107M needs investigation. The ftell-reported sizes jump 10x in 5M cells. Let me add a diagnostic that traces what happens at exactly that boundary — print the offset and count for each cell around 107M:


Edit

                if (cell_count <= 5) {
                    std::cerr << "    TRACE cell=" << cell_count << " geo_off=" << geo_off << " off=" << off << " n=" << id_buf.size() << std::endl;
                }

                if (cell_count >= 107000000 && cell_count <= 107000010 && geo_off == 8) {
                    std::cerr << "    TRACE cell=" << cell_count << " cid=" << cid << " oi=" << oi
                              << " off=" << off << " n=" << id_buf.size();
                    if (id_buf.size() > 0) std::cerr << " first_id=" << id_buf[0];
                    std::cerr << std::endl;
                }


                if (cell_count >= 107000000 && cell_count <= 107000010 && geo_off == 8) {
                    std::cerr << "    TRACE cell=" << cell_count << " cid=" << cid << " oi=" << oi
                              << " off=" << off << " n=" << id_buf.size();
                    if (id_buf.size() > 0) std::cerr << " first_id=" << id_buf[0];
                    std::cerr << std::endl;
                }

// geocoder-patch v4: Low-memory streaming patch application.
//
// Key design: mmap old files (zero copy), stream output via fwrite (never accumulate),
// process entry pipeline cell-by-cell. Target: <1 GiB peak RSS for planet.
//
// Usage: geocoder-patch <current-dir> <patch-file> -o <output-dir>

#include <algorithm>
#include <cstdint>
#include <cstdio>
#include <cstdlib>
#include <cstring>
#include <ctime>
#include <fcntl.h>
#include <iomanip>
#include <iostream>
#include <malloc.h>
#include <string>
#include <sys/mman.h>
#include <sys/stat.h>
#include <unordered_map>
#include <unordered_set>
#include <unistd.h>
#include <vector>

#include "patch_format.h"

enum MergeOp : uint8_t { OP_MATCH_RUN = 0, OP_INSERT_RUN = 1, OP_DELETE_RUN = 2 };

// --- Helpers ---
static double now_ms() {
    struct timespec ts; clock_gettime(CLOCK_MONOTONIC, &ts);
    return ts.tv_sec * 1000.0 + ts.tv_nsec / 1e6;
}
static size_t get_rss_mb() {
    FILE* f = fopen("/proc/self/statm", "r");
    if (!f) return 0;
    size_t dummy, rss; fscanf(f, "%zu %zu", &dummy, &rss); fclose(f);
    return rss * 4096 / (1024*1024);
}
static void log_phase(const char* label, double start) {
    std::cerr << "  [" << std::fixed << std::setprecision(1) << (now_ms() - start) / 1000.0
              << "s, " << get_rss_mb() << " MiB] " << label << std::endl;
}

// mmap a file read-only. Returns {pointer, size}. Caller must munmap.
struct MappedFile { const char* data; size_t size; };
static MappedFile mmap_file(const std::string& path) {
    int fd = open(path.c_str(), O_RDONLY);
    if (fd < 0) return {nullptr, 0};
    struct stat st; fstat(fd, &st);
    size_t sz = st.st_size;
    if (sz == 0) { close(fd); return {nullptr, 0}; }
    void* p = mmap(nullptr, sz, PROT_READ, MAP_PRIVATE, fd, 0);
    close(fd);
    if (p == MAP_FAILED) return {nullptr, 0};
    return {static_cast<const char*>(p), sz};
}
static void unmap_file(MappedFile& f) {
    if (f.data) { munmap(const_cast<char*>(f.data), f.size); f.data = nullptr; f.size = 0; }
}

// Detect stride from file size
static size_t detect_stride(const std::string& path, std::initializer_list<size_t> candidates) {
    struct stat st; if (stat(path.c_str(), &st) != 0) return *candidates.begin();
    for (size_t s : candidates) if (st.st_size % s == 0 && st.st_size > 0) return s;
    return *candidates.begin();
}

int main(int argc, char* argv[]) {
    if (argc < 5 || std::string(argv[3]) != "-o") {
        std::cerr << "Usage: geocoder-patch <current-dir> <patch-file> -o <output-dir>" << std::endl;
        return 1;
    }
    std::string cur_dir = argv[1], patch_path = argv[2], out_dir = argv[4];
    ensure_dir(out_dir);
    std::string tmpdir = "/tmp/geocoder-patch-" + std::to_string(getpid());
    ensure_dir(tmpdir);
    double t_start = now_ms();

    // --- Phase 1: Decompress + mmap patch ---
    std::string raw_path = tmpdir + "/patch.raw";
    {
        std::string cmd = "zstd -d '" + patch_path + "' -o '" + raw_path + "' -f --quiet 2>/dev/null";
        if (system(cmd.c_str()) != 0) { std::cerr << "Failed to decompress" << std::endl; return 1; }
    }
    MappedFile patch_map = mmap_file(raw_path);
    if (!patch_map.data) { std::cerr << "Failed to mmap patch" << std::endl; return 1; }
    remove(raw_path.c_str());
    const char* P = patch_map.data;
    size_t patch_size = patch_map.size;
    std::cerr << "Patch: " << patch_size << " bytes" << std::endl;
    log_phase("Decompress", t_start);

    size_t pos = 0;
    auto ru32 = [&]() -> uint32_t { uint32_t v; memcpy(&v, P+pos, 4); pos += 4; return v; };
    auto ru64 = [&]() -> uint64_t { uint64_t v; memcpy(&v, P+pos, 8); pos += 8; return v; };

    // Header
    if (memcmp(P, GCPATCH_MAGIC, 8) != 0) { std::cerr << "Bad magic" << std::endl; return 1; }
    pos = 8;
    uint32_t ver = ru32(); if (ver != 2) { std::cerr << "Bad version" << std::endl; return 1; }
    ru32(); // flags

    // --- Phase 2: String rebuild ---
    std::unordered_map<uint32_t, uint32_t> str_remap;
    {
        uint32_t marker = ru32();
        if (marker == 0xFFFFFFF7) {
            uint32_t n_added = ru32(), n_deleted = ru32();
            std::vector<std::string> added;
            for (uint32_t i = 0; i < n_added; i++) { const char* s = P+pos; added.push_back(s); pos += strlen(s)+1; }
            std::vector<uint32_t> del_idx(n_deleted);
            for (uint32_t i = 0; i < n_deleted; i++) del_idx[i] = ru32();
            std::unordered_set<uint32_t> del_set(del_idx.begin(), del_idx.end());

            MappedFile old_pool = mmap_file(cur_dir + "/strings.bin");
            std::vector<std::pair<uint32_t, std::string>> old_strs;
            size_t sp = 0;
            while (sp < old_pool.size) {
                old_strs.push_back({(uint32_t)sp, std::string(old_pool.data + sp)});
                sp += strlen(old_pool.data + sp) + 1;
            }
            std::vector<std::string> merged;
            for (size_t i = 0; i < old_strs.size(); i++)
                if (!del_set.count(i)) merged.push_back(old_strs[i].second);
            for (auto& s : added) merged.push_back(s);
            std::sort(merged.begin(), merged.end());

            // Write new pool and build remap
            FILE* fp = fopen((out_dir + "/strings.bin").c_str(), "wb");
            std::unordered_map<std::string, uint32_t> new_offs;
            uint32_t wpos = 0;
            for (auto& s : merged) {
                new_offs[s] = wpos;
                fwrite(s.c_str(), 1, s.size() + 1, fp);
                wpos += s.size() + 1;
            }
            fclose(fp);
            for (auto& [old_off, s] : old_strs) {
                auto it = new_offs.find(s);
                if (it != new_offs.end() && it->second != old_off) str_remap[old_off] = it->second;
            }
            unmap_file(old_pool);
            std::cerr << "  Strings: +" << n_added << " -" << n_deleted << " → " << merged.size()
                      << ", " << str_remap.size() << " remapped" << std::endl;
            marker = ru32();
        }
        if (marker == 0xFFFFFFFE) { uint32_t c = ru32(); for (uint32_t i = 0; i < c; i++) { uint32_t a = ru32(), b = ru32(); str_remap[a] = b; } }
        else pos -= 4;
    }
    log_phase("Strings", t_start);

    // Detect strides
    size_t way_stride = detect_stride(cur_dir + "/street_ways.bin", {12, 9});
    size_t interp_stride = detect_stride(cur_dir + "/interp_ways.bin", {24, 20, 18});
    size_t admin_stride = detect_stride(cur_dir + "/admin_polygons.bin", {24, 20, 19});

    // String remap applier (works on mutable copy)
    auto apply_str_remap = [&](std::vector<char>& data, uint32_t fid, size_t stride) {
        if (str_remap.empty()) return;
        std::vector<size_t> offs;
        if (fid == (uint32_t)PatchFileId::ADDR_POINTS) offs = {8, 12};
        else if (fid == (uint32_t)PatchFileId::STREET_WAYS) offs = {(stride == 12) ? 8ul : 5ul};
        else if (fid == (uint32_t)PatchFileId::INTERP_WAYS) offs = {(stride >= 20) ? 8ul : 5ul};
        else if (fid == (uint32_t)PatchFileId::ADMIN_POLYGONS) offs = {8};
        else return;
        for (size_t i = 0; i + stride <= data.size(); i += stride)
            for (size_t o : offs) { uint32_t v; memcpy(&v, data.data()+i+o, 4); auto it = str_remap.find(v); if (it != str_remap.end()) memcpy(data.data()+i+o, &it->second, 4); }
    };

    // --- Phase 3: Merge replays (streaming output) ---
    // ID remaps: file_id → vector<uint32_t> where remap[old_idx] = new_idx
    std::unordered_map<uint32_t, std::vector<uint32_t>> id_remaps;
    std::vector<uint64_t> geo_added, geo_removed, admin_added, admin_removed;
    std::unordered_map<uint64_t, uint8_t> flag_corrections;
    struct CellCorr { uint64_t cell_id; std::vector<uint32_t> ids; };
    std::unordered_map<uint32_t, std::vector<CellCorr>> entry_corrections;

    while (pos < patch_size) {
        uint32_t file_id = ru32();
        if (file_id == 0xFFFFFFFF) break;

        // --- Metadata sections ---
        if (file_id == 0xFFFFFFF7) {
            // String diff in main loop — process it (may be the only occurrence)
            uint32_t n_added = ru32(), n_deleted = ru32();
            std::vector<std::string> added;
            for (uint32_t i = 0; i < n_added; i++) { const char* s = P+pos; added.push_back(s); pos += strlen(s)+1; }
            std::vector<uint32_t> del_idx(n_deleted);
            for (uint32_t i = 0; i < n_deleted; i++) del_idx[i] = ru32();
            std::unordered_set<uint32_t> del_set(del_idx.begin(), del_idx.end());
            MappedFile old_pool = mmap_file(cur_dir + "/strings.bin");
            std::vector<std::pair<uint32_t, std::string>> old_strs;
            size_t sp = 0;
            while (sp < old_pool.size) { old_strs.push_back({(uint32_t)sp, std::string(old_pool.data+sp)}); sp += strlen(old_pool.data+sp)+1; }
            std::vector<std::string> merged;
            for (size_t i = 0; i < old_strs.size(); i++) if (!del_set.count(i)) merged.push_back(old_strs[i].second);
            for (auto& s : added) merged.push_back(s);
            std::sort(merged.begin(), merged.end());
            FILE* fp = fopen((out_dir + "/strings.bin").c_str(), "wb");
            std::unordered_map<std::string, uint32_t> new_offs;
            uint32_t wpos = 0;
            for (auto& s : merged) { new_offs[s] = wpos; fwrite(s.c_str(), 1, s.size()+1, fp); wpos += s.size()+1; }
            fclose(fp);
            for (auto& [old_off, s] : old_strs) { auto it = new_offs.find(s); if (it != new_offs.end() && it->second != old_off) str_remap[old_off] = it->second; }
            unmap_file(old_pool);
            std::cerr << "  Strings (loop): +" << n_added << " -" << n_deleted << " → " << merged.size() << std::endl;
            continue;
        }
        if (file_id == CELL_CHANGES_GEO_MARKER) {
            uint32_t na = ru32(), nr = ru32();
            geo_added.resize(na); geo_removed.resize(nr);
            for (uint32_t i = 0; i < na; i++) { memcpy(&geo_added[i], P+pos, 8); pos += 8; }
            for (uint32_t i = 0; i < nr; i++) { memcpy(&geo_removed[i], P+pos, 8); pos += 8; }
            std::cerr << "  Geo cells: +" << na << " -" << nr << std::endl;
            continue;
        }
        if (file_id == CELL_CHANGES_ADMIN_MARKER) {
            uint32_t na = ru32(), nr = ru32();
            admin_added.resize(na); admin_removed.resize(nr);
            for (uint32_t i = 0; i < na; i++) { memcpy(&admin_added[i], P+pos, 8); pos += 8; }
            for (uint32_t i = 0; i < nr; i++) { memcpy(&admin_removed[i], P+pos, 8); pos += 8; }
            std::cerr << "  Admin cells: +" << na << " -" << nr << std::endl;
            continue;
        }
        if (file_id == CELL_FLAGS_MARKER) {
            uint32_t c = ru32();
            for (uint32_t i = 0; i < c; i++) { uint64_t cid; memcpy(&cid, P+pos, 8); pos += 8; flag_corrections[cid] = P[pos++]; }
            std::cerr << "  Flag corrections: " << c << std::endl;
            continue;
        }
        if (file_id == SECONDARY_REMAP_MARKER) {
            uint32_t nf = ru32();
            for (uint32_t f = 0; f < nf; f++) {
                uint32_t fid = ru32(), np = ru32();
                auto& rm = id_remaps[fid];
                for (uint32_t i = 0; i < np; i++) { uint32_t o = ru32(), n = ru32(); if (o < rm.size()) rm[o] = n; }
                std::cerr << "  Secondary remap " << fid << ": " << np << " pairs" << std::endl;
            }
            continue;
        }
        if (file_id == ENTRY_CORRECTION_MARKER) {
            uint32_t fid = ru32(), c = ru32();
            auto& list = entry_corrections[fid];
            for (uint32_t i = 0; i < c; i++) {
                uint64_t cid; memcpy(&cid, P+pos, 8); pos += 8;
                uint16_t ec; memcpy(&ec, P+pos, 2); pos += 2;
                std::vector<uint32_t> ids(ec);
                if (ec > 0) { memcpy(ids.data(), P+pos, ec*4); pos += ec*4; }
                list.push_back({cid, std::move(ids)});
            }
            std::cerr << "  Entry corrections " << fid << ": " << c << std::endl;
            continue;
        }

        // --- Merge sequence replay (streaming) ---
        uint32_t stride = ru32();
        uint64_t old_size = ru64(), new_size = ru64();
        if (file_id >= (uint32_t)PatchFileId::COUNT) { std::cerr << "Unknown file " << file_id << std::endl; break; }
        const char* fname = patch_file_names[file_id];

        if (stride == 0) {
            // Full replacement — write directly from patch mmap
            uint32_t nf = ru32(); (void)nf; uint64_t ds = ru64();
            FILE* fp = fopen((out_dir + "/" + fname).c_str(), "wb");
            fwrite(P+pos, 1, ds, fp); fclose(fp);
            pos += ds;
            std::cerr << "  " << fname << ": full replace " << ds << " bytes" << std::endl;
            continue;
        }
        if (stride == 0xFE) { uint32_t nf = ru32(); (void)nf; uint64_t ds = ru64(); pos += ds; continue; }

        size_t actual_stride = stride;
        if (file_id == (uint32_t)PatchFileId::STREET_WAYS) actual_stride = way_stride;
        else if (file_id == (uint32_t)PatchFileId::INTERP_WAYS) actual_stride = interp_stride;
        else if (file_id == (uint32_t)PatchFileId::ADMIN_POLYGONS) actual_stride = admin_stride;

        // Load old file into mutable buffer (need to apply remap + fixups)
        // For large files (nodes/vertices), this is the peak memory per file.
        // Files are processed sequentially, so only one is in memory at a time.
        auto old_data = read_file(cur_dir + "/" + std::string(fname));

        // Zero padding
        if (file_id == (uint32_t)PatchFileId::ADMIN_POLYGONS && actual_stride == 24)
            for (size_t i = 0; i + actual_stride <= old_data.size(); i += actual_stride) memset(old_data.data()+i+13, 0, 3);
        if (file_id == (uint32_t)PatchFileId::INTERP_WAYS && actual_stride == 24)
            for (size_t i = 0; i + actual_stride <= old_data.size(); i += actual_stride) { memset(old_data.data()+i+5, 0, 3); memset(old_data.data()+i+21, 0, 3); }

        apply_str_remap(old_data, file_id, actual_stride);

        // Apply fixups
        uint32_t n_fixups = ru32();
        if (n_fixups > 0) {
            uint32_t delta_size = ru32();
            uint32_t pi = 0, pv = 0; size_t fp = pos;
            for (uint32_t i = 0; i < n_fixups; i++) {
                uint32_t idx = pi + read_varint(P, fp);
                uint32_t val = pv + read_varint(P, fp);
                if ((size_t)idx * actual_stride + 4 <= old_data.size())
                    memcpy(old_data.data() + (size_t)idx * actual_stride, &val, 4);
                pi = idx; pv = val;
            }
            pos = fp;
        }

        // Replay merge sequence — stream output directly to file
        uint64_t seq_size = ru64();
        size_t seq_end = pos + seq_size;
        FILE* outf = fopen((out_dir + "/" + fname).c_str(), "wb");

        // Track ID remap (only for data files that need entry reconstruction)
        bool track = (file_id == (uint32_t)PatchFileId::ADDR_POINTS ||
                      file_id == (uint32_t)PatchFileId::STREET_WAYS ||
                      file_id == (uint32_t)PatchFileId::INTERP_WAYS ||
                      file_id == (uint32_t)PatchFileId::ADMIN_POLYGONS);
        std::vector<uint32_t> id_map;
        if (track) id_map.assign(old_data.size() / actual_stride, 0xFFFFFFFF);

        size_t old_rec = 0, new_rec = 0, old_bytes = 0, written = 0;
        while (pos < seq_end) {
            uint8_t op = P[pos++];
            uint32_t count; memcpy(&count, P+pos, 4); pos += 4;
            if (op == OP_MATCH_RUN) {
                size_t bytes = count * actual_stride;
                fwrite(old_data.data() + old_bytes, 1, bytes, outf);
                written += bytes;
                if (track) for (uint32_t k = 0; k < count; k++) if (old_rec+k < id_map.size()) id_map[old_rec+k] = new_rec+k;
                old_rec += count; new_rec += count; old_bytes += bytes;
            } else if (op == OP_INSERT_RUN) {
                size_t bytes = count * actual_stride;
                fwrite(P+pos, 1, bytes, outf);
                written += bytes; pos += bytes; new_rec += count;
            } else if (op == OP_DELETE_RUN) {
                old_rec += count; old_bytes += count * actual_stride;
            }
        }
        fclose(outf);

        if (track) {
            id_remaps[file_id] = std::move(id_map);
            size_t matched = 0;
            for (auto v : id_remaps[file_id]) if (v != 0xFFFFFFFF) matched++;
            std::cerr << "  " << fname << ": " << written << " bytes (mapped " << matched << "/" << id_remaps[file_id].size() << ")" << std::endl;
        } else {
            std::cerr << "  " << fname << ": " << written << " bytes" << std::endl;
        }

        // Free old data immediately — don't let glibc hold the pages
        { std::vector<char>().swap(old_data); }
        malloc_trim(0);
    }
    log_phase("Merge replays", t_start);

    // Free string remap (no longer needed)
    { std::unordered_map<uint32_t,uint32_t>().swap(str_remap); }
    malloc_trim(0);

    // --- Phase 4: Streaming entry pipeline ---
    // Walk old geo_cells, remap IDs, apply corrections, write output directly to files.
    {
        double t_entry = now_ms();
        std::cerr << "Entry pipeline..." << std::endl;

        auto get_rm = [&](PatchFileId fid) -> const std::vector<uint32_t>& {
            static std::vector<uint32_t> empty;
            auto it = id_remaps.find((uint32_t)fid); return it != id_remaps.end() ? it->second : empty;
        };
        auto& w_rm = get_rm(PatchFileId::STREET_WAYS);
        auto& a_rm = get_rm(PatchFileId::ADDR_POINTS);
        auto& i_rm = get_rm(PatchFileId::INTERP_WAYS);

        // Build correction lookup maps
        std::unordered_map<uint64_t, const std::vector<uint32_t>*> cs, ca, ci;
        for (auto& [fid, list] : entry_corrections) {
            auto* m = (fid == (uint32_t)PatchFileId::STREET_ENTRIES) ? &cs :
                      (fid == (uint32_t)PatchFileId::ADDR_ENTRIES) ? &ca :
                      (fid == (uint32_t)PatchFileId::INTERP_ENTRIES) ? &ci : nullptr;
            if (m) for (auto& c : list) (*m)[c.cell_id] = &c.ids;
        }

        // mmap old files (zero-copy read)
        MappedFile old_geo = mmap_file(cur_dir + "/geo_cells.bin");
        MappedFile old_se = mmap_file(cur_dir + "/street_entries.bin");
        MappedFile old_ae = mmap_file(cur_dir + "/addr_entries.bin");
        MappedFile old_ie = mmap_file(cur_dir + "/interp_entries.bin");

        // Build derived cell list
        std::unordered_set<uint64_t> rm_set(geo_removed.begin(), geo_removed.end());
        struct CR { uint64_t cid; int32_t idx; };
        std::vector<CR> cells;
        cells.reserve(old_geo.size / 20 + geo_added.size());
        for (size_t i = 0; i < old_geo.size / 20; i++) {
            uint64_t cid; memcpy(&cid, old_geo.data + i*20, 8);
            if (!rm_set.count(cid)) cells.push_back({cid, (int32_t)i});
        }
        for (uint64_t cid : geo_added) cells.push_back({cid, -1});
        std::sort(cells.begin(), cells.end(), [](const CR& a, const CR& b) { return a.cid < b.cid; });
        log_phase("  Cell list", t_entry);

        // Open output files for streaming write
        FILE* f_geo = fopen((out_dir + "/geo_cells.bin").c_str(), "wb");
        FILE* f_se = fopen((out_dir + "/street_entries.bin").c_str(), "wb");
        FILE* f_ae = fopen((out_dir + "/addr_entries.bin").c_str(), "wb");
        FILE* f_ie = fopen((out_dir + "/interp_entries.bin").c_str(), "wb");
        uint32_t no_data = 0xFFFFFFFF;

        // Reusable buffer
        std::vector<uint32_t> id_buf;
        id_buf.reserve(256);

        auto parse_ids = [](const MappedFile& f, uint32_t off, std::vector<uint32_t>& buf) {
            buf.clear();
            if (off == 0xFFFFFFFF || off + 2 > f.size) return;
            uint16_t c; memcpy(&c, f.data+off, 2);
            if (c > 10000 || off+2+(size_t)c*4 > f.size) return; // sanity: no cell has >10K entries
            buf.resize(c); memcpy(buf.data(), f.data+off+2, c*4);
        };
        auto remap_buf = [](std::vector<uint32_t>& buf, const std::vector<uint32_t>& rm) {
            for (auto& id : buf) if (id < rm.size() && rm[id] != 0xFFFFFFFF) id = rm[id];
            std::sort(buf.begin(), buf.end());
        };
        auto write_ids = [](FILE* f, const uint32_t* ids, size_t n) -> uint32_t {
            if (n == 0) return 0xFFFFFFFF;
            uint32_t off = (uint32_t)ftell(f);
            uint16_t c = (uint16_t)n;
            fwrite(&c, 2, 1, f); fwrite(ids, 4, n, f);
            return off;
        };

        size_t sc = 0, ac = 0, ic = 0;
        size_t cell_count = 0;
        for (auto& [cid, oi] : cells) {
            if (++cell_count % 5000000 == 0) {
                std::cerr << "    " << cell_count << "/" << cells.size()
                          << " se=" << ftell(f_se)/1024/1024 << "M ae=" << ftell(f_ae)/1024/1024
                          << "M ie=" << ftell(f_ie)/1024/1024 << "M rss=" << get_rss_mb() << "M" << std::endl;
            }
            // Process one entry type at a time using shared id_buf
            auto process = [&](const MappedFile& old_e, size_t geo_off,
                               const std::vector<uint32_t>& rm,
                               const std::unordered_map<uint64_t, const std::vector<uint32_t>*>& corr,
                               FILE* outf, size_t& cc) -> uint32_t {
                auto cit = corr.find(cid);
                if (cit != corr.end()) { cc++; return write_ids(outf, cit->second->data(), cit->second->size()); }
                if (oi < 0) return no_data;
                uint32_t off; memcpy(&off, old_geo.data + oi*20 + geo_off, 4);
                parse_ids(old_e, off, id_buf);
                if (id_buf.empty()) return no_data;
                if (cell_count <= 5) {
                    std::cerr << "    TRACE cell=" << cell_count << " geo_off=" << geo_off << " off=" << off << " n=" << id_buf.size() << std::endl;
                }
                remap_buf(id_buf, rm);
                return write_ids(outf, id_buf.data(), id_buf.size());
            };

            // Determine flags
            bool hs = false, ha = false, hi = false;
            if (oi >= 0) {
                uint32_t s, a, ip; memcpy(&s, old_geo.data+oi*20+8, 4); memcpy(&a, old_geo.data+oi*20+12, 4); memcpy(&ip, old_geo.data+oi*20+16, 4);
                hs = (s != no_data); ha = (a != no_data); hi = (ip != no_data);
            }
            auto fc = flag_corrections.find(cid);
            if (fc != flag_corrections.end()) { hs = fc->second & 1; ha = fc->second & 2; hi = fc->second & 4; }
            if (cs.count(cid)) hs = true;
            if (ca.count(cid)) ha = true;
            if (ci.count(cid)) hi = true;

            uint32_t so = hs ? process(old_se, 8, w_rm, cs, f_se, sc) : no_data;
            uint32_t ao = ha ? process(old_ae, 12, a_rm, ca, f_ae, ac) : no_data;
            uint32_t io = hi ? process(old_ie, 16, i_rm, ci, f_ie, ic) : no_data;
            if (cell_count == 1) {
                // Count how many old cells have each flag
                size_t os = 0, oa = 0, oi_c = 0;
                for (size_t ci = 0; ci < old_geo.size / 20; ci++) {
                    uint32_t sv, av, iv;
                    memcpy(&sv, old_geo.data + ci*20+8, 4); memcpy(&av, old_geo.data + ci*20+12, 4); memcpy(&iv, old_geo.data + ci*20+16, 4);
                    if (sv != no_data) os++; if (av != no_data) oa++; if (iv != no_data) oi_c++;
                }
                std::cerr << "    DEBUG: old cells=" << old_geo.size/20 << " has_street=" << os << " has_addr=" << oa << " has_interp=" << oi_c << std::endl;
            }

            fwrite(&cid, 8, 1, f_geo); fwrite(&so, 4, 1, f_geo); fwrite(&ao, 4, 1, f_geo); fwrite(&io, 4, 1, f_geo);
        }
        fclose(f_geo); fclose(f_se); fclose(f_ae); fclose(f_ie);
        unmap_file(old_geo); unmap_file(old_se); unmap_file(old_ae); unmap_file(old_ie);
        std::cerr << "  Geo: " << cells.size() << " cells, corrections: " << sc << " st, " << ac << " ad, " << ic << " in" << std::endl;
        log_phase("  Geo entries", t_entry);

        // Admin: small, use existing rebuild + corrections
        {
            std::unordered_map<uint32_t,uint32_t> ad_rm;
            auto ait = id_remaps.find((uint32_t)PatchFileId::ADMIN_POLYGONS);
            if (ait != id_remaps.end()) {
                for (uint32_t i = 0; i < ait->second.size(); i++)
                    if (ait->second[i] != 0xFFFFFFFF) ad_rm[i] = ait->second[i];
            }
            auto old_ac = read_file(cur_dir + "/admin_cells.bin");
            auto old_adme = read_file(cur_dir + "/admin_entries.bin");
            auto admin = rebuild_admin_from_remap(old_ac, old_adme, ad_rm, admin_added, admin_removed);

            auto ecit = entry_corrections.find((uint32_t)PatchFileId::ADMIN_ENTRIES);
            if (ecit != entry_corrections.end()) {
                std::unordered_map<uint64_t, const std::vector<uint32_t>*> ac_corr;
                for (auto& c : ecit->second) ac_corr[c.cell_id] = &c.ids;
                size_t n = admin.admin_cells_data.size() / 12;
                FILE* fac = fopen((out_dir + "/admin_cells.bin").c_str(), "wb");
                FILE* fae = fopen((out_dir + "/admin_entries.bin").c_str(), "wb");
                uint32_t ae_wpos = 0;
                for (size_t i = 0; i < n; i++) {
                    uint64_t cid; memcpy(&cid, admin.admin_cells_data.data()+i*12, 8);
                    fwrite(&cid, 8, 1, fac);
                    auto cit = ac_corr.find(cid);
                    if (cit != ac_corr.end()) {
                        uint32_t off = cit->second->empty() ? no_data : ae_wpos;
                        fwrite(&off, 4, 1, fac);
                        if (!cit->second->empty()) {
                            uint16_t c = cit->second->size();
                            fwrite(&c, 2, 1, fae); fwrite(cit->second->data(), 4, c, fae);
                            ae_wpos += 2 + c*4;
                        }
                    } else {
                        uint32_t old_off; memcpy(&old_off, admin.admin_cells_data.data()+i*12+8, 4);
                        if (old_off != no_data && old_off+2 <= admin.admin_entries_data.size()) {
                            uint32_t new_off = ae_wpos; fwrite(&new_off, 4, 1, fac);
                            uint16_t c; memcpy(&c, admin.admin_entries_data.data()+old_off, 2);
                            fwrite(admin.admin_entries_data.data()+old_off, 1, 2+c*4, fae);
                            ae_wpos += 2+c*4;
                        } else {
                            fwrite(&no_data, 4, 1, fac);
                        }
                    }
                }
                fclose(fac); fclose(fae);
                std::cerr << "  Admin: " << n << " cells, " << ac_corr.size() << " corrections" << std::endl;
            } else {
                write_file(out_dir + "/admin_cells.bin", admin.admin_cells_data);
                write_file(out_dir + "/admin_entries.bin", admin.admin_entries_data);
            }
            log_phase("  Admin", t_entry);
        }
    }

    // Cleanup
    unmap_file(patch_map);
    { std::string cmd = "rm -rf '" + tmpdir + "'"; system(cmd.c_str()); }
    log_phase("Total", t_start);
    std::cerr << "Patch applied. Output in " << out_dir << std::endl;
    return 0;
}

-                    std::cerr << "    TRACE cell=" << cell_count << " geo_off=" << geo_off << " off=" << off << " n=" << id_buf.size() << std::endl;

+                    std::cerr << "    TRACE cell=" << cell_count << " cid=" << cid << " oi=" << oi

+                    if (id_buf.size() > 0) std::cerr << " first_id=" << id_buf[0];

+                    std::cerr << std::endl;


Bash

make -j$(nproc) geocoder-patch 2>&1 | tail -3 && sshpass -p 'elon-is-insane' ssh -o ProxyCommand="tailscale nc %h %p" -o StrictHostKeyChecking=no michtest@100.113.88.121 "rm -f /home/michtest/geocoder-patch" 2>&1 && sshpass -p 'elon-is-insane' scp -o ProxyCommand="tailscale nc %h %p" -o StrictHostKeyChecking=no /root/traccar-geocoder/builder/build/geocoder-patch michtest@100.113.88.121:/home/michtest/ 2>&1 && sshpass -p 'elon-is-insane' ssh -o ProxyCommand="tailscale nc %h %p" -o StrictHostKeyChecking=no michtest@100.113.88.121 'cd /home/michtest && rm -rf patched-planet-stream/ && timeout 600 ./geocoder-patch planet-B/ validated-patches/planet-mar16-to-mar23-v2.gcpatch -o patched-planet-stream/ 2>&1 | grep -E "TRACE|105.*se=|110.*se=|cells/" | head -20; rm -rf patched-planet-stream/' 2>&1

Trace the explosion point


Command running in background with ID: b10wdjd12. Output is being written to: /tmp/claude-0/-root-traccar-geocoder/af1c4baa-0f9a-4420-a8b3-ce20dc28c8b9/tasks/b10wdjd12.output


sleep 60 && cat /tmp/claude-0/-root-traccar-geocoder/af1c4baa-0f9a-4420-a8b3-ce20dc28c8b9/tasks/b10wdjd12.output 2>/dev/null

Check trace output 1 min


[ 50%] Building CXX object CMakeFiles/geocoder-patch.dir/tools/geocoder_patch.cpp.o
[100%] Linking CXX executable geocoder-patch
[100%] Built target geocoder-patch
    105000000/322726761 se=720M ae=224M ie=0M rss=13413M
    TRACE cell=107000000 cid=5135971574926540800 oi=106913503 off=767887024 n=1 first_id=21893149
    TRACE cell=107000001 cid=5135971575060758528 oi=106913504 off=767887030 n=1 first_id=20001305
    TRACE cell=107000002 cid=5135971575194976256 oi=106913505 off=767887036 n=2 first_id=20001305
    TRACE cell=107000003 cid=5135971575597629440 oi=106913506 off=767887046 n=1 first_id=21893149
    TRACE cell=107000004 cid=5135971576268718080 oi=106913507 off=767887052 n=1 first_id=21893149
    TRACE cell=107000005 cid=5135971576402935808 oi=106913508 off=767887058 n=1 first_id=21893149
    TRACE cell=107000006 cid=5135971576671371264 oi=106913509 off=767887064 n=1 first_id=21893149
    TRACE cell=107000007 cid=5135971576805588992 oi=106913510 off=767887070 n=1 first_id=21893149
    TRACE cell=107000008 cid=5135971576939806720 oi=106913511 off=767887076 n=1 first_id=21893149
    TRACE cell=107000009 cid=5135971578013548544 oi=106913512 off=767887082 n=1 first_id=20621427
    TRACE cell=107000010 cid=5135971578147766272 oi=106913513 off=767887088 n=1 first_id=20621427
    110000000/322726761 se=2342M ae=2653M ie=3M rss=13439M


The cells at 107M look NORMAL — 1-2 IDs each, offsets incrementing by 6. The explosion happens between 107M and 110M. The ftell shows se=720M at 105M and se=2342M at 110M — a jump of 1622M in 5M cells.

But the trace shows normal-looking cells at 107M. The explosion must be AFTER 107M. Let me trace the exact point by adding a check:


Edit

                if (cell_count >= 107000000 && cell_count <= 107000010 && geo_off == 8) {
                    std::cerr << "    TRACE cell=" << cell_count << " cid=" << cid << " oi=" << oi
                              << " off=" << off << " n=" << id_buf.size();
                    if (id_buf.size() > 0) std::cerr << " first_id=" << id_buf[0];
                    std::cerr << std::endl;
                }

                // Log cells with anomalously many entries (>500)
                if (id_buf.size() > 500) {
                    std::cerr << "    BIG cell=" << cell_count << " cid=" << cid << " oi=" << oi
                              << " geo_off=" << geo_off << " off=" << off << " n=" << id_buf.size() << std::endl;
                }