Kinematic Positioning Benchmark (PPC-Dataset)¶

This benchmark evaluates MRTKLIB's kinematic (vehicle-mounted) positioning performance using the open-data PPC2024 (Precise Positioning Challenge) dataset collected for the contest organised by the Institute of Navigation Japan (測位航法学会). It covers six urban driving runs (Nagoya × 3, Tokyo × 3) and supports four positioning modes. Each benchmark config sets the correction axis explicitly:

Note: This benchmark is intentionally excluded from the regular CTest suite because it requires large external datasets. Run it on demand.

Disclaimer: The configuration parameters used here have not been tuned or optimised for maximum accuracy. Results are provided for reference only and should not be taken as representative of the best achievable performance.

Dataset¶

Overview¶

• Vehicle receiver: Septentrio mosaic-X5, 5 Hz, RINEX 3.04
• Reference station: Trimble NetR9/Alloy, 1 Hz, RINEX 3.04
• Ground truth: Applanix POS LV 220, provided as reference.csv

Manual Download¶

The PPC-Dataset (~200 MB) must be downloaded manually:

1. Visit the dataset page or use the direct SharePoint link provided by the PPC2024 organiser.
2. Download and unzip the archive.
3. Place the contents so that the layout matches:

data/benchmark/
  nagoya/
    run1/
      rover.obs
      base.nav
      base.obs
      reference.csv
      ...
    run2/ ...
    run3/ ...
  tokyo/
    run1/ ...
    ...

The data/benchmark/ directory is listed in .gitignore and will not be committed.

reference.csv Format¶

GPS TOW (s), GPS Week, Latitude (deg), Longitude (deg), Ellipsoid Height (m),
ECEF X (m), ECEF Y (m), ECEF Z (m), Roll (deg), Pitch (deg), Heading (deg),
East Velocity (m/s), North Velocity (m/s), Up Velocity (m/s)

Prerequisites¶

1. Build MRTKLIB (rnx2rtkp binary):

cmake --preset default
cmake --build build

1. Extract test data (ATX, ISB, ERP, etc. required by the CLAS conf):

cd build && ctest -R setup --output-on-failure

1. Install Python dependencies:

cd scripts && python -m venv .venv && source .venv/bin/activate
pip install numpy matplotlib

Quick Start¶

Step 1 — Download L6 correction data¶

cd scripts/benchmark
python download_l6.py --mode both

This downloads the CLAS L6D and MADOCA L6E files for all six runs into data/benchmark/l6/. Use --dry-run to preview the URLs without downloading.

single mode needs no correction data — it uses only rover.obs + base.nav (broadcast ephemeris). Skip this step and run run_benchmark.py --mode single --skip-download.

Options:

Step 2 — Run the benchmark¶

python scripts/benchmark/run_benchmark.py

This calls rnx2rtkp for each run × mode combination, saves NMEA output to data/benchmark/results/, and prints a summary table.

Results are cached: if the NMEA output file is newer than all its inputs, the positioning step is skipped. Use --force to re-run.

Full options:

Step 3 — Compare a single result (optional)¶

cd scripts/benchmark
python compare_ppc.py \
    --ref ../../data/benchmark/nagoya/run1/reference.csv \
    ../../data/benchmark/results/nagoya_run1_clas.nmea

v0.3.3 Benchmark Results¶

Results recorded on MRTKLIB v0.3.3, GNSS-only (no IMU), --skip-epochs 60.

Solution Quality Tiers¶

CLAS and RTK produce integer-fix solutions and are broken down into three tiers. MADOCA-PPP never produces an integer fix and is reported as a single PPP tier.

Rate% column: - FIX / FF rows: fraction of that tier among all matched epochs - PPP row: fraction of epochs with 2D horizontal error < 30 cm

TTFF column: - CLAS / RTK: first epoch of a ≥30-consecutive Q=4 run (shown on FIX row) - MADOCA: first epoch of a ≥30-consecutive sub-30 cm run (shown on PPP row)

Nagoya¶

Tokyo¶

Notes¶

• CLAS FIX accuracy (Q=4 only): RMS 2D is 0.3–1.1 m across all runs — consistent PPP-RTK performance under urban multipath.
• CLAS FF vs ALL gap: the large ALL RMS (13–627 m) reflects SPP fallback epochs (Q=1) that dominate in dense canyons. FF tier (24–78%) excludes these.
• RTK FF = ALL: RTK outputs Q=4 (fix) or Q=5 (float) only — no SPP fallback — so FF always equals ALL.
• RTK FIX quality: with triple-frequency AR (l1+2+3, ionoopt=off), most runs achieve sub-metre FIX RMS (e.g. nagoya_run3: 0.31 m, tokyo_run3: 0.29 m). tokyo_run2 FIX RMS remains elevated (18 m) despite tiny 1σ/95% values, indicating a small number of wrongly-fixed epochs that dominate the RMS at this ~27 km baseline. This false-fix issue is resolved in v0.4.0 (see below).
• MADOCA PPP N: outputs Q=3 (PPP float) or Q=0 (no solution); Q=0 is filtered, so N reflects epochs where the filter produced a solution. In nagoya_run1, only 27% of rover epochs have a valid solution (heavy urban multipath). In nagoya_run2 and tokyo_run2 (more open sky), coverage is ~100%. This is a real performance limitation — not a software or configuration issue.
• CLAS nagoya_run2 TTFF=0 means the very first epoch after the skip window was already in a sustained fix run.
• RTK baselines range from ~13 km (Nagoya) to ~27 km (Tokyo), which is long for kinematic RTK; float solutions dominate accordingly.

v0.4.0 RTK Benchmark Results (demo5 Integration)¶

MRTKLIB v0.4.0 (feat/demo5-integration) integrates several RTK algorithm improvements from rtklibexplorer's demo5 fork. The CLAS and MADOCA engines are unaffected by these changes.

Algorithm changes¶

RTK configuration changes¶

Three new parameters are enabled in conf/benchmark/rtk.conf:

pos2-arminfixsats  = 4    # min satellite pairs for a valid PAR fix
pos2-arfilter      = on   # exclude newly-locked sats that degrade ratio
pos2-thresdop      = 1.0  # Doppler slip threshold (cyc/s, 0=off)

RTK comparison: v0.3.3 → v0.4.0¶

--skip-epochs 60, FIX tier (Q=4) only.

Key findings (Phase 1A–3B)¶

• tokyo_run2 false-fix elimination: v0.3.3 had RMS_2D(fix) = 17.993 m despite tiny 1σ/95% values — a small cluster of wrongly-fixed epochs dominated the RMS at this ~27 km baseline. The rejc<2 guard (Phase 3B) prevents premature phase-bias reset on a single transient outlier. Combined with arminfixsats=4, the false fix is eliminated and RMS_2D(fix) drops to 0.095 m — a 189× improvement.
• nagoya_run2 fix rate: +12 pp (16.1% → 28.0%); PAR finds valid partial subsets on a run with fewer simultaneously-clean satellite pairs.
• 5/6 runs show improved RMS_2D(fix); 4/6 runs show improved Fix%.
• nagoya_run3 RMS_2D(fix) increases (0.307 → 0.837 m) while Fix% rises slightly, suggesting the fix window now includes noisier epochs that were previously rejected.

v0.4.1 RTK Benchmark Results (demo5 Phase 4A–4F)¶

A second deep-diff pass against demo5 identified additional correctness fixes (Phases 4A–4F). Results below use city conf (nagoya.conf / tokyo.conf) for precise base-station coordinates.

Additional algorithm changes (Phase 4A–4F)¶

RTK comparison: v0.4.0 → v0.4.1¶

--skip-epochs 60, FIX tier (Q=4) only. v0.4.0: no city conf (base position from RINEX header). v0.4.1: city conf applied (precise base-station LLH coordinates).

Fix rate¶

Fix rate vs v0.4.0 is mixed (Phase 4D conditional lock reverted), but all runs remain equal to or better than the v0.3.3 baseline except nagoya_run1 (27.4% vs 29.7%).

Accuracy (FIX epochs only)¶

Key findings (Phase 4A–4F)¶

• Fix rate vs v0.3.3 baseline: 4/6 runs improved (nagoya_run2 +12 pp, tokyo_run2 +4 pp, tokyo_run3 +3 pp, nagoya_run3 +2 pp); 1/6 slightly below (nagoya_run1 −2 pp); 1/6 nearly unchanged (tokyo_run1 −0.5 pp).
• 1σ accuracy restored: Phase 4D's conditional lock increment caused false-fix persistence (holdamb cycles in urban canyons), inflating nagoya_run3 1σ to 1.373 m. Phase 4F reverts this, restoring 1σ = 0.135 m — matching the v0.3.3 baseline (0.128 m).
• Mechanism: once holdamb() fires with wrong integers, it constrains ambiguity P to VAR_HOLDAMB=0.001 cy² (process noise would need ~10 M epochs to overcome). The conditional lock froze lock counters on nfix=0 epochs, preventing satellite-set diversification and causing immediate re-selection of the same wrong integers. Reverting to unconditional lock++ allows the eligible satellite set to evolve between fix attempts, enabling eventual escape from the false-fix cycle.
• TTFF: nagoya_run1 regresses slightly (797 s vs 64 s with Phase 4D conditional lock), but the underlying accuracy is preserved. The Phase 4D accel-coupling gate (D1) is retained; it improves state convergence without the false-fix side-effect.

v0.4.2 PPP-RTK / PPP Benchmark Results (demo5 port to CLAS/MADOCA engines)¶

MRTKLIB v0.4.2 (feat/ppp-rtk-demo5-improvements) extends the demo5 algorithm improvements from the RTK engine to the CLAS (PPP-RTK) and MADOCA (PPP) engines. The RTK engine is unchanged.

Algorithm changes¶

Note on D2 and PPP: The adaptive outlier inflation was found to cause a severe regression in undifferenced PPP (tokyo_run1 MADOCA RMS: 1.8 m → 199 m) because PPP phase residuals are not double-differenced and can legitimately reach tens of metres at initialisation. D2 is applied only to PPP-RTK, whose sigma-normalised test already incorporates the large initial bias variance via Q.

CLAS comparison: v0.3.3 → v0.4.2¶

--skip-epochs 60, FIX tier (Q=4) only.

MADOCA comparison: v0.3.3 → v0.4.2¶

MADOCA results are unchanged across all 6 runs. The algorithms applied to the PPP engine (Phase A GLO taun, Phase C detslp_dop/code, Phase D1 EFACT) produced no measurable difference on this dataset, and Phase D2 was explicitly excluded from PPP to avoid undifferenced-observation regression.

Key findings (v0.4.2)¶

• CLAS tokyo accuracy: 2/6 runs show improved FIX RMS_2D (tokyo_run1: −14%, tokyo_run2: −13%). The D1 EFACT expansion provides slightly more accurate variance modelling for multi-constellation measurements.
• CLAS nagoya_run2 fix rate: −3.5 pp (26.9% → 23.4%). This run's FF rate simultaneously improved (+4 pp: 63.9% → 67.9%) and 1σ FIX accuracy improved significantly (0.717 m → 0.461 m), indicating that fewer but higher-quality fixes are produced. The likely mechanism: Phase C detslp_code resets biases on code-switching events, increasing FLOAT residency but reducing false fixes.
• CLAS 4/6 runs: Fix rate and FIX accuracy unchanged.
• MADOCA unchanged: The demo5 algorithmic improvements have no measurable impact on MADOCA PPP for this dataset. Future gains may require PPP-specific improvements (e.g., adaptive iono/trop Q tuning, wider MADOCA signal selection).

v0.6.10 Single-Point Positioning (SPP) Benchmark Results¶

MRTKLIB v0.6.10 (#116) adds opt-in, default-off accuracy improvements to the single-point (single) engine — historically a near-verbatim RTKLIB 2.4.3 snapshot WLS solver:

1. C/N0 (Sigma-ε) pseudorange weighting — down-weight low-C/N0 signals.
2. IGG-III robust re-weighting + a pre-robust acceptance gate that keeps the chi-square test effective under weighting (without it, weighting inflates the error tail).
3. TDCP velocity (time-differenced carrier phase, mm/s-class) with Doppler fallback, a SPP-local cycle-slip detector, and a jump-rejection QC that drops epochs whose code position change disagrees with the TDCP displacement.

The full design rationale, staged per-feature analysis, and the deferred work are in docs/design/spp-accuracy.md.

Configuration¶

The features are gated under [positioning] / [kalman_filter.measurement_error] and shipped enabled together in conf/benchmark/single.toml:

[positioning]
mode = "single"
correction = "none"
robust = "igg3"     # IGG-III robust + pre-robust gate
tdcp = true         # TDCP velocity + jump rejection
tdcp_jump = 5.0     # m

[slip_detection]
doppler = 1.0       # Doppler-vs-phase slip threshold (cyc/s)

[kalman_filter.measurement_error]
snr_max = 50.0      # dB-Hz
snr_error = 0.5     # m  (0 = C/N0 weighting off)

prcopt_default keeps all of these off, so any other configuration is bit-identical to the previous snapshot WLS.

Results: baseline (off) → v0.6.10 (enabled)¶

--mode single --skip-epochs 60. Rate% is the fraction of epochs with 2D horizontal error < 2 m.

Key findings (v0.6.10)¶

• All six runs improve on every aggregate metric: <2 m rate +20 pp, median −50 %, p95 −30 %, RMS −56 %.
• The jump-rejection QC collapses the extreme tail — e.g. tokyo_run3 RMS 36.02 → 4.84 m — by removing code position spikes that the precise TDCP displacement contradicts.
• C/N0 weighting and robust estimation alone improve the bulk (rate, median) but defeat the snapshot chi-square gate; the pre-robust gate is what turns them into a clean win rather than a tail-inflating one.
• The residual tail is consistent NLOS bias, which snapshot methods cannot detect. P5 (common-mode clock-jump correction) and P6 (position EKF) need a smartphone dataset where clock jumps and large jitter occur, and are deferred to the GSDC benchmark follow-up (#165).

Metrics Definitions¶

ENU errors are computed per-epoch using the corresponding ground-truth coordinate as the reference origin (moving-base projection).

Configuration¶

The benchmark uses layered configuration files. Each run is processed with mrtk post -k <mode>.toml -k <city>.toml, so the city conf overrides only the keys it specifies.

Mode confs (common settings per algorithm):

• conf/benchmark/clas.toml — CLAS PPP-RTK
• conf/benchmark/madoca.toml — MADOCA PPP
• conf/benchmark/rtk.toml — Baseline RTK
• conf/benchmark/single.toml — SPP (single-point), with the v0.6.10 accuracy features enabled

City confs (antenna types and reference-station coordinates):

• conf/benchmark/nagoya.toml — Nagoya overrides
• conf/benchmark/tokyo.toml — Tokyo overrides

Key differences from the standard test configurations:

Known Limitations¶

• Urban multipath: The Nagoya and Tokyo datasets include heavy shadowing and multipath. Fix rates and 3D accuracy will be lower than open-sky tests.
• No antenna calibration: rover RINEX antenna field is "Unknown", so no PCV correction is applied. This may affect Up accuracy by a few centimetres.
• No ISB correction: Septentrio mosaic-X5 is not in the ISB table. QZSS inter-system biases will be slightly mis-corrected.
• MADOCA float only: MADOCA-PPP mode does not produce integer AR fixes (Q=4) — Fix% and RMS_2D(fix) will be nan.
• IMU not used: Only GNSS-only positioning is evaluated.

Acknowledgements¶

The PPC-Dataset was collected for the Precise Positioning Challenge 2024 (高精度測位チャレンジ2024) organised by the Institute of Navigation Japan (測位航法学会), and is kindly made available as open data by:

Taro Suzuki, Chiba Institute of Technology PPC-Dataset — GNSS/IMU driving data for precise positioning research https://github.com/taroz/PPC-Dataset

We gratefully acknowledge the Institute of Navigation Japan for organising PPC2024 and Prof. Suzuki for making the dataset publicly available. Please cite the PPC2024 materials when publishing results derived from this dataset.

References¶

• PPC2024 overview (Japanese)
• PPC2024 results (Japanese)
• PPC-Dataset GitHub