PLATFORM-COMPARISON Benchmark Results

Comparing geospatial analysis workflows across models, backends, and cost dimensions.

11
Problems
7
Gold Specs
31
Total Runs
2
Pareto-Optimal

Workflow Comparison

Workflow Model Backend Correct Avg Cost Avg Latency
exec gold folia-rust 10/10 --- 104ms
exec gold gee 11/11 $0.000745 13.4s
exec gold qgis 9/10 --- 395ms

Pareto Frontier: Cost vs Effectiveness

Points on the frontier (highlighted) represent configurations where no other option achieves more correct answers at equal or lower cost.

Pass Rate by Category

CategoryPassedTotalRate
unknown 30 31 97%

By Model

Model Attempted Valid Specs Correct Avg Token Cost Avg Gen Latency
gold 31 0 30 --- ---

By Backend

Backend Runnable Correct Avg Exec Cost Avg Latency
folia-rust 10 10 --- 104ms
gee 11 11 $0.000745 13.4s
qgis 10 9 --- 395ms

Problems

IDTitleDifficultyCategory
fire-burn-severity Fire Burn Severity via dNBR (2020 Creek Fire, CA) intermediate change-detection
hansen-forest-loss Hansen Forest Loss 2020 (Rondonia, Brazil) easy change-detection
harmonic-phenology Harmonic NDVI Phenology (Kansas Cropland, 2020-2023) intermediate phenology
landsat-composite Landsat 8 Annual Median Composite (Yellowstone, 2022) intermediate temporal-analysis
landtrendr-change LandTrendr Forest Disturbance Detection (PNW, 1985-2023) difficult change-detection
morphological-urban Morphological Urban Footprint Extraction (Phoenix) intermediate morphological
sentinel2-ndvi Sentinel-2 NDVI (Iowa Farmland, Summer 2023) easy index-calculation
supervised-classification Supervised RF Classification (Sacramento Valley, CA) difficult classification
terrain-derivatives Terrain Derivatives from SRTM (Slope/Aspect/Hillshade) easy terrain-analysis
weighted-overlay Solar Siting Weighted Overlay (Utah) intermediate siting-analysis
zonal-stats Mean Elevation by County (Colorado) easy zonal-analysis