Calibrated from GEE execution on 2026-03-29 (actual: 58.1%). Will be validated against QGIS and folia outputs. Percentage of AOI classified as urban after morphological cleanup (open then close). ESA WorldCover class 50 = built-up. Morphological operations clean small isolated pixels and fill gaps.
| Workflow | Model | Backend | Status | Answer | Error | Cost | Latency |
|---|---|---|---|---|---|---|---|
| exec | gold | gee | PASS | 58.09722400806599 | 0.2% | $0.000017 | 307ms |
| exec | gold | folia-rust | PASS | 44.02705729007721 | 24.1% | --- | 405ms |
| exec | gold | qgis | PASS | 64.19145113315517 | 10.7% | --- | 800ms |
Known-correct folia spec for this problem. This is the reference implementation used for backend quality testing.
# Platform Comparison: Morphological Urban Extraction — Gold Spec
#
# Extract urban footprint from ESA WorldCover 2021 over Phoenix metro
# using morphological filtering. Ground truth: ~58% urban.
#
# This spec is designed to run through both the Python backend and
# browser-wasm executor (folia bench exec -b browser-wasm).
name: morphological-urban
version: "1.0"
description: >
Extract the urban footprint from ESA WorldCover 2021 (10m) over the
Phoenix metro area. Threshold built-up pixels (class 50), apply
morphological opening (radius=2) then closing (radius=3), and report
the percentage of the AOI classified as urban.
Ground truth: ~58% (calibrated from GEE 2026-03-29).
settings:
default_bbox: [-112.2, 33.3, -111.8, 33.6]
default_crs: EPSG:4326
layers:
# ============================================================
# SOURCE LAYERS
# ============================================================
source/worldcover:
uri: stac://earth-search/esa-worldcover
type: raster
description: >
ESA WorldCover 2021 (10m). Class 50 = built-up.
params:
bbox: [-112.2, 33.3, -111.8, 33.6]
# ============================================================
# COMPUTE: URBAN EXTRACTION PIPELINE
# ============================================================
compute/urban-threshold:
type: raster
description: >
Binary mask of built-up pixels (WorldCover class 50 = 1, else 0).
compute:
op: raster_threshold_mask
inputs:
data: { layer: source/worldcover }
params:
threshold: 50
operator: eq
compute/opened:
type: raster
description: >
Morphological opening (erode then dilate) with radius=2.
Removes isolated built-up pixels smaller than the kernel.
compute:
op: morph_open
inputs:
data: { layer: compute/urban-threshold }
params:
radius: 2
compute/urban-footprint:
type: raster
description: >
Morphological closing (dilate then erode) with radius=3.
Fills small gaps in the urban fabric.
compute:
op: morph_close
inputs:
data: { layer: compute/opened }
params:
radius: 3
# ============================================================
# RESULT: URBAN PERCENTAGE
# ============================================================
result/urban-pct:
type: table
description: >
Percentage of the AOI classified as urban after morphological cleanup.
Ground truth: ~58%.
compute:
op: analysis_zonal_stats
params:
stats: [mean]
inputs:
raster: { layer: compute/urban-footprint }
The prompt given to LLMs in single-shot workflow benchmarks.
Problem: Extract the urban footprint from ESA WorldCover over the
Phoenix metro area using morphological filtering.
Method:
- Load ESA WorldCover 2021 (10m resolution)
- Threshold to extract built-up class (class 50 = 1, else 0)
- Apply morphological opening (radius=2) to remove noise
- Apply morphological closing (radius=3) to fill gaps
- Report percentage of AOI classified as urban
Study area: -112.2, 33.3, -111.8, 33.6 (Phoenix metro, AZ).
Expected answer: approximately 58% urban area.