Datasets docs

Datasets overview

geolith Datasets is in development, not deployed: raster datasets such as rainfall turned into a map layer and numbers per area. What a raster dataset is, its two outputs, the first datasets, the live rainfall pages and the self-hosted engine command.

By · Updated

Warning

Status: In development, not deployed. The Datasets API is not a product yet. There is no dataset endpoint on api.geolith.app and no downloads. The hosted platform offers geocoding and vector tiles (India tile set today; planet tiles in development). See Status.

The rainfall pages on geolith.app are the first public pages built with the engine: one page per India district and state, with embeds and share cards.

A raster dataset is a grid of values over the Earth, such as rain per cell. geolith turns one raster and one set of area polygons into two outputs.

Two outputs

OutputWhat it is
Map layerTiles coloured by value, drawn over the basemap. The layer for India is cut to the Survey of India outline.
Numbers per areaPer area: the value, the normal for that time of year, the difference, and a short series of past periods.

First datasets

DatasetSourcePeriodWhen
RainfallCHIRPS v3Every 5 daysFirst
Greenness (NDVI)Copernicus CLMS NDVI 300 m (v3)Every 10 daysLater

When Datasets launches, file downloads (CSV, Parquet, GeoParquet) will need the Scale plan, the highest plan.

The areas are districts, from Overture Maps divisions. India comes first. The engine and the file formats work for any country. See Data sources for the licences and the exact attribution lines.

Self-host

The engine command is geolith dataset, part of the geolith builder. It runs on one machine and writes static files. It is on the main branch of geolith/geolith, not in a geolith release yet; build it from source. Its flags can still change.

It has three steps:

# District areas from Overture divisions
geolith dataset areas --overture <dir> --countries IN \
    --india-outline <geojson> --out <dir>
# Per-district baseline averages for the given MM-P slots
geolith dataset normals --config <json> --rasters <dir> --areas <geojson> \
    --slots 06-1,06-2 --out <normals.csv>
# Numbers and map layer of every period on disk
geolith dataset build --config <json> --rasters <dir> --areas <geojson> --out <dir>
  • areas: --countries is a comma list. Optional: --subtype (default county), --india-outline, --release, --max-zoom (default 8) and --allow-districtless <codes>. It writes areas.geojson (India cut to the Survey of India outline), gaps.geojson, areas-report.json and areas.pmtiles. Every Indian district gets exactly one coded state or UT.
  • normals: --baseline (default 1991-2020). It writes the normals CSV and normals-stats.json beside it.
  • build: optional --normals <csv>, --india-outline <geojson>, --published <csv> and --crosswalk <csv>. It writes one directory per build with areas/<id>.json, index.json, numbers.csv, tiles.pmtiles (the map layer) and manifest.json.

--india-outline (Survey of India GeoJSON) is required when areas lists IN, PK, CN or an X* country, and when the areas given to build include India.

The district crosswalk

--crosswalk <csv> gives each area its official LGD (Local Government Directory) district code. The header is overture_id,region_code,overture_name,lgd_state_code,lgd_district_code,lgd_name,match_kind,note; # lines are comments.

  • An area gets its code in lgd_code and approximate: false only from an exact or alias row.
  • split, merged, none and areas without a row keep lgd_code: null and approximate: true.
  • The build fails on an unknown overture_id, on a region_code or overture_name that differs from the areas, or on two exact or alias rows with one LGD code.
  • When any area gets a code, the area JSON and index.json carry codes_attribution: "Source: Local Government Directory, Government of India".

The India crosswalk is geolith/data/datasets/in/districts-crosswalk.csv in the geolith/geolith repo.

The operator script

scripts/datasets/datasets.sh (pnpm datasets <command>) runs the whole dataset: download, build, verify, status and run. run downloads, builds and verifies, and stops before any upload. CROSSWALK_CSV sets the crosswalk; it defaults to geolith/data/datasets/<cc>/districts-crosswalk.csv when present, empty means none, and a set path that is missing fails.