# Datasets overview

> geolith Datasets is in development, not deployed: raster datasets such as rainfall turned into a map layer and numbers per area. What a raster dataset is, its two outputs, the first datasets, the live rainfall pages and the self-hosted engine command.

::warning
**Status: In development, not deployed.** The Datasets API is not a product yet. There is no dataset endpoint on api.geolith.app and no downloads. The hosted platform offers [geocoding](/search) and [vector tiles](/tiles) (India tile set today; planet tiles in development). See [Status](/datasets/status).
::

The rainfall pages on geolith.app are [the first public pages built with the engine](https://geolith.app/data/in/rainfall): one page per India district and state, with embeds and share cards.

A raster dataset is a grid of values over the Earth, such as rain per cell. geolith turns one raster and one set of area polygons into two outputs.

## Two outputs

| Output           | What it is                                                                                                  |
| ---------------- | ----------------------------------------------------------------------------------------------------------- |
| Map layer        | Tiles coloured by value, drawn over the basemap. The layer for India is cut to the Survey of India outline. |
| Numbers per area | Per area: the value, the normal for that time of year, the difference, and a short series of past periods.  |

## First datasets

| Dataset          | Source                          | Period        | When  |
| ---------------- | ------------------------------- | ------------- | ----- |
| Rainfall         | CHIRPS v3                       | Every 5 days  | First |
| Greenness (NDVI) | Copernicus CLMS NDVI 300 m (v3) | Every 10 days | Later |

When Datasets launches, file downloads (CSV, Parquet, GeoParquet) will need the Scale plan, the highest plan.

The areas are districts, from Overture Maps divisions. India comes first. The engine and the file formats work for any country. See [Data sources](/datasets/data-sources) for the licences and the exact attribution lines.

## Self-host

The engine command is `geolith dataset`, part of the [geolith builder](/builder). It runs on one machine and writes static files. It is on the `main` branch of geolith/geolith, not in a geolith release yet; build it from source. Its flags can still change.

It has three steps:

```bash
# District areas from Overture divisions
geolith dataset areas --overture <dir> --countries IN \
    --india-outline <geojson> --out <dir>
# Per-district baseline averages for the given MM-P slots
geolith dataset normals --config <json> --rasters <dir> --areas <geojson> \
    --slots 06-1,06-2 --out <normals.csv>
# Numbers and map layer of every period on disk
geolith dataset build --config <json> --rasters <dir> --areas <geojson> --out <dir>
```

- `areas`: `--countries` is a comma list. Optional: `--subtype` (default `county`), `--india-outline`, `--release`, `--max-zoom` (default 8) and `--allow-districtless <codes>`. It writes `areas.geojson` (India cut to the Survey of India outline), `gaps.geojson`, `areas-report.json` and `areas.pmtiles`. Every Indian district gets exactly one coded state or UT.
- `normals`: `--baseline` (default `1991-2020`). It writes the normals CSV and `normals-stats.json` beside it.
- `build`: optional `--normals <csv>`, `--india-outline <geojson>`, `--published <csv>` and `--crosswalk <csv>`. It writes one directory per build with `areas/<id>.json`, `index.json`, `numbers.csv`, `tiles.pmtiles` (the map layer) and `manifest.json`.

`--india-outline` (Survey of India GeoJSON) is required when `areas` lists `IN`, `PK`, `CN` or an `X*` country, and when the areas given to `build` include India.

### The district crosswalk

`--crosswalk <csv>` gives each area its official LGD (Local Government Directory) district code. The header is `overture_id,region_code,overture_name,lgd_state_code,lgd_district_code,lgd_name,match_kind,note`; `#` lines are comments.

- An area gets its code in `lgd_code` and `approximate: false` only from an `exact` or `alias` row.
- `split`, `merged`, `none` and areas without a row keep `lgd_code: null` and `approximate: true`.
- The build fails on an unknown `overture_id`, on a `region_code` or `overture_name` that differs from the areas, or on two `exact` or `alias` rows with one LGD code.
- When any area gets a code, the area JSON and `index.json` carry `codes_attribution`: "Source: Local Government Directory, Government of India".

The India crosswalk is `geolith/data/datasets/in/districts-crosswalk.csv` in the geolith/geolith repo.

### The operator script

`scripts/datasets/datasets.sh` (`pnpm datasets <command>`) runs the whole dataset: `download`, `build`, `verify`, `status` and `run`. `run` downloads, builds and verifies, and stops before any upload. `CROSSWALK_CSV` sets the crosswalk; it defaults to `geolith/data/datasets/<cc>/districts-crosswalk.csv` when present, empty means none, and a set path that is missing fails.
