THE EXPORT
This site is a static build over a set of JSON files generated straight from the harvester’s database. They are served as-is. Schema version 4, generated by datafromgov 0.1.0.
Terms
This export is a derived work: it records what each government portal's own catalogue said when this harvester read it. Dataset titles and identifiers remain the material of the portals they were read from and stay subject to those portals' terms, which differ by country and are linked per portal in portals.json. Counts, states and timestamps are this harvester's own measurements.
Each portal’s entry in portals.json carries its homepage and, where its terms have been read and dated, the credit it requires, quoted as that portal words it. A null there means the terms have not been read yet, so check the portal before reusing anything from it. Ireland is the case to copy: it requires the notice “Contains Irish Public Sector Data licensed under a Creative Commons Attribution 4.0 International (CC BY 4.0) licence”.
The counts, states and timestamps are this site’s own measurements, offered with no warranty: they describe what the harvester read at the stated moment and nothing more. This is not an official source for any government, and it is not affiliated with any of them.
Files
Per-portal detail (31 with data)
Shape
Catalogues are sharded at 250 datasets per page, at datasets/<portal_id>/page-0000.json. Each portal detail file states how many pages it has.
| Field | Type | Meaning |
|---|---|---|
| portal_id | string | Stable identifier used across every file. |
| state | enum | not_started, discovering, discovered, harvesting, complete. |
| datasets | number | Datasets we enumerated ourselves. |
| reported_total | number | null | The portal's own claim about its catalogue size. Null when unknown. |
| resources | number | Downloadable resources across those datasets. |
| downloaded | number | Resources mirrored to disk and type-validated. |
| retrying | number | Transient failures awaiting another attempt. Outstanding work, not failures. |
| failed | number | Terminal failures: hard errors, HTML in place of data, empty responses. |
| skipped | number | Excluded by policy: over the per-file cap, over budget, or with no fetchable URL. |
| bytes | number | Bytes actually written to disk. |
Individual resource rows are not exported: no per-file URLs, no filesystem paths, and no error text.