Open Data Catalog
Dataset listings harvested from government open-data portals running CKAN.
open_data_catalog — the dataset name to pass to the Obscura API.
What one row means
One CKAN package-listing on a government open-data portal (Data.NSW or Queensland Open Data) as returned by that portal's `package_search` API — a discovery-layer index row (title, publishing org, resource count, last-modified) describing a dataset LISTING, not the underlying data itself.
One row per (portal, dataset_id).
Point-in-time availability
Every Obscura dataset carries available_date: the calendar day the publisher made the row
available, day-of, with no session rounding. It is the one column a backtest filters on, and it means the
same thing on every dataset in the catalog.
For open_data_catalog: published: available_date = the UTC date-part of event_time (CKAN's metadata_modified for the listing).
Refresh cadence
Obscura refreshes open_data_catalog weekly — the most frequent scheduled job that re-collects or re-exports it. This is Obscura's own pipeline cadence, not the upstream publisher's release schedule; when a row became public is recorded per row in available_date.
Schema — 12 columns
The full public column list for open_data_catalog, with the meaning of every field. The same
schema is served unauthenticated at https://api.obscura.trade/v1/catalog/open_data_catalog.
| Column | Type | Description |
|---|---|---|
| portal | text · not null | Portal discriminator: `nsw` (data.nsw.gov.au) or `qld` (data.qld.gov.au); first half of the composite primary key. |
| dataset_id | text · not null | CKAN package 'name' slug, the portal's own permanent identifier for the listing; second half of the composite primary key. |
| title | text · not null | Human-readable dataset title; falls back to the dataset_id slug when CKAN omits a title. |
| organization | text | Publishing agency/department display name; NULL when CKAN returns no organization object. |
| num_resources | integer · not null | Count of downloadable resources (files/API endpoints/formats) attached to the CKAN package. |
| num_tags | integer · not null | Count of searchable topic tags on the package (raw `num_tags`); the array of tag names itself is deferred to a child table. |
| event_time | timestamp with time zone | CKAN's metadata_modified timestamp for the package — the moment the portal itself last edited/republished this listing. |
| metadata_created | timestamp with time zone | CKAN's metadata_created timestamp — the moment this listing was first published on the portal (dataset age / first-appearance; distinct from metadata_modified/event_time). Raw `metadata_created`. |
| notes | text | Dataset description/abstract — the main free-text summary (rich HTML). Raw `notes`. |
| license_id | text | License identifier for reuse/openness filtering, e.g. `cc-by`, `notspecified`. Raw `license_id`. |
| license_title | text | Human-readable license title, e.g. "License Not Specified". Raw `license_title`. |
| available_date | date | PUBLIC-availability date: STORED generated column = UTC date-part of event_time — the day the catalog listing last changed on the portal. DB-generated, read-only. |
Access open_data_catalog
Two delivery paths, one identifier. Both require an Obscura account and an active subscription; the catalog entry and the schema above are public.
import obscura
client = obscura.Client("obs_live_…")
df = client.query(
dataset="open_data_catalog",
start="2024-01-01",
)Create a free account Browse all 95 datasets
Frequently asked questions
What is in the open_data_catalog dataset?
Dataset listings harvested from government open-data portals running CKAN. One CKAN package-listing on a government open-data portal (Data.NSW or Queensland Open Data) as returned by that portal's `package_search` API — a discovery-layer index row (title, publishing org, resource count, last-modified) describing a dataset LISTING, not the underlying data itself.
How do I avoid look-ahead bias with open_data_catalog?
Filter on open_data_catalog.available_date, the day the publisher made the row public. For this dataset that date is derived as follows — published: available_date = the UTC date-part of event_time (CKAN's metadata_modified for the listing). A query of the form WHERE available_date <= '<as-of date>' never sees a row before it existed.
In what formats can I get open_data_catalog?
As a Parquet bulk export (POST https://api.obscura.trade/v1/download) or as JSON from the typed query API (POST https://api.obscura.trade/v1/query), both with dataset="open_data_catalog". The column schema is public at https://api.obscura.trade/v1/catalog/open_data_catalog.
How often is open_data_catalog updated?
Obscura refreshes open_data_catalog on a weekly schedule — that is the most frequent scheduled job that re-collects or re-exports the table. It is Obscura's own pipeline cadence, not the upstream publisher's release schedule; when the publisher makes a row available is described by the availability rule above, and is recorded per row in available_date.