Hacker News Stories
Front-page stories from Hacker News.
hn_stories — the dataset name to pass to the Obscura API.
What one row means
One daily snapshot of a single Hacker News story's front-page rank/score/ comment-count as observed on one calendar day of collection: the collector fetches topstories.json plus each item document once per run and upserts on (story_id, captured_date), so a story that stays on the front page for several days produces one row per day, each carrying that day's refreshed score/comments/rank.
One row per (story_id, captured_date).
Point-in-time availability
Every Obscura dataset carries available_date: the calendar day the publisher made the row
available, day-of, with no session rounding. It is the one column a backtest filters on, and it means the
same thing on every dataset in the catalog.
For hn_stories: published: available_date = captured_date, the UTC calendar day the collector's snapshot run captured this row (STORED generated column, generated from captured_date itself since migration 000175 — previously synced_at::date, which is the same day on a live run but the REPLAY day on a bronze replay).
Refresh cadence
Obscura refreshes hn_stories daily — the most frequent scheduled job that re-collects or re-exports it. This is Obscura's own pipeline cadence, not the upstream publisher's release schedule; when a row became public is recorded per row in available_date.
Schema — 12 columns
The full public column list for hn_stories, with the meaning of every field. The same
schema is served unauthenticated at https://api.obscura.trade/v1/catalog/hn_stories.
| Column | Type | Description |
|---|---|---|
| story_id | bigint · not null | Hacker News Firebase item id for the story (e.g. 48905028); stable across all daily snapshots of the same story. First half of the composite primary key. Sourced from the Firebase topstories.json id list and confirmed via item/{id}.json's 'id' field (parse_item). |
| captured_date | date · not null | UTC calendar day (Utc::now().date_naive() at task start) on which this snapshot run captured the story's rank/score/comments. Second half of the composite primary key/OnConflict target; this is the column the ORM upserts on, and is the true public-availability day for this row's data — the day this specific score/comment/rank snapshot became knowable. |
| rank | integer · not null | 1-based position of the story within the top_n slice of topstories.json at capture time (i=index+1 in hackernews.rs::run); updated on every re-capture (part of the upsert's update_columns), so it reflects rank as of captured_date, not the story's rank history. |
| title | text | Story headline as submitted to HN, from the item document's 'title' field; refreshed on every re-capture (the upsert replaces every column the batch writes), so it reflects the title as of the LAST capture on captured_date — an HN moderator retitle lands here. |
| score | integer | HN points/upvote score at capture time, from the item document's 'score' field; refreshed on every re-capture (in update_columns) so each day's row shows that day's current score, the primary attention/virality signal for this dataset. |
| comments | integer | Total descendant comment count at capture time, from the item document's 'descendants' field (parse_item maps descendants -> comments); nullable because Ask HN / some item types omit descendants. Refreshed on every re-capture. |
| author | text | HN username of the story submitter, from the item document's 'by' field. NOT refreshed on upsert — captured only at first snapshot of the story. |
| url | text | External URL the story links to, from the item document's 'url' field; nullable/absent for text-only submissions such as Ask HN posts (parse_item leaves it None when the field is missing). NOT refreshed on upsert. |
| event_time | timestamp with time zone | The story's original HN submission instant — the real-world event — parsed from the item document's unix 'time' field (DateTime::from_timestamp). Constant across every daily snapshot row for the same story_id (submission never changes); this is the period/event moment, distinct from available_date which tracks when each day's snapshot became knowable. |
| item_type | text | Hacker News item kind from the item document's 'type' field — `story` / `job` / `poll` / `ask` (etc.). topstories.json mixes real stories with job and poll posts, so this is the field that lets a consumer separate a job/poll row from a genuine story. Captured at first snapshot (item type never changes). |
| text | text | HTML body of a text-only submission, from the item document's 'text' field — the entire post content for Ask HN / Show HN text posts / job posts, where 'url' is null. Null for link submissions. Not derivable from any other kept field; captured at first snapshot. |
| available_date | date | PUBLIC-availability date = captured_date. STORED generated column (migration 000175), read-only; the point-in-time column to filter/join on — the UTC day this specific snapshot row's rank/score/comments became knowable, NOT the story's original submission day. |
Access hn_stories
Two delivery paths, one identifier. Both require an Obscura account and an active subscription; the catalog entry and the schema above are public.
import obscura
client = obscura.Client("obs_live_…")
df = client.query(
dataset="hn_stories",
start="2024-01-01",
)Create a free account Browse all 95 datasets
Frequently asked questions
What is in the hn_stories dataset?
Front-page stories from Hacker News. One daily snapshot of a single Hacker News story's front-page rank/score/ comment-count as observed on one calendar day of collection: the collector fetches topstories.json plus each item document once per run and upserts on (story_id, captured_date), so a story that stays on the front page for several days produces one row per day, each carrying that day's refreshed score/comments/rank.
How do I avoid look-ahead bias with hn_stories?
Filter on hn_stories.available_date, the day the publisher made the row public. For this dataset that date is derived as follows — published: available_date = captured_date, the UTC calendar day the collector's snapshot run captured this row (STORED generated column, generated from captured_date itself since migration 000175 — previously synced_at::date, which is the same day on a live run but the REPLAY day on a bronze replay). A query of the form WHERE available_date <= '<as-of date>' never sees a row before it existed.
In what formats can I get hn_stories?
As a Parquet bulk export (POST https://api.obscura.trade/v1/download) or as JSON from the typed query API (POST https://api.obscura.trade/v1/query), both with dataset="hn_stories". The column schema is public at https://api.obscura.trade/v1/catalog/hn_stories.
How often is hn_stories updated?
Obscura refreshes hn_stories on a daily schedule — that is the most frequent scheduled job that re-collects or re-exports the table. It is Obscura's own pipeline cadence, not the upstream publisher's release schedule; when the publisher makes a row available is described by the availability rule above, and is recorded per row in available_date.