Web Popularity
Daily website popularity rank of each public company's domain, from four independent top-sites lists (Majestic Million, Cisco Umbrella, Tranco and Cloudflare Radar). Lower rank means more popular; ranks are only comparable within one source.
web_popularity — the dataset name to pass to the Obscura API.
What one row means
One website-popularity rank for one company domain on one day, from one of four independent top-sites lists — Majestic Million (referring subnets), Cisco Umbrella Top 1M (DNS resolver traffic), Tranco (a research list combining five providers over a 30-day window) and Cloudflare Radar (resolver traffic). Each list is downloaded whole and filtered to the domains of our covered equity universe, so a row exists only where a ranked domain is a public company's website: `apple.com` at rank 43 becomes one row for AAPL. Rank is ORDINAL and lower means more popular; the four sources measure different things and are never comparable rank-for-rank, which is why `source` is part of the key rather than a footnote.
One row per (source, domain, date).
Point-in-time availability
Every Obscura dataset carries available_date: the calendar day the publisher made the row
available, day-of, with no session rounding. It is the one column a backtest filters on, and it means the
same thing on every dataset in the catalog.
For web_popularity: published: available_date is the day the source's own list became downloadable — the day the PUBLISHER made it available, never an adjustment for when a consumer could act on it — computed per source by a STORED generated CASE over `source` (m20260702_000172, corrected by m20260702_000184, reconciled by m20260702_000197). majestic publishes only a LATEST list (no dated archive), so the collector stamps the capture day and available_date = date. tranco states its publication instant in its list-metadata API (`created_on` = 22:00 UTC on the list's own end date, verified across three consecutive dates), i.e. the list for day D exists on D, so available_date = date; that 22:00 UTC is 18:00 ET, after the US close, is deliberately NOT a reason to add a day (an interim build did, under a convention since withdrawn, and m20260702_000197 takes it back off). umbrella keeps a DATED archive, and its file for day D is written by a stable daily job at ~18:00 Pacific / ~21:00 Eastern on D+1, so the file does not EXIST on day D and available_date = date + 1. WHY +1 and not +2: the archive was first measured from a single Last-Modified (`top-1m-2026-07-24.csv.zip` at 2026-07-26 01:20 GMT) read as a UTC calendar day; re-probed across five consecutive list days every write lands on D+1 on the publisher's own clock, and only crosses midnight into D+2 in UTC — an evening job on a US clock filed under tomorrow's UTC date. Availability is the publisher's day (corrected by m20260702_000206). This +1 is a REAL publication delay (the file genuinely does not exist yet), not an adjustment for when a consumer could act. Note this differs from the FETCH lag of 2, which is when a UTC-scheduled collector can first GET the file — a different question about a different actor. cloudflare radar aggregates a full closed UTC day, so the ranking for D cannot exist until D+1 and available_date = date + 1 — conservative and UNAUDITED, because no API token is configured and the table holds no cloudflare rows to measure against.
Refresh cadence
Obscura refreshes web_popularity daily — the most frequent scheduled job that re-collects or re-exports it. This is Obscura's own pipeline cadence, not the upstream publisher's release schedule; when a row became public is recorded per row in available_date.
Schema — 6 columns
The full public column list for web_popularity, with the meaning of every field. The same
schema is served unauthenticated at https://api.obscura.trade/v1/catalog/web_popularity.
| Column | Type | Description |
|---|---|---|
| source | text · not null | Which top-sites list this rank came from: 'majestic' (Majestic Million, ranked by count of referring /24 subnets in Majestic's web crawl), 'umbrella' (Cisco Umbrella Top 1M, ranked by DNS query volume at Cisco's public resolvers), 'tranco' (the Tranco research list, a Dowdall-rule combination of crux/farsight/majestic/radar/umbrella over a rolling 30-day window) or 'cloudflare' (Cloudflare Radar domain rankings from 1.1.1.1 resolver traffic). Part of the composite primary key: the same domain on the same day appears once per source, and ranks are NEVER comparable across sources. |
| domain | text · not null | Registered company website host, lowercased with scheme, 'www.', port and path stripped (`https://www.Apple.com/investor` -> `apple.com`) by `web_traffic::normalize_domain`. Sourced from the covered equity universe's homepage field and matched against the raw list entry after the same normalization, so `www.x` and `x` collapse to one row. Part of the composite primary key. |
| date | date · not null | The list's OWN date: for tranco and cloudflare the dated list requested from the source's per-date API; for umbrella the day its dated archive file is named for; for majestic the UTC capture day, because Majestic publishes only a latest snapshot with no dated archive. This is the period the ranking describes, NOT a guaranteed availability date — use `available_date` for any point-in-time filter or join. Part of the composite primary key. |
| ticker | text | Stock ticker of the company owning this domain, resolved from the covered equity universe's (ticker, homepage) mapping at ingest; the first ticker wins when several companies share a homepage. Nullable for rows whose domain no longer resolves to a covered ticker, and refreshed on every re-ingest of the same (source, domain, date) so a ticker change overwrites the prior value. |
| rank | bigint · not null | The source's own ordinal position for this domain in its list on this date — 1 is the most popular site the source ranks, and the lists this collector reads are all top-1,000,000, so values run 1..=1000000. LOWER IS MORE POPULAR. Not a traffic count and not comparable across sources: Majestic ranks by referring subnets, Umbrella and Cloudflare by DNS query volume, Tranco by a combination of five providers. A domain missing from a list on a date has no row rather than a large rank. |
| available_date | date | PUBLIC-availability date — the day the source published this rank, and the ONLY column to filter or join on for point-in-time work. DB-generated (STORED) and read-only: `date` for majestic (a latest-only list stamped with the capture day) and for tranco (whose API reports the list is created at 22:00 UTC on its own end date, so the list for D exists on D — the 18:00 ET hour does not move the day), `date + 1` for umbrella (its dated archive file for day D is written by a daily job at ~18:00 Pacific on D+1, so it does not exist on D; corrected from +2, which was that same write read as a UTC calendar day), `date + 1` for cloudflare (the ranking for D aggregates a full closed UTC day, so it cannot exist until D+1; conservative, pending audit once a token exists). |
Access web_popularity
Two delivery paths, one identifier. Both require an Obscura account and an active subscription; the catalog entry and the schema above are public.
import obscura
client = obscura.Client("obs_live_…")
df = client.query(
dataset="web_popularity",
symbols=["NVDA", "AAPL"],
start="2024-01-01",
)Create a free account Browse all 95 datasets
Frequently asked questions
What is in the web_popularity dataset?
Daily website popularity rank of each public company's domain, from four independent top-sites lists (Majestic Million, Cisco Umbrella, Tranco and Cloudflare Radar). Lower rank means more popular; ranks are only comparable within one source. One website-popularity rank for one company domain on one day, from one of four independent top-sites lists — Majestic Million (referring subnets), Cisco Umbrella Top 1M (DNS resolver traffic), Tranco (a research list combining five providers over a 30-day window) and Cloudflare Radar (resolver traffic). Each list is downloaded whole and filtered to the domains of our covered equity universe, so a row exists only where a ranked domain is a public company's website: `apple.com` at rank 43 becomes one row for AAPL. Rank is ORDINAL and lower means more popular; the four sources measure different things and are never comparable rank-for-rank, which is why `source` is part of the key rather than a footnote.
How do I avoid look-ahead bias with web_popularity?
Filter on web_popularity.available_date, the day the publisher made the row public. For this dataset that date is derived as follows — published: available_date is the day the source's own list became downloadable — the day the PUBLISHER made it available, never an adjustment for when a consumer could act on it — computed per source by a STORED generated CASE over `source` (m20260702_000172, corrected by m20260702_000184, reconciled by m20260702_000197). majestic publishes only a LATEST list (no dated archive), so the collector stamps the capture day and available_date = date. tranco states its publication instant in its list-metadata API (`created_on` = 22:00 UTC on the list's own end date, verified across three consecutive dates), i.e. the list for day D exists on D, so available_date = date; that 22:00 UTC is 18:00 ET, after the US close, is deliberately NOT a reason to add a day (an interim build did, under a convention since withdrawn, and m20260702_000197 takes it back off). umbrella keeps a DATED archive, and its file for day D is written by a stable daily job at ~18:00 Pacific / ~21:00 Eastern on D+1, so the file does not EXIST on day D and available_date = date + 1. WHY +1 and not +2: the archive was first measured from a single Last-Modified (`top-1m-2026-07-24.csv.zip` at 2026-07-26 01:20 GMT) read as a UTC calendar day; re-probed across five consecutive list days every write lands on D+1 on the publisher's own clock, and only crosses midnight into D+2 in UTC — an evening job on a US clock filed under tomorrow's UTC date. Availability is the publisher's day (corrected by m20260702_000206). This +1 is a REAL publication delay (the file genuinely does not exist yet), not an adjustment for when a consumer could act. Note this differs from the FETCH lag of 2, which is when a UTC-scheduled collector can first GET the file — a different question about a different actor. cloudflare radar aggregates a full closed UTC day, so the ranking for D cannot exist until D+1 and available_date = date + 1 — conservative and UNAUDITED, because no API token is configured and the table holds no cloudflare rows to measure against. A query of the form WHERE available_date <= '<as-of date>' never sees a row before it existed.
In what formats can I get web_popularity?
As a Parquet bulk export (POST https://api.obscura.trade/v1/download) or as JSON from the typed query API (POST https://api.obscura.trade/v1/query), both with dataset="web_popularity". The column schema is public at https://api.obscura.trade/v1/catalog/web_popularity.
Can I filter web_popularity by company or symbol?
Yes. web_popularity carries ticker, the column the API's symbols filter resolves against.
How often is web_popularity updated?
Obscura refreshes web_popularity on a daily schedule — that is the most frequent scheduled job that re-collects or re-exports the table. It is Obscura's own pipeline cadence, not the upstream publisher's release schedule; when the publisher makes a row available is described by the availability rule above, and is recorded per row in available_date.