Job postings are collected from multiple sources via automated ingestion adapters (RSS, JSON API, XML feeds, and direct integrations). Each source runs on a configurable schedule (typically every 6 hours). Jobs are deduplicated using a SHA-256 hash of the normalized title, company name, and location.
Raw skill strings from job postings are normalized against a canonical skills registry. The normalizer matches on exact name, slug, or alias. Unrecognized skills are auto-created as ungrouped entries for administrative triage. A join table (job_skills) provides fast, indexed skill-dimension queries.
At the start of each compute run, dimensions are discovered dynamically from the data. Only dimensions meeting the minimum sample threshold (20 jobs in the 90-day window) are considered. Dimensions include: skills, locations, companies, job types, experience levels, and remote/onsite status.
Salary data undergoes several transformations:
Confidence reflects data reliability. It combines sample size with source quality weighting:
A stat page is published when its dimension reaches 30+ jobs. Below this threshold, the page exists but is marked as "below_threshold" and not linked publicly. Pages can be manually unpublished by administrators with a reason noted.
Quarterly snapshots are created using UTC boundaries and are immutable once written. Q1 = Jan 1 00:00:00 UTC to Mar 31 23:59:59 UTC. This ensures reproducibility and prevents timezone-related off-by-one errors in multi-region deployments.