Alternative Data Pipeline
Sep 2025 – Apr 2026
an alternative-data pipeline for equity research: it tracks hiring as a signal by scraping job postings from the applicant-tracking systems that big companies actually run on. everything is infrastructure-as-code under AWS SAM, and the whole thing runs for less than a streaming subscription. the interesting engineering is not the scraping, it is surviving Lambda's 15-minute wall without dropping work.
175,000+ postings a day from 6 ATS platforms covering 61% of the S&P 500, into PostgreSQL and an S3/Athena lake for under $15/month
Lambda timeouts from ~37 per cycle to zero: time-budgeted scrapes, 16-way concurrent fetching, and SQS self-continuation where workers commit progress and re-enqueue past the 15-minute ceiling
a shared ats_runner with per-platform adapters took coverage from 2 platforms to 6, each adapter 50-130 lines
what i built
- per-ATS adapters (Ashby, Lever, Phenom, SmartRecruiters, SuccessFactors, Oracle Cloud) behind one shared runner
- an SQS self-continuation loop so a worker that runs out of clock commits what it has and re-enqueues itself
- EventBridge scheduling, CloudWatch alarms on source degradation, and Alembic-managed Postgres schema
- an S3/Athena lake for the historical snapshots the Postgres tables do not keep
how it works
- 1a dispatcher fans companies out to per-ATS scrape jobs on SQS
- 2each worker runs against a time budget, fetching 16 pages concurrently
- 3when the budget runs out it commits progress and re-enqueues its own cursor, so nothing is lost at the 15-minute ceiling
- 4rows land in Postgres for querying and in S3 for Athena over the history
results
- ✓175,000+ postings synced daily across 61% of the S&P 500
- ✓Lambda timeouts reduced from ~37 per cycle to zero
- ✓under $15/month all-in
what's next
- more ATS coverage
- backfill the lake further