Alternative Data Pipeline
Sep 2025 – Apr 2026
🔒 private repo — NU Systematic Alpha
- Python
- AWS Lambda
- SQS
- EventBridge
175,000+job postings synced a day
self-reported, private repo61%of the S&P 500 covered by six applicant-tracking platforms
self-reported, private repo~37 to 0Lambda timeouts per cycle, before and after time budgets and SQS self-continuation
self-reported, private repo
an alternative-data pipeline for equity research: it tracks hiring as a signal by scraping job postings from the applicant-tracking systems that big companies actually run on. everything is infrastructure-as-code under AWS SAM, and the whole thing runs for less than a streaming subscription. the interesting engineering is not the scraping, it is surviving Lambda's 15-minute wall without dropping work.
what i built
- per-ATS adapters (Ashby, Lever, Phenom, SmartRecruiters, SuccessFactors, Oracle Cloud) behind one shared runner
- an SQS self-continuation loop so a worker that runs out of clock commits what it has and re-enqueues itself
- EventBridge scheduling, CloudWatch alarms on source degradation, and Alembic-managed Postgres schema
- an S3/Athena lake for the historical snapshots the Postgres tables do not keep
how it works
- 1a dispatcher fans companies out to per-ATS scrape jobs on SQS
- 2each worker runs against a time budget, fetching 16 pages concurrently
- 3when the budget runs out it commits progress and re-enqueues its own cursor, so nothing is lost at the 15-minute ceiling
- 4rows land in Postgres for querying and in S3 for Athena over the history
what's next
- more ATS coverage
- backfill the lake further