Skip to content

Exa

Software Engineer (Web Crawling)

Posted Updated

Description

  • As a Web Crawler engineer, you’d be responsible for crawling the entire web. Basically build Google-scale crawling!
  • Build a distributed crawler that can handle 100M+ pages per day
  • Optimize crawl politeness and rate limiting across thousands of domains
  • Design systems to detect and handle dynamic content, JavaScript rendering, and anti-bot measures
  • Create intelligent crawl scheduling and prioritization algorithms for maximum coverage efficiency
  • You have extensive experience building and scaling web crawlers, or would be excited to ramp up very quickly
  • You care about the problem of finding high quality knowledge and recognize how important this is for the world
  • You have experience with some high performance language (C++, Rust, etc.)
  • You are familiar with TypeScript, Playwright, modern web design, CDP (Chrome DevTools Protocol)
  • You’re comfortable optimizing a system to an exceptional degree

Skills

Apply

Apply to Software Engineer (Web Crawling) at Exa

Create a free account first, or go straight to the employer.

Free RoleWatch account

  • Get an email when new jobs match your search
  • Save jobs and track every application
  • See how well your CV matches each job

Already have one?

No account needed. Opens the employer's own site in a new tab.

See also

Software Engineering jobs by country — openings, pay and top skills →