Exa is building a search engine from scratch to serve every AI application. We build massive-scale infrastructure to crawl the web, train state-of-the-art embedding models to index it, and develop super high performant vector databases in Rust to search over it. We also own a $5m H200 GPU cluster and routinely run batch jobs with tens of thousands of machines. This isn't your average startup :)

As a Web Crawler engineer, you'd be responsible for crawling the entire web. Basically build Google-scale crawling!

Desired Experience

• You have extensive experience building and scaling web crawlers, or would be excited to ramp up very quickly

• You have experience with some high performance language (C++, Rust, etc.)

• You’re comfortable optimizing a system to an exceptional degree

• You care about the problem of finding high quality knowledge and recognize how important this is for the world

Example Projects

• Build a distributed crawler that can handle 100M+ pages per day

• Optimize crawl politeness and rate limiting across thousands of domains

• Design systems to detect and handle dynamic content, JavaScript rendering, and anti-bot measures

• Create intelligent crawl scheduling and prioritization algorithms for maximum coverage efficiency

This is an in-person opportunity in San Francisco. We're happy to sponsor international candidates (e.g., STEM OPT, OPT, H1B, O1, E3).

Experience building and scaling web crawlers; proficiency in high-performance languages (C++, Rust); ability to optimize systems for efficiency; passion for high-quality knowledge gathering.

Key Responsibilities

• crawl web
• scale crawler
• optimize system

Skills & Tools

C++, Rust

Job Details

• Category: Software Development
• Seniority: No Prior Experience Required
• Commitment: Full Time
• Workplace: Remote — North America
• Salary: USD 150,000 – 300,000 / year
• Languages: English

About Exa

A company building a scalable web crawling and AI-powered search infrastructure. — Industry: Information Technology