Senior Software Engineer (Data Acquisition)
Posted Updated
Description
- We are looking for individuals who can balance extreme ownership with a “one-team, one-dream” mindset
- Our customers are trying to solve complex problems, and we only help them achieve their goals as a team
- Our Data Engineering & Acquisition Team ensures our customers have standardized and high quality data to build upon
- You will be crucial in accelerating our efforts to build standalone data products that enable data teams and independent developers to create innovative solutions at massive scale
- In this role, you will be working with a team to continuously improve our existing datasets as well as pursuing new ones
- If you are looking to be part of a team discovering the next frontier of data-as-a-service (DaaS) with a high level of autonomy and opportunity for direct contributions, this might be the role for you
- We like our engineers to be thoughtful, quirky, and willing to fearlessly try new things
- Failure is embraced at PDL as long as we continue to learn and grow from it
- Contribute to the architecture and improvement of our data acquisition and processing platform, increasing reliability, throughput, and observability
- Use and develop web crawling technologies to capture and catalog data on the internet
- Build, operate, and evolve large-scale distributed systems that collect, process, and deliver data from across the web
- Design and develop backend services that manage distributed job orchestration, data pipelines, and large-scale asynchronous workloads
- Structure and model captured data, ensuring high quality and consistency across datasets
- Continuously improve the speed, scalability, and fault-tolerance of our ingestion systems
- Partner with data product and engineering teams to design and implement new data products powered by the data you help collect, and enhance and improve upon existing products
- Learn and apply domain-specific knowledge in web crawling and data acquisition, with mentorship from experienced teammates and access to existing systems
Benefits
- Remote-first work, always: As a remote-first company, we believe in flexible work that fits your life. It’s that simple.
- Unlimited PTO: No checking how many vacation days you have left! Rest, recharge, and take time off when you need it.
- Monthly Employee Stipends: We provide monthly stipends automatically for employees. Think internet bills bills, running shoes, your new pickleball obsession, etc. — use it as you wish!
- Medical, Dental, Vision Insurance: We provide multiple insurance plan options to make sure you and your family are covered!
- Experience with streaming data systems (Kafka, Pub/Sub, Spark Streaming, etc.)
- Scope and break down complex projects into deliverable milestones, and communicate progress, risks, and blockers effectively
- Proficiency with SQL and data warehousing (Snowflake, Redshift, BigQuery, or similar)
- Experience as a Red Teamer
- Balance pragmatism with craftsmanship, shipping reliable systems while continuously improving them
- Communicate clearly and thoughtfully in writing (Slack, docs, design proposals)
- Degree in a quantitative field such as computer science, mathematics, or engineering
- Familiarity with network architecture and debugging (HTTP, DNS, proxies, packet capture and analysis)
- Solid understanding of distributed systems concepts: parallelism, asynchronous programming, backpressure, and message-driven design
- Familiarity with message queues, orchestration, and distributed task systems (Kafka, SQS, Airflow, etc.)
- Experience working on large-scale data ingestion, crawling, or indexing systems
- Experience evaluating and monitoring data quality, ensuring consistency, completeness, and reliability across releases
- Strong grasp of software architecture and backend fundamentals; you can reason clearly about concurrency, scalability, and fault tolerance
- Proficiency with Linux / Unix command-line tools and system resource management
- Experience designing or maintaining resilient data ingestion, API integration, or ETL systems
- Solid programming experience in Python, Go, Rust, or similar, including experience with async / await, coroutines, or concurrency frameworks
- Work independently in a fast-paced, remote-first environment, proactively unblocking themselves and collaborating asynchronously
- Experience with cloud platforms (AWS preferred, GCP or Azure also great)
- Experience with Apache Spark, Databricks, or other distributed data platforms
- Solid understanding of browser rendering pipeline, web application architecture (auth, cookies, http request / response)
- 7+ years of professional experience building or operating backend or infrastructure systems at scale
- Write and maintain technical design documents, including pipeline design, schema design, and data flow diagrams
- Knowledge of modern data design and storage patterns (e.g., incremental updating, partitioning and segmentation, rebuilds and backfills)
- Experience building and maintaining data pipelines on modern big-data or cloud platforms (Databricks, Spark, or equivalent)
- Understanding of modern data storage and design patterns (parquet, Delta Lake, partitioning, incremental updates)
Skills
Sign in to read and join the discussion.