Skip to content

SpyCloud

Data Engineer

Posted Updated

Description

  • This Data Engineer II role sits on the Engineering team, reporting to the Senior Director of Data Engineering.
  • Your focus will include evolving our data platform using technologies such as Databricks and Spark, building scalable data pipelines, and serving customers through purpose-built data stores including RDS, Elasticsearch, and DynamoDB.
  • This is a hands-on individual contributor role with ownership of complex data engineering initiatives that support the scalability, reliability, and continued evolution of SpyCloud’s data platform
  • Design, build, and improve scalable data systems with a focus on performance, and reliability
  • Use Databricks and Spark to prepare, transform, and serve high-quality data to customers and downstream systems
  • Build, optimize, and maintain real-time and batch data pipelines using AWS technologies such as Lambda, S3, DynamoDB, ECS, EKS, and Terraform, as well as develop and maintain pipelines within the Databricks ecosystem
  • Identify and implement improvements to the cost and performance of data ingestion, transformation, and storage workflows
  • Be available for on-call support to troubleshoot and remediate data pipeline failures outside of standard business hours
  • Partner with Product Management, SpyCloud Labs, and Engineering teams to understand requirements and develop effective technical solutions
  • Evaluate and implement modern tooling for data processing, ML/AI data workflows, and observability
  • Contribute to architectural and technical design decisions and help implement scalable solutions aligned with engineering standards
  • Promote engineering best practices through collaboration, code reviews, documentation, and knowledge sharing
  • Identify and resolve complex data quality, performance, security, and reliability issues across production systems
  • Independently own complex data engineering initiatives from design and development through deployment and production support
  • Contribute to the continued evolution of SpyCloud’s data platform as product, customer, and AI requirements grow

Benefits

  • Health benefits: From medical to dental, we’ve got you covered.
  • Work/life balance: We offer generous PTO and a remote-friendly culture.
  • Compensation: Our talented employees enjoy competitive salaries and equity.
  • 401k matching: Investing in the future of our employees is a no-brainer.
  • SpyCares: We’re only as strong as the communities we’re a part of and giving back is in our DNA.
  • Practical experience with DynamoDB and other AWS-native data services
  • Experience building and operating Lakehouse, Databricks, or large-scale distributed data architectures in AWS
  • Strong development skills in Python, particularly when building data-intensive systems
  • 5+ years of professional experience in data engineering, software engineering, or a closely related discipline
  • Experience troubleshooting complex data pipeline, performance, and data quality issues in production environments
  • Experience preparing and managing data for machine learning and LLM workflows
  • Strong communication and collaboration skills across technical and non-technical stakeholders
  • Hands-on experience designing, building, and maintaining ETL and data-processing pipelines
  • Experience building and managing Elasticsearch/OpenSearch data stores
  • Experience with infrastructure-as-code tools such as Terraform
  • Ability to independently own complex technical projects from design and development through deployment and production support
  • Strong understanding of distributed systems as they relate to data storage, processing, and computing
  • Ability to evaluate technical trade-offs and contribute effectively to architectural and design decisions
  • Familiarity with AI/ML tools and technologies such as SageMaker, LangChain, or vector databases
  • Understanding of data governance, privacy, and compliance considerations within AI and ML environments
  • Experience with observability and monitoring tools for large-scale data platforms
  • Experience with Go or C programming languages
  • Comfort working in fast-paced environments with evolving product and technical requirements
  • Experience working with cybersecurity or security-related datasets and technologies
  • Bachelor’s or Master’s degree in Computer Science, Engineering, or a related STEM field

Skills

Apply

Apply to Data Engineer at SpyCloud

Create a free account first, or go straight to the employer.

Free RoleWatch account

  • Get an email when new jobs match your search
  • Save jobs and track every application
  • See how well your CV matches each job

Already have one?

No account needed. Opens the employer's own site in a new tab.

See also

Data Engineering jobs by country — openings, pay and top skills →