Lead Data Software Engineer with Databricks, Apache Kafka, Apache Spark, Kubernetes

We are seeking a Lead Data Software Engineer to drive the migration of an existing data analytics platform, originally built on Azure resources such as DataFactory, Databricks, EventHub, and Cassandra, to a cloud-agnostic platform capable of on-premises deployment.

Responsibilities

• Implement streaming and batch Spark pipelines running on K8S
• Develop objects of the data generator framework
• Deploy and test solutions on local and development environments
• Enhance and refine the current actively evolving solution
• Investigate and resolve bugs
• Conduct unit testing to ensure code quality
• Participate in refinement, planning, and demo sessions
• Guide and mentor team members while syncing with the team lead

Requirements

• 5+ years of experience with Apache Spark, Databricks, and Kubernetes (K8S)
• Proficiency in Python
• Experience with Confluent/Apache Kafka
• Completed EPAM Data Engineering course or equivalent qualification
• Capability to quickly adapt and dive into the active implementation phase of a project
• Ability to work independently while coordinating with a team lead
• Strong communication skills and a proactive, hands-on, team-player attitude
• English proficiency at B2 level or higher

Nice to have

• Knowledge of Java
• Familiarity with Kafka Connect, KSQL, and Kafka Streams
• Background in Ansible and Argo CD
• Understanding of TDD

Benefits

CONTINUOUS UPSKILLING, LEARNING & DEVELOPMENT

• Diversity of tasks and projects
• Assessment center for objective review of competency level
• Personal development plan
• Mentoring programs and leadership development
• Certification and professional development support
• Access to learning platforms including more than 2,500 internal courses
• English courses taught by certified teachers

CORPORATE BENEFITS

• Extra leave days
• Referral bonuses

COMPENSATION PACKAGE

• Competitive compensation paid in USD
• Regular salary and performance reviews

MEDICAL & HEALTHCARE

• Private health insurance
• Well-being events

WORKING ENVIRONMENT

• Recreation areas and kitchens
• Tea, coffee and snacks
• Sports equipment and game consoles
• IT Equipment
• Microsoft’s Software Assurance Home Use Program (HUP)