Databricks Data Engineer

# Databricks Data Engineer ## Company Overview *Company details are not specified.* ## Job Summary We are seeking an experienced **Databricks Data Engineer** with a proven track record in building and optimizing big data pipelines. The ideal candidate will leverage strong expertise in **Apache Spark**, **Databricks**, and **AWS** to contribute to our organization's data engineering goals and best practices. ## Responsibilities - Design, build, and maintain **scalable data pipelines** using **Databricks** and **Apache Spark**. - Develop and optimize **ETL/ELT processes** for both structured and unstructured data. - Implement and manage **Lakehouse architecture** for data storage, processing, and analytics. - Orchestrate data workflows using **Databricks Workflows**, **Jobs API**, and **Airflow**. - Work with **AWS data services** (S3, Lambda, CloudWatch) for integration and automation. - Optimize query performance using **pushdown techniques**, **indexing**, and **partitioning**. - Ensure data security and compliance through **Unity Catalog**, governance policies, and access controls. - Monitor, troubleshoot, and improve **Databricks jobs**, clusters, and pipelines. - Participate in end-to-end **data migration projects** to **AWS Cloud**. - Collaborate using **Git/GitHub** for code versioning and **CI/CD**. ## Qualifications - **5–9 years** of experience in data engineering roles. - Strong hands-on experience in **Databricks**, **Apache Spark**, **AWS**, and **Python**. - Expertise in **data orchestration** and **workflow management**. - Proficient in **SQL**, performance optimization, and troubleshooting. - Excellent analytical, problem-solving, and communication skills. - **Bachelor's degree** in Computer Science, Information Technology, or a related field (preferred). ## Preferred Skills - Familiarity with machine learning frameworks. - Experience with data governance tools and practices. - Knowledge of additional programming languages or frameworks (e.g., Java, Scala). ## Experience - Minimum of **5 years** of relevant experience in data engineering, with a focus on big data technologies. ## Environment This position is **remote**, allowing for flexible work arrangements while requiring effective collaboration with team members in various locations. ## Salary Estimated budget for the position is **₹1.2 LPM**. ## Growth Opportunities *Growth opportunities within the company are not specified.* ## Benefits *Benefits associated with this role are not specified.*