Azure Databricks Data Engineer

We are seeking a hands-on Azure Databricks Data Engineer to support the implementation of a modern data platform on Microsoft Azure to enable self-serve data analytics for various business users. This role is technical execution-focused, responsible for building data pipelines, implementing structured data layers, and supporting the deployment of Databricks Genie capabilities based on predefined architecture and design.

Job Description

Responsibilities:

1. Azure Databricks Platform Implementation

1. Develop and maintain data solutions using Azure Databricks (notebooks, Delta Lake)
2. Configure and optimize Databricks jobs, clusters, and workloads
3. Integrate with Azure Data Lake (ADLS Gen2)

1. Databricks Genie Implementation Support

1. Support implementation of Databricks Genie based on predefined architecture
2. Prepare curated datasets for Genie consumption
3. Optimize data structures for query performance

1. Data Pipeline Development

1. Build ETL/ELT pipelines using Databricks and Azure tools
2. Develop ingestion pipelines from APIs, databases, and external systems
3. Ensure pipelines are reliable, monitored, and production-ready
    

1. Data Management & Governance

1. Apply data quality checks and controls
2. Implement access control and data organization practices
3. Document pipelines and datasets
    

1. Production Readiness & Optimization

1. Ensure scalability and performance
2. Optimize queries and pipeline efficiency
3. Improve Genie performance to deliver faster, more efficient results

Job requirements:

1. Hands-on skills of Azure Databricks (Delta Lake, notebooks), Azure Data Lake (ADLS Gen2), Azure Data Factory or Synapse
2. Good SQL and ETL experience and medallion architecture familiarity
3. Exposure to Databricks Genie, familiarity with Python/PySpark, Unity Catalog or data governance tools are advantageous
4. Good execution focus with the ability to deliver high-quality work independently
5. High attention to details, particularly in data quality and accuracy
6. Good problem-solving skills, with the ability to troubleshoot data and pipeline issues
7. Receptive to feedback and able to iterate quickly based on technical guidance
8. Able to collaborate effectively within a technical team environment