Data Engineer (Azure Databricks)

Responsibilities:

1. Azure Databricks Platform Implementation

1. Develop and maintain data solutions using Azure Databricks (notebooks, Delta Lake)
2. Configure and optimize Databricks jobs, clusters, and workloads
3. Integrate with Azure Data Lake (ADLS Gen2)

1. Databricks Genie Implementation Support

1. Support implementation of Databricks Genie based on predefined architecture
2. Prepare curated datasets for Genie consumption
3. Optimize data structures for query performance

1. Data Pipeline Development

1. Build ETL/ELT pipelines using Databricks and Azure tools
2. Develop ingestion pipelines from APIs, databases, and external systems
3. Ensure pipelines are reliable, monitored, and production-ready
    

1. Data Management & Governance

1. Apply data quality checks and controls
2. Implement access control and data organization practices
3. Document pipelines and datasets
    

1. Production Readiness & Optimization

1. Ensure scalability and performance
2. Optimize queries and pipeline efficiency.
3. Improve Genie performance to deliver faster, more efficient results

Skills & Experiences:

1. Hands-on skills of Azure Databricks (Delta Lake, notebooks), Azure Data Lake (ADLS Gen2), Azure Data Factory or Synapse
2. Solid SQL and ETL experience and medallion architecture familiarity
3. Exposure to Databricks Genie, familiarity with Python/PySpark, Unity Catalog or data governance tools are advantageous
4. Solid execution focus with the ability to deliver high-quality work independently
5. High attention to details, particularly in data quality and accuracy
6. Solid problem-solving skills, with the ability to troubleshoot data and pipeline issues
7. Receptive to feedback and able to iterate quickly based on technical guidance
8. Able to collaborate effectively within a technical team environment