About the role
structured by ORIAs a Lead Data Engineer, you will leverage Azure technologies to drive data transformation, analytics, and machine learning. You will design scalable Databricks data pipelines using PySpark, transforming raw data into actionable insights.
What you will do
- Design scalable Databricks data pipelines using PySpark, transforming raw data into actionable insights
- Build, deploy, and maintain machine learning models using MLlib or TensorFlow
- Optimize cloud data integration from Azure Blob Storage, Data Lake, and SQL/NoSQL sources
- Execute large-scale data processing using Spark Pools, fine-tuning configurations for efficiency
What they are looking for
- Bachelor's or Master's in Computer Science, Data Science, or a related field
- 7+ years in data engineering
- 5+ years specializing in Azure Databricks, PySpark, and Spark Pools
- Proficiency in Python PySpark, Pandas, NumPy, SciPy, Spark SQL, DataFrames, RDDs, Delta Lake, Databricks Notebooks, and MLflow
- Hands-on experience in Azure Data Lake, Blob Storage, and Synapse Analytics
Full posting text
As a Lead Data Engineer, you will leverage Azure technologies to drive data transformation, analytics, and machine learning. You will design scalable Databricks data pipelines using PySpark, transforming raw data into actionable insights. Your role includes building, deploying, and maintaining machine learning models using MLlib or TensorFlow while optimizing cloud data integration from Azure Blob Storage, Data Lake, and SQL/NoSQL sources.
You will execute large-scale data processing using Spark Pools, fine-tuning configurations for efficiency. The ideal candidate holds a Bachelors or master?s in computer science, Data Science, or a related field, with 7+ years in data engineering and 5+ years specializing in Azure Databricks, PySpark, and Spark Pools. Proficiency in Python PySpark, Pandas, NumPy, SciPy, Spark SQL, DataFrames, RDDs, Delta Lake, Databricks Notebooks, and MLflow is required, along with hands-on experience in Azure Data Lake, Blob Storage, and Synapse Analytics