IN_ Associate _Azure Databricks Pyspark Developer -_Data & Analytics _Advisory _Mumbai

Job Description & Summary : A career within Technology Consulting services, will provide you with the opportunity to bring our clients a competitive advantage through defining their technology objectives, assessing solution options, and devising architectural solutions that help them achieve both strategic goals and meet operational requirements. We help build software and design data platforms, manage large volumes of client data, develop compliance procedures for data management, and continually researching new technologies to drive innovation and sustainable change.

Job Position Title: IN_ Associate _Azure Databricks Pyspark Developer -_Data & Analytics _Advisory _Mumbai

Responsibilities:

Key Responsibilities:

Data Lake and Lakehouse Implementation:

Design, implement, and manage Data Lake and Lakehouse architectures. (Must have)

Develop and maintain scalable data pipelines and workflows. (Must have)

Utilize Azure Data Lake Services (ADLS) for data storage and management. (Must have)

Knowledge on Medalion Architecture, Delta Format. (Must have)

Data Processing and Transformation:

Use PySpark for data processing and transformations. (Must have)

Implement Delta Live Tables for real-time data processing and analytics. (Good to have)

Ensure data quality and consistency across all stages of the data lifecycle. (Must have)

Data Management and Governance:

Employ Unity Catalog for data governance and metadata management. (Good to have)

Ensure robust data security and compliance with industry standards. (Must have)

Data Integration:

Extract, transform, and load (ETL) data from multiple sources (Must have) including SAP (Good to have), Dynamics 365 (Good to have), and other systems.

Utilize Azure Data Factory (ADF) and Synapse Analytics for data integration and orchestration. (Must have)

Performance Optimization of the Jobs. (Must have)

Data Storage and Access:

Implement and manage Azure Data Lake Storage (ADLS) for large-scale data storage. (Must have)

Optimize data storage and retrieval processes for performance and cost-efficiency. (Must have)

Collaboration and Communication:

Work closely with data scientists, analysts, and other stakeholders to understand data requirements. (Must have)

Provide technical guidance and mentorship to junior team members. (Good to have)

Continuous Improvement:

Stay updated with the latest industry trends and technologies in data engineering and cloud computing. (Good to have)

Continuously improve data processes and infrastructure for efficiency and scalability. (Must have)

Mandatory skill sets:

Technical Skills:

Proficient in PySpark and Python for data processing and analysis.

Strong experience with Azure Data Lake Services (ADLS) and Data Lake architecture.

Hands-on experience with Databricks for data engineering and analytics.

Knowledge of Unity Catalog for data governance.

Expertise in Delta Live Tables for real-time data processing.

Familiarity with Azure Fabric for data integration and orchestration.

Proficient in Azure Data Factory (ADF) and Synapse Analytics for ETL and data warehousing.

Experience in pulling data from multiple sources like SAP, Dynamics 365, and others.

Soft Skills:

Excellent problem-solving and analytical skills.

Strong communication and collaboration abilities.

Ability to work independently and as part of a team.

Attention to detail and commitment to data accuracy and quality.

Preferred skill sets:

Handson ETL / Datalake is good to have..

Certifications required:

Certification in Azure Data Engineering or relevant Azure certifications.

DP203 (Must have)

Certification in Databricks.

Databricks certified Data Engineer Associate (Must have)

Databricks certified Data Engineer Professional (Good Have)

Years of experience required: 2-4 Years

Educational Qualification B.E. / B.Tech / MCA/ M.E/ M.TECH/ MBA/ PGDM. All qualifications should be in regular full-time mode with no extension of course duration due to backlogs.

Seniority level: Entry level

Employment type: Full-time

Industries: IT Services and IT Consulting