Lead Data Engineer - Databricks

Role:

As one of the founding members of Kanerika's Databricks delivery practice, you will be a hands-on builder responsible for designing, developing, and optimizing data pipelines on Databricks for client engagements. You'll work closely with the Practice Lead to translate architecture into working solutions, build reusable accelerators ahead of client demand, and help establish engineering best practices for the growing team.

Key Responsibilities

• Design, build, and optimize ETL/ELT data pipelines on Databricks using PySpark and Spark SQL.
• Build and maintain incremental ingestion pipelines using Auto Loader and structured streaming, including checkpoint management and schema evolution handling.
• Implement and maintain Delta Lake tables (including Change Data Feed and liquid clustering), Unity Catalog structures, and Databricks Workflows for client and internal projects.
• Build reusable accelerators, templates, and demo environments to support pre-sales and speed up future client delivery.
• Collaborate with the Practice Lead/Architect on solution design for client engagements, providing engineering-level input on feasibility and effort.
• Perform data quality checks, performance tuning, and cost optimization on Databricks clusters and jobs.
• Participate in client-facing technical discussions as needed, including discovery sessions and technical walkthroughs.
• Write clean, well-documented, and testable code following team engineering standards.
• Mentor junior engineers as the team scales, and contribute to internal knowledge-sharing and best-practice documentation.
• Troubleshoot and resolve production data pipeline issues across client environments.
• Stay current with Databricks platform releases across Unity Catalog, Lakeflow Declarative Pipelines, and share learnings with the team through internal knowledge sessions and documentation.

Required Skills & Experience

• 5–9 years of data engineering experience, with at least 2 years of hands-on Databricks experience in production.
• Strong proficiency in PySpark and/or Spark SQL for large-scale data processing.
• Practical experience with Delta Lake, Unity Catalog, and Databricks Workflows/job orchestration.
• Solid SQL skills and experience with data modeling for analytics/lakehouse architectures.
• Experience with at least one major cloud platform (Azure, AWS, or GCP); Azure strongly preferred.
• Experience with Python for data engineering tasks beyond Spark (scripting, automation, testing).
• Familiarity with CI/CD practices for data pipelines (Git-based workflows, automated testing/deployment).
• Strong debugging and performance-tuning skills for Spark jobs (partitioning, caching, cluster sizing).
• Good communication skills and comfort working directly with client stakeholders when needed.

PEFERRED/ NICE TO HAVE

• Databricks Certified Data Engineer Associate/Professional certification.
• Experience with Lakeflow Declarative Pipelines (formerly Delta Live Tables), Lakeflow Connect for managed ingestion, MLflow for experiment tracking and model registry, Databricks SQL, and AI/BI Genie.
• Exposure to data governance and security frameworks (row/column-level security, data masking).
• Prior experience in a consulting/IT services environment delivering to multiple clients.
• Familiarity with orchestration tools (Airflow) and ingestion tools (Fivetran, Kafka, Azure Data Factory).

What Success Looks Like

• Within 3 months: Comfortable with Kanerika's delivery standards; has built or contributed to at least one reusable accelerator/demo asset.
• Within 6 months: Independently delivering core engineering work on client engagement(s) with minimal oversight.
• Within 12 months: Recognized as a go-to senior engineer on the team, mentoring newer hires and contributing to architecture decisions.
• Leading databricks partnership upgrade to Gold level.