About the role
from listingRole Overview:
Lead the design, development, and optimization of large-scale, reliable, and secure data pipelines and data lake architecture on AWS.
Architect and implement end-to-end data solutions, including data ingestion, storage, transformation, and analytics using AWS services (Glue, Redshift, S3, Lambda, EMR, Kinesis, Athena, RDS, etc.).
Mentor and guide a team of data engineers, conducting code reviews and fostering best practices in data engineering and cloud architecture.
Collaborate with data scientists, analysts, and business stakeholders to translate requirements into scalable and maintainable solutions.
Oversee migration of data from legacy systems to AWS-based data lakes and data warehouses.
Develop and enforce standards for data quality, security, and governance.
Drive the adoption of DevOps, CI/CD, and infrastructure-as-code practices within the data engineering team.
Ensure solutions are cost-effective, performant, and aligned with enterprise data strategy.
Stay current with advancements in AWS technologies and data engineering trends and evaluate new tools and frameworks for potential adoption.
Troubleshoot complex data issues and provide technical leadership in problem resolution.
Key Responsibilities & Skillsets:
Common Skillsets:
Superior analytical and problem solving skills
Should be able to work on a problem independently and prepare client ready deliverable with minimal or no supervision
Good communication skill for client interaction
Application development Skillsets:
Strong ability to debug complex data workflows, optimize application and ETL code, and automate data transformation processes
Systematic, analytical problem‑solving approach with strong ownership over data quality, performance, and delivery
Ability to quickly evaluate new AWS data and analytics services and determine fit for data pipelines or application architecture
Hands‑on experience developing data workflows and infrastructure using IaC frameworks such as CloudFormation or Terraform (as needed)
Working knowledge of CI/CD pipelines primarily to support data application deployments (Jenkins, CodePipeline, etc.)
Proficient with Git for versioning data processing code, libraries, and application components
Skilled in writing production‑grade code in Python, Bash, PowerShell, or similar languages, focusing on data processing and backend development
Experience using Docker for packaging applications and data-processing workloads, with exposure to containerized data services (ECS, EKS, etc.)
Comfortable developing and troubleshooting in Linux environments
Solid understanding of key AWS data services and application primitives such as S3, EC2, Glue, EMR, Lambda, RDS, DynamoDB, CloudWatch, and VPC networking concepts
Strong knowledge of AWS security and IAM as it relates to data pipelines, encryption (KMS), secure data access (IAM roles/policies), and audit controls
Hands‑on experience with ETL, distributed compute, and big data frameworks such as Spark, Glue, Hadoop/EMR, Impala, or similar tooling
Deep understanding of relational databases, SQL optimization, and application‑to‑database interaction patterns
Familiarity with log analytics and observability platforms (Splunk, ELK, Prometheus, Grafana) as they relate to monitoring data pipelines and applications
Candidate Profile:
Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
10+ years of experience in data engineering, with at least 3 years in technical leadership or lead engineer role.
Extensive hands-on experience with AWS data services (Glue, Redshift, S3, Lambda, EMR/Spark, Kinesis, Athena, RDS, API Gateway, etc.).
Proficient in programming languages such as Python and SQL; experience with Shell scripting and Scala is a plus.
Strong experience designing, implementing, and managing data lakes, data warehouses, and data ingestion pipelines on AWS.
Proven experience with ETL/ELT processes, data modeling, and big data frameworks.
Demonstrated ability to lead, mentor, and coach engineers in a collaborative team environment.
Experience with DevOps practices, CI/CD pipelines, and infrastructure-as-code tools (e.g., CloudFormation, Terraform).
Excellent problem-solving, communication, and organizational skills.
Originally posted on Himalayas