Note: The job is a remote job and is open to candidates in USA. HMG AMERICA LLC is seeking a Lead Data Engineer (Databricks) to lead end-to-end cloud data initiatives for clients. The role involves architecting AWS Databricks Lakehouse platforms, advising client stakeholders, modernizing legacy data warehouses, and guiding engineering teams.

Responsibilities

• Lakehouse Architecture: Architect, design, and implement scalable, enterprise-grade data platforms on AWS using the Databricks Lakehouse architecture
• Medallion Architecture: Establish robust data architecture patterns across Bronze, Silver, and Gold layers for both real-time streaming and batch processing pipelines
• Data Governance & Security: Implement fine-grained access control, governance, data lineage, and audit trails using Databricks Unity Catalog, AWS IAM, and KMS
• Performance & Cost Optimization: Optimize cluster configurations, auto-scaling parameters, and Delta Lake performance techniques (Z-Ordering, Liquid Clustering, compaction)
• Ecosystem Integration: Seamlessly integrate AWS-native services (S3, AWS Glue, Lambda, EventBridge, Redshift, Athena, Secrets Manager) with Databricks
• Stakeholder Management: Lead discovery workshops, technical assessments, and architecture reviews with C-level executives, VPs of Engineering/Data, and Enterprise Architects
• Technical Roadmaps: Translate complex business requirements into solution blueprints, high-level architecture diagrams, technical roadmaps, and Statements of Work (SOWs)
• Legacy Modernization: Drive client migration strategies from legacy data warehouses (e.g., Teradata, Netezza, Snowflake, On-Prem Hadoop) to AWS Databricks
• Best Practices Guidance: Educate clients on cloud data best practices, FinOps/cost optimization strategies, and modern Lakehouse paradigms
• Technical Mentorship: Guide, direct, and mentor cross-functional teams of Data Engineers, Analytics Engineers, and DevOps Engineers
• Infrastructure as Code (IaC): Enforce IaC best practices using Terraform or CloudFormation for automated enterprise workspace and cluster deployments
• DevOps & Automation: Build and maintain CI/CD pipelines for Databricks notebooks, Delta Live Tables (DLT), and Spark jobs across multi-environment setups (Dev, Test, Prod)

Skills

• Bachelor's or Master's degree in Computer Science, Data Engineering, Information Technology, or a related field
• 8+ years in Data Engineering with at least 3+ years in a Lead/Architect capacity
• Deep experience with Databricks , PySpark/Scala, Delta Lake, Unity Catalog, and Delta Live Tables (DLT)
• Hands-on experience with AWS Cloud Data Ecosystem (S3, Glue, Lambda, EMR, Redshift, IAM)
• Proficiency in Python , SQL , and Spark SQL performance tuning
• Strong experience with Infrastructure-as-Code ( Terraform ) and CI/CD tools (GitHub Actions, Azure DevOps, Jenkins)
• Exceptional verbal and written communication skills with the ability to articulate complex technical ideas to executive audiences
• Strong problem-solving skills and client-facing advisory experience

Benefits

• Remote

Company Overview