Note: The job is a remote job and is open to candidates in USA. Management Science and Innovation (MSI) Consulting, LLC is a mid-size management consultancy serving industry, government, non-profit, and higher-education clients. The Senior Databricks Engineer will architect and implement scalable data pipelines and platforms using Databricks, PySpark, and Delta Lake while establishing engineering standards, governance, CI/CD, and data quality practices. The role also partners with cross-functional teams and mentors junior engineers to deliver production-ready data products.

Responsibilities

• Architect and implement high-performance ETL/ELT pipelines using Databricks, PySpark, and Delta Lake
• Optimize data workflows for scalability, reliability, and cost efficiency
• Partner with data scientists, analytics teams, business stakeholders, and product teams to deliver trusted, production-ready data products
• Implement CI/CD, testing, and data quality processes
• Define data platform architecture standards, engineering best practices, and governance frameworks across Databricks environments
• Mentor junior engineers and conduct code reviews

Skills

• 7+ years of data engineering experience, including 3+ years hands-on Databricks experience
• Deep expertise with Databricks Lakehouse Platform, Delta Lake, and Apache Spark
• Strong Spark performance tuning and optimization skills
• Advanced proficiency in SQL and Python; Scala preferred
• Experience designing batch and streaming data pipelines
• Experience building and deploying production Databricks AI/Agent solutions, ML engineering workflows, GenAI applications, and enterprise-grade dashboards and data products on the Databricks Lakehouse Platform
• Experience implementing CI/CD and automated testing for data engineering solutions
• Strong understanding of Unity Catalog, RBAC, lineage, auditing, and governance
• Experience designing and operating data platforms on AWS, Azure, or GCP, including security, networking, storage, and cost optimization considerations
• Familiarity with Terraform, CloudFormation, ARM, or Bicep
• Experience designing and implementing Medallion (Bronze/Silver/Gold) Lakehouse architectures and scalable data domain models
• Experience supporting Federal government clients
• Bachelor's (Required)
• Databricks Certified Data Engineer Professional
• Experience supporting RFP/RFQ/RFI responses and solution estimation
• Experience leading platform migration or modernization initiatives
• Experience in regulated environments and cloud cost optimization
• Certifications in other skills listed in the Qualifications section

Benefits

• 401(k)
• Dental insurance
• Health insurance
• Life insurance
• Vision insurance
• Paid time off
• Parental leave
• Remote work

Company Overview

Company H1B Sponsorship