FULLY REMOTE ScolerTec is seeking Data Platform Engineer – Databricks & AWS to design, administer, optimize, secure, and support an enterprise Databricks Lakehouse environment. This role is heavily hands-on and will support scalable data engineering, analytics, governance, performance, reliability, and platform operations. The ideal candidate has strong production experience administering Databricks on AWS, including Unity Catalog, clusters, SQL Warehouses, Workflows, Delta Lake, Spark, Terraform, monitoring, security, and performance optimization.

Key Responsibilities

Design, configure, administer, and maintain enterprise Databricks Lakehouse environments. Manage Databricks workspaces, compute, clusters, SQL Warehouses, serverless capabilities, and workload management. Implement and administer Unity Catalog, including catalogs, schemas, tables, groups, RBAC, permissions, and governance controls. Develop and optimize data pipelines and ETL/ELT workloads using Apache Spark, Delta Lake, Delta Live Tables, and Databricks Workflows. Automate platform provisioning, configuration, and deployment using Terraform, Python, SQL, Databricks APIs, and Asset Bundles. Perform Spark and SQL performance tuning, scalability testing, and troubleshooting. Implement monitoring, logging, alerting, observability, capacity planning, and operational health dashboards. Optimize Databricks platform costs through autoscaling, auto-termination, right-sizing, workload isolation, and usage analysis. Integrate Databricks with AWS services, enterprise data sources, governance platforms, BI tools, and AI/ML environments. Implement IAM, RBAC, encryption, networking, auditing, security, and compliance controls. Support high availability, disaster recovery, incident response, root-cause analysis, upgrades, patching, and release management. Establish reusable platform standards, reference architectures, runbooks, and engineering practices. Provide technical guidance to data engineers and platform users.

Qualifications

Strong hands-on experience implementing and administering Databricks Lakehouse Platform in production. Significant experience with Unity Catalog, cluster management, SQL Warehouses, and Databricks job/workflow orchestration. Strong experience operating Databricks in AWS environments. Strong SQL and Apache Spark experience, including performance tuning and workload optimization. Experience with Delta Lake and modern lakehouse architectures. Hands-on experience with Terraform or comparable Infrastructure as Code. Strong Python and Bash scripting experience. Experience with CI/CD tools such as Jenkins, GitHub Actions, GitLab CI, or Azure DevOps. Experience with CloudWatch, Datadog, Prometheus, Grafana, Elastic, or comparable observability tools. Understanding of AWS IAM, networking, access controls, security, auditing, and resource governance. Experience troubleshooting platform, pipeline, compute, and data-processing issues. Knowledge of ETL/ELT, distributed data processing, data ingestion, and lakehouse architecture. Federal or regulated-environment experience is preferred. Preferred Certifications Databricks Certified Data Engineer Professional or Associate Databricks Certified Associate Platform Administrator Databricks Certified Associate Developer for Apache Spark AWS Certified Data Engineer AWS Certified Solutions Architect – Professional AWS SysOps Administrator Certified Kubernetes Administrator Relevant DevOps/DevSecOps certification