Note: The job is a remote job and is open to candidates in USA. Scicom Infrastructure Services is seeking a Databricks Data Operations Specialist to support the stability, reliability, performance, and security of a large-scale Databricks data platform. The role monitors production data pipelines, troubleshoots incidents and platform issues, coordinates deployments, and ensures data products meet availability, quality, and service-level requirements.

Responsibilities

• Monitor the health, availability, performance, and capacity of Databricks workspaces, jobs, pipelines, clusters, SQL warehouses, and related cloud services
• Provide day-to-day operational support for production Databricks environments
• Monitor scheduled and event-driven data pipelines for failures, delays, data-quality issues, and performance degradation
• Troubleshoot Databricks jobs, Spark workloads, cluster failures, notebook errors, connectivity issues, and data-processing exceptions
• Restart, rerun, recover, or coordinate remediation of failed workflows according to documented procedures
• Support Databricks Workflows, Lakeflow, Delta Live Tables, Auto Loader, Structured Streaming, and batch-processing workloads
• Review driver and executor logs, Spark UI metrics, cluster event logs, audit logs, and application telemetry to identify root causes
• Support configuration and maintenance of Databricks clusters, serverless compute, instance pools, cluster policies, and SQL warehouses
• Monitor resource utilization and recommend changes to cluster sizing, autoscaling, workload scheduling, and compute policies
• Help maintain consistent configurations across development, testing, staging, and production environments
• Support platform upgrades, runtime-version changes, library updates, and configuration changes
• Coordinate with Databricks support and cloud-service providers when vendor assistance is required
• Monitor ETL, ELT, streaming, API, and file-based data-integration processes
• Validate that scheduled data loads complete within required processing windows
• Investigate missing, late, duplicated, incomplete, or inconsistent data
• Perform controlled data reprocessing, backfills, reconciliation, and recovery activities
• Support ingestion and transformation processes involving JSON, CSV, XML, Parquet, APIs, databases, message queues, and cloud storage
• Monitor Bronze, Silver, and Gold data layers within a medallion architecture
• Validate source-to-target record counts, control totals, schema consistency, and business-rule compliance
• Coordinate with data engineers to resolve recurring pipeline or transformation defects
• Maintain operational runbooks for common failures, recovery procedures, escalation paths, and recurring support tasks
• Ensure production interventions are documented and performed in accordance with change-control procedures
• Serve as a first or second level of support for production Databricks and data-pipeline incidents
• Triage incidents based on severity, business impact, affected systems, and service-level commitments
• Coordinate incident response among engineering, cloud, security, governance, and business teams
• Provide timely status updates during active incidents
• Escalate issues to technical leads, architects, vendors, or program leadership when appropriate
• Conduct or support root-cause analyses for production incidents
• Document incident timelines, contributing factors, corrective actions, and preventive measures
• Track recurring problems and recommend permanent engineering or process improvements
• Participate in post-incident reviews and ensure assigned corrective actions are completed
• Help develop automated remediation for common operational failures
• Maintain incident, problem, and service-request records in the organization’s ticketing system
• Develop and maintain dashboards, alerts, and operational reports for Databricks and associated data services
• Monitor job success rates, pipeline latency, processing duration, cluster utilization, data freshness, data quality, and compute consumption
• Configure actionable alerts that minimize unnecessary notifications while identifying material operational issues
• Integrate Databricks monitoring data with enterprise observability platforms
• Support tools such as Azure Monitor, Log Analytics, CloudWatch, Datadog, Splunk, Grafana, Prometheus, or comparable platforms
• Define operational health indicators, service-level indicators, and service-level objectives
• Produce daily, weekly, and monthly operational metrics for program leadership
• Identify performance trends and operational risks before they result in production failures
• Maintain dashboards showing system availability, incident volume, recovery time, pipeline status, and data freshness
• Monitor compliance with established data contracts between data producers and consumers
• Validate schema, field, data-type, frequency, freshness, completeness, and quality requirements
• Detect schema drift, unexpected source-system changes, and contract violations
• Coordinate resolution of data-contract issues with business analysts, data owners, and engineering teams
• Execute automated and manual data-quality controls
• Monitor data-quality rules related to accuracy, completeness, uniqueness, consistency, validity, and timeliness
• Track exceptions and ensure data-quality issues are assigned to the appropriate owner
• Support reconciliation of contracting, procurement, financial, and operational datasets
• Assist with validation of data structured according to the Open Contracting Data Standard when applicable
• Maintain operational evidence supporting data quality, lineage, audit, and governance requirements
• Support the release and deployment of Databricks notebooks, jobs, workflows, libraries, configurations, and infrastructure changes
• Coordinate deployments across development, testing, staging, and production environments
• Verify deployment readiness, approvals, testing results, dependencies, and rollback plans
• Participate in release validation and post-deployment monitoring
• Support CI/CD pipelines using Azure DevOps, GitHub Actions, GitLab CI, Jenkins, or comparable tools
• Assist with infrastructure-as-code deployments using Terraform or similar technologies
• Maintain deployment logs, implementation records, configuration documentation, and release notes
• Ensure emergency changes follow established approval and documentation requirements
• Support environment comparisons and investigate configuration drift
• Coordinate production changes with technical and business stakeholders to minimize operational disruption
• Support Databricks identity, access, and entitlement administration
• Assist with user onboarding, offboarding, group membership, workspace permissions, and service-principal access
• Support Unity Catalog permissions, catalogs, schemas, tables, volumes, external locations, and storage credentials
• Apply role-based and least-privilege access-control principles
• Monitor access failures, unusual activity, audit events, and policy violations
• Coordinate access requests with security, governance, and data-ownership teams
• Support secret management through Databricks secrets, Azure Key Vault, AWS Secrets Manager, or comparable services
• Maintain operational documentation supporting security reviews and audits
• Assist with data retention, archival, backup, recovery, and disaster-recovery procedures
• Ensure production-support activities comply with applicable security and privacy requirements
• Monitor Databricks and cloud-resource consumption
• Identify idle, oversized, inefficient, or improperly configured compute resources
• Analyze job duration, cluster utilization, query performance, storage consumption, and workload patterns
• Recommend cluster right-sizing, autoscaling, scheduling, caching, partitioning, and workload-isolation improvements
• Support implementation of compute policies, tagging standards, budgets, and cost alerts
• Produce usage and cost reports for technical and program leadership
• Work with engineers and architects to reduce unnecessary compute consumption
• Track the operational impact of optimization initiatives
• Create and maintain operational procedures, troubleshooting guides, escalation matrices, and support runbooks
• Document platform configurations, dependencies, schedules, service accounts, integrations, and recovery requirements
• Maintain an accurate inventory of production jobs, pipelines, data products, and system interfaces
• Identify opportunities to automate repetitive monitoring, support, recovery, and reporting activities
• Participate in operational-readiness reviews for new data products and pipelines
• Ensure new solutions include appropriate monitoring, alerts, logging, support procedures, and ownership assignments before production deployment
• Share lessons learned and common troubleshooting practices across the Databricks team
• Contribute to platform standards, operational policies, and continuous-improvement initiatives

Skills

• Bachelor's degree in computer science, information technology, data engineering, information systems, or a related field, or equivalent professional experience
• At least four years of experience in data operations, production support, data engineering, cloud operations, DevOps, platform support, or a related technical role
• At least two years of hands-on experience supporting Databricks environments or production Databricks workloads
• Experience supporting production ETL, ELT, batch, or streaming data pipelines
• Working knowledge of: + Databricks + Apache Spark + PySpark or Python + SQL + Delta Lake + Databricks Workflows or Jobs + Cloud object storage + Data pipeline monitoring + Log analysis and troubleshooting
• Experience investigating failed jobs, delayed pipelines, schema issues, data discrepancies, and performance problems
• Familiarity with medallion architecture and Bronze, Silver, and Gold data layers
• Experience with at least one major cloud platform: Microsoft Azure, AWS, or Google Cloud
• Experience using ticketing and service-management tools such as ServiceNow, Jira, or a comparable platform
• Experience with monitoring, alerting, logging, or observability tools
• Understanding of incident, problem, change, and release-management processes
• Ability to write and maintain technical procedures and operational documentation
• Strong analytical, organizational, communication, and troubleshooting skills
• Ability to work effectively with engineers, business analysts, architects, security teams, and nontechnical stakeholders
• Ability to support multiple production priorities in a structured and timely manner
• Strong operational ownership and attention to detail
• Ability to remain calm and organized during production incidents
• Clear written and verbal communication
• Methodical troubleshooting and root-cause analysis
• Ability to distinguish symptoms from underlying technical causes
• Strong documentation and process-discipline skills
• Ability to prioritize incidents based on business and technical impact
• Collaborative approach to working with engineering and business teams
• Commitment to automation, reliability, security, and continuous improvement
• Willingness to take ownership of issues through resolution
• Databricks Certified Data Engineer Associate or Databricks Certified Data Engineer Professional
• Databricks Certified Associate Developer for Apache Spark
• Microsoft Azure, AWS, or Google Cloud certification
• Experience with Unity Catalog, Delta Live Tables, Lakeflow, Auto Loader, Structured Streaming, or Databricks SQL
• Experience with Azure Data Factory, Azure Data Lake Storage, Azure Monitor, Log Analytics, AWS Glue, Amazon S3, CloudWatch, or comparable cloud services
• Experience with Terraform, Git, Azure DevOps, GitHub Actions, Jenkins, or other CI/CD tools
• Experience with Splunk, Datadog, Grafana, Prometheus, or similar observability platforms
• Experience supporting APIs, Kafka, Event Hubs, Kinesis, Airflow, dbt, Snowflake, or Power BI
• Knowledge of data contracts, schema registries, metadata management, data lineage, and data-quality frameworks
• Familiarity with OCDS, procurement data, contracting data, financial data, or public-sector data standards
• Experience supporting government, regulated, or high-security environments
• Experience operating systems with formal service-level agreements and 24-hour support requirements
• Experience with disaster recovery, backup, archival, and operational-resilience testing

Benefits

• Remote work

Company Overview

Company H1B Sponsorship