Role:-Databricks Engineer Location:-Remote Duration:-6+ Months Here are some bullets on what they called out in my first conversation with them:

• Databricks experience required; they are building out data lakes in commercial, technical operations, etc.- would ideally like someone with Commercial experience and other areas
• Work is split between monitoring databricks pipelines, remedying issues, performing upgrades, etc. as well as some development work as needed
• Tie-ins to IDMC and PowerCenter
• Consulting firm experience could be good (e.g. Cognizant, TATA, Mindtree, etc. as long as have had both technical and functional experience)
• Does not need to lead the Databricks function, but has to be able to communicate with non-technical audiences and fellow team members -- strong comm skills required.

Required Experience · 8+ years data/analytics engineering experience · Preferred Databricks & Informatica certifications · Experience in Pharma or Regulated industry preferred · Experience utilizing AI for data lifecycle optimization Required Tools / Platforms Experience: · Databricks, Informatica Power center & IDMC · SQL, Python · Azure Data Factory

Responsibilities

Day-to-Day Responsibilities:- Job Responsibilities: Design and develop scalable ETL/ELT pipelines for data extraction, ingestion, transformation, and processing using Apache Spark, Spark-Scala, PySpark, Spark SQL, and Databricks. Design and orchestrate Azure Data Factory (ADF) pipelines and cloud solutions to migrate and process data from on-premises RDBMS platforms to Azure Data Lake, Delta Lake, and Snowflake. Develop Python-based automation solutions for batch processing, data movement, workflow automation, monitoring, and alerting. Develop scalable real-time and streaming pipelines using Databricks Lakeflow Declarative Pipelines, Structured Streaming, PySpark, and Delta Lake. Develop and automate Snowflake data transformation pipelines using dbt, enabling scalable, reusable, version-controlled, and testable SQL workflows. Implement data profiling, data quality, validation, and standardization based on Enterprise Data Management (EDM) standards. Implement data governance using Databricks Unity Catalog, including catalogs, schemas, tables, external locations, access controls, permissions, and data lineage. Design databases, Data Lakes, and Data Warehouse solutions using Azure, Databricks, Delta Lake, Snowflake, and Microsoft SQL Server. Apply statistical and machine-learning techniques for predictive analytics and develop Business Intelligence dashboards using Power BI and Tableau. Collaborate with business stakeholders, architects, analysts, and application teams to understand data requirements, data sources, transformations, and data movement across enterprise systems.