Note: The job is a remote job and is open to candidates in USA. 24-MAG LLC is offering a full-time remote opportunity for an experienced Data Engineer; the hiring client is not identified. The role builds and operates scalable distributed data pipelines and AWS data infrastructure supporting analytics, experimentation, and AI/ML research and products.

Responsibilities

• Design, build, and maintain large-scale pipelines for structured and unstructured data
• Develop distributed processing workflows using Apache Spark or comparable frameworks
• Optimise transformations, partitioning strategies, and computational workloads
• Identify and resolve performance bottlenecks across high-volume data systems
• Support downstream analytics, experimentation, and model-development requirements
• Design scalable AWS-based data architectures across SQL and NoSQL systems
• Build reliable ingestion, transformation, storage, and distribution workflows
• Write efficient Python and SQL for production data processing
• Evaluate storage and database technologies against workload requirements
• Improve scalability, maintainability, accessibility, and operational efficiency
• Implement monitoring, validation, and automation across data workflows
• Identify failures, anomalies, and data-quality issues
• Maintain integrity and reliability throughout pipelines and storage layers
• Collaborate with AI researchers, data scientists, and engineering teams
• Support data infrastructure for AI/ML training, evaluation, and experimentation

Skills

• Strong professional experience in data engineering or distributed data systems
• Advanced proficiency in Python and SQL
• Hands-on experience with Apache Spark or comparable distributed-processing frameworks
• Strong experience with AWS data services and cloud-native architecture
• Experience with SQL and NoSQL databases
• Demonstrated experience processing large-scale datasets
• Strong understanding of partitioning, performance optimisation, and scalable architecture
• Familiarity with orchestration, automation, monitoring, and data-quality workflows
• Exposure to AI/ML or research environments is advantageous
• Familiarity with LLM training, evaluation, or experimentation datasets is beneficial
• Experience with data-visualisation tools such as Matplotlib, Seaborn, or Plotly is a plus

Benefits

• Fully remote

Company Overview