Note: The job is a remote job and is open to candidates in USA. 24-MAG LLC offers a part-time consulting opportunity for an experienced Big Data Engineer supporting an advanced AI development project. The role focuses on designing and maintaining scalable data pipelines and distributed systems, developing Python-based processing and integration workflows, and optimizing databases and storage.

Responsibilities

• Design, build, and maintain scalable data pipelines and big-data architectures
• Develop reliable workflows for ingesting, transforming, processing, and delivering large volumes of data
• Translate technical and business requirements into robust data-engineering solutions
• Apply appropriate architectural patterns for distributed and high-volume data systems
• Ensure data workflows remain scalable, maintainable, and suitable for production-oriented environments
• Develop data-processing, automation, and integration solutions using Python
• Implement ETL and data-transformation workflows across structured and unstructured data sources
• Work with distributed processing technologies and modern big-data frameworks
• Build reproducible and efficient programmatic workflows for large-scale data operations
• Identify opportunities to improve processing efficiency, reliability, and maintainability
• Develop, manage, and optimise relational, NoSQL, distributed database, and storage systems
• Apply data-modelling and data-warehousing principles to support scalable analytical and operational workloads
• Monitor data-system performance and identify bottlenecks, failures, and reliability issues
• Troubleshoot performance, availability, and integration problems across distributed data environments
• Optimise storage, queries, and processing workflows for efficiency and resilience
• Apply appropriate data-quality, security, and governance standards across engineering workflows
• Collaborate with technical and cross-functional stakeholders to understand data requirements and delivery priorities
• Document architectures, data flows, implementation decisions, and technical trade-offs
• Communicate complex engineering concepts clearly to both technical and non-technical audiences
• Support consistent engineering standards across remote and distributed project environments

Skills

• Proven professional experience in big-data engineering with hands-on responsibility for large-scale data pipelines and architectures
• Advanced proficiency in Python for data processing, automation, integration, and related engineering workflows
• Deep understanding of relational and NoSQL databases, including optimisation, management, and performance considerations
• Practical experience with distributed data-processing frameworks such as Hadoop, Spark, Flink, or comparable technologies
• Strong foundation in data modelling, ETL processes, data integration, and data-warehousing principles
• Ability to troubleshoot distributed systems, database performance, pipeline failures, and data-quality issues
• Excellent written and verbal communication skills
• Ability to explain technical concepts clearly to both engineering and non-technical stakeholders
• Strong attention to detail, initiative, and ability to work independently in remote environments
• Experience supporting fast-paced, startup-like, or globally distributed teams is advantageous
• Familiarity with cloud-based data platforms such as AWS, GCP, or Azure is advantageous
• Exposure to machine-learning operations, data-science workflows, or adjacent AI infrastructure is beneficial

Benefits

• Fully remote

Company Overview