Note: The job is a remote job and is open to candidates in USA. 24-MAG LLC is offering an independent contractor opportunity to evaluate and improve AI coding systems. The role involves reviewing AI-generated code in real-world repositories, developing evaluation rubrics and infrastructure, and sharing findings with research and engineering teams.

Responsibilities

• Review code produced by AI coding agents
• Assess implementations for correctness, robustness, and maintainability
• Determine whether agents selected appropriate technical approaches
• Identify subtle implementation errors and engineering weaknesses
• Explain clearly why generated solutions succeed or fail
• Design and refine technical evaluation rubrics
• Define criteria for assessing coding-agent performance
• Review and label preference and evaluation data
• Apply consistent quality standards across repeated assessments
• Capture nuanced differences between competing implementations
• Work within substantial real-world software repositories
• Analyse code changes in their broader architectural context
• Evaluate repository-level behaviour rather than isolated snippets
• Review implementation quality using senior engineering judgement
• Identify recurring failure patterns across coding tasks
• Build and maintain pipelines supporting data generation and evaluation
• Improve infrastructure for collecting and reviewing model outputs
• Support scalable evaluation and feedback workflows
• Contribute to reliable processes for repeated technical assessment
• Help translate qualitative engineering judgement into structured systems
• Collaborate with research and engineering teams
• Summarise evaluation findings in clear written reports
• Provide actionable recommendations based on observed model behaviour
• Help improve evaluation methodologies and coding-agent benchmarks
• Communicate complex technical issues in concise, structured language

Skills

• 5+ years of hands-on software engineering experience
• Strong proficiency in Python, TypeScript, JavaScript, Go, Java, or another major production language
• Experience working in substantial real-world codebases
• Strong code-review and technical-analysis skills
• Ability to assess implementation correctness and maintainability
• Strong understanding of modern software-engineering practices
• Experience with GitHub-based development workflows
• Familiarity with CI/CD and production engineering processes
• Comfortable analysing unfamiliar repositories and code changes
• Ability to identify subtle technical issues others may overlook
• Strong written communication and structured technical reasoning
• Experience using modern LLMs or AI-assisted coding tools
• Familiarity with open-source development is valuable
• LLM evaluation or coding-agent experience is advantageous
• Experience with RLHF, preference data, rubric design, or post-training is useful but not required

Benefits

• Remote — North America only
• Flexible commitment of approximately 20–40 hours per week

Company Overview