Note: The job is a remote job and is open to candidates in USA. Randstad Digital Americas is recruiting for a Senior Product Quality Analyst for an unnamed client. The role leads human annotation projects, develops annotation frameworks and guidelines, manages vendors, and improves annotation quality to support evaluation and fine-tuning of AI and LLM-based products.

Responsibilities

• Lead end-to-end execution of human annotation projects, from intake and scoping to delivery
• Maintain a roadmap of annotation needs, prioritizing requests and coordinating the appropriate internal or external annotation resources
• Manage vendors and coordinate contracts and related reviews with legal, security, privacy, finance, and accounting
• Develop and maintain reusable annotation frameworks and guidelines that can be adopted across teams, and consult on annotation task and rubric design
• Run feedback sessions with modeling teams to identify model strengths, failure patterns, and evaluation gaps
• Evaluate annotation quality, identify sources of disagreement, and work with teams and vendors to improve guidelines and outputs

Skills

• Background in product quality, evaluation, data operations, or related roles
• Hands-on experience designing, coordinating, or reviewing data annotation workflows for product development
• Familiarity with ML-driven products and human evaluation of AI/LLM-powered features
• Ability to translate product and modeling needs into clear annotation guidelines and evaluation rubrics
• Skilled at managing vendors and coordinating delivery across internal teams
• Excellent communication and stakeholder management skills, including setting expectations and resolving ambiguous requirements

Benefits

• Medical, prescription, dental, vision, AD&D, and life insurance offerings (based on eligibility)
• Short-term disability (based on eligibility)
• A 401K plan (based on eligibility)

Company Overview

Company H1B Sponsorship