Note: The job is a remote job and is open to candidates in USA. Netflix is hiring a Program Manager to design and execute human evaluation programs that measure and improve recommendation quality across its homepage and discovery experience. The role partners with Product Management and cross-functional teams to build evaluation frameworks, manage vendors and raters, and deliver consistent, scalable evaluations.

Responsibilities

• Lead end-to-end execution of human evaluation and data operations initiatives, from intake and scoping to delivery
• Partner with Product Management to translate recommendation strategy and roadmap priorities into evaluation designs, success metrics, and timelines
• Manage human annotation vendors, including capacity planning, SLAs, quality performance, and escalations
• Drive operational efficiency across evaluation programs, including cost per task, throughput, turnaround time, and speed to IRR
• Develop clear, scalable guidelines and scoring rubrics to ensure consistent rater judgment, including for subjective, taste-based questions
• Oversee rater onboarding, calibration, and QA workflows
• Pilot and refine evaluation tasks to improve clarity, inter-rater reliability, and feedback quality
• Anticipate and resolve bottlenecks and blockers
• Act as the connective tissue across product, ML, data science, and vendor partners to ensure alignment and effective execution of evaluations at scale

Skills

• 4+ years of experience working in human evaluations, data collection, labeling, or annotation operations in ML environments
• Track record of implementing process improvements or quality control systems for data collection needs
• Prior experience managing human annotation vendors, raters, or data labeling teams, including vendor performance and cost management
• Experience improving operational efficiency of annotation programs (throughput, turnaround, cost)
• Strong understanding of evaluation design, including guidelines, rubrics, and scoring protocols
• Working fluency in film and TV content, including genres, formats, and talent
• Proven ability in end-to-end management of complex, cross-functional programs, demonstrating strong Program Management skills and clear accountability for successful delivery
• Experience with human labeling platforms
• Excellent written and verbal communication skills
• Ability to synthesize feedback into clear recommendations and process improvements
• Familiarity with how rater subjectivity and bias affect taste-based judgments, and how to account for them in evaluation design
• Strong organizational skills and executional focus; ability to track details while seeing the bigger picture
• Experience with personalization, recommender systems, search, or ranking evaluation
• Familiarity with how offline human evaluation complements online experimentation (A/B tests) and engagement metrics
• Experience evaluating at the list or page level, not only the individual item level
• Experience partnering with Product Managers and Data Scientists on metric definition

Benefits

• Health Plans
• Mental Health support
• A 401(k) Retirement Plan with employer match
• Stock Option Program
• Disability Programs
• Health Savings and Flexible Spending Accounts
• Family-forming benefits
• Life and Serious Injury Benefits
• Paid leave of absence programs
• Full-time hourly employees accrue 35 days annually for paid time off to be used for vacation, holidays, and sick paid time off.
• Full-time salaried employees are immediately entitled to flexible time off.
• Remote work

Company Overview

Company H1B Sponsorship