Note: The job is a remote job and is open to candidates in USA. Ellipsis Health is hiring a Senior QA Engineer, Forward Deployed, to ensure the reliability, stability, and quality of its conversational AI product across client workflows. The role focuses on automated testing, prompt validation, root cause analysis of LLM behavior, and debugging production voice AI deployments in collaboration with engineering and client teams.

Responsibilities

• Workflow Mapping & Test Case Generation: Deeply analyze assigned client workflows to design robust, comprehensive positive and negative test cases that safeguard system stability
• AI-Driven Test Automation: Build and execute automated test scenarios by configuring shadow agents
• Prompt Evaluation & Optimization: Apply a strong understanding of prompt awareness to draft, refine, and evaluate prompts used within the testing framework to accurately simulate user behaviors and edge cases
• End-to-End Testing Execution: Strategically deploy specific testing methodologies including Sanity, Smoke, Regression, and Functional testing - determining the exact environment (staging, pre-production, production) and timing for each execution
• Deployment Cadence & Cross-Functional Collaboration: Partner closely with engineering teams during release cycles to proactively identify, triage, and unblock technical roadblocks, ensuring the product is continuously deployment-ready
• Daily LLM Defect RCA: Perform rigorous, daily root cause analysis on LLM-specific failures inherent to generative AI, including hallucinations, high latency, and logic deviations
• Live Production Call Debugging: Investigate live customer calls and production incidents in real time to unblock critical production use cases
• Audio & Transcription Validation: Query and analyze historical call transcripts, system behaviors, and audio data pipelines to pinpoint where a conversational workflow broke down
• Speech-to-Speech (S2S) Pipeline Monitoring: Monitor and evaluate the end-to-end voice AI pipeline. This involves analyzing Automatic Speech Recognition (ASR) accuracy, managing audio-to-text latency issues, and understanding general Speech-to-Speech mechanics alongside the stability of the core Knowledge Base feeding the AI
• Advanced Evaluation Frameworks: Maintain a strong conceptual understanding of advanced LLM evaluation paradigms and tools such as LLM-as-a-judge - to remain aware of how AI response quality and accuracy are programmatically graded at scale
• Telephony & Call Flow Awareness: Possess a foundational understanding of real-world call management and telephony routing concepts, including how the system is expected to navigate warm transfers, blind transfers, and voicemail detection workflows

Skills

• Strong background in software quality assurance, with a proven track record of designing, executing, and managing end-to-end test strategies (Smoke, Sanity, Regression, Functional)
• Hands-on experience or deep technical familiarity with troubleshooting LLM behaviors, diagnosing hallucinations, managing latency, and using LLM call-tracing tools
• Ability to comfortably write SQL queries (specifically PostgreSQL) to pull data logs and navigate cloud infrastructure logs (such as GCP) to perform rapid root-cause analysis
• Foundational understanding of Speech-to-Speech pipelines, including Automatic Speech Recognition (ASR), audio-to-text workflows, and core knowledge base integrations
• Basic familiarity with enterprise telephony routing, call management mechanics (warm/blind transfers), and voicemail detection systems
• Strong communication skills and the professional agility required to manage UAT timelines, coordinate with client stakeholders, and support rapid production deployments

Benefits

• 401(k) matching
• Health, vision, and dental insurance
• Very flexible paid time off

Company Overview

Company H1B Sponsorship