Note: The job is a remote job and is open to candidates in USA. SoTalent is seeking a Data Scientist to lead evaluation, benchmarking, and reliability efforts for an AI copilot embedded in a trading platform. The role builds quality frameworks, testing and monitoring systems, and improvement processes to support safe, reliable AI behavior in financial workflows.

Responsibilities

• Design and build evaluation frameworks for AI-powered trading assistants
• Establish quality metrics for:
• Correctness
• Accuracy
• Safety
• Reliability
• Latency
• Hallucination Risk
• Regression Detection
• Create repeatable evaluation methodologies that measure AI performance across financial workflows
• Define acceptance criteria and release quality standards for production AI systems
• Develop and maintain benchmark datasets and evaluation suites
• Create:
• Golden Datasets
• Scenario-Based Test Suites
• Adversarial Test Cases
• Edge Case Libraries
• Market Regime Validation Sets
• Continuously refresh benchmark corpora to reflect evolving market conditions and trading scenarios
• Validate AI performance across diverse trading and portfolio management use cases
• Build automated quality gates that prevent releases when critical metrics degrade
• Implement regression testing frameworks for LLMs, AI agents, retrieval systems, and tool-using assistants
• Evaluate model updates, prompt changes, workflow modifications, and tool integrations before deployment
• Establish testing methodologies for production AI systems operating in regulated financial environments
• Own AI improvement cycles through structured evaluation and experimentation
• Develop:
• Error Taxonomies
• Data Collection Strategies
• Labeling Frameworks
• Quality Assurance Processes
• Identify root causes of model failures and implement corrective improvements
• Support:
• Prompt Engineering
• Fine-Tuning
• Preference Optimization
• Retrieval Improvements
• Tool-Use Optimization
• Measure and validate performance improvements against benchmark standards
• Partner with Product and Engineering teams to establish safe AI interaction patterns
• Design safeguards for:
• Trading Recommendations
• Order Placement Workflows
• Portfolio Analysis
• Risk Management Functions
• Support deterministic previews, user confirmations, auditability, and safe action execution
• Ensure AI outputs are predictable, explainable, and trustworthy
• Build monitoring and observability solutions for AI systems
• Establish telemetry, alerting, and operational dashboards
• Detect performance degradation, reliability issues, and production anomalies
• Lead root cause investigations and incident response activities
• Develop "fix-forward" processes to rapidly remediate AI-related production issues
• Develop expertise in:
• Trading Workflows
• Portfolio Management
• Order Execution
• Margin Accounts
• Short Selling
• Risk Management
• Portfolio Margin
• Market Structure
• Ensure AI systems accurately communicate financial concepts and trading risks
• Validate AI-generated explanations and recommendations within financial contexts
• Partner with:
• AI Engineers
• Software Engineers
• Product Managers
• Quantitative Teams
• UX Teams
• Platform Engineers
• Influence product decisions using evaluation data and model performance insights
• Support development of safe and intuitive AI-powered user experiences
• Drive continuous improvement in AI engineering practices

Skills

• 8+ years of experience delivering production software
• Strong software engineering fundamentals and systems design expertise
• Experience building:
• Evaluation Frameworks
• Testing Infrastructure
• Benchmark Suites
• Automated Quality Systems
• Experience working with:
• LLMs
• AI Agents
• Search Systems
• Retrieval Systems
• Recommendation Systems
• Ranking Systems
• Proven experience improving model performance through structured experimentation
• Strong debugging, troubleshooting, and analytical problem-solving skills
• Ability to work across backend systems, AI infrastructure, and product workflows
• Highly self-motivated with strong ownership and execution capabilities
• Large Language Models (LLMs)
• Agentic AI
• AI Evaluation
• Prompt Engineering
• Model Benchmarking
• Fine-Tuning
• RLHF / Preference Optimization
• Retrieval-Augmented Generation (RAG)
• Systems Design
• Distributed Systems
• API Development
• Production Software Development
• Testing Frameworks
• Performance Engineering
• AI Monitoring
• Observability
• Telemetry
• Incident Response
• Model Performance Tracking
• Evaluation Pipelines
• Experimental Design
• A/B Testing
• Benchmark Analysis
• Error Analysis
• Metrics Frameworks
• Data Labeling Pipelines
• Trading Platforms
• Portfolio Analysis
• Market Data
• Order Management
• Risk Analytics
• Financial Workflows
• Rust
• TypeScript
• PostgreSQL
• React
• React Native
• LLM APIs
• Model Serving Platforms
• AI Evaluation Platforms
• AI Evaluation
• LLM Quality Assurance
• Agentic AI
• Benchmark Development
• AI Safety
• Prompt Engineering
• RAG
• Testing Frameworks
• MLOps
• Observability
• Production AI Systems
• Trading Technology
• Quantitative Analysis
• Systems Design
• Rust
• TypeScript
• PostgreSQL
• Model Optimization
• Incident Management
• AI Governance

Benefits

• Remote work (US)

Company Overview