Senior AI Quality Engineer

ECCO Select is a talent acquisition and consulting company specializing in people, process and technology solutions. We provide the talent behind the technology enabling our clients to achieve their goals. For more information about ECCO Select, visit us at www.eccoselect.com. Job Title: Senior Technical Consultant - AI Quality Engineering Location: 100% remote; Overview: We are seeking a Senior Technical Consultant to support the testing, evaluation, and monitoring of AI-powered applications for an automotive client. This person will use Python and LangSmith to assess application quality, trace execution, analyze model responses, and identify reliability and performance issues. The ideal candidate brings strong technical testing skills and practical experience evaluating applications built with large language models. Job Description Responsibilities • Develop and maintain Python-based testing and evaluation workflows for LLM applications. • Use LangSmith for application testing, evaluation, observability, tracing, performance monitoring, and quality validation. • Analyze model responses and application traces to identify quality issues and investigate failures. • Collaborate with developers and stakeholders to communicate findings and recommend improvements. • Document test results, defects, and quality trends. • Support validation of AI application reliability, accuracy, and performance. Required Qualifications • Proficiency in Python, including the ability to write and troubleshoot testing and evaluation code. • Hands-on experience using LangSmith for LLM application testing, evaluation, tracing, observability, or performance monitoring. • Understanding of how LLM-powered applications operate and how to assess their quality. • Strong analytical and troubleshooting skills. • Ability to explain technical findings clearly and collaborate with delivery teams. Preferred Qualifications • Experience designing and executing test strategies for AI applications. • Experience validating retrieval-augmented generation (RAG) applications and agent workflows. • Experience analyzing model responses, identifying recurring quality issues, and evaluating application accuracy and reliability. • Experience building repeatable automated evaluations and comparing results across application changes. ECCO Select is committed to hiring and retaining a diverse workforce. Our policy is to provide equal opportunity to all people without regard to race, color, religion, national origin, ancestry, marital status, veteran status, age, disability, pregnancy, genetic information, citizenship status, sex, sexual orientation, gender identity or any other legally protected category. Veterans of our United States Uniformed Services are specifically encouraged to apply for ECCO Select opportunities. Equal Employment Opportunity is The Law This Organization Participates in E-Verify