Senior Automation Test Engineer - AI Systems

We are looking for a Senior QA Engineer to join our team and contribute to building an AI-powered operating system for healthcare marketing. In this role, you will go beyond traditional end-to-end testing, owning quality, evaluation, and monitoring for complex AI systems. You will work on real-world production-grade products that create monetizable value, ensuring the stability of our full-stack architecture while designing and scaling AI evaluation pipelines and ML drift monitoring systems. If you are passionate about developing robust test and evaluation frameworks for modern systems powered by AI and ML, we want to hear from you. Your tasks Designing, building, and maintaining evaluation frameworks for Large Language Models (LLMs) and custom agentic harnessesDeveloping automated scoring mechanisms, assertions, and test suites to measure agent accuracy, reasoning reliability, hallucination rates, and tool-use correctnessSimulating complex, multi-step agent workflows to uncover edge cases in autonomous decision-makingCollaborating with engineering teams to set up robust monitoring frameworks for data drift, concept drift, and model performance degradation in productionImplementing telemetry and alerting systems to track latency, token usage, cost efficiency, and output quality over timeOwning the end-to-end testing strategy across our entire stack, ensuring seamless integration between the Angular frontend, Node.js backend services, and Python-based AI microservicesWriting clean, maintainable, and scalable automated test scripts in Python and JavaScript/TypeScriptPerforming rigorous API testing, integration testing, and performance/load testing for high-throughput microservicesIntegrating AI evaluation and testing harnesses into our CI/CD pipelines to ensure continuous quality gate enforcement before deploymentDefining and tracking key quality metrics (KPIs) for software reliability and AI model behavior Requirements Minimum 5 years of experience in software testing or quality engineering, with at least 2 years focused on testing or evaluating AI/ML applications, LLM workflows, or agentic systemsProficiency in Python for backend/AI testing scripts and familiarity with JavaScript/TypeScript for Node.js and Angular testing ecosystemsExperience with evaluation frameworks such as LangSmith, Langfuse, EvidentlyAI, Ragas, or MLflow, and the ability to select appropriate metrics for classification or generation-quality problemsFamiliarity with ML drift monitoring concepts, metrics tracking, and observability tools like Datadog or GrafanaStrong system-level design skills and the ability to build production-grade test and evaluation infrastructure from the ground upAdvanced level of English Nice to have Understanding of agentic design patterns and their decomposition into testable unitsExperience working under the regulatory conditions of healthcare systems Job no. JOB-GTXQF Sii ensures that all hiring decisions are made solely on the basis of qualifications and competence. We are committed to equal and fair treatment of all, regardless of legally protected characteristics. At Sii, we promote a diverse and inclusive work environment, in full compliance with applicable anti-discrimination laws. Benefits For You Great Place to WorkSolid financial situationContracts with the biggest brandsCentre of internal trainingsMany experts you can learn fromOpen and accessible management teamProfit sharingPassion Sponsorship programRegular integration events and tripsComfortable and well-equipped officesMySii appMedical care