Research Engineer - AI Benchmarks

Experience: Technical aptitude and learning potential matter more than years Pay: $100k-$200k, full time Location: San Francisco, CA, US or Singapore (on-site) Visa: Relocation and visa support available for strong candidates The role Build high-quality benchmarks for evaluating frontier agents on domain-specific tasks. You will create benchmarks that are technically rigorous, practically useful, and credible to frontier labs. What you'll do Own the design, implementation, and quality of HUD's internal agent benchmarksWork with subject-matter experts to define realistic domain-specific tasksBuild infrastructure to run models and agents reliably against benchmark tasksDevelop metrics and analyses for benchmark difficulty, reliability, and failure modesValidate whether benchmark performance correlates with real-world evaluations, customer needs, and lab expectationsWrite clear documentation and benchmark reports for technical audiences What we're looking for Proficiency in Python, Docker, and Linux environmentsPublished papers or technical blogs about benchmarks, model failure modes, or related topicsA strong understanding of what makes a benchmark realistic, reliable, and usefulExperience with environments and evaluationsCuriosity and the ability to understand how workflows in unfamiliar domains actually workFirst-principles reasoning about task design, scoring, and failure modes Strong signals Detail orientation and an eye for subtle inconsistencies and edge casesIndependent work in unstructured problem spacesEarly-stage startup experienceStrong communication across time zones