HHiring Reality
← OpenTeams

Senior AI/ML Test and Evaluation Engineer

OpenTeams
Location
Washington, DC Metro | Denver, CO Metro | Colorado Springs, CO - Hybrid/Remote
Posted
18 days ago
Department
Technology
Salary
$145,000–$250,000
What they actually want (must-haves)
  • U.S. citizenship and eligibility to obtain and maintain a U.S. security clearance
  • 6+ years of software engineering or machine learning engineering experience, including 3+ years evaluating, benchmarking, or deploying ML models in production or applied research environments
  • Strong Python proficiency in a machine learning or data science context
  • Hands-on experience with common ML frameworks and tooling, such as PyTorch and the Hugging Face ecosystem
  • Experience developing or using model evaluation harnesses, benchmark suites, or test and evaluation frameworks
  • Experience designing evaluation metrics and applying appropriate statistical rigor when interpreting and reporting results
Nice to have
  • Active U.S. security clearance
  • Prior AI/ML evaluation or test and evaluation experience supporting the Department of Defense, Intelligence Community, or another federal customer
  • Experience designing human-in-the-loop evaluations, measuring inter-rater reliability, or facilitating structured expert adjudication
  • Experience defining or implementing benchmark interchange formats or evaluation standards used across multiple organizations
  • Familiarity with intelligence analysis workflows or other high-stakes analytical domains
What the job really is

The Senior AI/ML Test and Evaluation Engineer will focus on building and operating benchmarking and evaluation capabilities for AI models. This role involves designing evaluation methodologies, creating automated metrics, and documenting limitations and failure modes of models. The engineer will also produce evaluation reports for senior stakeholders and support integration with partner organizations.

Benefits
  • Medical, Dental & Vision – 100% paid for employees, 75% for dependents
  • 401(k) Match – Up to 5% with full vesting after 2 years
  • Unlimited PTO – With a required minimum of 15 days off annually
  • Fully Remote Setup – Includes up to $3,000 equipment reimbursement
  • Continuous Education – Includes up to $500 reimbursement
  • Disability & Life Insurance – 100% employer-paid
Things to weigh
  • Role is contingent upon contract award
  • Travel of up to 15% may be required, primarily to Government facilities and between company locations
  • Focus on evaluating what models get wrong rather than what they get right may not appeal to all candidates
  • Hands-on engineering with an open-source toolchain may require adaptability to evolving technologies
Job score4.4/5
Benefits4/5
Freshness4/5
Career value4/5
Role clarity5/5
Pay transparency5/5

Applying to OpenTeams?

See how your résumé matches this role — and tailor it from what actually gets interviews.

Calculate resume scoreGenerate resume to apply
Job Link →