About the role
structured by ORI← All rolesMember of Technical Staff (Evals)Make our AI Employees measurably better over time.Apply on Work at a StartupLocationSan FranciscoTeamEngineering, EvalsJob typeFull-timeExperience3+ years of experienceVisaUS citizen or visa holder onlyAbout JarminJarmin builds AI Employees for human-level roles. An AI…
What you will do
- Evaluation systems: datasets, offline replay, scorers, and regression alerts
- Feedback loops for models and agents
- Autonomous failure analysis
- Defining what good, degraded, and failed runs look like, and alerting on them
What they are looking for
- 3+ years of experience
- US citizen or visa holder only
- Have built evaluation or measurement systems, such as AI evals, experimentation, ranking, or search quality
- Can turn ambiguous quality questions into concrete metrics and decisions
- Have strong software engineering fundamentals and ship production systems
- Are comfortable where there are few established patterns
- Want to work in person in San Francisco
Nice to have
- You've shipped LLM systems to production and learned where they break
- You have strong data instincts and work well with researchers
- You contribute to open source projects
Before you apply
- Applications are handled on Work at a Startup.
Full posting text
← All rolesMember of Technical Staff (Evals)Make our AI Employees measurably better over time.Apply on Work at a StartupLocationSan FranciscoTeamEngineering, EvalsJob typeFull-timeExperience3+ years of experienceVisaUS citizen or visa holder onlyAbout JarminJarmin builds AI Employees for human-level roles. An AI Employee learns its responsibilities, understands your data, embeds into your systems, and becomes part of your team's day-to-day workflow.
Companies hire AI ML Engineers, Account Executives, Paralegals, Financial Analysts, Compliance Analysts, and more.Our founding team is from Meta Superintelligence Labs (Staff Engineers), Apple (Machine Learning), AWS (scaling infrastructure), Lockheed Martin (Research), and JPMorganChase, with experience building AI and ML systems and automating the complex work of researchers, data scientists, and ML engineers.We are a Y Combinator Fall 2025 company in San Francisco.About the roleAs a Member of Technical Staff on Evals, you'll own how we measure AI Employee performance on the job.Areas you may work inEvaluation systems: datasets, offline replay, scorers, and regression alertsFeedback loops for models and agentsAutonomous failure analysisDefining what good, degraded, and failed runs look like, and alerting on themYou may be a good fit if youHave built evaluation or measurement systems, such as AI evals, experimentation, ranking, or search qualityCan turn ambiguous quality questions into concrete metrics and decisionsHave strong software engineering fundamentals and ship production systemsAre comfortable where there are few established patternsWant to work in person in San FranciscoEven betterYou've shipped LLM systems to production and learned where they breakYou have strong data instincts and work well with researchersYou contribute to open source projectsHow we workOur core team is in person, resourceful, and building the future we want to live in.We give feedback early and often, so you always know where you stand and where you're headed.Interested?
Applications are handled on Work at a Startup. If you have a question before applying, write to careers@jarmin.ai.Apply on Work at a Startup