返回职位列表

Expert Talent Pool, Evals and Benchmarks (Associate)

🌍 全球 · 全职 · 2 天前 发布

职位描述

About the Role

Join a team that designs and runs human-led evaluations of AI models. You will help measure model capabilities by creating rigorous questions, rubrics, and comparative assessments, contributing to better understanding of AI performance.

What You'll Do

  • Write questions and tasks that test specific model capabilities.
  • Develop rubrics that define strong, partial, and weak answers.
  • Create reference answers and edge-case scenarios to challenge models.
  • Grade model outputs against rubrics and explain each score clearly.
  • Analyze incorrect answers, group error patterns, and document findings.
  • Compare multiple models using the same prompts.
  • Validate benchmarks and flag questions that may no longer provide reliable measurements.
  • Support small studies of how people evaluate or interact with model outputs.

What We're Looking For

  • Zero to five years of professional experience; prior AI-data work is not required.
  • Experience or training in QA, test engineering, assessment design, psychometrics, UX research, data analysis, ML evaluation, research support, or a subject area involving rigorous question writing.
  • Experience writing test cases, exam questions, rubrics, or evaluation tasks, and reviewing work consistently against written criteria.
  • Ability to use spreadsheets or Python to organize and analyze results, and to identify and document error causes.
  • Experience with evaluation tools such as Inspect, lm-eval-harness, HELM, or promptfoo, model comparisons, or user and rating studies is useful.
  • Clear, professional written communication.

Compensation & Benefits

Compensation is $30 to $60 per hour, depending on project, field, and experience. Payment is for hours worked.

Location

Fully remote. Projects typically run from one week to several months.

Originally posted on Himalayas

前往申请