Russian Audio Generalist Evaluator Expert (San Francisco Bay Area)
Mercor is seeking a Russian Audio Generalist Evaluator Expert to contribute to a high-impact audio AI research project with a leading research lab. In this role, you will work on transcription, annotation, and evaluation tasks that help train and benchmark advanced language models. This is a short-term, structured engagement ideal for candidates with strong academic or analytical backgrounds who are fluent in Russian and English and are **based in the San Francisco Bay Area**, enabling occasional in-person collaboration if required.
## Job Responsibilities
### Transcribe and Optimise Audio & Video
- Listen to, analyse, and transcribe audio and video content in Russian, following detailed constraints and instructions.
- Produce high-quality written outputs in Russian, with supporting work in English when required.
- Ensure clarity, accuracy, and strict adherence to formatting and stylistic guidelines.
Sourced from Mercor. Apply through our listing to get matched.
What we look for
- Listen to, analyse, and transcribe audio and video content in Russian, following detailed constraints and instructions.
- Produce high-quality written outputs in Russian, with supporting work in English when required.
- Ensure clarity, accuracy, and strict adherence to formatting and stylistic guidelines.
- Capture nuances such as tone, intent, formal vs. informal register, regional expressions, dialectal variations, and contemporary Russian usage where relevant.
- Establish clear expectations for correct and high-quality responses in general consumer audio contexts.
- Develop detailed evaluation rubrics and grading guidelines in Russian and English.
- Document standards to ensure consistency across reviewers and model evaluations.
- Identify linguistic nuances, grammatical complexities, colloquialisms, and edge cases specific to Russian.
- Run prompts through language models and assess generated outputs.
- Evaluate responses against predefined criteria for accuracy, completeness, fluency, and instructional clarity.
- Provide structured feedback to improve model performance in Russian audio tasks.
- Participate in QA and review cycles to ensure tasks, rubrics, and outputs meet Mercor’s quality bar.
How it works
Apply once and complete a short skills assessment. Once approved you are matched to projects that fit your background, set your own weekly hours, and are paid on a regular cycle for the work you complete.