LLM Evaluator – Model Response Analyst (Remote)

About the role

We are looking for a detail‑oriented LLM Evaluator to assess and improve the performance of large language models. Working remotely with a global team, you will analyze AI‑generated content for accuracy, coherence, safety, and alignment with our guidelines.

Key responsibilities

• Evaluate and rank model‑generated text using complex rubrics covering factuality, coherence, safety, instruction‑following, and creativity.
• Compare multiple model responses to the same prompt and justify the preferred output.
• Provide concise feedback to modeling and training teams on recurring failure patterns.
• Design adversarial prompts to expose biased, harmful, or insecure model behavior.
• Collaborate with quality‑assurance to refine evaluation guidelines and handle edge cases.
• Participate in cross‑checking sessions to ensure inter‑rater reliability.
• Investigate underperforming outputs, hypothesise root causes, and flag novel behaviors for research.

Required profile

• Minimum 2 years of professional experience in computational linguistics, data analysis, technical writing, NLP/AI quality assurance, or cognitive science.
• Bachelor’s degree in Computer Science or a related field.
• Strong understanding of prompt engineering and the ability to explain why a response is good or bad.
• Experience with Reinforcement Learning from Human Feedback (RLHF) data collection.
• Proven track record of maintaining consistency across evaluation teams and conducting calibration sessions.
• Experience sourcing, cleaning, and annotating datasets for LLM fine‑tuning or evaluation.
• Knowledge of data distribution impacts, A/B testing concepts, and experimental design for model comparison.

Required skills

What we offer

• Fully remote work with flexible hours.
• Opportunity to influence cutting‑edge AI research and product development.
• Collaboration with a diverse, international team of AI experts.