Position: LLM – AI Quality Analyst (Personalization) – Turkish
Type: Short-Term Contract
Location: Remote
Commitment: 30–40 hours/week, 4-hour overlap with PST
Engagement Length: 2 months
Start Date: Immediate
Role Responsibilities
- Design multi-turn conversational prompts (1–5 turns) using personal context
- Evaluate personalized AI responses for grounding, integration, and helpfulness
- Assess whether personalization is applied correctly and naturally
- Identify flawed inferences, hallucinations, and incorrect personalization
- Review integration quality to ensure responses are not robotic or over-narrated
- Conduct side-by-side (SxS) evaluation and ranking of model responses
- Analyze subtle differences in tone, clarity, and usefulness
- Write clear, structured rationales referencing specific conversation turns
- Extract and verify “Debug Info” to confirm correct usage of data sources
- Maintain strict data hygiene by deleting evaluation conversations
Requirements
- Turkish proficiency (reading and writing)
- Strong experience in data annotation, AI quality evaluation, content moderation, or related roles
- Strong analytical thinking for evaluating nuanced and ambiguous AI responses
- Experience with creative prompt engineering and multi-turn conversations
- Understanding of personalization concepts and evaluation methodologies
- High attention to detail for SxS comparisons
- Excellent written communication and feedback documentation skills
- BS/BA degree or equivalent experience in a relevant field
- Willingness to use a primary personal Google account with enabled personal data sources
- Self-motivated and able to work independently in a remote environment
- Desktop/laptop with reliable internet connection
Application Process
- Fill out the application form
- Complete the ICF
- Complete the assessment