Position: LLM AI Quality Analyst (Personalization) Romanian
Type: Short-Term Contract (1 month)
Compensation: $11.5-$15.3 per hour
Location: Remote
Commitment: Part-time availability required (30-40 hours/week with 4 hours overlap with PST)
Role Responsibilities
- Evaluate a personalization feature for Gemini
- Design and execute multi-turn conversational prompts that require the AI to utilize personal information and experiences
- Assess how effectively the model uses past conversations, Gmail, Google Search, and YouTube activity to generate relevant and helpful responses
- Evaluate model responses based on intent and appropriate personalization
- Analyze responses for grounding issues, including flawed inferences or hallucinations
- Assess integration quality to ensure personal data is incorporated naturally into responses without overnarration
- Perform side-by-side evaluations and stack-rank model responses based on helpfulness, naturalness, and usability
- Write clear rationales referencing specific conversation turns
- Extract and verify debug information to confirm correct use of summaries and data sources
- Maintain data hygiene by deleting evaluation conversations after completion
Requirements
- Experience in data annotation, AI quality evaluation, content moderation, or related roles
- Strong Romanian proficiency (reading and writing)
- BS/BA degree or equivalent experience in a relevant field (Policy, Law, Ethics, Linguistics, Journalism, Computer Science, or similar)
- Willingness to use a primary personal Google account and enable personal data sources for assessment
- Strong analytical thinking and attention to detail
- Experience with creative prompt engineering and personalization concepts
- Ability to evaluate nuanced and ambiguous AI responses
- Ability to provide structured feedback and clear written explanations
- Excellent written communication and collaboration skills
- Ability to work independently in a remote environment
- Desktop or laptop with a stable internet connection
Application Process
- Apply/Easy Apply and check email for application form
- Fill Google form
- Assessment Link (After shortlisting to be completed within 24 hours)
- Language vetting