Senior ML Data Linguist
Feb 2026 – PresentServiceNow, LUNAR team (Linguistic Unit for NLU Analysis & Research)
- Co-designed a 10-dimension annotation schema that breaks TTS “naturalness” into word-level, prosodic, and paralinguistic dimensions, and built an 860-utterance benchmark labeled by trained linguist raters (EMNLP 2026, co-first author).
- Compared eight automated speech evaluators (four MOS predictors, four audio-LLM judges) against human labels under four prompting conditions, MOS predictors mostly tracked acoustic signal quality, and audio-LLM judges caught some error types depending on the prompt. Neither reliably detected the full range of linguistic errors.
- Designed and ran a multilingual TTS vendor comparison. Wrote test sentences drawn from ITSM and CSM language and text normalization edge cases (symbols, math, units, URLs), defined a rubric pairing holistic 1–5 ratings with per-category head-to-head judgments, and adapted it for Japanese with a native-speaker linguist.
- Built a rating framework for AI voice agent conversations covering speech fluency, tool calling, task completion, and helpfulness, and wrote its annotation guidelines.
- Ran the labeling efforts that produced human benchmarks for customer-facing voice fluency judges: wrote the guidelines, onboarded labelers, and presented results across agents, separating ASR-driven failures (such as repeated confirmations) from conversation-flow and voice-style problems.
- Leading success labeling for document question answering and building the matching LLM judge.
- Built and maintain the internal labeling tools used on these projects, and the team’s project-tracking app.