AI Product Operation Analyst

About Us Founded in 2012, ByteDance's mission is to inspire creativity and enrich life. With a suite of more than a dozen products, including TikTok, Lemon8, CapCut and Pico as well as platforms specific to the China market, including Toutiao, Douyin, and Xigua, ByteDance has made it easier and more fun for people to connect with, consume, and create content. Why Join ByteDance Inspiring creativity is at the core of ByteDance's mission. Our innovative products are built to help people authentically express themselves, discover and connect - and our global, diverse teams make that possible. Together, we create value for our communities, inspire creativity and enrich life - a mission we work towards every day. As ByteDancers, we strive to do great things with great people. We lead with curiosity, humility, and a desire to make impact in a rapidly growing tech company. By constantly iterating and fostering an "Always Day 1" mindset, we achieve meaningful breakthroughs for ourselves, our Company, and our users. When we create and grow together, the possibilities are limitless. Join us. Diversity & Inclusion ByteDance is committed to creating an inclusive space where employees are valued for their skills, experiences, and unique perspectives. Our platform connects people from across the globe and so does our workplace. At ByteDance, our mission is to inspire creativity and enrich life. To achieve that goal, we are committed to celebrating our diverse voices and to creating an environment that reflects the many communities we reach. We are passionate about this and hope you are too. Responsibilities About the Team Flow is ByteDance's AI-native innovation business team, focusing on developing products in emerging fields such as LLM. Our product portfolio includes Doubao (China) and Cici (Global). The Model Evaluation Team plays a key role within Flow, providing accurate and timely assessment signals for product iterations. We are responsible for the entire evaluation process, from designing methodology to executing evaluations, identifying issues, and proposing feasible improvement suggestions. Join us to explore innovative opportunities in AI, grow in a dynamic team environment, and create real value for users Responsibilities 1. Evaluation Project Management: Lead execution of evaluation projects on LLM and other GenAI domains with a highly hands-on approach. Apply deep evaluation know-hows and domain expertise to solve sophisticated challenges in LLM capability assessment, and ensure timely and quality delivery of results 2. Data Set Curation: Build human and/or automated evaluation data sets that represent authentic user scenarios across key segments and modalities. Ensure automated evaluation results align with expert judgment and reliably predict online user preferences 3. Evaluation Results Analysis: Deliver evaluation reports with clear insights into model and product performance, converting nuanced case analyses into generalizable patterns. Analyze model performance & user signals to identify gaps in existing evaluation framework and propose actionable optimization directions 4. Cross-Functional Collaboration: Coordinate effectively with internal evaluators and external vendors to ensure project goals are met on time. Communicate effectively across product, engineering and other functions to build consensus around project objectives, methodologies and results Qualifications Minimum Qualifications 1. Bachelor's degree or above in Science, Arts & Humanities, or related fields, demonstrated ability to learn and synthesize knowledge across diverse domains 2. Excellent English writing and critique skills; OR domain expertise in a specific knowledge field (e.g., Science, Humanities) with adequate language critique ability 3. 2+ years of experience in product operations, project management or strategy roles in technology companies 4. Strong critical thinking, numerical, and analytical skills are required 5. Proven project management capabilities with both internal and external stakeholders; comfortable handling ambiguous requirements and adapting to changes while maintaining project momentum 6. Comfortable with meeting deadlines and generating creative solutions, especially Prompt Engineering techniques, to enhance the quality of both individual and team outputs Preferred Qualifications 1. Hands-on experience with LLM, GenAI and search product performance evaluation is strongly preferred 2. A deep interest in LLMs is essential; the ideal candidate is an enthusiastic learner who finds engagement with diverse case studies stimulating