AI Product Management and Operation

AI Product Management and Operation (Evaluation) - Flow Flow is ByteDance's AI-native innovation business team, focusing on developing products in emerging fields such as LLM. Our product portfolio includes Doubao (China) and Cici (Global). The Model Evaluation Team plays a key role within Flow, providing accurate and timely assessment signals for product iterations. We are responsible for the entire evaluation process, from designing methodology to executing evaluations, identifying issues, and proposing feasible improvement suggestions. Join us to explore innovative opportunities in AI, grow in a dynamic team environment, and create real value for usersResponsibilities Evaluation Framework Establishment: Design evaluation frameworks for the Cici App including standards, test cases, and workflows. Create metrics that measure performance across key dimensions and enable data-driven decisions in existing and upcoming use scenarios. Human-Machine Evaluation Integration: Build workflows combining human expert and automated evaluation. Ensure automated processes align with human judgment, identify performance gaps, and increase testing coverage to supplement human evaluation. Evaluation Results Analysis: Interpret and analyze evaluation findings with depth, transforming nuanced case analysis into actionable insights. Develop data production and product design strategies aimed at enhancing model capabilities and/or user experience. Cross-functional Collaboration: Align product, engineering and evaluation teams across cultural and professional boundaries. Facilitate communication, resolve differences, and build consensus around overall evaluation methodologies. Evaluation Methodology Research: Stay at the forefront of LLM evaluation by researching industry-leading assessment methodologies and reports. Integrate insights with practical applications to continuously improve our evaluation framework. Qualifications Bachelor's degree or above in Computer Science, Data Science, Business Analytics, or related fields. 3+ years of experience as a Product Manager, Business Analyst, or Operations Specialist in technology companies or other highly dynamic environments. Strong data analysis skills with experience in designing metrics for product performance. Project management capabilities with the ability to balance priorities and handle ambiguous requirements. Quick adaptation to new tools, technologies and concepts, with demonstrated learning agility. Genuine passion for AI/LLM technologies. Preferred: Experience in AI/ML product management, particularly with conversational AI. Hands‐on experience with LLM evaluation methodologies. Background in developing metrics frameworks for measuring product performance. Excellent stakeholder management and communication skills. About Us Founded in 2012, ByteDance's mission is to inspire creativity and enrich life. With a suite of more than a dozen products, including TikTok, Lemon8, CapCut, Pico, Toutiao, Douyin, and Xigua, ByteDance has made it easier and more fun for people to connect, consume, and create content. Why Join ByteDance Inspiring creativity is at the core of ByteDance's mission. Our innovative products are built to help people authentically express themselves, discover and connect — and our global, diverse teams make that possible. Together, we create value for our communities, inspire creativity and enrich life—a mission we work towards every day. As ByteDancers, we strive to do great things with great people. We lead with curiosity, humility, and a desire to make impact in a rapidly growing tech company. By constantly iterating and fostering an "Always Day 1" mindset, we achieve meaningful breakthroughs for ourselves, our Company, and our users. When we create and grow together, the possibilities are limitless. Diversity & Inclusion ByteDance is committed to creating an inclusive space where employees are valued for their skills, experiences, and unique perspectives. Our platform connects people from across the globe and so does our workplace. At ByteDance, our mission is to inspire creativity and enrich life. To achieve that goal, we are committed to celebrating our diverse voices and to creating an environment that reflects the many communities we reach. Seniority Level Mid-Senior level Employment Type Full-time Job Function Product Management and Marketing Industries Technology, Information and Internet #J-18808-Ljbffr