GenAI / AgentOps Engineer

Overview RegASK is an AI-driven solution enabling end-to-end intelligent regulatory workflow orchestration: automating regulatory intelligence with curated content and actionable insights, smart documentation and a connected community of over 1,700+ subject matter experts for streamlined regulatory execution globally. RegASK augments Regulatory Affairs teams, combining AI efficiency and community-driven intelligence to streamline compliance across the product lifecycle. The RegASK team is highly diverse, with more than 10 nationalities across 3 continents, entrepreneurial and very agile. We are currently recruiting highly motivated individuals to expand its team and support its ambitious growth plans. If you are a doer who enjoys working across industries and continents, autonomous in your working style, and eager to problem solve, we would love to hear from you. Position overview We are looking for a mid-to-senior GenAI / AgentOps Engineer to join our Data Science team. This role is focused on building and operating LLM-powered agentic systems using frameworks like LangGraph, LangChain, and LangSmith, along with scalable evaluation, observability, and deployment pipelines. You will work cross-functionally with engineering and product teams to design, monitor, and optimize production-grade GenAI applications that leverage embeddings, retrieval, and fine-tuned prompts for high accuracy and reliability. Responsibilities AgentOps & LangGraph: Design and implement agent workflows using LangGraph/LangChain, with strong focus on orchestration, observability, debugging, and evaluation Evaluation Frameworks: Develop automated LLM evaluation pipelines (factuality, robustness, hallucination detection, guardrails) to ensure production readiness and reliability Embeddings & Retrieval: Optimize embedding models, vector store integrations, and RAG pipelines for domain-specific unstructured data Prompt & Workflow Optimization: Maintain prompt pipelines and fine-tune prompts to balance cost, performance, and accuracy in production Data Systems: Architect data ingestion and vectorization pipelines across SQL/NoSQL databases and large-scale unstructured content sources Production Deployment: Lead end-to-end LLMOps/MLOps workflows, from experimentation to deployment, monitoring, and continuous improvement (preferably in Azure: ML Studio, Azure OpenAI, Azure AI Foundry) Research & Innovation: Stay up-to-date with GenAI/AgentOps advances (LangGraph, LangSmith, eval frameworks, retrieval strategies) and bring best practices into production Collaboration & Mentorship: Partner with stakeholders to translate requirements into agentic pipelines, and mentor junior engineers on LangGraph workflows, AgentOps best practices, and evaluation design Requirements Hands-on expertise with LangGraph, LangChain, and LangSmith (or similar agent frameworks) Strong experience with AgentOps: orchestration, monitoring, debugging, and evaluation of LLM-based workflows Proficiency in embedding models, vector databases (Pinecone, Weaviate, FAISS, Mongo Atlas Vector Search, etc.), and retrieval optimization Proven track record deploying agentic GenAI systems into production with robust evaluation/monitoring pipelines Solid engineering background with Python and libraries such as Transformers, FastAPI, spaCy Experience with SQL and NoSQL databases for data modeling and retrieval Strong understanding of LLMOps/MLOps practices (CI/CD, monitoring, scaling, cost optimization) Excellent communication skills, ability to clearly explain evaluation results and system behaviors to technical and non-technical stakeholders Good to have Experience with LLM fine-tuning/adaptation (LoRA, QLoRA, RLHF, DPO) as a complement to embeddings/prompting Experience with graph databases (Neo4j, ArangoDB, TigerGraph) for hybrid RAG pipelines Familiarity with domain-specific ontologies or knowledge graphs for structured + unstructured reasoning Knowledge of continuous evaluation & A/B testing frameworks for agents in production Strong knowledge of data streaming frameworks (Kafka, Flink, Spark) for real-time retrieval & evaluation Experience with AI safety, interpretability, explainability or responsible AI practices What We Offer Flexible working arrangements (hybrid) Opportunity to work in a high impact role at the intersection on AI, SaaS and Compliance/ Regulatory intelligence Continuous learning and professional development How to Apply If you are excited about this opportunity and believe you have the skills and qualifications to excel as our GenAI / AgentOps Engineer, please submit your resume for our consideration. We appreciate all applications, but only selected candidates will be contacted for an interview. Thank you for considering joining the RegASK team. We look forward to reviewing your application #J-18808-Ljbffr