Senior AI Developer
Posted Updated
Description
- We’re looking for a highly skilled and experienced Senior AI Engineer to join our ClickUp Engineering team, with a focus on AI evaluations
- You’ll own how we measure and improve the quality of ClickUp’s AI: building the eval systems that tell us how well our AI is performing, the core agent harness, and the AI that performs well on those evals
- You’ll work on backend systems that make AI-powered capabilities scalable, reliable, and secure, while working hands-on with LLMs to solve real user problems and drive product innovation
- Design, build, and own ClickUp’s AI evaluation system, including golden datasets, LLM-as-judge and automated scorers, regression detection, and quality tracking across AI features
- Build and evolve the core agent harness, and the AI that performs well on those evals, partnering closely with product and engineering to define what “great” looks like
- Build self-improvement loops where agents modify other agents or prompts, rerun evals, and iterate automatically
- Benchmark new models as they’re released and own model selection, routing, and fallback strategies across providers (e.g., OpenAI, Anthropic, Google, open-weight), balancing quality, latency, and cost
- Architect, design, and implement scalable AI platform services that support the deployment, orchestration, and lifecycle management of LLMs and other AI models
- Build and maintain robust APIs and backend systems that integrate AI-powered features into ClickUp’s core platform
- Develop infrastructure for model serving, monitoring, logging, and automated evaluation to ensure high reliability and performance of AI services in production
- Drive best practices in AI privacy, security, and compliance, including data anonymization, secure data handling, and regulatory adherence
- Collaborate cross-functionally with product, frontend, and data teams to deliver seamless, reliable, and user-centric AI experiences
Benefits
- 16” MacBook Pro
- Moving Expenses
- $200/mo commuter benefits
- Team lunches each week
- 401(k) match up to 2%
- Unlimited snacks, kombucha, and cold brew
- Health insurance
- Dental insurance
- ClickUp swag
- Teammate recognition rewards
- Flexible PTO
- Passion for staying at the forefront of AI and applying new technologies to solve real-world problems at scale
- Proven track record of applying LLMs and AI models to real-world, customer-facing product features
- Familiarity with search and retrieval technologies and their integration into AI-driven applications
- Proficiency in orchestration frameworks and workflow engines (e.g., LangGraph, Airflow, Ray, or similar)
- Deep expertise in backend engineering, distributed systems, and cloud-native technologies (e.g., Kubernetes, Docker, AWS/GCP/Azure)
- Hands-on experience building LLM evaluation systems in production (golden datasets, LLM-as-judge, judge calibration, offline and online evals), tied to real product quality
- Proven experience integrating and managing multiple LLMs, including benchmarking, routing, and cost/latency/quality tradeoffs
- Strong programming skills in Python (preferred), Go, TypeScript, or similar languages used for backend and AI platform development
- Strong experiment design and statistical rigor, including A/B testing, offline vs. online metrics, and separating real regressions from noisy or variable results
- 5+ years of software engineering experience, including 2+ years building production LLM or ML systems
- Excellent collaboration and communication skills, with a track record of working effectively in cross-functional teams
- Background in data science, applied science, or machine learning engineering
- Experience with eval tooling (e.g., Braintrust, LangSmith, Weave, Arize, or in-house frameworks)
- Experience with model routing, AI gateways, or inference optimization
Skills
Sign in to read and join the discussion.