Senior Staff Software Engineer (AI Platform)
Posted
Description
- As a Senior Staff Software Engineer on the AI Platform team, you will shape and scale Pinterest’s AI/ML evaluation platform into a unified, reliable system adopted company-wide
- You will identify modernization opportunities, elevate the roadmap, and execute technical strategy of a durable evaluation system for GenAI and ML workloads at Pinterest, partnering across engineering, research, product, and safety to improve quality, launch decisions, and trust in AI-powered experiences
- Identify modernization opportunities and scale the roadmap for a company-wide evaluation system in partnership with technical leadership
- Define cross-team technical strategy, align partner teams, and drive broad adoption of shared evaluation capabilities
- Make durable architectural decisions that improve the user experience and reliability of Pinterest’s AI/ML evaluation platform
- Establish evaluation methodologies and standards that balance automated metrics, expert judgment, human feedback, safety, and business outcomes
- Lead complex cross-team workstreams, resolve technical tradeoffs and risks, and deliver impact
- Deep familiarity with GenAI and RecSys evaluation libraries and frameworks, AutoEvals, OpenEvals, Ragas, etc
- 3+ years of deep, hands-on experience designing and operating evaluation systems for GenAI and complex ML systems, including benchmark design, golden datasets, automated metrics, human evaluation, model-based grading, and failure analysis
- Strong programming skills in Python coupled with a solid grasp of distributed systems principles, and comfortable iterating with AI tools on coding and design
- Bachelor’s degree in Computer Science, Engineering, or a related field, or equivalent experience
- Strong judgment around GenAI risks, including robustness, bias, privacy, security, and the limitations of automated evaluation signals
- 5+ years of experience building scalable platforms for distributed experimentation, data and model versioning, orchestration, observability, release management, and production monitoring