Senior Data Scientist, Customer Experience and Business Trends
Posted Updated
Description
Amazon's Customer Experience and Business Trends (CXBT) organization is hiring a Senior Data Scientist for its Benchmarking, Economics, Analytics and Measurement (BEAM) team. BEAM's mission is to improve customer experience across every Amazon business by turning fragmented internal and external data into decision-grade intelligence for senior leaders.
This role spans two of BEAM's core products. The Topline View of Retail is an always-on competitive intelligence capability that fuses third-party transaction panels with Amazon's internal signals to answer how Amazon is winning or losing share of wallet, across segments, categories and time. The Intelligence Engine is an LLM-powered analytics assistant that lets teams across Amazon ask business questions in natural language and get trustworthy, data-backed answers. You will not just analyze metrics. You will decide which metrics should exist and build the models that generate, forecast and explain them. And as the Intelligence Engine scales, you will own how we measure its quality and drive its accuracy forward: defining evaluation standards, diagnosing where and why it fails, building the labeled data that improves it, and running the experiments, including fine-tuning, that prove which changes actually work.
We are looking for someone with exceptional business judgment and metric intuition, paired with deep hands-on modeling skills: a strong point of view on what to measure, and the technical ability to measure it at scale, whether the subject is customer spending or the quality of an LLM's answers. You should be equally comfortable pressure-testing a model's assumptions and explaining its "so what" to a VP in two sentences.
Key job responsibilities
• Own the definition, methodology and evolution of competitiveness metrics such as share of wallet and segment penetration, including validation and calibration of third-party panel data against internal sources.
• Design and validate causal inference, forecasting, segmentation and anomaly detection models that explain what drives shifts in customer spending.
• Define the evaluation framework for the Intelligence Engine: representative benchmark question sets, accuracy and groundedness metrics, LLM-as-judge methods validated against human labels, and clear launch bars.
• Run systematic error analysis on LLM outputs to identify failure patterns, and turn those insights into targeted improvements across knowledge base content, prompts and model behavior.
• Curate and maintain high-quality labeled datasets from expert review and user feedback, and use them to design and measure the impact of model improvement experiments, such as supervised fine-tuning and preference-based tuning, in partnership with applied scientists and engineers.
• Build reproducible pipelines that combine panel and internal data, and productionize recurring analyses and evaluations with data engineers.
• Translate ambiguous business questions into defensible metrics and answers, and communicate findings to senior leadership through written narratives and monthly reporting.
Set modeling, evaluation and scientific rigor standards for the team, and mentor scientists and analysts.
About the team
CXBT is a group of diverse functions dedicated to understanding, improving, and influencing customer experience globally, across all of Amazon. We are builders who develop products, services, and data-driven approaches that shape offerings for nearly every Amazon business and customer type — consumers, developers, sellers, brands, employees, investors, streamers, and gamers. We work backwards from customer needs, combine technical and non-technical methods, and track industry and business trends. Our roles span Product Managers, Data Scientists, Economists, Business Intelligence Engineers, Data Engine, Applied Scentist, and Product Manager.
This role spans two of BEAM's core products. The Topline View of Retail is an always-on competitive intelligence capability that fuses third-party transaction panels with Amazon's internal signals to answer how Amazon is winning or losing share of wallet, across segments, categories and time. The Intelligence Engine is an LLM-powered analytics assistant that lets teams across Amazon ask business questions in natural language and get trustworthy, data-backed answers. You will not just analyze metrics. You will decide which metrics should exist and build the models that generate, forecast and explain them. And as the Intelligence Engine scales, you will own how we measure its quality and drive its accuracy forward: defining evaluation standards, diagnosing where and why it fails, building the labeled data that improves it, and running the experiments, including fine-tuning, that prove which changes actually work.
We are looking for someone with exceptional business judgment and metric intuition, paired with deep hands-on modeling skills: a strong point of view on what to measure, and the technical ability to measure it at scale, whether the subject is customer spending or the quality of an LLM's answers. You should be equally comfortable pressure-testing a model's assumptions and explaining its "so what" to a VP in two sentences.
Key job responsibilities
• Own the definition, methodology and evolution of competitiveness metrics such as share of wallet and segment penetration, including validation and calibration of third-party panel data against internal sources.
• Design and validate causal inference, forecasting, segmentation and anomaly detection models that explain what drives shifts in customer spending.
• Define the evaluation framework for the Intelligence Engine: representative benchmark question sets, accuracy and groundedness metrics, LLM-as-judge methods validated against human labels, and clear launch bars.
• Run systematic error analysis on LLM outputs to identify failure patterns, and turn those insights into targeted improvements across knowledge base content, prompts and model behavior.
• Curate and maintain high-quality labeled datasets from expert review and user feedback, and use them to design and measure the impact of model improvement experiments, such as supervised fine-tuning and preference-based tuning, in partnership with applied scientists and engineers.
• Build reproducible pipelines that combine panel and internal data, and productionize recurring analyses and evaluations with data engineers.
• Translate ambiguous business questions into defensible metrics and answers, and communicate findings to senior leadership through written narratives and monthly reporting.
Set modeling, evaluation and scientific rigor standards for the team, and mentor scientists and analysts.
About the team
CXBT is a group of diverse functions dedicated to understanding, improving, and influencing customer experience globally, across all of Amazon. We are builders who develop products, services, and data-driven approaches that shape offerings for nearly every Amazon business and customer type — consumers, developers, sellers, brands, employees, investors, streamers, and gamers. We work backwards from customer needs, combine technical and non-technical methods, and track industry and business trends. Our roles span Product Managers, Data Scientists, Economists, Business Intelligence Engineers, Data Engine, Applied Scentist, and Product Manager.