"Alerted.org

Position Summary...

What you'll do...

Key Responsibilities

AI Evaluation & Transparency

+ Manage dashboards connecting **capability metrics (ACI)** with **business metrics (Scorecard)** to surface what’s working and what isn’t.

+ Maintain **golden datasets** for benchmarking model relevance, execution accuracy, personalization, and compliance.

+ Partner with AI Tech and Tech Ops to calibrate **LLM-as-Judge** and ensure automated evaluation remains aligned with human judgment.

Shopper Experience Quality & Audits

+ Conduct periodic audits of shopper-facing AI experiences (answers, nudges, recommendations, in-shop chat).

+ Define and monitor KPIs for intent recognition, execution accuracy, and personalization precision.

+ Run cross-journey evaluations (Events, Essentials, Discovery, Raptor baskets) to identify friction or divergence patterns.

SOP Governance & Continuous Improvement

+ Own and evolve **SOPs** for annotation, labeling, and evaluation to reflect changing features and customer contexts.

+ Capture and codify new edge cases (multimodal queries, accessibility, household profiles) into updated labeling guides.

+ Ensure all SOPs align with shopper-first outcomes and compliance (ADA, HIPAA, privacy).

Human + AI Divergence Monitoring

+ Track variance between human labels, automated judgments, and live model outputs.

+ Create structured escalation paths for high-risk failures (irrelevant, unsafe, or biased responses).

+ Quantify divergence trends and drive remediation with Product and ML Engineering.

Vendor & Partner Management

+ Manage external labeling partners, ensuring throughput, accuracy, and compliance SLAs.

+ Deliver training grounded in real Sparky interactions and golden dataset examples.

+ Audit vendor output and hold partners accountable for quality metrics.

Closed-Loop Feedback & Model Integration

+ Operate the **feedback flywheel** linking:

+ Shopper signals (thumbs up/down, drop-offs, basket edits) →

+ Human review & annotation →

+ Automated LLM evaluation →

+ Product & ML updates.

+ Ensure insights translate into measurable improvement in personalization, nudge effectiveness, and conversational trust.

Qualifications

+ 2–4 years in **AI operations, ML evaluation, or eCommerce data quality** ; experience managing annotation or human-in-the-loop programs preferred.

+ Working knowledge of **AI evaluation frameworks** (LLM-as-Judge, golden datasets, A/B testing).

+ Skilled in **KPI design, audit processes, and vendor oversight** .

+ Analytical storyteller able to link model metrics to business outcomes.

+ Obsessed with **customer trust, transparency, and continuous improvement.**

Act with Integrity Follows the law our code of conduct and company policies and sets expectations for others to do the same promotes an environment where associates feel comfortable sharing concerns and reinforces our culture of non retaliation listens to concerns raised by associates takes action and encourages others to do the same holds self and others accountable for achieving results in a way that is consistent with our values

Act with Integrity Acts as an altruistic servant leader and is consistently humble self-aware honest and transparent

Serve our Customers and Members Delivers expected business results while putting the customer member first and consistently applying an omni merchant mindset and acts with an everyday Low-cost mindset to drive value and every day Low Prices for customers members

Serve our Customers and Members Adopts a holistic perspective that considers data analytics customer member insights and different parts of the business when making plans and shaping the teams strategy

Strive for Excellence Consistently raises the bar and seeks to improve demonstrates curiosity and a growth mindset seeks feedback asks thoughtful questions fosters an environment that supports learning innovation and learning from mistakes and intelligent risk-taking and exhibits resilience in the face of setbacks

Strive for Excellence Seeks and implements continuous improvements and encourages the team to leverage new digital tools and ways of working

At Walmart, we offer competitive pay as well as performance-based bonus awards and other great benefits for a happier mind, body, and wallet. Health benefits include medical, vision and dental coverage. Financial benefits include 401(k), stock purchase and company-paid life insurance. Paid time off benefits include PTO (including sick leave), parental leave, family care leave, bereavement, jury duty, and voting. Other benefits include short-term and long-term disability, company discounts, Military Leave Pay, adoption and surrogacy expense reimbursement, and more. You will also receive PTO and/or PPTO that can be used for vacation, sick leave, holidays, or other purposes. The amount you receive depends on your job classification and length of employment. It will meet or exceed the requirements of paid sick leave laws, where applicable. For information about PTO, see https://one.walmart.com/notices . Live Better U is a Walmart-paid education benefit program for full-time and part-time associates in Walmart and Sam's Club facilities. Programs range from high school completion to bachelor's degrees, including English Language Learning and short-form certificates. Tuition, books, and fees are completely paid for by Walmart.

Eligibility requirements apply to some benefits and may depend on your job classification and length of employment. Benefits are subject to change and may be subject to a specific plan or program terms.

For information about benefits and eligibility, see One.Walmart (https://one.walmart.com/) .

Bentonville, Arkansas US-09401: The annual salary range for this position is $90,000.00 - $180,000.00

San Bruno, California US-08848 | Sunnyvale, California US-11657: The annual salary range for this position is $117,000.00 - $234,000.00 Additional compensation includes annual or quarterly performance bonuses. Additional compensation for certain positions may also include :

- Stock

ㅤ

ㅤ

ㅤ

ㅤ

‎

Minimum Qualifications...

_Outlined below are the required minimum qualifications for this position. If none are listed, there are no minimum qualifications._

Bachelor's degree in information technology, computer science, or related area and 5 years’ experience in eCommerce merchandising, site operations, business management, or related area OR 7 years’ experience in eCommerce merchandising, site operations, business management, information technology, computer science or related area.

2 years’ supervisory experience.

Preferred Qualifications...

_Outlined below are the optional preferred qualifications for this position. If none are listed, there are no preferred qualifications._

ECommerce Merchandising, Site Operations, Business Management, or related area, Master's degree in Information Technology, Computer Science, or related area

Primary Location...

702 Sw 8Th St, Bentonville, AR 72716, United States of America

Walmart and its subsidiaries are committed to maintaining a drug-free workplace and has a no tolerance policy regarding the use of illegal drugs and alcohol on the job. This policy applies to all employees and aims to create a safe and productive work environment.

Walmart, Inc. is an Equal Opportunity Employer- By Choice. We believe we are best equipped to help our associates, customers, and the communities we serve live better when we really know them. That means understanding, respecting, and valuing diversity- unique styles, experiences, identities, abilities, ideas and opinions- while being inclusive of all people.