"Alerted.org

Sr. Engineer, AI Model Evaluation

General Information

Req #

WD00086178

Career area:

Software Engineering

Country/Region:

United States of America

State:

California

City:

San Jose

Date:

Tuesday, August 5, 2025

Working time:

Full-time

**Additional Locations** :

* United States of America - California - San Jose

Why Work at Lenovo

We are Lenovo. We do what we say. We own what we do. We WOW our customers.

Lenovo is a US$69 billion revenue global technology powerhouse, ranked #196 in the Fortune Global 500, and serving millions of customers every day in 180 markets. Focused on a bold vision to deliver Smarter Technology for All, Lenovo has built on its success as the world’s largest PC company with a full-stack portfolio of AI-enabled, AI-ready, and AI-optimized devices (PCs, workstations, smartphones, tablets), infrastructure (server, storage, edge, high performance computing and software defined infrastructure), software, solutions, and services. Lenovo’s continued investment in world-changing innovation is building a more equitable, trustworthy, and smarter future for everyone, everywhere. Lenovo is listed on the Hong Kong stock exchange under Lenovo Group Limited (HKSE: 992) (ADR: LNVGY).

**Lenovo’s AI Technology Center (LATC)** is the global engine powering our hybrid-AI vision. As Lenovo’s AI Center of Excellence, we’re building a world-class team to define the next era of computing—powered by AI.

With Lenovo’s unmatched product ecosystem—spanning Moto smartphones and wearables, ThinkPad laptops, PCs, workstations, servers, and cloud—we have a unique platform to experiment, deploy, and scale AI across every layer of technology. Few companies can match the breadth of our canvas.

We’re tackling some of the most exciting challenges in AI:

+ Scaling and deploying foundation models in real-world environments

+ Advancing agentic computing across mobile, edge, and cloud

+ Seamlessly orchestrating intelligent systems to collaborate everywhere

This is a generational shift—and we’re moving fast. At LATC, you’ll join a team of bold innovators building transformative platforms at the intersection of AI and real-world impact. If you’re ready to push boundaries and help shape the future of hybrid AI, **this is where you belong.** Let’s build what’s next—together.

Description and Requirements

Summary:

We are seeking a highly motivated and skilled Sr. AI Model Evaluation Engineer to join our rapidly growing AI team. You will play a critical role in assessing the performance, robustness, and safety of large language models (LLMs), large vision models (LVMs), and large multimodal models (LMMs). This is a challenging yet rewarding opportunity to contribute to cutting-edge research and development in generative AI. You’ll be working with a collaborative team to push the boundaries of what’s possible with AI models and deploy them into innovative products. If you are passionate about making Smarter Technology For All, come help us realize our Hybrid AI vision!

Responsibilities:

+ Design, implement, and evaluate comprehensive evaluation pipelines for large generative AI models, encompassing various metrics and methodologies.

+ Evaluate the performance of publicly available models, and discuss their relative advantages and disadvantages.

+ Establish and maintain benchmarks for evaluating model performance across a range of tasks and datasets.

+ Conduct thorough error analysis to identify patterns in model failures and provide actionable insights for improvement.

+ Design and implement methods to detect and mitigate biases in model outputs, ensuring fairness and equitable performance.

+ Develop and execute robustness tests to assess model resilience against adversarial inputs, noise, and variations in real-world data.

+ Assess model safety, including identifying and mitigating harmful or inappropriate outputs.

+ Experiment with various evaluation techniques, metrics, and datasets to optimize model quality and reliability.

+ Contribute to the development and refinement of evaluation metrics that accurately reflect model performance and desired characteristics.

+ Clearly communicate evaluation results and insights to engineers, researchers, and stakeholders.

+ Identify potential partnerships with third parties.

+ Develop and maintain evaluation tools and infrastructure.

+ Monitor and analyze model performance in production environments, identify degradation, and propose solutions.

+ Stay up-to-date with the latest advancements in large language and multi-modal models, model evaluation techniques, metrics, and related technologies.

+ Contribute to the development of internal tools and infrastructure for model evaluation and monitoring.

Required Qualifications:

+ Bachelor's or Master's degree in Computer Science, Machine Learning, or a related field.

+ 12+ years of development experience

+ Strong programming skills in Python and experience with deep learning frameworks like PyTorch.

+ Deep understanding of machine learning evaluation principles, including various metrics (e.g., BLEU, ROUGE, perplexity, F1-score) and methodologies.

+ Proven ability to design and conduct rigorous experiments, analyze data, and draw meaningful conclusions.

+ Familiarity with large language models, transformer architectures, and related concepts.

+ Experience with data processing tools and techniques (e.g., Pandas, NumPy).

+ Experience working with Linux systems and/or HPC cluster job scheduling (e.g., Slurm, PBS).

Preferred Qualifications:

+ Ph.D. in Computer Science, Machine Learning, or a related field.

+ Excellent communication, collaboration, and problem-solving skills.

+ Experience with automated model evaluation frameworks and tools.

+ Experience with techniques for detecting and mitigating bias in AI models.

+ Experience with safety and alignment evaluation methodologies.

+ Experience with A/B testing and online evaluation techniques.

The base salary range budgeted for this position in CA, CO, Jersey City - NJ, NV, Ithaca - NY, NYC, WA, is $220k - $300k. Individuals may also be considered for bonus and/or commission. Lenovo’s various benefits can be found here: https://www.lenovobenefits.com/enrolling-in-benefits/why-join-lenovo/

\#LATC

_We are an Equal Opportunity Employer and do not discriminate against any employee or applicant for employment because of race, color, sex, age, religion, sexual orientation, gender identity, national origin, status as a veteran, and basis of disability or any federal, state, or local protected class._

**Additional Locations** :

* United States of America - California - San Jose

* United States of America

* United States of America - California

* United States of America - California - San Jose