"Alerted.org

1. Strong Communication/Can interface with senior Eng and DS stakeholders

2. Computational Linguist or Linguistic Eng background

3. Experience setting up guidelines and golden sets, measuring quality.

4.Python, SQL, NLP

Description

Data Accuracy Guidelines & Audits: Define stringent data quality and accuracy guidelines and create clear user interface (UI) instructions for data collection efforts. You will conduct comprehensive manual data audits and reratings, meticulously assessing data against defined criteria to ensure high accuracy and alignment among all involved groups (e.g., vendors, GHD QA, research).

Complex Data Analysis & Problem Solving: Perform in-depth manual data audits and reratings of incoming data. You will expertly identify systematic patterns of data inaccuracy or subtle quality issues, particularly those stemming from complex subject matters or inconsistent guideline application. Conduct ad hoc investigations and generate detailed reports with actionable insights.

Feedback for Accuracy: Provide constructive and precise feedback to human raters, highlighting strengths and weaknesses, offering clear examples for improvement, and thoroughly explaining justifications for ratings to directly enhance data accuracy.

Workflow Improvement & Tool Leverage: Contribute to streamlining data collection workflows by identifying opportunities for tooling improvements and automation that enhance data accuracy and quality monitoring. Leverage existing tools and dashboards effectively for detailed data exploration and analysis.

Cross-Functional Team Collaboration: Collaborate closely with diverse teams including requesters, data scientists, and engineers. Establish clear communication channels, participate in regular meetings, and share detailed audit results. Gain a deep understanding of each stakeholder's unique data quality needs and complex subject matters to ensure optimal outcomes.

Rater Performance Management: Contribute to the review and identification of high-performing raters for potential promotion. Provide constructive and precise feedback to raters, highlighting strengths and weaknesses, offering concrete examples for improvement, and clearly explaining justifications for ratings to drive performance improvements.

Workflow and Tooling Enhancement: Actively contribute to streamlining data collection workflows, including rater offboarding processes and comprehensive rater status tracking. Collaborate closely with engineering teams to enhance QA tooling, such as GHD QA tasks and quality monitoring dashboards. Seek opportunities to automate quality reports and data filtering processes where feasible.

Cross-Functional Collaboration and Communication: Serve as a pivotal point of contact and collaborate effectively with diverse stakeholders including requesters, data scientists, software engineers, and external vendors. Establish clear communication channels, participate in regular meetings, share detailed audit results, and gain a deep understanding of each stakeholder's unique data quality needs to ensure optimal outcomes. Facilitate vendor alignment meetings as needed.

About you

We look for the following skills and experience:

Deep Understanding of Data Quality: Demonstrated strong grasp of what constitutes high-quality data, particularly concerning accuracy, consistency, and relevance, for AI/LLM training datasets. Familiarity with common data quality issues like imprecise prompts, inconsistent guideline adherence, factual inaccuracies, and suboptimal rater performance.

Analytical Acumen: Proficiency in both quantitative and qualitative analysis techniques to identify patterns and trends in large datasets. Ability to interpret key metrics such as inter-rater agreement, average rating scores, and data non-compliance rates. Competence in interpreting data visualizations (histograms, line graphs, bar charts) to inform decisions.

Attention to Detail and Accuracy: Exceptional precision and thoroughness in assessing data quality, including prompt specificity, detail, relevance, presentation, and response accuracy (e.g., factuality, adherence to system instructions, user request fulfillment). Strong understanding of the importance of comprehensive justifications for ratings.

Training and Instruction Design: Proven ability to create clear, unambiguous instructions and comprehensive training materials for human raters, including effective examples of both high and low-quality data. Appreciation for the role of system instructions in guiding AI assistant behavior for consistent responses.

Problem-Solving and Intervention: Ability to effectively investigate complex data quality issues, diagnose root causes, and propose actionable interventions (e.g., guideline revisions, targeted rater retraining, data filtering strategies).

Skills

Data science, Sql, Linguistics, Computational Linguistics

Top Skills Details

Data science, Sql, Linguistics, Computational Linguistics

Additional Skills & Qualifications

See above

Experience Level

Expert Level

FAQ California Assembly Bill 701 – Warehouse Distribution Centers

What is California Assembly Bill 701 - Warehouse Distribution Centers?

CA Assembly Bill 701 mandates new protections for workers in Warehouse and Distribution Centers. The

bill aims to reduce injuries and improve workplace safety.

What are employers required to do to comply with CA AB 701?

Employers are required to obtain performance related quotas and disseminate those quotas to

employees, which includes the quantified number of tasks or materials to be produced or handled,

within the defined time period, and any potential adverse employment action that could result from

failure to meet the quota.

What Is a “Quota” Under AB 701?

A “quota” under AB 701 is defined as the work standard under which an employee is assigned or

required to perform at a specified productivity speed, or perform a quantified number of tasks, or to

handle or produce a quantified amount of material, within a defined time period, and under which

the employee may suffer an adverse employment action if they fail to complete the performance

standard.

What do Sales Teams who support Clients in CA Warehouse and Distribution environments need to

do?

• A client letter will be provided to active Warehouse & Distribution Center clients by

Marketing Communications

o Sales teams should obtain Performance Quotas for all employees at the client

worksite (where quotas are required)

o Clients must complete the Performance Quota form for each position/requisition for

all levels of that position.

• All quotas obtained from the client must be attached to the CRG

• Quota information should be stored in the Performance Expectations field of the Connected

job req

o Copy and paste Performance Quota template into ‘Performance Expectations’ field

and fill in applicable details

o If you are limited by characters in the ‘Performance Expectations’ field, please enter

remaining quota details into the ‘Additional Information’ field

o When submitting an ESF for a contractor with a Performance Quota, please note ‘CA

Warehouse Information Required’

**Quotas must be obtained for all active employees for clients that require them, who work

in a warehouse or distribution center environment, as early in the req-qualifying process as

possible *

Pay and Benefits

The pay range for this position is $55.00 - $55.00/hr.

Eligibility requirements apply to some benefits and may depend on your job classification and length of employment. Benefits are subject to change and may be subject to specific elections, plan, or program terms. If eligible, the benefits available for this temporary role may include the following:

• Medical, dental & vision • Critical Illness, Accident, and Hospital • 401(k) Retirement Plan – Pre-tax and Roth post-tax contributions available • Life Insurance (Voluntary Life & AD&D for the employee and dependents) • Short and long-term disability • Health Spending Account (HSA) • Transportation benefits • Employee Assistance Program • Time Off/Leave (PTO, Vacation or Sick Leave)

Workplace Type

This is a fully remote position.

Application Deadline

This position is anticipated to close on Oct 20, 2025.

h4>About TEKsystems:

We're partners in transformation. We help clients activate ideas and solutions to take advantage of a new world of opportunity. We are a team of 80,000 strong, working with over 6,000 clients, including 80% of the Fortune 500, across North America, Europe and Asia. As an industry leader in Full-Stack Technology Services, Talent Services, and real-world application, we work with progressive leaders to drive change. That's the power of true partnership. TEKsystems is an Allegis Group company.

The company is an equal opportunity employer and will consider all applications without regards to race, sex, age, color, religion, national origin, veteran status, disability, sexual orientation, gender identity, genetic information or any characteristic protected by law.

About TEKsystems and TEKsystems Global Services

We’re a leading provider of business and technology services. We accelerate business transformation for our customers. Our expertise in strategy, design, execution and operations unlocks business value through a range of solutions. We’re a team of 80,000 strong, working with over 6,000 customers, including 80% of the Fortune 500 across North America, Europe and Asia, who partner with us for our scale, full-stack capabilities and speed. We’re strategic thinkers, hands-on collaborators, helping customers capitalize on change and master the momentum of technology. We’re building tomorrow by delivering business outcomes and making positive impacts in our global communities. TEKsystems and TEKsystems Global Services are Allegis Group companies. Learn more at TEKsystems.com.

The company is an equal opportunity employer and will consider all applications without regard to race, sex, age, color, religion, national origin, veteran status, disability, sexual orientation, gender identity, genetic information or any characteristic protected by law.