"Alerted.org

Job Description

Responsibilities:

• Validate that AI models meet performance benchmarks (accuracy, precision, recall, etc.).

• Detect and mitigate bias in datasets and model predictions.

• Identify potential risks in AI systems (e.g., harmful outputs, adversarial attacks).

• Implement safeguards to prevent unintended consequences.

• Ensure compliance with safety standards and regulations

• Continuously monitor AI systems in production for anomalies or drift.

• Audit decision-making processes for transparency and explainability.

• Document testing procedures and safety measures for accountability.

• Ensure AI systems align with ethical principles (fairness, accountability, transparency).

• Work with legal and compliance teams to meet regulatory requirements.

• Partner with data scientists, ML engineers, and product teams to integrate safety checks.

Day to Day:

One of Insight Global’s clients is seeking an AI Quality and Safety Engineer to join their team. In this role, the engineer will review system dashboards to monitor AI models in production, evaluating them for anomalies, performance drift, and safety alerts. They will assess potential risks within AI systems, including harmful outputs or adversarial attacks, and implement safeguards to prevent unintended consequences. They will also conduct stress tests on new models to validate accuracy, fairness, and robustness, as well as perform risk assessments to identify potential failure scenarios and document mitigation strategies. Additional responsibilities include ensuring models comply with regulatory standards such as GDPR, maintaining audit logs, and updating safety documentation to support transparency. Additionally, they will work closely with developers to implement safety guardrails such as content filters or fallback mechanisms and will provide feedback to guide model improvements. Finally, they will report findings to stakeholders, highlighting critical issues and areas for improvement, while dedicating time to continuous learning to stay current with emerging AI safety frameworks and tools.

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to [email protected] learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/.

Skills and Requirements

Must Have:

• 3-5 years of experience

• Extensive experience with eval frameworks for LLM quality and safety

• Deep understanding of LLMs (Gemini, OpenAI, Llama, etc.)

• Experience with prompt engineering and fine-tuning

• Strong Python skills and experience with LangChain or similar frameworks • Bachelor’s degree in related field