"Alerted.org

Job Description

Role Overview

We are looking for a Data/LLM Scientist to join our LLM Monitoring Team, which is dedicated to enhancing and maintaining our Large Language Model (LLM) systems. This role focuses on integrating advanced features such as use case onboarding, business dashboards, and various scorers. You will be responsible for developing connectors for diverse data sources, enhancing user support, and ensuring robust monitoring through the MARS UI. Key features like drift analysis, explainability, and robust validators are central to providing comprehensive oversight and efficient resource management.

Key Responsibilities

Use Case Onboarding:

• Develop and implement procedures for integrating new use cases into existing LLM systems.

• Work closely with business units to understand their needs and tailor LLM functionalities to meet business objectives.

Business Dashboards & Scorers:

• Design and deploy business dashboards that provide real-time insights into LLM performance and business impact.

• Develop and maintain various scorers to evaluate the effectiveness and accuracy of LLM predictions.

Data Integration & User Support:

• Create connectors for a variety of data sources to ensure seamless data integration and processing.

• Enhance user support mechanisms to facilitate efficient and effective use of LLM applications.

Monitoring through MARS UI:

• Implement comprehensive monitoring strategies using the MARS UI to track LLM performance and user engagement.

• Continuously update and refine monitoring tools to capture relevant data and insights.

Advanced Analytics Features:

• Conduct drift analysis to identify and correct deviations in LLM behavior over time.

• Utilize explainability tools to make the decision-making processes of LLMs transparent and understandable.

• Develop robust validators to ensure the accuracy and reliability of LLM outputs.

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to [email protected] learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/.

Skills and Requirements

Required Technical Skills

• Advanced proficiency in Python, especially with data science and machine learning libraries.

• Experience in developing and deploying machine learning models, particularly LLMs.

• Strong background in data integration and managing large datasets.

• Proficiency in monitoring tools and user interface development, preferably with experience in MARS UI.

• Knowledge of machine learning model validation techniques and performance metrics.

Soft Skills

• Analytical Thinking: Strong problem-solving skills with a keen analytical mindset.

• Detail-Oriented: High attention to detail with a focus on accuracy and quality.

• Communication: Excellent communication skills to effectively collaborate with team members and stakeholders.

• Proactive: Ability to anticipate needs and propose solutions with minimal supervision.

This position is suited for individuals who are passionate about data science, machine learning, and operational excellence, and who thrive in a dynamic and innovative environment.