"Alerted.org

Job Description

DAY TO DAY:

Currently implementing vendor tools that provide Guardrails and Observability for Generative AI applications. We are seeking a GenAI Engineer with proven expertise in GenAI Ops — operationalizing, monitoring, and scaling LLM and RAG-powered applications with robust guardrails and observability.

The role will focus on leveraging LLMs as Judges and Specialized Models (SMLs) to measure and score guardrail metrics such as chunk attribution, context adherence, prompt injection detection, tone, sexism, bias, and PII leakage. Success requires strong skills in annotation, fine-tuning, and alignment techniques to calibrate these judge models, and in bringing all of this into an operational framework for enterprise readiness.

Data Science and AI Engineering skillset required for enablement of AI Technology Strategy. Data Architecture Strategy Lead performs flawless, end to end execution of cross functional, high impact strategic data related initiatives and/or large programs that have significant influence on how the company manages data. Execution plans outline multi-year strategic outcomes (based on target state AI Technology products and tools) and the activities required to support achievement of those outcomes. Communicates, influences and negotiates both vertically and horizontally to obtain or leverage necessary resources. Knowledgeable in the agile framework, demonstrate a strong combination of strategic thinking, tactical planning and project management skills along with the ability to lead and influence project teams without direct management.

Responsibilities:

• Operationalize guardrails and observability across vendor-based RAG and LLM applications.

• Set up GenAI Ops workflows to continuously monitor inference latency, throughput, quality, and safety metrics.

• Define, track, and analyze RAG guardrail metrics using LLMs as Judges and SMLs (e.g., attribution, grounding, prompt injection, tone, PII leakage).

• Implement annotation, structured feedback loops, fine-tuning, and alignment methods to calibrate judge models.

• Use LangChain to orchestrate guardrail checks, manage prompt versioning, and integrate judge model scoring workflows.

• Work with OpenShift to deploy, scale, and monitor containerized GenAI services.

• Build observability dashboards and alerts (Grafana or equivalent) for AI reliability.

• Contribute to emerging agentic evaluation and guardrails as autonomous AI workflows expand.

• Ability to lead cross-functional work and motivate cross-functional teams to achieve business objectives and process improvements.

• Ability to translate needs from data stakeholders into solutions

Required Qualifications

• Bachelor’s or master’s degree in Data Science, Computer Science, MIS, related field, or equivalent experience.

• Proven hands-on experience in GenAI Ops — operationalizing LLM and RAG applications in production.

• Strong hands-on experience with the LangChain framework.

• Experience specifically with the OpenAI API, chat completions, embeddings, etc.

• Have a solid awareness on TensorRT and VLLM implementation.

• Strong proficiency in Python and data science libraries (NumPy, Pandas, Scikit-learn, PyTorch/TensorFlow).

• Proven experience applying guardrails and observability to LLM or RAG-powered applications.

• Experience with LLMs as Judges and SMLs for evaluation (attribution, adherence, bias, PII, etc.).

• Hands-on experience with OpenShift (or Kubernetes) for containerized AI workloads.

• Experience measuring and optimizing inference latency.

• Strategic and innovative thinker, able to solve complex problems and develop solutions incorporating data/research to support recommendations and risk/reward tradeoffs.

• Ability to lead and drive change and deliver results in a heavily matrixed environment.

• Demonstrated ability to influence a wide variety of stakeholders including senior executives.

• Self-starter with an exceptional drive for results and success; conveys a sense of urgency to achieve outcomes and exceed expectations; persists despite obstacles, setbacks and competing influences.

• Outstanding communication and presentation skills (verbal and written) across all management levels. Must be able to concisely summarize key observations, clearly articulate considerations, propose solutions.

• Solution design - demonstrates expertise in solution design across multiple technologies; can identify opportunities for technical training and coaching across the organization.

• Demonstrated expertise in AI strategies and tool.

• Can identify opportunities for technical training and coaching across the organization.

• 10+ years of experience designing and developing AI solution architectures to scale.

• Design and develop new proof of concept projects to enhance current AI systems to handle increased traffic and larger datasets with cutting edge.

• Experience in Agile frameworks and Agile at scale.

• Ability to provide input for financial planning, tracking, and managing budget variances.

Desired Qualifications

• Self-motivated and can collaborate across a broad team and support groups.

• Experience with annotation pipelines, feedback loops, fine-tuning, and alignment techniques to improve judge model calibration.

• Familiarity with prompt versioning and lifecycle management.

• Exposure to Grafana or similar observability dashboards.

• Prior experience with vendor-provided guardrail and observability solutions.

• Knowledge of GenAI frameworks (e.g., LangChain, LlamaIndex, Haystack) and tools for agentic solutions.

• Understanding of agentic evaluation and guardrails for multi-step or autonomous GenAI workflows

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to [email protected] learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/.

Skills and Requirements

MUST HAVES:

• Data Science and AI Engineering

• Proven hands-on experience in GenAI Ops — operationalizing LLM and RAG applications in production.

• Strong hands-on experience with the LangChain framework.

• Experience specifically with the OpenAI API, chat completions, embeddings, etc.

• Have a solid awareness on TensorRT and VLLM implementation.

• Strong proficiency in Python and data science libraries (NumPy, Pandas, Scikit-learn, PyTorch/TensorFlow).

• Proven experience applying guardrails and observability to LLM or RAG-powered applications.

• Experience with LLMs as Judges and SMLs for evaluation (attribution, adherence, bias, PII, etc.).

• Design and develop new proof of concept projects to enhance current AI systems to handle increased traffic and larger datasets with cutting edge. NICE TO HAVES:

• Hands-on experience with OpenShift (or Kubernetes) for containerized AI workloads.