"Alerted.org

PENNYMAC

Pennymac (NYSE: PFSI) is a specialty financial services firm with a comprehensive mortgage platform and integrated business focused on the production and servicing of U.S. mortgage loans and the management of investments related to the U.S. mortgage market.

At Pennymac, our people are the foundation of our success and at the heart of our dynamic work culture. Together, we work towards a unified goal of helping millions of Americans achieve aspirations of homeownership through the complete mortgage journey.

Job Overview

We’re looking for an experienced, forward-thinking Observability Engineer to expand our Observability team in Core Services Engineering, and strengthen our observability capabilities across Pennymac environments. In this role, you will be responsible for designing, implementing, and maintaining our observability platform, with a strong focus on New Relic. You will leverage your expertise in Infrastructure as Code (IaC) to automate and manage our monitoring and alerting infrastructure, ensuring our systems are reliable, performant, and transparent. You will work closely with Core Services, DevOps, Development and Operation teams to foster a culture of proactive monitoring and data-driven decision-making. If you’re passionate about automation, cloud-native patterns, and making systems run smarter and safer, we want to hear from you.

A Typical Day

The Sr Observability Engineer will:

+ Design & Implement Observability Solutions: Architect, build, and scale comprehensive monitoring solutions using the New Relic platform , including APM, Infrastructure, Logs, Synthetics, and custom instrumentation (NRQL).

+ Automate with IaC: Develop, manage, and maintain observability configurations—including alerts, dashboards, and synthetic checks—using Infrastructure as Code (IaC) tools such as Terraform/OpenTofu .

+ Develop Dashboards & Alerts: Create and refine insightful dashboards and actionable alerting policies in New Relic to provide real-time visibility into infrastructure and application health.

+ Promote Best Practices: Act as a subject matter expert on observability, guiding teams on best practices for logging, metrics, and tracing to improve system reliability and reduce mean time to resolution (MTTR).

+ Troubleshoot & Optimize: Analyze performance data and telemetry to identify bottlenecks, troubleshoot production issues, and drive performance optimization efforts across the stack.

What You’ll Bring

+ Bachelor's degree in Computer Science, Engineering, or a related field (or equivalent experience).

+ 7+ years of experience in a Cloud Engineering role (Observability, DevOps, SRE, etc).

+ Proven New Relic Expertise: 3+ years of hands-on experience with the New Relic platform, including deep knowledge of Dashboards, NRQL, APM and setting up effective alerting.

+ Strong IaC Proficiency: 3+ years of experience managing infrastructure and configurations with IaC tools like Terraform/OpenTofu (preferred), AWS CDK, CloudFormation, Chef or Ansible.

+ Cloud Platform Experience: Extensive hands-on experience working with major cloud providers such as AWS (preferred), GCP , or Azure .

+ Scripting Skills: Proficiency in a scripting language such as Python , Go, or Bash for automation and tooling.

+ System Knowledge: Strong understanding of cloud architecture, networking principles, Windows/Linux Server administration, microservices in a SaaS context, containerization ( Docker , Kubernetes ), and CI/CD principles.

+ Security & Compliance experience : Deep understanding of security best practices and their implementation in cloud infrastructure and CI/CD pipelines. Experience working in complexly regulated environments.

+ Excellent problem-solving and troubleshooting skills.

+ Strong communication and collaboration skills.

Why You Should Join

As one of the top mortgage lenders in the country, Pennymac has helped over 4 million lifetime homeowners achieve and sustain their aspirations of home. Our vision is to be the most trusted partner for home. Together, 4,000 Pennymac team members across the country are guided by our core values: to be Accountable, Reliable and Ethical in all that we do. Pennymac is committed to conducting a business that makes positive contributions and promotes long-term sustainable growth and to fostering an equitable and inclusive environment, where all employees and customers feel valued, respected and supported.

Benefits That Bring It Home: Whether you're looking for flexible benefits for today, setting up short-term goals for tomorrow, or planning for long-term success and retirement, Pennymac's benefits have you covered. Some key benefits include:

+ Comprehensive Medical, Dental, and Vision

+ Paid Time Off Programs including vacation, holidays, illness, and parental leave

+ Wellness Programs, Employee Recognition Programs, and onsite gyms and cafe style dining (select locations)

+ Retirement benefits, life insurance, 401k match, and tuition reimbursement

+ Philanthropy Programs including matching gifts, volunteer grants, charitable grants and corporate sponsorships

To learn more about our benefits visit: https://pennymacnews.page.link/benefits

For residents with state required benefit information, additional information can be found at: https://www.pennymac.com/additional-benefits-information

Compensation: Individual salary may vary based on multiple factors including specific role, geographic location / market data, and skills and experience as defined below:

+ Lower in range - Building skills and experience in the role

+ Mid-range - Experience and skills align with proficiency in the role

+ Higher in range - Experience and skills add value above typical requirements of the role

Some roles may be eligible for performance-based compensation and/or stock-based incentives awarded to employees based on company and individual performance.

Salary

$95,000 - $155,000

Work Model

OFFICE