• Staff Software Engineer - Kora Orchestration

    Confluent (Ontario, CA)


    Location:

    CA Remote Ontario

    Employment Type:

    FullTime

     

    Department

    Engineering

     

    Overview

     

    We’re not just building better tech. We’re rewriting how data moves and what the world can do with it. With Confluent, data doesn’t sit still. Our platform puts information in motion, streaming in near real-time so companies can react faster, build smarter, and deliver experiences as dynamic as the world around them.

     

    It takes a certain kind of person to join this team. Those who ask hard questions, give honest feedback, and show up for each other. No egos, no solo acts. Just smart, curious humans pushing toward something bigger, together.

     

    One Confluent. One Team. One Data Streaming Platform.

    About the Role:

    Our team is responsible for managing the metadata that makes Kafka a reliable, fault-tolerant, and high-performance distributed streaming platform. This includes coordinating the election of leaders, monitoring broker availability, persisting and distributing metadata, coordinating cluster-wide actions, and more. In this role, you will be working on building a highly available, reliable, secure, scalable, and multi-tenant metadata layer. You will be instrumental in driving the technical solutions end-to-end, working closely with the team and key stakeholders to design, architect, and develop top-notch solutions. The team recently completed a major initiative to implement its own consensus protocol based on the Raft protocol. This new protocol, named Kraft, has allowed the team to remove the dependency on Zookeeper which allows for more scalable and robust metadata management as well as a more simple deployment model for Kafka. Here are examples of some of the challenges we've taken on in recent years:

     

    Kora: A Cloud-Native Event Streaming Platform For Kafka (https://www.vldb.org/pvldb/vol16/p3822-povzner.pdf) KRaft: Apache Kafka Without ZooKeeper

     

    Why ZooKeeper Was Replaced with KRaft – The Log of All Logs (https://www.confluent.io/blog/why-replace-zookeeper-with-kafka-raft-the-log-of-all-logs/) KIP-866: ZooKeeper to KRaft Migration

     

    KIP-853: KRaft Controller Membership Changes (https://cwiki.apache.org/confluence/display/KAFKA/KIP-853%3A+KRaft+Controller+Membership+Changes)

    What You Will Do:

    As a Staff Software Engineer, you will be a technical leader whose work significantly impacts the product, engineering organization, and customer success.

     

    + Drive the end-to-end delivery of complex, multi-quarter technical projects and critical infrastructure components from concept to production, setting a high bar for operational excellence, scalability, reliability, and security.

    + Act as a primary technical escalation point for complex, elusive production issues and architectural challenges. Proactively identify systemic weaknesses and drive large-scale refactoring or architectural changes to prevent future recurrence.

    + Exert influence across multiple teams and organizational boundaries (e.g., product, support, engineering) to align technical direction, mitigate cross-cutting risks, and ensure cohesive, high-quality solution delivery.

    + Serve as a technical voice of the customer, anticipating future needs and translating ambiguous, high-level business problems into clear, executable technical strategies and architectures that drive significant customer value and product adoption.

    + Act as a force multiplier by actively mentoring and coaching engineers and elevating the overall technical acumen and best practices across the organization.

    What You Will Bring:

    + BS, MS, or PhD in computer science or a related field, or equivalent work experience.

    + 8+ years of progressive experience in designing, building, and operating large-scale, mission-critical distributed systems (or equivalent expertise).

    + Deep, expert-level understanding of distributed systems fundamentals, including consensus protocols (e.g., Raft/Paxos), consistency models (e.g., strong, eventual), fault-tolerance, and high-availability architecture.

    + Experience building and operating large-scale systems in the Cloud.

    + Solid understanding of basic systems operations (disk, network, operating systems, etc.).

    + Demonstrated history of mentoring engineers and driving significant improvement in engineering practices and operational rigor across multiple teams.

    + Proficiency in Java, Scala, C/C++, Go or other statically typed languages.

    What Gives You an Edge:

    + Expert-level proficiency in Java and the JVM ecosystem, including threading, performance tuning, and diagnostics.

    + Hands-on experience with consensus algorithms (Raft, Paxos, etc.), or experience operating large-scale coordination services (ZooKeeper, etcd, Consul).

    + A strong background in distributed storage systems or databases.

    + Experience/knowledge with public clouds (AWS, Azure or GCP).

    + Interest in evangelism (giving talks at tech conferences, writing blog posts evangelizing Kafka).

     

    Ready to build what's next? Let’s get in motion.

     

    Come As You Are

     

    Belonging isn’t a perk here. It’s the baseline. We work across time zones and backgrounds, knowing the best ideas come from different perspectives. And we make space for everyone to lead, grow, and challenge what’s possible.

     

    We’re proud to be an equal opportunity workplace. Employment decisions are based on job-related criteria, without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, veteran status, or any other classification protected by law.