"Alerted.org

Job Description

We’re seeking a highly skilled Senior Spark Architect/Engineer with deep expertise in Apache Spark and Kafka to lead the design and development of scalable data solutions within our AWS cloud infrastructure. This is a hands-on role that blends architecture, engineering, and leadership to drive innovation in big data processing.

Key Responsibilities

Architect, develop, test, and deploy datalake/warehouse solutions using modern development tools in AWS.

Design and implement robust data ingestion pipelines for high-volume datasets.

Lead data migration and transformation initiatives with precision and efficiency.

Develop and integrate software applications using industry-standard methodologies and architectural patterns, with a strong focus on performance and security.

Collaborate with Business Analysts, Architects, and Senior Developers to define and establish the AWS application framework.

Automate end-to-end ETL processes for diverse datasets within the big data ecosystem.

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to [email protected] learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/.

Skills and Requirements

Proficiency in SQL, Spark/PySpark, Spark Structured Streaming, Kafka, Linux/RHEL 8 , Python, Java, and REST APIs.

Strong interpersonal skills with the ability to communicate complex technical concepts to both technical and non-technical stakeholders.

Proven experience in leading, coaching, and mentoring technical teams.

Hands-on experience with:

Spark SQL, Spark Structured Streaming, and Java Spark Framework for large-scale data processing.

Machine learning capabilities via API

Performance tuning and optimization of Spark jobs.

Designing high-throughput, low-latency pipelines using Kafka.

AWS EMR for distributed data processing.

Experience with Kubernetes or like framework to deploy and manage Spark clusters and applications.

Experience with Hybrid Cloud Architectures

Delta Table Experience Familiarity with HL7 FHIR specifications is a plus.

Experience with Serverless Architectures. null

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal employment opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment without regard to race, color, ethnicity, religion,sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military oruniformed service member status, or any other status or characteristic protected by applicable laws, regulations, andordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or the recruiting process, please send a request to [email protected].