• Remote Data Engineer

    Insight Global (Chesterfield, MO)


    Job Description

    Insight Global is looking to add 2 Data Engineers to our client's team who are headquartered in the St. Louis area but can sit remotely. This individual will be working with a team of 2 Data Engineers and a hands on Lead Data Engineer. This individual will be supporting a brand new project standing up an Enterprise Data warehouse. They will be responsible for modernizing ETL processes within an on-premises Cloudera Data Platform (CDP) environment, leveraging technologies like Apache Spark, Apache Iceberg, and Apache Airflow for scalable, efficient, and reliable data transformation and management. Daily, they will be responsible for ETL development and troubleshooting needs, along with participating in production support environments. They will contribute to development efforts for ETL pipelines in the Enterprise Data Warehouse (EDW). They will support and rebuild legacy ETL jobs (currently not using ACID transactions) with modern solutions using Apache Spark and Apache Iceberg to support ACID transactions. This person will transform and integrate EBCDIC Mainframe data into Hive and Impala tables using Precisely Connect for Big Data. This individual will optimize data transformation processes for performance, scalability, and reliability and ensure data consistency, accuracy, and quality across the ETL pipelines. Development responsibilities will be 50% of this role. Additionally, this person will be sharing weekly 24/7 production support with the managed service vendor on a 4-week rotation, monitoring, ETL workflows and troubleshooting issues to ensure smooth production operations along with research and resolve user requests and issues. Production support is 20% of this role.

     

    We are a company committed to creating inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity employer that believes everyone matters. Qualified candidates will receive consideration for employment opportunities without regard to race, religion, sex, age, marital status, national origin, sexual orientation, citizenship status, disability, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to Human Resources Request Form (https://airtable.com/app21VjYyxLDIX0ez/shrOg4IQS1J6dRiMo) . The EEOC "Know Your Rights" Poster is available here (https://www.eeoc.gov/sites/default/files/2023-06/22-088\_EEOC\_KnowYourRights6.12ScreenRdr.pdf) .

     

    To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/ .

     

    Skills and Requirements

     

    5+ years of Data Engineering experience

     

    3+ years pf ETL development

     

    3+ years of advanced SQL experience

     

    3+ years in Python scripting

     

    Experience working in a Linux environment.

     

    Expertise in using Apache Spark, Apache Iceberg, and Apache Airflow for ETL pipelines.

     

    Strong familiarity with version control systems, specifically Azure DevOps

     

    Will have hand in production support - would prefer experience with being on a rotation before

     

    Familiarity with data modeling, schema design, and building data models for reporting needs Hands-on experience with Cloudera Data Platform (CDP), including Hive and Impala

     

    Extensive knowledge of Precisely Connect for Big Data or similar tools for mainframe data transformation. null

     

    We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal employment opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment without regard to race, color, ethnicity, religion,sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military oruniformed service member status, or any other status or characteristic protected by applicable laws, regulations, andordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or the recruiting process, please send a request to [email protected].