Data Engineer General

Company: Altair
Location: Dearborn
Posted on: June 21, 2022

Job Description:

Transforming the Future with the Convergence of Simulation and Data Data Engineer GeneralJob Summary:Our client in Dearborn, MI is looking for a Data Engineer General.This is a W-2, contract position.What You Will Do:

• Data Engineer Design and build production data engineering solutions to deliver data pipeline patterns using following Google Cloud Platform (GCP) services:
• In-depth understanding of Google's product technology and underlying architectures.
• BigQuery - Warehouse/ data marts - Through understanding of Big Query internals to write efficient queries for ELT needs, creation of views/materialized views, creation of reusable store procedures etc.
• DataFlow (Apache Beam) - reusable Flex templates/ data processing frameworks using Java for both batch and stream needs.
• Pub/Sub, Kafka, Confluent Kafka - Real time streaming of database changes or events.
• Experience of designing, building, and deploying production-level data pipelines using Kafka.
• Strong experience working on Event Driven Architecture.
• Strong knowledge of the Kafka Connect framework, with experience using several connector types HTTP REST proxy, JMS, File, SFTP, JDBC etc.
• Experience in handling huge volumes of streaming messages from Kafka.
• Cloud Composer (Apache Airflow) - to build, monitor and orchestrating the pipeline.
• Knowledge on BigTable.
• Cloud SQL, Compute Engine, Cloud Function, Cloud Run and App Engine, Cloud Storage.
• Experience with open-source distributed storage and processing utilities in the Apache Hadoop family.
• Extensive knowledge on processing various file formats orc, Avro, csv, json, xml etc.
• Knowledge/experience in any ETL tools like DataStage/Informatica - Ability to understand existing on-premises ETL workflows and redesign them in GCP.
• Experience and expertise on Terraform to deploy the GCP's in CI/CD.
• Knowledge/ Experience on connecting to on-prem API's from google cloud.Basics:
  • GCP
  • Big Query Terraform
  • Kafka
  • Cloud run
  • MUST be willing to work as a W-2How You Will Be Successful:
    • Envision the Future
    • Communicate Honestly and Broadly
    • Seek Technology and Business "First"
    • Embrace Diversity and Take RisksWhat We Offer:
      • Competitive Salary
      • Comprehensive Benefit Package
      • 401(k) with matching contributions
      • Paid Time Off
      • Employee Discounts
      • Free training on all Altair productsWhy Work with Us:Altair is a global technology company providing software and cloud solutions in the areas of data analytics, product development, and high-performance computing (HPC). Altair enables organizations in nearly every industry to compete more effectively in a connected world, while creating a more sustainable future. With more than 3,000 engineers, scientists, and creative thinkers in 25 countries, we help solve our customer's toughest challenges and deliver unparalleled service, helping the innovators innovate, drive better decisions, and turn today's problems into tomorrow's opportunities.Our vision is to transform customer decision making with data analytics, simulation, and high-performance computing.For more than 30 years, we have been helping our customers integrate electronics and controls with mechanical design to expand product value, develop AI, simulation and data-driven digital twins to drive better decisions, and deliver advanced HPC and cloud solutions to support unlimited idea exploration. To learn more, please visit altair.comReady to go? #ONLYFORWARD

Keywords: Altair, Dearborn , Data Engineer General, Engineering , Dearborn, Michigan