DearbornRecruiter Since 2001
the smart solution for Dearborn jobs

GCP Data Engineer

Company: PamTen, Inc.
Location: Dearborn
Posted on: June 21, 2022

Job Description:

Job Description Primary skill BigQuery, SQL, DataFlow/Apache Beam - core Java Primary skill Worked in both Batch and streaming pipeline with Pub/sub, Kafka & using APIs Secondary skill Apache Airflow (composer)- Python Secondary skill Experience in ETL/ELT/CDC tools like DataStage, Informatica, Talend etc preferred Preferred Certification Google Data Engineer professional Design and build production data engineering solutions to deliver data pipeline patterns using following Google Cloud Platform (GCP) services: + In-depth understanding of Google's product technology and underlying architectures + BigQuery - Warehouse/ data marts - Through understanding of Big Query internals to write efficient queries for ELT needs, creation of views/materialized views, creation of reusable store procedures etc. + DataFlow (Apache Beam) - reusable Flex templates/ data processing frameworks using Java for both batch and stream needs. + Pub/Sub, Kafka, Confluent Kafka - Real time streaming of database changes or events. + Experience of designing, building, and deploying production-level data pipelines using Kafka; Strong experience working on Event Driven Architecture + Strong knowledge of the Kafka Connect framework, with experience using several connector types HTTP REST proxy, JMS, File, SFTP, JDBC etc. + Experience in handling huge volumes of streaming messages from Kafka + Cloud Composer (Apache Airflow) - to build, monitor and orchestrating the pipeline + Knowledge on BigTable + Cloud SQL, Compute Engine, Cloud Function, Cloud Run and App Engine, Cloud Storage + Experience with open-source distributed storage and processing utilities in the Apache Hadoop family. + Extensive knowledge on processing various file formats orc, Avro, csv, json, xml etc. + Knowledge/experience in any ETL tools like DataStage/Informatica - Ability to understand existing on-premises ETL workflows and redesign them in GCP. + Experience and expertise on Terraform to deploy the GCP's in CI/CD. + Knowledge/ Experience on connecting to on-prem API's from google cloud.

Keywords: PamTen, Inc., Dearborn , GCP Data Engineer, Engineering , Dearborn, Michigan

Click here to apply!

Didn't find what you're looking for? Search again!

I'm looking for
in category

Log In or Create An Account

Get the latest Michigan jobs by following @recnetMI on Twitter!

Dearborn RSS job feeds