Data Engineer

Tech Stack

CLOUD
PYTHON
BIG DATA
SNOWFLAKE
DATABRICKS
SCALA
APACHE SPARK
AIRFLOW
DOCKER
KAFKA

Job Description

Datumo specializes in providing Big Data and Cloud consulting services to clients from all over the world, primarily in Western Europe, Poland and the USA.

Core industries we support include e-commerce, telecommunications and life science.

Our team consists of exceptional people whose commitment allows us to conduct highly demanding projects.

Our team members tend to stick around for more than 3 years, and when a project wraps up, we don't let them go - we embark on a journey to discover exciting new challenges for them.

It's not just a workplace; it's a community that grows together!

What we expect: Must-have: at least 3 years of commercial experience in Big Dataproven record with a selected cloud provider GCP preferred, Azure or AWSgood knowledge of JVM languages - Scala or Java or Kotlingood knowledge of Pythongood knowledge of SQLunderstanding of Apache Spark or similar distributed data processing frameworkexperience with BigQuery, Snowflake, Hive or similar distributed datastoredesigning and implementing Big Data systems following best practicesensuring solution quality through automatic tests, CI / CD and code reviewproven collaboration with businessesEnglish proficiency at B2 level, communicative in PolishNice to have:experience in Snowflake/Databricks platformfamiliarity with Apache Airflow or similar pipeline orchestrator knowledge of Apache Kafka, Docker and Kubernetes technologiesknowledge of real-time data processingexperience in Machine Learning projectsexperience in Apache Flinkwillingness to share knowledge (conferences, articles, open-source projects)What’s on offer:100% remote work, with workation opportunity 20 free daysonboarding with a dedicated mentorproject switching possible after a certain periodindividual budget for training and conferencesbenefits: Medicover private medical care, co-financing of the Medicover Sport cardopportunity to learn English with a native speakerregular company trips and informal get-togethersDevelopment opportunities in Datumo:participation in industry conferencesestablishing Datumo's online brand presencesupport in obtaining certifications (e.g. GCP, Azure, Snowflake)involvement in internal initiatives, like building technological roadmapstraining budgetaccess to internal technological training repositories Discover our exemplary projects:IoT data ingestion to cloud The project integrates data from edge devices into the cloud using Azure services.

The platform supports data streaming via either the IoT Edge environment with Java or Python modules, or direct connection using Kafka protocol to Event Hubs.

It also facilitates batch data transmission to ADLS.

Data transformation from raw telemetry to structured tables is done through Spark jobs in Databricks or data connections and update policies in Azure Data Explorer.Petabyte-scale data platform migration to Google CloudThe goal of the project is to improve scalability and performance of the data platform by transitioning over a thousand active pipelines to GCP.

The main focus is on rearchitecting existing Spark applications to either Cloud Dataproc or Cloud BigQuery SQL, depending on the Client’s requirements and automate it using Cloud Composer.Data analytics platform for investing companyThe project centers on developing and overseeing a data platform for an asset management company focused on ESG investing.

Databricks is the central component.

The platform, built on Azure cloud, integrates various Azure services for diverse functionalities.

The primary task involves implementing and extending complex ETL processes that enrich investment data, using Spark jobs in Scala.

Integrations with external data providers, as well as solutions for improving data quality and optimizing cloud resources, have been implemented.

Realtime Consumer Data PlatformThe initiative involves constructing a consumer data platform (CDP) for a major Polish retail company.

Datumo actively participates from the project’s start, contributing to planning the platform’s architecture.

The CDP is built on Google Cloud Platform (GCP), utilizing services like Pub/Sub, Dataflow and BigQuery.

Open-source tools, including a Kubernetes cluster with Apache Kafka, Apache Airflow and Apache Flink, are used to meet specific requirements.

This combination offers significant possibilities for the platform.

Recruitment process:Quiz - 15 minutesSoft skills interview - 30 minutesTechnical interview - 60 minutesFind out more by visiting our website - you like what we do and you dream about creating this world with us - don’t wait, apply now!