Data Engineer (Spark)

Technologies we use

About the project

Your responsibilities

• Develop and maintain a high-performance data processing platform for automotive data, ensuring scalability and reliability.
• Design and implement data pipelines that process large volumes of data in both streaming and batch modes.
• Optimize data workflows to ensure efficient data ingestion, processing, and storage using technologies such as Spark, Cloudera, and Airflow.
• Work with data lake technologies (e.g., Iceberg) to manage structured and unstructured data efficiently.
• Collaborate with cross-functional teams to understand data requirements and ensure seamless integration of data sources.
• Monitor and troubleshoot the platform, ensuring high availability, performance, and accuracy of data processing.
• Leverage cloud services (AWS) for infrastructure management and scaling of processing workloads.
• Write and maintain high-quality Python (or Java/Scala) code for data processing tasks and automation.

Our requirements

• At least 3 years of commercial experience implementing, developing, or maintaining Big Data systems, data governance and data management processes.
• Strong programming skills in Python (or Java/Scala): writing a clean code, OOP design.
• Hands-on with Big Data technologies like Spark, Cloudera, Kafka, Data Platform, Airflow, NiFi, Docker, and Iceberg,.
• Excellent understanding of dimensional data and data modeling techniques.
• Experience implementing and deploying solutions in cloud environments.
• Consulting experience with excellent communication and client management skills, including prior experience directly interacting with clients as a consultant.
• Ability to work independently and take ownership of project deliverables.
• Fluent English (at least C1 level).
• Bachelor’s degree in technical or mathematical studies.

Optional

• Experience with an MLOps framework such as Kubeflow or MLFlow.
• Familiarity with Databricks and/or dbt.

Division of working time

This is how we organize our work

This is how we work

Team members

Development opportunities we offer

What we offer

• Work in a supportive team of passionate enthusiasts of AI & Big Data.
• Engage with top-tier global enterprises and cutting-edge startups on international projects.
• Enjoy flexible work arrangements, allowing you to work remotely or from modern offices and coworking spaces.
• Accelerate your professional growth through career paths, knowledge-sharing initiatives, language classes, and sponsored training or conferences, including a partnership with Databricks, which offers industry-leading training materials and certifications.
• Choose from various employment options: B2B, employment contracts, or contracts of mandate.
• Make use of 20 fully paid days off available for B2B contractors and individuals under contracts of mandate.
• Participate in team-building events and utilize the integration budget.
• Celebrate work anniversaries, birthdays, and milestones.
• Access medical and sports packages, eye care, and well-being support services, including psychotherapy and coaching.
• Get full work equipment for optimal productivity, including a laptop and other necessary devices.
• With our backing, you can boost your personal brand by speaking at conferences, writing for our blog, or participating in meetups.
• Experience a smooth onboarding with a dedicated buddy, and start your journey in our friendly, supportive, and autonomous culture.

Benefits