Lead Data Engineer (Databricks)

Technologies we use

About the project

Your responsibilities

• Implement a scalable architecture capable of handling the high volume of simulation data
• Build a flexible data preprocessing pipelines that are extensible and that can be integrated into customer’s existing platform
• Define KPIs to measure the improved reusability and automation of the new pipelines and test their performance in an end-to-end setting with model training
• Develop and implement processes and best practices for data management and governance
• Optimize and enhance system setup and improve data structures following industry best practices
• Collaborate effectively with data engineering team members while partnering closely with analytics and data science teams to meet user needs
• Collaborate with business stakeholders and technical teams to understand data requirements, translate business needs into technical solutions
• Lead technical discussions and solution design sessions with clients or internal stakeholders, presenting complex data engineering concepts in accessible ways

Our requirements

• Master’s degree in computer science, Data Engineering, AI, or a related field
• 4+ years of professional experience in Data Engineering and Big Data, building production-grade data platforms and pipelines
• Proven experience with Databricks platform (Azure Databricks or Databricks on AWS)
• Strong experience with Apache Spark (PySpark), including performance optimization, and large-scale data processing
• Proficiency in Python and SQL for data transformations and analytics workloads
• Experience with data pipeline orchestration (Airflow, Dagster, Prefect or similar)
• Experience with data governance and quality frameworks
• Strong problem-solving skills and ability to work independently
• Fluency in English, both written and spoken

Optional

• PhD degree in computer science, Data Engineering, AI, or a related field (completed or in progress)
• Advanced Databricks expertise (Unity Catalog, Delta Lake and Lakehouse architecture)
• Data Engineer (Databricks) certificate
• Experience processing high-volume Automotive or IoT data (e.g., telemetry, sensor, or event-based data)
• Experience in pre-sales and consulting activities
• Experience with streaming tools like Kafka or Azure Event Hubs, etc.
• Experience with Infrastructure as Code (Terraform, CloudFormation)

This is how we organize our work

This is how we work

Team members

Development opportunities we offer

What we offer

• Cutting-Edge Technology: Drive innovation with the latest tech solutions
• Work Your Way: Enjoy complete flexibility in choosing your ideal work environment – office, hybrid or remote.
• Non-corporate work environment
• Equipment of your choice
• Language lessons (English, German, and Polish)
• LuxMed private medical care
• Weekly Lunch & Learn where we meet up in the office, lunch together, and share our knowledge
• Employee referral program
• Rewards for the Success of the Month and Year awarded by our employees
• G-Man work anniversary awards
• Integration activities

Benefits