Düsseldorf, North Rhine-Westphalia, Germany
Data Platform Team Lead | Cloud & Big Data Enthusiast | Team Player & Mentor I’m a Data Platform Team Lead with a passion for building scalable, efficient, and future-ready data solutions. Over the years, I’ve worked across Azure, AWS, and open-source technologies, architecting high-performance data platforms that empower businesses to make better decisions. Beyond the technical side, I genuinely enjoy working with people. I believe that strong teams, open communication, and shared learning are just as important as the technology we use. Leading a team is not just about guiding projects—it’s about supporting people, helping them grow, and creating an environment where we all succeed together. I believe that great teams are built on trust, flexibility, and continuous learning. Whether it's mentoring junior engineers, brainstorming solutions with stakeholders, or navigating tough technical challenges, my goal is always to listen, support, and find the best way forward together. What I Bring to the Table ✅ Building Scalable Data Platforms – Designing solutions that handle massive datasets efficiently. ✅ Cloud Expertise (Azure & AWS) – Optimizing cost, performance, and security for modern data platforms. ✅ ETL, Streaming & Real-Time Analytics – Bringing data to life with pipelines that process billions of records. ✅ Collaboration & Mentorship – Helping my team grow and learn while delivering great results. ✅ Solving Challenges with a Calm Mindset – Breaking down complex problems into simple, actionable steps.
Primary Responsibility: Architecting a Cutting-Edge Data Migration and Modernization Initiative I am spearheading a transformative initiative aimed at modernizing our data infrastructure. This entails a comprehensive migration strategy from our existing, legacy systems to a state-of-the-art data solution, ensuring seamless data integration, enhanced scalability, and optimized performance. Leveraging the principles of the modern analytics architecture, as outlined in Microsoft's documentation (Azure Databricks Modern Analytics Architecture), my objective is to: -Design and Implement a Scalable Data Architecture: Develop a robust, flexible data architecture that supports real-time analytics, machine learning, and data science capabilities. -Ensure a Smooth Migration Path: Craft a detailed migration plan that minimizes downtime and data loss, ensuring a smooth transition from legacy systems to the new platform. This includes data cleansing, validation, and incremental migration strategies to maintain data integrity and business continuity. -Promote Data-Driven Decision Making: By modernizing our data platform, enable more sophisticated, real-time analytics that can drive strategic business decisions. The new architecture will provide stakeholders with timely insights, leveraging big data and AI capabilities to uncover new opportunities and optimize operations. -Foster Innovation and Agility: The new data solution will not only address current needs but also provide a scalable foundation for future growth. It will empower our teams to experiment with new data models, analytics techniques, and AI applications, thereby fostering a culture of innovation and agility within the organization. -Enhance Data Security and Compliance: Prioritize data security and compliance throughout the migration process and in the new architecture. Implement best practices and Azure's built-in security features to protect sensitive information and meet regulatory requirements.
I am at the forefront of delivering cutting-edge solutions in the Azure Cloud environment, leveraging a suite of advanced tools and technologies. My role involves intricate collaboration with multiple teams to develop and integrate robust cloud data and analytics solutions. Responsibilities: - Spearheading the development of high-performance, cost-efficient data infrastructures using Azure technologies such as Azure Data Factory (ADF), Azure Databricks, Azure Data Lake Storage (ADLS), Azure Synapse Analytics, and Apache Kafka. - Expertly building and managing complex ETL pipelines, ensuring efficient data integration and transformation. My proficiency in PySpark has been instrumental in this regard. - Leading the design and implementation of Modern Data Warehouse solutions, utilizing the full spectrum of the Azure Stack to achieve optimized data storage, processing, and analytics capabilities. Pioneering in the realm of data wrangling, adeptly handling heterogeneous data sets to derive actionable insights and support business intelligence and machine learning initiatives. - Championing the use of Azure DevOps and CI/CD methodologies to streamline development processes, enhance collaboration, and ensure the delivery of high-quality, reliable data solutions. - Actively participating in the development of cloud-based data services, focusing on scalability, reliability, and security to meet diverse business needs. My role as a Senior Data Engineer is characterized by a relentless pursuit of excellence in leveraging Azure's capabilities to build state-of-the-art data solutions. My expertise in complex data infrastructure development and my commitment to efficient, cost-effective solutions have been key drivers of success in my current position.
- Big Data Cluster Migration: Successfully led the transition from Oracle to a comprehensive big data cluster. This strategic shift involved the integration of Kafka, Spark, Hadoop, Hive, Kudu, Druid, and Presto. This migration resulted in significant annual cost savings and a notable boost in performance. - Real-Time Data Pipeline Design and Implementation: Engineered a robust real-time data pipeline, capable of processing both structured and semi-structured data. This innovative solution integrated 150 billion raw records from over 20 diverse data sources, utilizing Kafka and Spark (Scala and Spark-SQL) for efficient data handling. - AWS Integration for Enhanced Storage and Computing: Expanded the data infrastructure to include AWS services, utilizing S3 buckets for scalable data storage and EC2 for computing, specifically for managing Spark clusters. This integration significantly improved the scalability and flexibility of our data processing capabilities. - Data Product Development: Focused on building user-centric data products, including applications and APIs, ensuring that they are intuitive, reliable, and meet the evolving needs of users. - Continuous Technology Evaluation: Committed to the ongoing evaluation of emergent and successful tech stacks, libraries, frameworks, and tools, fostering a culture of continuous improvement and innovation within the team. - Team Leadership and Strategic Collaboration: Oversaw a team of 5 data engineers, playing a pivotal role in guiding their development and project contributions. Collaborated closely with company management to recommend strategic changes, informed by comprehensive data history analysis and rigorous testing. Tools and Technologies: Big Data Tools: Hadoop, Hive, Airflow, Spark, Kafka, Flume, Zookeeper, Presto, Kudu, Zeppelin Programming Languages: Scala, Python Database Management: MongoDB Cloud Computing: AWS S3 (for storage), AWS EC2 (for computing and managing Spark clusters)
- Designed and implemented ROLAP and MOLAP by using SSAS and Oracle OLAP - Created more than 30 ETLs from scratch to identify operational impact, trends and opportunities - Developed custom data and integrated them with Python, Tableau, and Power BI to visualize the data. - Created real-time data pipelines to provide insights to debug system integrations and improve operating efficiency - Developed the expertise, proficiency, and overall talent of junior members of the team by providing guidance, training, and mentoring
- Built basic ETL that ingested transactional and event data from an ERP system with 500,000 daily transactions. - Maintained and revised data warehouse procedures. - Identified and leveraged opportunities to continuously improve data management processes, standards, and procedures. - Worked with clients to understand business needs and produce actionable reports in Tableau or Power BI. - Developed test scenarios and test plans for testing ETLs and reports.