London, England, United Kingdom
A Lead Data Engineer with a passion for building scalable, automated, and future-proof data platforms. With over 15 years of experience, I currently lead the Data Engineering team at a mortgage software company, where I’ve driven the transformation from a legacy SQL Server / SSIS solution to a modern, cloud-native data platform built on Azure Data Factory, Azure Databricks, and Delta Lake. I combine hands-on technical expertise with strategic vision having architected a metadata-driven pipeline framework, CI/CD automation, and bespoke Delta Lake schema evolution tooling. My work also includes advanced data validation solutions using Great Expectations, Pydantic, and custom JSON-driven configurations to ensure data quality and flexibility at scale. I’m particularly interested in modern data architectures, data quality frameworks, and engineering practices that bring software development rigour to data platforms - from automated testing to version control and deployment pipelines. Ultimately, I enjoy turning complex data challenges into elegant, maintainable solutions that empower teams and deliver real business value.
As Lead Engineer for the Data Engineering team, I’m responsible for the design and development of our enterprise data platform. I’ve overseen the transition from a traditional SQL Server / SSIS-based system to a cloud-native data platform built on Azure Data Factory, Azure Databricks, and Delta Lake - leading both the architectural vision and hands-on implementation. Key achievements include: - Delivering a metadata-driven pipeline framework supporting late schema binding for flexible, scalable data ingestion. - Implementing CI/CD pipelines with PyTest for automated testing and deployment. - Building a bespoke Delta Lake schema evolution and source control solution. - Integrating Great Expectations for data validation and developing a JSON-driven row validation engine with Pydantic-based configuration checks. - Creating a robust logging framework with outputs to console, Delta Lake, and ADLS. - Contributing domain expertise to CAIS (CRA reporting), MLAR, PSD001 & PSD007 regulatory reporting.
Administration and development of the company’s Business Intelligence solutions, utilising SQL Server 2005/2008 and the full Microsoft BI stack (SSIS, SSAS and SSRS). Responsibilities include: • Development and administration of Data Warehouses using dimensional modelling techniques such as star/snowflake schemas and incorporating Ralph Kimball best practices, Database performance tuning through effective index strategies, index defragmentation, creation and optimisation of T-SQL Stored Procedures, Assemblies (using C#), Views and Functions, maintenance of backup processes, archiving of data and security management. • Create and administer ETL processes to load data from various disparate data sources, using SSIS Packages, Stored Procedures & Functions to cleanse and load the data. Data sources include flat files and OLTP systems. • Development and maintenance of very large OLAP Cubes. Includes the creation of Facts, Dimensions, calculated measures, partition slicing and aggregations. Some of the larger cubes are in excess of 1 TB and so require an effective strategy for processing objects and optimising query performance. Recently implemented an SSIS package to perform incremental processing of measure groups and dimensions, yielding considerable time savings in OLAP Processing.
Involved with a number of technologies including MSSQL Server 2000, 2005 & 2008, SSIS (2008), SSAS (2008) and .NET 3.5 (C#). Implemented a Microsoft Business Intelligence solution for Sales Analysis. I was involved in all aspects of the development lifecycle, from prototyping and demonstrating to stakeholders, to the creation of a full scale solution. Entailed developing a Data Warehouse, ETL processes to load data from disparate data sources utilising SQL Stored Procedures, Functions & SSIS packages and the design and development of an OLAP cube. Includes partition slicing and aggregation strategies, dimension attribute relationships and user hierarchies, KPI's and calculated measures. Used SSIS packages to automate processing of SSAS objects, including the creation of partitions and aggregations and execution of cache warming scripts. I also had some exposure with SSRS, migrating a number of existing reports to this service. .NET projects undertaken included developing automated file importers for data stored in various formats from specified repositories, parsing, verifying and cleansing data for insertion into OLTP databases. I also developed applications to automate the generation of reconciliation feeds.
Responsibilities included creating reports in a variety of formats for end users, including ad-hoc SQL Reports, Stored Procedures, Crystal Reports and other bespoke reporting solutions. Working in a high volume transactional environment emphasised the need to produce efficient and optimised T-SQL procedures. Other responsibilities included support tasks such as the setup of new products and services, liaising with customers and suppliers, management of electronic stock levels, ensuring availability of services and outage resolutions. I was also responsible for managing the team’s relations with our Polish offices on a weekly and ad-hoc basis to manage projects and gather requirements, ensuring their workload was prioritised alongside UK commitments. Some of the more complex tasks undertaken for Poland involved integrating a new billing model, for which I visited the Polish offices to gain a better understanding of their specific business model and operations. This was a very beneficial trip, helping me prepare for other client facing tasks. This led to other opportunities to attend client facing meetings, including the launch of new products to retailers and service review meetings with suppliers. During my time in this role I looked to improve upon a number of processes. This includes streamlining existing reporting procedures to run more quickly and efficiently, thus reducing Server overhead. I created a number of analysis reports; one such instance monitored index usage on our Reporting Server and identified under utilised and potentially unnecessary indexes. Showing an interest in .NET, I was also given the opportunity to design and build an application to automate the import of Accounting data into our reporting database for UK and other international subsidiaries.