Bengaluru, Karnataka, India
Data Engineer with over 6 years of experience in designing and developing scalable data pipelines, real-time streaming systems and cloud-based data warehousing solutions. Skilled in translating complex business requirements into efficient, high-impact data solutions that power advanced analytics, drive product innovation, and fuel sustainable business growth. Collaborates effectively with cross-functional teams/stakeholders to deliver actionable insights and foster long-term organizational growth. Proactive mentor to junior engineers, promoting continuous learning.
As part of the Data & AI Engineering team (Data/ML/AI/Software engineers), contributing across diverse data-engineering projects with a growing focus on ML/AI engineering. ➢ Led migration of Beacon's property and borrower data layer from a premium provider (Cherre) to a cost-effective alternative (Forecasa), targeting ~$100K/year in savings while preserving functionality across two user-facing portals — via a strangler-fig pattern behind a feature flag enabling a zero-downtime cutover with one-switch rollback. ➢ Built a resilient Forecasa REST client (token auth, retry/backoff, pagination) integrating four external data sources (Forecasa, FEMA flood data, US Census ACS, FHFA indices) behind a unified provider interface, an in-house Automated Valuation Model (AVM, hybrid HPI/tax-assessor) at $0 cost replacing the vendor's paid feed, and test-driven parity-comparison tooling (Python/FastAPI + React/TypeScript) with field-level diff confirming 99% data parity. ➢ Re-architected the company's data-products Airflow platform from monolithic files into a standardized, layered package structure that extracts business logic out of DAG definitions — shipped with zero production breakage. ➢ Improved reliability and scalability across 20+ production DAGs: an S3-backed XCom layer (Parquet/Snappy) replacing native metadata-DB XCom to eliminate database bloat and OOM failures, a resource-monitoring and Slack alerting system that profiles per-task CPU, memory, and OOM-threshold breaches and pushes real-time breach alerts to the team, and a DAG-staggering strategy to smooth scheduler load on MWAA — plus standardized Alembic migration tooling on ephemeral EC2 GA runners. ➢ Led platform-wide standardization of the automated loan-underwriting check library, migrating 100+ legacy rules into a single dbt format with unified outcomes, powering real-time underwriting decisions and the exception-tracking workflow. Established automated unit testing (TDD) as the standard for new checks.
As part of Supply Chain Analytics team, I collaborated with multiple Data Engineering & Analytics professionals. This role gave me exposure to various technologies used to process and transform large-scale data within minimal time, and helped me understand the overall journey of a product. ➢ Designed and developed scalable ETL pipelines to identify functional and systemic breaches across the supply chain, spanning warehouses (FC), mother hubs (MH), delivery hubs (DH), and transport/line haul (LH), enabling real-time anomaly detection and faster issue resolution, ultimately improving delivery reliability and customer satisfaction. ➢ Developed and optimised 60+ batch pipelines using Apache Spark, integrating 10+ diverse data sources (including snapshots, journals, dimension tables, fact tables, static uploads etc.) to efficiently process terabytes of data, enhancing operational visibility across teams and reducing dependency on manual reporting. ➢ Reduced report generation time from 3 days (D-3) to 1 day (D-1) by improving pipeline runtimes by up to 7x (average 3x), enabling stakeholders to make faster, data-driven decisions that directly impacted daily business operations. ➢ Optimised resource utilisation, cutting compute costs by 5x on average and up to 40x in peak cases, while improving memory efficiency during read operations, significantly reducing infrastructure spend and increasing processing throughput. ➢ Automated data validation across all batch pipelines using DQA scripts, ensuring consistent data quality and integrity across teams, minimising the risk of reporting errors and increasing trust in analytics outputs. ➢ Implemented a near-real-time data warehousing solution using Flink CDC and Spark Structured Streaming to stream data from OLTP (RDBMS) systems to OLAP (Google BigQuery), enabling live analytics and dashboards that empowered business leaders with up-to-date insights for decision-making.
As part of a Data Science R&D team, I had the privilege of working with a diverse group of professionals, Including Data Engineers, Analysts, Scientists & Subject Matter Experts, all under the guidance of an exceptional Manager. This experience allowed me to gain exposure to various technologies, engage in cross-functional collaboration & contribute to the end-to-end development of multiple products. ➢ Designed and developed ETL pipelines on Google Search data and built dashboards for in-depth analysis of user intent, historical and geographical market trends. Leveraged resulting insights to inform and drive new product development strategies, aligning business initiatives with market demands. ➢ Designed and developed ETL workflows by utilising YouTube API data and built dashboards to support influencer marketing by identifying high-impact collaborations with relevant influencers closely aligned with relevant brands and categories. ➢ Developed batch pipelines to clean and analyse competitor's products data, enabling market research, gap analysis, and strategic new product development, ensuring alignment with competitive landscape. ➢ Developed a custom NER (Named Entity Recognition) model with over 95% accuracy to extract entities such as brands, categories, ingredients, benefits, concerns, product types, grammage, and more from product titles and descriptions. This enabled us to Identify gaps at entity level leading to more effective product development strategies. Utilised it in accurate SKU classification as well. ➢ Developed a price-tracking tool to monitor competitors, empowering business analysts to assess market competition and optimise sales by effective pricing and promotional strategies. ➢ Built automated data scraping tools to extract data from e-commerce platforms for market research, and from social media platforms like Youtube, Instagram, and reddit for influencer marketing.
As part of a skilled team of Full-Stack Python Developers, I got the opportunity to learn and work with a range of cutting-edge technologies. This collaborative experience enabled me to confidently design and develop web applications that are scalable, user-friendly, and capable of handling high traffic. Furthermore, I honed my ability to quickly adapt to new technologies, which allowed me to continuously enhance my professional capabilities. ➢ Ingested IPL datasets pulled from Kaggle into PostgreSQL, processed and cleaned it using Python, Pandas, and SQL, then created visualisations using Matplotlib to deliver actionable insights on player and team performance. ➢ Designed and developed a Python-Django based Quiz application enabling authenticated users to navigate through quiz categories, partake in a range of quizzes, create own custom quizzes, receive immediate question-level feedback, and track scores via a leaderboard. ➢ Built a fully functional video streaming platform like Youtube using Python and Flask, featuring secure authentication, video content creation and upload, and interactive user engagement through views, likes, dislikes, and comments. ➢ Successfully completed the training period, and subsequently deployed to their client Purplle in Mumbai.