Noida, Uttar Pradesh, India
Data Engineering Intern Jan 17, 2022 – Jun 15, 2022 (6 months) Built and maintained scalable data pipelines using Apache Airflow and Apache NiFi for ETL orchestration. Developed data processing solutions using Python, PySpark, and SQL to handle large datasets efficiently. Designed and implemented configuration-driven pipelines using external config files (YAML/JSON) for flexible and reusable workflows. Generated and monitored metric/log files to track pipeline performance, failures, and data quality. Wrote and optimized complex SQL queries for data extraction, transformation, and validation. Used Bitbucket for version control and collaborative development. Worked on PyCharm for coding, debugging, and performance tuning. Managed pipelines in an on-premises environment, ensuring secure data processing. Performed data validation, troubleshooting, and root cause analysis to maintain data integrity.