Frisco, Texas, United States
• Having 8+ Years of strong experience in Software Development Life Cycle (SDLC) including Requirements Analysis, Design Specification and Testing as per Cycle in both Waterfall and Agile methodologies. • Have very strong inter-personal skills and the ability to work independently and with the group, can learn quickly and easily adaptable to the working environment. • Good working experience on Spark (spark streaming, spark SQL) with Scala and Kafka. Worked on reading multiple data formats on HDFS using Scala. • Extensive knowledge in writing Hadoop jobs for data analysis as per the business requirements using Hive and worked on HiveQL queries for required data extraction, join operations, writing custom UDF's as required and having good experience in optimizing Hive Queries. • Experience in importing and exporting the data using Sqoop from HDFS to Relational Database systems and vice-versa and load into Hive tables, which are partitioned. • Developed custom Kafka producer and consumer for different publishing and subscribing to Kafka topics. • Good understanding of distributed systems, HDFS architecture, Internal working details of MapReduce and Spark processing frameworks. • Worked with various formats of files like delimited text files, click stream log files, Apache log files, Avro files, JSON files, XML Files. Mastered in using different columnar file formats like RC, ORC and Parquet formats. Has good understanding of various compression techniques used in Hadoop processing like G-zip, Snappy, LZO etc. • Created continuous integration and continuous delivery (CI/CD) pipeline on AWS that helps to automate steps in software delivery process • Experience in extracting files from MongoDB through Sqoop and placed in HDFS and processed. • Working Experience with Amazon Web Services (AWS) Cloud Platform which includes services like EC2, S3, VPC, ELB, IAM, DynamoDB, Cloud Front, Cloud Watch, Route 53, Elastic Beanstalk (EBS), Auto Scaling, Security Groups, EC2 Container Service (ECS), Code Commit, Code Pipeline, Code Build, Code Deploy, Dynamo DB, Auto Scaling, Security Groups, Red shift, CloudWatch, CloudFormation, CloudTrail, Ops Works, Kinesis, IAM, SQS, SNS, SES. • Excellent programming skills with experience in Java, C, SQL and Python Programming. • Involved in converting Hive/SQL queries into Spark transformations using Spark Data frames and Scala. • Experience in using Kafka and Kafka brokers to initiate spark context and processing livestreaming.
• Developed Spark code using Scala and Spark-SQL/Streaming for faster processing of data. • Developed Automation Regressing Scripts for validation of ETL process between multiple databases like AWS Redshift, Oracle, MongoDB, T-SQL, and SQL Server using Python. • Involved as primary on-site ETL Developer during the analysis, planning, design, development, and implementation stages of projects using IBM Web Sphere software (Quality Stage v9.1, Web Service, Information Analyzer, Profile Stage) • Worked in AWS environment for development and deployment of custom Hadoop applications. • Developing Spark programs with Python, and applied principles of functional programming to process the complex structured data sets. • Use SparkSQL to load JSON data and create Schema RDD and loaded it into Hive Tables and handled structured data using SparkSQL. • Designed and developed architecture for data services ecosystem spanning Relational, NoSQL, and Big data technologies. Extracted Mega Data from Amazon Redshift, AWS, and Elastic Search engine using SQL Queries to create reports. • Used Talend for Big Data Integration using Spark and Hadoop. • Used Kafka and Kafka brokers, initiated the spark context and processed live streaming information with RDD and Used Kafka to load data into HDFS and NoSQL databases. • Used Zookeeper to store offsets of messages consumed for a specific topic and partition by a specific Consumer Group in Kafka. • Reduced access time by refactoring information models, query streamlining and actualized Redis store to help Snowflake. • Strong experience in working with ELASTIC MAPREDUCE(EMR) and setting up environments on Amazon AWS EC2 instances. • Utilized Spark, Scala, Hadoop, HBase, Cassandra, MongoDB, Kafka, Spark Streaming, a broad variety of machine learning methods including classifications, regressions, dimensionally reduction etc. • Prepared Data Mapping Documents and Design the ETL jobs based on the DMD with required Tables in the Dev Environment
• Worked in Agile environment, and used rally tool to maintain the user stories and tasks. • Implement ad-hoc analysis solutions using Azure Data Lake Analytics/Store, HDInsight • Implemented Apache Sentry to restrict the access on the hive tables on a group level. • Employed AVRO format for the entire data ingestion for faster operation and less space utilization. • Designed SSIS Packages to extract, transfer, load (ETL) existing data into SQL Server from different environments for the SSAS cubes (OLAP) • Developed visualizations and dashboards using PowerBI • Good Exposure on Map Reduce programming using Java, PIG Latin Scripting and Distributed Application and HDFS. • Experienced in using Tidal enterprise scheduler and Oozie Operational Services for coordinating the cluster and scheduling workflows. • Designed and implemented by configuring Topics in new Kafka cluster in all environment. • Created multiple dashboards in tableau for multiple business needs. • Implemented Partitioning, Dynamic Partitions and Buckets in HIVE for efficient data access. • Architect & implement medium to large scale BI solutions on Azure using Azure Data Platform services (Azure Data Lake, Data Factory, Data Lake Analytics, Stream Analytics, Azure SQL DW, HDInsight/Databricks, NoSQL DB). • Implemented Composite server for the data virtualization needs and created multiples views for restricted data access using a REST API. • Exported the analyzed data to the relational databases using Sqoop for visualization and to generate reports for the BI team Using Tableau. • Installed Kerberos secured Kafka cluster with no encryption on Dev and Prod. Also set up Kafka ACL's into it • Developed Apache Spark applications by using spark for data processing from various streaming sources.
• Developed Python utility to validate HDFS tables with source tables • Conduct systems design, feasibility and cost studies and recommend cost-effective cloud solutions such as Amazon Web Services (AWS). • Loaded data into S3 buckets using AWS Glue and PySpark. • Automated all the jobs for pulling data from FTP server to load data into Hive tables using Oozie workflows • Implement code in Python to retrieve and manipulate data. • Designed ETL Process using Informatica to load data from Flat Files, and Excel Files to target Oracle Data Warehouse database. • Configured AWS Identity and Access Management (IAM) Groups and Users for improved login authentication. • Responsible for developing Python wrapper scripts which will extract specific date range using Sqoop by passing custom properties required for the workflow • Involved in filtering data stored in S3 buckets using Elasticsearch and loaded data into Hive external tables. • Designed and developed UDF'S to extend the functionality in both PIG and HIVE • Import and Export of data using Sqoop between MySQL to HDFS on regular basis • Developed a shell script to create staging, landing tables with the same schema as the source and generate the properties which are used by Oozie Jobs • Worked with NoSQL databases like HBase in creating HBase tables to load large sets of semi structured data coming from various sources • Developed Oozie workflows for executing Sqoop and Hive actions • Built various graphs for business decision making using Python matplotlib library.
• Installed and configured Hadoop Map Reduce, HDFS, developed multiple MapReduce jobs in java and Scala for data cleaning and preprocessing. • Supported MapReduce Programs those are running on the cluster. Involved in loading data from UNIX file system to HDFS. • Experienced in installing, configuring and using Hadoop Ecosystem components. • Installed and configured Hive and written Hive UDFs and Used Map Reduce and Junit for unit testing. • Queried and analyzed data from DataStax Cassandra for quick searching, sorting and grouping. • Experienced in working with various kinds of data sources such as Teradata and Oracle. Successfully loaded files to HDFS from Teradata, and load loaded from HDFS to hive and impala. • Used Yarn Architecture and MapReduce in the development cluster for POC. • Designed and implemented a product search service using Apache Solr/Lucene. • Involved in various NOSQL databases like Hbase, Cassandra in implementing and integration. • Experienced in Importing and exporting data into HDFS and Hive using Sqoop. • Participated in development/implementation of Cloudera Hadoop environment. • Implemented Kafka consumers to move data from Kafka partitions into Cassandra for near real-time analysis • Worked in installing cluster, commissioning & decommissioning of Data nodes, Name node recovery, capacity planning, and slots configuration.
• Responsible for gathering requirements from Business Analysts and identifying the data sources required for the requests. • Wrote SAS Programs to convert Excel data into Teradata tables. • Worked on importing/exporting large amounts of data from files to Teradata and vice versa. • Created multi-set tables and volatile tables using existing tables and collected statistics on table to improve the performance. • Wrote Python scripts to parse files and load the data in database, used Python to extract weekly information from the files, Developed Python scripts to clean the raw data. • Worked on data pre-processing and cleaning the data to perform feature engineering and performed data imputation techniques for the missing values in the dataset using Python. • Developed Teradata SQL scripts using OLAP functions like rank and rank () Over to improve the query performance while pulling the data from large tables. • Designed and developed weekly, monthly reports related to the Logistics and manufacturing departments using Teradata SQL.