Dallas-Fort Worth Metroplex
Professional Summary ● Domains: Big data, Analytics, Apache Spark, Hadoop, Scala, and Python ● Hands on experience in developing data pipeline to perform ETL operation/aggregation on huge volume data in near real time that can be fed to downstream applications. ● Experience in using Apache Spark, Hive, SQOOP, HBASE, KAFKA. ● Built a Data lake that is handling billions of records using HBASE AND S3. Experience in Designing, creating, testing and maintaining the complete data management & processing systems. Experience in Creating a complete solution by integrating a variety of programming languages & tools together into the existing system to make it more efficient.
Responsibilities Develop and implement complex big data solutions with a focus on ingesting, parsing, transforming, analyzing, and querying large data sets that produce valuable business insights and discoveries. Driving innovation through collaboration across our data science teams and help push Business to the next level with technologies like Big Data, Machine Learning, Artificial Intelligence, Cloud Computing, SQL/NoSQL Databases, DevOps. Build robust automated ingestion pipelines using Python, Scala, Spark, Hive, Hadoop, Sqoop, Parquet, HDFS, Kafka to help enable the business to access all of our data sets in Data lake and Data Warehouse. Operationalize machine learning solutions that are deployed on Scalable Infrastructures like Kubernetes, Docker, and Serverless Amazon Cloud applications Support our Data Scientists by helping enhance their modeling jobs to be more scalable when modeling across the entire data set. Build complex solution to Expose the data to data services platform according to the API standards which can supports multiple version and delta functionality of data. Build the data lake of parquet files in amazon s3 that can handle incremental data as well as historical data based On the requirement given by data science team. Design and implement automated data pipelines, Data structures, Algorithms, APIs, data quality checks, CI/CD design, and SQL interfaces. Moving existing ETL jobs from traditional SQL database processing to the cloud based big data processing platform, ensure that the jobs are designed to scale. Provide technical support for optimize and performance tuning of ETL jobs in distributed cloud environment that result in cost saving and robust data management & processing platform.
Responsibilities Support our Data engineer by Moving traditional based data management application to the Microservices , Spark based near real time streaming application. Analyzed the data by performing Hive queries (Hive QL) and Worked on Partitioning, Bucketing, Join Optimizations and query optimizations in Hive. Ingested and Aggregated large amounts of data using Kafka and Spark. Responsible for designing, tuning and optimizing SQL queries based on functional specifications. Provide technical support to data analytics functions as they relate to varied business units, and technical expertise on the selection, development, and implementation of various reporting.
Responsibilities Analyzed and prioritized user and business requirements as system requirements that must be included while developing the software. Designed and implemented Stored Procedures while working with Oracle SQL server. Created database designs and proper database maintenance to maximize the database availability and high performance. Actively monitor the data for its accuracy and capacity to take corrective actions when needed. Reports status to all relevant management on the development of the project.