Seattle, Washington, United States
- Competency with hands-on experience in both Hadoop cluster administration and Big Data application development. - Managed GPU Kubernetes cluster and provided MLOps support to Data Scientists - Solid understanding and practical experience with AWS and Hadoop ecosystem (YARN/MapReduce/Hive/Spark/Oozie/Airflow) - Developed Spark applications to process Kafka stream transaction data to populate sales report - Adjusted Hadoop cluster configurations to achieve performance improvement and resolved system environment issues.
Data Platform Team, Senior Software Engineer II - Owned and managed big-data infrastructure, including EMR, Athena (Data Lake), and Snowflake (Data Warehouse). - Leveraged industry-standard technologies such as Airflow, Hudi, and Terraform to build scalable, maintainable, and production-grade data systems. - Built and operated an in-house Temporal platform to support mission-critical workflows with high reliability and fault tolerance. - Led the architecture and operations of the company’s service log pipeline, powering customer-facing analytics and downstream data workflows. - Designed and developed a scalable event-logging framework that serves as the foundation of the organization’s SIEM logging pipeline.
Data Infrastructure Team, Senior Backend Engineer - Processed sales transaction data from Kafka stream to populate sales report in near real-time (NRT) by leveraging Spark with Airflow scheduler - Managed GPU Kubernetes cluster and provided MLOps support offering Feature Engineering/ML tools to Data Scientists - Gatekeeper to manage and monitor all the company's EMR (Hadoop) clusters (more than a hundred) - Provided general guidance to users to troubleshoot and tune Hive/Spark applications - Designed and developed company's internal API service where users can easily spawn customized EMR clusters
Big Data Platform Team, Sr. Data Engineer - Administered 24/7 multi-tenant CDH Hadoop cluster - Supported stakeholders located globally (Palo Alto, Seattle, Shanghai) who use the Hadoop data and cluster for mission critical applications - Managed Sqoop ETL applications to collect and refine data from various data sources (OLTP, Data Mart, Weblogs) - Heavily participated on company-wide cloud migration project, especially focused on Big Data platform part - Migrated on-premise CDH cluster to EMR clusters - Automated EMR cluster provisioning process with DevOps tools (Ansible, Bigtop-puppet).
Data Platform Team, Hadoop Engineer - Developed Storm application to calculate main web page latency statistics in nearly real-time (NRT) - Collaborated building Weblog data ingestion pipeline which collects user’s behavior data and send to Kafka and HDFS - Managed Web Coupon generating system (consists of MapReduce, Hbase, and Redis) which expose seller items with competitive price compared to other e-commerce website --------------------------- Intranet Team, Software Engineer - Developed in-house code deployment web application integrated with Git version control system - Developed intranet task management application integrating with other in-house administration tools