São Paulo, São Paulo, Brazil
Data professional currently focused on Data Engineering and Data Platforms, with a track record from early-stage startups, enterprises, and consultancy firms. Experienced in developing and maintaining data lakes a data warehouses, particularly using SQL, Spark and Python in AWS and GCP clouds. Also skilled in leading cross-team initiatives and aligning with business stakeholders to drive impact through data. Architects and implements data solutions leveraging concepts such as Data Lakes, Data Warehousing, Data Streaming, BI, DataOps and Data Governance. Techs: Python, SQL, AWS, GCP, Spark, BigQuery, Athena, Databricks, Kafka, Airflow, Terraform, CI/CD, Kubernetes, PostgreSQL, Pulumi. Full Professional Proficiency in English
Activities: • Design, develop, and manage data processing solutions on GCP (BigQuery, Cloud Storage, Cloud SQL, BigTable, Dataflow, Dataproc, Composer) and AWS (Redshift, Glue, RDS, Athena, Kinesis) along with Dbt to support Analytics and AI/ML workflows; • Build and optimize streaming pipelines using Apache Kafka, Apache Beam, Cloud SQL, BigTable and GCS; • Migrate legacy Data Quality and Governance solutions (Informatica IDQ, PowerCenter, MDM) to cloud-native architectures, leveraging tools such as GCP Dataform; DLP and Dataplex; • Deploy and maintain data products (Streamlit and FastAPI) as containers, running on GKE and Cloud Runs; • Maintains AI workflows with Fuel IX and Vertex AI Search; • Monitor and observability with dashboards and alerts using Grafana, Prometheus, Datadog and GCP Cloud Logging/Monitoring; • Implement CI/CD pipelines with GCP Cloud Build and Artifact Registry; • Manage GitOps workflows with GitHub Actions; • Define and provision infrastructure with Terraform (IaC); • Align technically with SRE, Infrastructure, Security and Networking teams to ensure ongoing maintenance, reliability, and compliance of the data platform/products; • Provide technical leadership and mentorship to data engineers, guiding best practices in cloud architectures and data pipelines; • Collaborate with business teams to design, deliver, and operationalize data products that add values to the business.
Cora is a fintech that provides digital banking services for small and medium-sized businesses (SMBs). It offers services including business accounts, payment solutions, credit and financial management tools. Activities: • Alignment and refinement with projects stakeholders to ensure data solutions meet both business needs and technical requirements; • Develop and maintain the data lake environment (Airflow, Debezium CDC, AWS Lake Formation, Spark (PySpark, SparkSQL), Athena, S3, GCS, Glue, Kinesis, Lambda, BigQuery); • Creates and maintain relational and non-relational data pipelines for batch and streaming data platforms; • Maintains internal libraries and implements Unit and Integration Tests to achieve data quality; • Provision of data artifacts using Terraform (IaC); • CI/CD and GitOps with ArgoCD and GitHub Actions; • Monitoring, Observability and Alerts with Prometheus and Grafana; • Evaluates and provides technical guidance to juniors peers. Projects: • Led the migration from a legacy Airflow setup to a fully customized, self-hosted Kubernetes deployment fully controlled by the data team, achieving SLA Uptime of 99%, allowing the company to orchestrate data with reliability and confidence; • Designed and implemented a real-time, low-latency data stack achieving latency P95 of ~300ms, using Kafka (Confluent), Apache Pinot and Flink, deployed on a GKE Kubernetes cluster, which enabled: - Real-time ML feature engineering for fraud-prevention models; - Launch of a paid feature “Extrato Inteligente” in the end-user app for the entire customer base which consumes real-time data; • Led the internalization/replacement of expensive third-party software components by implementing a solution with postgREST, postgreSQL, Cloud Run, Cloud Functions and Retool (low-code UI), allowing the company to reduce costs by ~82% with licenses and servers. • Currently conducting a POC to ingest CDC NRT data with Apache Iceberg and S3 Tables.
Enjoei is a Brazilian online peer-to-peer marketplace where users can buy and sell new and second-hand items, primarily focused on fashion, accessories and home decor. • Led the deployment and maintenance of Apache Amundsen and internal libraries to improve Data • Governance and Data Discovery, using AWS EKS, ECS and Neo4J (graph DB). This initiative documented and enforced data quality to 91% of the tables in the golden layer of the data lake, allowing the company to make the best usage of its data; • Developed and maintained data pipelines using BigQuery, DBT, Airflow, Dataform and Embulk; • Worked along with the DevOps/SRE team to establish a dedicated AWS account for the data team.
• Designed and maintained pipelines to ingest and process ~20PB (daily) of structured and unstructured data from hundreds of external sources. These data were later used on the company’s commercial data products either as features (variables) or ML models predictions for both Individuals and Legal Entities. Used mainly Spark on GCP Dataproc, BigQuery, BigTable, Composer (Airflow) and GCS. GitOps with GitLab; • Maintained legacy pipelines on Cloudera clusters with Spark(Scala), Java and Impala.
Joined the Guiabolso data team to help on scaling the stack for processing open-finance data products. Activities: • Maintained the dala-lake pipelines using Databricks, EMR(Spark, HiveQL, Hive Metastore, HDFS), Athena, DMS, ECS, S3, EC2 and Glue, mainly with Python, Spark, SQL and shell script; • Modeled and developed tables for the Delta Lake environment using Medallion architecture; • Maintained and developed tables for the business data marts using AWS Redshift; • Developed data pipelines to ensure compliance with LGPD regulations; • Trouble-shoot of data intensive Spark workloads using SparkHistoryServer, SparkUI and Ganglia on Databricks and EMR clusters; • Developed a Kappa pipeline to ingest near real-time and scheduled data from MixPanel API, using API Gateway, Lambda, Kinesis Data Firehose, S3 and Athena, addressing the marketing team; • CI/CD: Jenkins, CircleCi. • Monitoring & Alerting: Datadog, NewRelic and AWS CloudWatch; • Worked with Databricks and AWS technical teams to resolve internal issues and conduct POCs to advance the company’s data platform. • Cloud costs optimization projects: - Migrated ETLs from Databricks Jobs Clusters to Athena, reducing the costs with Databricks DBUs and EC2 servers by ~34%; - Migrated Dashboards from Looker to open-source Redash, reducing costs with Looker seats.