João Rodrigues

João Rodrigues

Data Engineer
Lisbon, Portugal.

About

Accomplished Data Engineer with over 5 years of experience, specializing in architecting and optimizing scalable data pipelines and real-time data processing solutions. Proven track record in leveraging AWS, Terraform, and PySpark to drive efficiency, enhance workflows, and deliver actionable insights within complex cloud and big data environments.

Work

Critical Techworks - BMW Group
|

Data Engineer & Scrum Master

Lisbon, Lisbon, Portugal

Summary

Led data engineering initiatives and agile scrum processes, driving scalable data solutions and enhancing team efficiency within a dynamic cloud environment.

Highlights

Architected and deployed scalable real-time data ingestion and processing pipelines in AWS, leveraging Lambda, Glue, and S3 to ensure high data availability and integrity.

Developed comprehensive functional testing sets and a robust CI/CD pipeline with GitHub Actions, significantly improving deployment reliability and efficiency.

Built and maintained analytical tables on Apache Iceberg, adhering to best practices to optimize data accessibility and performance for critical business insights.

Automated infrastructure deployments using Terraform, reducing manual effort and accelerating deployment times for data solutions.

Drove team efficiency and alignment with BMW's data initiatives as Scrum Master, iteratively adapting delivery metrics and fostering continuous improvement.

BNP Paribas
|

Data Engineer / Data Analyst

Lisbon, Lisbon, Portugal

Summary

Designed and implemented robust ETL/ELT processes and analyzed large-scale datasets to generate actionable insights for credit risk management and regulatory compliance.

Highlights

Designed and implemented efficient ETL/ELT processes for critical credit risk datasets within a distributed HDFS environment, ensuring data integrity and availability.

Analyzed large-scale datasets using PySpark and Hive SQL, generating actionable insights crucial for EBA Stress Testing and other regulatory frameworks.

Developed interactive Tableau dashboards, empowering stakeholders to monitor key risk metrics and compliance KPIs, enhancing data-driven decision-making.

Celfinet
|

Software Developer Intern

Lisbon, Lisbon, Portugal

Summary

Developed an object-oriented Python application to automate mobile app testing, significantly reducing operational time.

Highlights

Developed an object-oriented Python application for automating app usage testing on iOS/Android devices, reducing testing time by more than 20%.

Education

Instituto Superior Técnico
Lisbon, Lisbon, Portugal

M.Sc.

Electrical and Computer Engineering

Grade: 16/20

Courses

Thesis: Time series analysis using restricted Boltzmann machines and dynamic Bayesian networks (Grade 18/20)

Télécom ParisTech
Paris, Île-de-France, France

ATHENS Programme

Université Catholique de Louvain
Louvain-la-Neuve, Walloon Brabant, Belgium

Erasmus Programme

Languages

Portuguese

Native

English

Proficient

Spanish

Proficient

French

Proficient

German

Proficient

Certificates

Professional Scrum Master I/II

Issued By

Scrum.org

Neural Networks and Deep Learning

Issued By

Coursera

Azure Data Fundamentals

Issued By

Microsoft

Spark and Python for Big Data with PySpark

Issued By

Udemy

Skills

Cloud Platforms

AWS, Azure Data Fundamentals, Lambda, Glue, S3.

Programming & Data Tools

Python, PySpark, SQL, Apache Iceberg, Spark, Hive SQL, Tableau, Hugo.

DevOps & Orchestration

Terraform, GitHub Actions, Docker, CI/CD.

Methodologies & Data Management

Scrum, Agile, ETL/ELT, HDFS, Data Ingestion, Data Processing, Data Pipelines.

Machine Learning & AI

Neural Networks, Deep Learning, Generative AI (GenAI), Restricted Boltzmann Machines, Dynamic Bayesian Networks.

Projects

Personal Blog

Summary

Developed and deployed a static website with Hugo and a customized theme to showcase technical skills and personal insights.