Rajat Gupta

Developed comprehensive schema strategies and metadata layers, including a Data Catalog, to enhance data discovery, lineage, and governance for Zahid Group.

Utilized Databricks and Apache Spark to cleanse and transform complex datasets, implementing a medallion architecture for structured and semi-structured data.

Streamlined data operations and automation by building reusable utilities, reducing manual effort and improving efficiency.

Created scalable Data Ingestion Pipelines from diverse RDBMS databases to a centralized storage layer, orchestrated by Cloud Composer DAGs.

Transformed complex business logic using Databricks Spark, running efficiently on Cloud Composer jobs to support critical financial operations.

Built an event-based framework utilizing Cloud Functions for near real-time processing of financial data, leveraging Databricks workflows for efficiency.

Designed and implemented a serverless architecture with reusable components, adopted by multiple product teams to process financial data related to debt and recovery.

Contributed to Python Spark notebooks, SQL queries, and workflows for daily ETL data loads, ensuring high data quality and availability.

Automated CI/CD pipelines for code deployment and migration across Dev, QA, and Production environments using Terraform and Jenkins Groovy script, reducing deployment time and errors.

Collaborated with business stakeholders to gather requirements and translate them into technical user stories, facilitating seamless data onboarding from multiple source teams.

Created data ingestion pipelines from various RDBMS databases to AWS S3, leveraging AWS DMS for efficient data transfer and integration.

Developed robust ETL data pipelines using PySpark, Glue Jobs, Lambda, DynamoDB, Athena, and S3 with Parquet files, optimizing data processing for the new data lake.

Built new infrastructure in AWS using Terraform and a CI/CD Pipeline with Bitbucket and TeamCity, enhancing deployment efficiency and reliability.

Led requirements gathering, design, and implementation for ETL pipelines on AWS cloud, ensuring alignment with business needs for data integration.

Developed and designed reusable libraries, frameworks, and utilities for ETL data pipelines, data transformations, warehousing, and validations.

Optimized performance for PySpark data pipelines, significantly improving data load times and overall system efficiency for daily reports on AWS Databricks.

Handled real-time data ingestion via Kafka with Spark Streaming for large-scale data processing, ensuring timely and efficient data availability.

Translated complex functional and technical requirements into detailed high and low-level designs, facilitating effective project execution.

Integrated HSM API with Kafka to provide hardware and software-level 256-bit encryption for secure debit/credit card transactions, enhancing data security.

Played a key role in data transformation using Spark scripts for structured and semi-structured data, improving data quality and usability.

Performed data analysis by implementing various machine learning algorithms via Spark ML, optimizing hyperparameter deployment with GridSearchCV for enhanced model performance.

Developed PySpark scripts to calculate traditional and ad-hoc KPIs from structured and semi-structured data, providing critical insights for tariff plan regression.

Managed data ingestion using Sqoop, cleaning, and manipulation of data with Spark scripts, ensuring data quality and readiness for analysis.

Performed ETL from ENIQ Oracle database to HDFS using Sqoop and processed data with Hive scripts for the new Datalake for LTE Network, supporting critical KPIs.

Migrated an old dashboard written in PHP scripts to a new Java-based dashboard, enhancing performance and scalability for 3G and 4G network KPIs.

Developed Data Entity Beans (POJOs) and established relationships between them, improving data integrity and application architecture for the portal.