Kopal Ghosh

Engineered and optimized Hive queries and Unix shell scripts for comprehensive data analysis and automated report generation, ensuring timely delivery to stakeholders.

Developed high-performance Spark solutions using Scala, Dataframes, and Spark SQL API to accelerate data processing for interactive queries and ingesting terabytes of data into MongoDB.

Designed and implemented advanced ETL processes and data models, improving reporting efficiency and data availability through Kafka-based real-time streaming solutions.

Analyzed structured, semi-structured, and unstructured data to ensure quality and completeness, extracting critical business value for downstream applications.

Built and maintained robust data pipelines using Apache Spark for structured and unstructured data, ensuring high availability and integrity for critical transaction data.

Developed REST APIs for seamless data access and integration with third-party platforms, improving system interoperability and data exchange capabilities.

Optimized database indexing and query performance, reducing response times by 20% for critical card transaction data analysis.

Processed and transformed large datasets (AVRO, PARQUET, ORC, CSV, JSON, XML) using Spark, providing real-time serviceability insights during crisis periods.

Managed CI/CD processes by monitoring Spark jobs via Spark UI, pushing code to GIT, conducting rigorous code reviews, and automating deployments with Jenkins.

Led end-to-end data integration for the Autowrite project, pulling data from Windows-based applications via API and establishing connectivity with Synapse, JTux, and JNI.

Automated data verification using Postgres, Toad, and Unix, streamlining the data processing pipeline for settlement information files.

Implemented and managed CI/CD processes, including Jenkins pipelines, Git code management, peer reviews, and Chef changes for efficient deployment.

Designed and deployed NIFI pipelines for automated, scheduled data ingestion, processing large volumes of structured and unstructured datasets.

Developed Spark code to cleanse, merge, and process diverse datasets, ensuring data quality and readiness for storage in Hive, HBase, or S3 for upstream consumption.

Automated end-to-end API testing for E-Wallet platforms (Samsung Pay, Apple Pay, Amazon Pay, Android Pay) using Cucumber and Java, ensuring robust functionality for digital payment services.

Developed and automated data validation processes for Oracle and PostgreSQL databases, enhancing data integrity for financial transactions.

Managed and monitored Jenkins jobs for continuous integration and deployment, improving delivery efficiency for E-Wallets and Core Billing applications.

Utilized the CRAFT framework to design and execute batch jobs for over 600 test cases on credit card operations via TS2, ensuring comprehensive backend application testing.

Supported core billing operations by handling complex calculations for minimum payments, finance charges, late fees, and membership fees, contributing to accurate billing cycles.