B Lavanya
Azure Data Engineer
Balaji Colony, 583101, Ballari, IN.About
Results-driven Azure Data Engineer with 1.5 years of experience in designing, developing, and deploying scalable data solutions. Proven expertise in building end-to-end data pipelines using Azure Data Factory, Azure Databricks, Spark (PySpark, SQL), and Python for large-scale data ingestion, processing, and migration. Adept at ensuring data quality, optimizing performance, and collaborating within Agile frameworks to deliver robust data solutions that drive business insights and operational efficiency.
Work
Chennai, Tamil Nadu, India
→
Summary
As a Data Engineer at Kion Group, contributed to the Jarvis_ML (Telecom) project by designing and implementing end-to-end data solutions, focusing on scalable data ingestion and processing within the Azure ecosystem.
Highlights
Designed and deployed end-to-end data solutions on Azure for the Jarvis_ML telecom project, encompassing storage, integration, and processing.
Developed and optimized data pipelines in Azure Data Factory, integrating diverse sources (Azure SQL, ADLS, Datawarehouses) for efficient ETL operations.
Engineered PySpark scripts for large-scale data transformation, pushing processed data to Azure Data Lake Storage Gen2, and implementing daily incremental load strategies.
Constructed complex distributed systems for high-volume data handling, collecting key metrics, and building robust data pipelines for analytics, while collaborating with Agile teams.
Chennai, Tamil Nadu, India
→
Summary
As an Associate at PWC, served as a Data Engineer on the CDW project, focusing on designing and optimizing data processing workflows and database solutions leveraging Azure services.
Highlights
Developed and deployed data processing workflows using Azure Databricks and Spark, facilitating large-scale data transformations for the CDW project.
Engineered and implemented efficient database solutions using Azure Data Lake for optimized data storage and retrieval.
Optimized data pipelines and Spark jobs in Azure Databricks, enhancing performance through configuration tuning, caching, and data partitioning techniques.
Established robust data lineage and metadata management solutions, improving data governance and traceability while collaborating with cross-functional teams.
Skills
Big Data Technologies
Spark, PySpark, Spark SQL, Data Partitioning, Performance Tuning, Distributed Systems.
Cloud Services
Azure Databricks, Azure Data Factory (ADF), Azure Data Lake, Azure Blob Storage, Azure Logic Apps, Azure Key Vault, Azure SQL Database, ADLS Gen2, On-Prem to Azure Migration.
Programming Languages
Python, SQL.
Databases
MySQL, SQL Server, Oracle, Data Warehousing.
Methodologies & Tools
Scrum, Agile, Jira, Git, Azure DevOps (ADO).
Core Data Engineering
Data Ingestion, Data Transformation, ETL, Data Quality, Data Migration, Data Storage Solutions, Data Pipelines, Data Lineage, Metadata Management, Troubleshooting, Data Analytics, Reporting, Documentation.