RAMYA AVALA

Designed and developed robust Azure Data Factory pipelines, ingesting diverse data from on-prem SQL databases, APIs, and flat files into Azure Data Lake Gen2, enhancing data accessibility.

Engineered PySpark-based transformation logic within Azure Databricks for cleansing, validating, and business-rule-driven transformations, ensuring data accuracy and usability.

Implemented efficient incremental data loading strategies using watermarking and Delta Lake merge operations, effectively managing high-volume datasets.

Integrated Kafka-based streaming ingestion for near real-time data feeds with Databricks Structured Streaming jobs, enabling immediate data processing and analysis.

Optimized Databricks clusters through executor memory tuning, auto-scaling policies, and job configurations, resulting in improved performance and reduced operational costs.

Streamlined deployment of ADF pipelines and Databricks notebooks across DEV, QA, and PROD environments using Azure DevOps CI/CD pipelines, accelerating release cycles.

Proactively monitored daily pipeline executions using Datadog dashboards and configured alerts for failures and SLA breaches, ensuring high data availability and reliability.

Managed incidents and change requests via ServiceNow, including performing root cause analysis and implementing permanent fixes to minimize recurrence and improve system stability.

Supported and maintained Azure-based data platforms by developing and optimizing ETL pipelines using Azure Data Factory, ensuring continuous and efficient data flow.

Developed Databricks notebooks with PySpark and SQL to transform raw data into curated datasets, supporting critical analytics and reporting needs.

Implemented robust data quality checks and reconciliation logic to ensure consistency and integrity between source and target systems, improving data reliability.

Optimized SQL performance through indexing strategies and query optimization, significantly improving downstream data consumption efficiency.

Managed version control, code reviews, and release management using Git and Azure DevOps, fostering collaborative development and ensuring code quality.

Provided critical production support for scheduled pipelines, efficiently resolving data failures and ensuring minimal downtime for business operations.