Profile Summary
Data Engineer with 11 years of experience, including 3 years in Java application development and 8 years in building scalable data pipelines and modernizing data warehouses into lakehouse architectures on AWS and Databricks. Skilled in Apache Spark, AWS, Terraform, data quality, governance, and CI/CD automation, with hands-on AI-driven development using Generative AI to accelerate pipeline delivery and enable AI-ready data platforms.
Comcast — Development Engineer 4
Oct. 2021 - Present
Engineered a GitHub-backed Agent skills registry to centralize reusable AI capabilities, accelerating pipeline development and driving efficient AI adoption across data engineering teams.
Designed and implemented a Salesforce data catalog using MuleSoft and Delta Live Tables (DLT), enabling streamlined data discovery and governance.
Implemented Monte Carlo based data quality checks and anomaly detection, reducing data incidents by 30% and enabling real-time alerts for 20+ data pipelines.
Built reusable common ETL framework and Terraform modules adopted by multiple teams, reducing pipeline development time by 40% through standardized CI/CD pipelines.
Designed and developed a portal for streamlined metadata management and access control across Amazon S3 and Unity Catalog.
Mindtree Ltd — Senior Software Engineer
Jul. 2015 - Oct. 2021
Orchestrated an insight-driven automotive business intelligence system using AWS, improving overall reporting efficiency by 70%.
Automated the ETL process by orchestrating data pipelines using AWS Lambda and processed terabytes of sales data using PySpark in AWS Glue.
Developed ETL framework in Azure Databricks for data cleansing, data wrangling, standardization, variable generation, and feature engineering used by product recommendation system.
Performed various optimizations in Apache Spark by monitoring Spark UI and analyzing query execution plans, thereby increasing throughput by 20%.
Automation testing utility to compare two Apache Spark DataFrames or datasets, generating detailed reports on matches, mismatches, and missing records.
Designed for data validation, ETL testing, and regression testing in Spark pipelines.
Technologies: Apache Spark, PySpark, Python, Data Validation
Programming Languages
Java, Scala, Python
Big Data & Lakehouse
Apache Spark, Databricks, Snowflake, Hadoop
Cloud & Platforms
AWS: Athena, CloudWatch, DynamoDB, ECS, EMR, Glue, Lambda, S3 & StepFunctions
Databases
PostgreSQL, Oracle, Microsoft SQL Server
DevOps & Automation
GitHub Actions, Terraform, Docker, Concourse CI
Project Management & Collaboration
Agile, JIRA, Confluence, GitHub
Artificial Intelligence & Emerging Tech
GitHub Copilot, Opencode, MCP (GitHub, Confluence, JIRA & OpenSearch), Generative AI
Mindtree Pillar Award — Bronze
Oct. 2017
An award recognizing the top 5% of employees for consistent high performance over two consecutive years.
Birla Institute of Technology and Science, Pilani
2022 - 2024
M.Tech in Data Science and Engineering
CGPA: 7.33
Panimalar Engineering College, Affiliated to Anna University
2011 - 2015
B.Tech in Information Technology
CGPA: 7.97
