Project Lumos
Platform-agnostic metadata automation layer for discovery, lineage, and governance without manual drift.
2023-2024
Data governance
Metadata
Lineage
Discovery
Effort saved
400+ hours per year
Engineers unblocked
10 FTE
Resume
Senior Data & Platform Engineer (9+ years) re-architecting and migrating multi-tenant SaaS data platforms across AWS and GCP (Databricks, Spark, Airflow) with FinOps, reliability, and performance focus (~15M customers / ~100M devices; ~50 PB lake). OSS founder (Project Lumos) and building OpsPilot (LLM reliability copilot) with eval gates and OpenTelemetry observability.
Focus areas
Systems-first engineering with evidence over hype.
Multi-cloud data platforms
Streaming and batch pipelines
Distributed systems reliability
Cost optimization and FinOps
Current seat
Plume Design Inc
Multi-cloud data platform architecture, streaming ingestion, and cost optimization at ISP scale.
Device Scale
100M+
Data Footprint
550PB+
Cost Savings
$15M+
Hours Saved
6000+/yr
Experience
Selected roles and outcomes
Curated role cards replace the denser timeline so the strongest operational wins land earlier in the reading flow.
Plume Design Inc
Staff Data & Platform Engineer
Aug 2023 - Present
Multi-cloud data platform architecture, streaming ingestion, and cost optimization at ISP scale.
Built a high-throughput Go microservice for 5G cellular backup telemetry; engineered dual-granularity fact materialization cutting mobile query latency from 15s to <80ms (-85%) and resolved an 11-month silent bug that distorted usage reporting by 55%.
Co-created Project Weave (cloud architecture explorer) for 50+ repos; won direct CTO endorsement ('this is amazing... put this in front of noc') and advanced it into a funded company initiative for developer onboarding and NOC testing.
Architected Plume's first GCP data platform (Databricks Mumbai) for tier-1 ISP (Jio); made tech lead after upstream delays, delivered prod in 15 days, and scaled to 100k customers at ISP scale (~15M customers / ~100M devices).
Redesigned Frontline (NOC analyst tool) Yugabyte ingestion by replacing Lambda fan-out with Spark Structured Streaming (~40 GB/hr), cutting infra cost ~30% and data-quality issues ~50%.
Milestone
Multi-cloud onboarding for 15M customers
Delivered AWS/GCP parity with portability and quality guardrails.
Morgan Stanley
Senior Data & Platform Engineer
May 2021 - Jul 2023
Built self-service data platforms and governance for global trade settlements.
Built self-service reporting platform (UI + Spring Boot query planner routing Spark vs Snowflake), cutting business reporting turnaround by ~1.5 weeks.
Saved ~120 hours/week by automating trade-fail ownership assignment using Drools (Spring Batch + CQRS) with test-driven, auditable rules.
Owned a critical trade-fails warehouse and improved reporting correctness while delivering ~$10M cost avoidance through data-quality, reconciliation, and governance controls.
Milestone
Technology Innovation Program (3rd place)
Built a Neo4J knowledge graph prototype for transactional fraud detection.
Fractal Analytics
Data Engineer
Apr 2019 - May 2021
Delivered national-scale public health platforms and enterprise data systems.
Built COVID-19 India data platform (Prime Minister's Office (PMO) / NITI Aayog / NASSCOM task force) with PII-safe AWS ingestion/curation, powering hospital readiness and spread dashboards during lockdown.
Led AWS/Infosys/Mapbox architecture discussions to design privacy-safe movement-data pipelines and security boundaries for PII-adjacent datasets.
Built AWS analytics platform; cut infra cost ~40% and improved Spark runtime ~3x (~45 min to ~12-13) via EMR + S3/Spark optimizations.
Milestone
COVID-19 India data platform
Delivered the public health data backbone for national dashboards.
Skills
Readable capability map
Languages
Python • Scala • Java • Go • SQL
Frameworks & Data
Apache Spark (batch/streaming) • Airflow (orchestration) • Kafka / Pub/Sub • Spring Boot (microservices) • FastAPI • Drools (rules engine) • Delta Lake / Parquet / Avro • Hadoop / Hive • Data modeling (SCD Type-2)
Cloud & DevOps
AWS (S3, Athena, Glue, Lambda, EMR, Secrets Manager) • GCP (BigQuery, Pub/Sub, GCS, Dataproc, Secret Manager) • Databricks • Kubernetes / EKS • Docker • CI/CD (Jenkins, Azure DevOps) • Opsgenie / Jira Service Management (alert routing & on-call)
Data & AI Infrastructure
Snowflake / BigQuery / Athena • YugabyteDB / PostgreSQL / Redis • SQL Server / DB2 (fintech) • Neo4j (graph) • DataHub / Collibra / Soda / Tonic.ai (governance, DQ, PII masking) • OpenTelemetry (observability) • pgvector (vector search) • RAG systems (retrieval, citations) • Eval-gated LLM deployments • MCP (tool orchestration)
Open Source
Projects that deepen the story
OpsPilot
Production-shaped LLM platform project: multi-tenant RAG + agent workflows for incident and cost operations.
2026
LLM platform
RAG
Agents
Gateway
Roadmap
Layers 0–3 with pass/fail gates
Quality gates
Gold set + eval harness