Role : Associate Director, Data Engineering

Location : Noida, India

Work Mode : 5 days WFO

About CloudKeeper :

CloudKeeper is a cloud cost optimization partner that combines the power of group buying and commitments management, expert cloud consulting and support, and an enhanced visibility and analytics platform to reduce cloud cost and help businesses maximize the value from AWS, Microsoft Azure, and Google Cloud.

A certified AWS Premier Partner, Azure Technology Consulting Partner, Google Cloud Partner, and FinOps Foundation Premier Member, CloudKeeper has helped 350+ global companies save an average of 20% on their cloud bills, modernize their cloud set-up and maximize value.

CloudKeeper Platform Suite is a leading enterprise FinOps platform that helps organizations optimize and govern their multi-cloud spend across AWS, Azure, and Google Cloud. Our platform processes billions of dollars in cloud spend, delivering actionable cost optimization recommendations, automated governance, and financial accountability to enterprises worldwide.

To know more - https : //www.cloudkeeper.com/platform-suite

The Opportunity :

CloudKeeper's products are federated, but they stand on one shared foundation : cost and usage data. Every CK-Lens dashboard, every CK-Commit purchase decision, every CK-Tuner recommendation, and every FinOps for AI insight is only as good as the billing data underneath it. Today that data flows through pipelines that grew up inside individual products, spanning the Stormus backend, Jenkins-driven data jobs, and Airflow DAGs.

We are hiring an Associate Director, Data Engineering to own this layer as a product in its own right. You will define the target architecture for a multi-cloud billing data platform, consolidate ingestion across AWS, Azure, and Google Cloud, set the warehouse and lakehouse strategy, and build the team that runs it. The bar is unforgiving : customers make financial decisions on these numbers, so every figure has to reconcile to the invoice.

What You'll Own :

Multi-Cloud Billing Data Platform :

- A single ingestion layer for AWS Cost and Usage Reports and Data Exports, Azure Cost Management exports and Billing APIs, and GCP BigQuery billing exports, on daily and intra-day schedules

- Normalization of provider-specific line items into a common cost model, aligned to the FinOps Foundation's FOCUS specification

- Correct handling of restatements, late-arriving usage, credits, discounts, marketplace charges, and consolidated billing across payer accounts, so reported cost always matches what the customer is invoiced

Warehouse and Lakehouse Architecture :

- The analytics engine behind 100+ CK-Lens dashboards, anomaly detection, custom reports, and LensGPT natural-language queries, which runs on Snowflake today

- The target architecture for the next three years : where a warehouse ends and a lakehouse begins, which workloads belong in each, and how the platform scales with customer count without scaling spend at the same rate

- Warehouse and lake cost governance. CloudKeeper sells data-platform cost optimization, and the platform should be the proof

Data Foundation for New Products :

- Ingestion for FinOps for AI : GPU instance usage, managed AI services such as Bedrock, Azure OpenAI, and Vertex AI, and LLM API consumption from providers such as OpenAI and Anthropic

- Kubernetes cost telemetry from Prometheus and OpenCost feeding CK-Tuner Kubernetes

- Cost allocation and virtual-tagging datasets that power unit economics reporting

- A governed semantic layer and data contracts so that LensGPT and the LensGPT MCP server can answer questions about cost data reliably

Responsibilities :

Data Engineering Leadership :

- Build and lead the data engineering pod : hire and mentor Data Engineers and Senior Engineers in data engineering, set coding, testing, and review standards, and grow the team as the product lines scale

- Own delivery against a data-platform roadmap agreed with Product Management, with explicit data contracts for each product and published service levels for data freshness and accuracy

- Run the platform like a product : define and report its metrics, including data freshness, reconciliation accuracy against source billing, pipeline reliability, and warehouse cost per customer, and review them monthly

- Represent data engineering in architecture, security, and customer conversations, including due-diligence and security questionnaires from enterprise customers

Technical Strategy :

- Define the target data architecture across batch and streaming, warehouse and lake, and three cloud providers, and sequence the migration from today's product-embedded pipelines without disrupting delivery

- Design the reconciliation and data-quality framework : every pipeline proves its output against the source invoice, and every restatement is traceable

- Establish data governance that satisfies CloudKeeper's SOC 2 Type 2 controls and UK GDPR obligations : access control, retention, lineage, and customer data isolation

- Build the integration points with CK-Lens for visibility, CK-Commit and CK-Tuner for optimization, and the FinOps for AI and Kubernetes products for new data domains

- Lead the team's adoption of AI-assisted development. CloudKeeper builds AI-first and expects measurable output gains from AI in the software development lifecycle

Technical Landscape You'll Navigate :

Cloud Billing and Usage Sources :

- AWS : Cost and Usage Reports, Data Exports, Organizations consolidated billing, S3, CloudWatch

- Azure : Cost Management exports, Billing and Consumption APIs, Azure Monitor

- GCP : BigQuery billing export, Cloud Billing APIs

- Standards : FinOps Foundation FOCUS specification

Warehouse and Lakehouse :

- Warehouse : Snowflake (current analytics platform); BigQuery, Redshift, or Synapse experience relevant

- Lake and table formats : S3, ADLS, or GCS object storage with Apache Iceberg, Delta Lake, or Hudi; Parquet

- Modeling and transformation : dimensional and semantic-layer modeling; dbt or equivalent transformation frameworks

Processing and Orchestration :

- Compute : Apache Spark (SQL, DataFrames, Structured Streaming); Hadoop ecosystem familiarity

- Orchestration : Apache Airflow; Jenkins-driven jobs in the current estate

- Messaging : Apache Kafka, RabbitMQ, AWS SQS

- Languages : Python plus Java or Scala

Operational Stack :

- Application data : MySQL, MongoDB, behind Java and Spring Boot services

- Observability : Prometheus, Grafana, OpenTelemetry, OpenCost

- Platform : Kubernetes, multi-tenant SaaS on AWS with Azure and GCP footprints

Role Requirements :

Experience :

- 10+ years in data engineering with 3+ years leading data engineering teams of 4 - 10 people building and shipping SaaS or data products. Leadership experience in services, consulting, or systems integration (SI) teams does not apply

- Hands-on production experience with the data ecosystems of at least two of AWS, Azure, and Google Cloud

- Has built and operated both a cloud data warehouse (Snowflake, BigQuery, Redshift, or Synapse) and a data lake or lakehouse (Iceberg, Delta Lake, or Hudi on object storage) at multi-terabyte scale, and can explain when to use which

- Track record of pipelines whose output reconciles to a financial source of truth, such as billing, payments, or ledgers, strongly preferred

- Experience with cloud billing data (Cost and Usage Reports, Azure cost exports, BigQuery billing export) or FinOps practice is a significant advantage

Technical Depth :

- Expert-level Spark and SQL; strong Python, with Java or Scala

- Warehouse performance and cost engineering : clustering, partitioning, materialization strategy, query cost governance

- Streaming with Kafka and batch orchestration with Airflow in production

- Data modeling for analytics : dimensional models, semantic layers, and data contracts consumed by multiple products

- Data quality, lineage, metadata, and governance tooling, and the judgment to apply them without slowing delivery

- Working understanding of SOC 2 and GDPR obligations as they apply to a data platform

Leadership :

- Ability to operate in a 'founding team' mode : small team, high ownership, rapid shipping

- Experience hiring and building data engineering teams from scratch

- Comfortable wearing multiple hats : architecture, coding, hiring, product input

- Strong communicator who can influence product, platform, security, and customer-facing teams, and can explain a reconciliation gap to a CFO as clearly as to an engineer

- Sets the expectation for AI-assisted development by using it personally

Why CloudKeeper :

- Market Opportunity : Cloud spend optimization is a $10B+ market growing rapidly as enterprises accelerate cloud adoption

- Data at the Core : The platform processes billions of dollars in cloud spend. Data engineering is the foundation every product stands on, not a support function

- Platform Ownership : Define and build the shared data platform for a federated product portfolio within a revenue-generating company : startup speed with enterprise stability

- New Domains : FinOps for AI, Kubernetes cost, and data-platform cost optimization are new data problems being solved for the first time, and this role owns their data foundation

- Impact : The north-star metric for the whole product line is savings delivered per customer per month. Every improvement in data accuracy, freshness, and coverage moves it directly

- Growth : Build the team and the platform from their current state to scale, with the metrics and ownership that come with running it as a product

CloudKeeper is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees.