Osdc Project Structure (PyTorch)

---
name: osdc-project-structure
description: >
  OSDC platform architecture, project organization, directory tree,
  key design decisions, external knowledge base, key files, and docs index.
  Applies to ~/meta/ci-infra/osdc.
  Load when working on OSDC to understand project layout and architecture.
metadata:
  origin: PyTorch
---

# OSDC Project Structure & Architecture

## What This Is

Modular Kubernetes infrastructure platform on AWS EKS. A shared `base/` provides the cluster (VPC, EKS, Harbor, GPU plugins, NodeLocal DNS), and optional `modules/` layer services on top (ARC, runners, BuildKit, future projects). One codebase drives multiple clusters across regions via `clusters.yaml`.

Working directory: `osdc/`. Run all commands from here.

## Project Organization — Critical

**Before making ANY changes, understand how this project is organized.** This project MUST be well-organized at all times:

1. **Base vs modules**: Shared infrastructure goes in `base/`. Optional services go in `modules/<name>/`. If every cluster needs it, it's base.
2. **Folder separation is mandatory**: Separate directories for `terraform/`, `kubernetes/`, `helm/`, `scripts/`, `docker/` within each component.
3. **Technology separation**: Don't mix Python, Terraform, Bash, Helm, and Kubernetes manifests in the same directory.
4. **Cloud separation**: Separate cloud-specific code (AWS/Kubernetes) from cloud-agnostic definitions (runner defs).

**Before planning changes:**
- **FIRST**: Examine the current project structure
- **THEN**: Plan changes to maintain or improve the organization
- **NEVER**: Mix unrelated files or technologies in the same directory

```
osdc/
├── clusters.yaml           # THE source of truth — clusters + module lists
├── justfile                # All operations (deploy, lint, test, setup)
├── mise.toml               # Tool versions (tofu, kubectl, helm, crane, ruff, etc.)
├── pyproject.toml          # Python deps + dev deps (managed by uv)
├── scripts/                # Orchestration helpers
│   ├── cluster-config.py   # Reads clusters.yaml, outputs values for just/shell
│   ├── bootstrap-state.sh  # Creates S3 + DynamoDB for tofu state
│   ├── mise-activate.sh    # Sourceable helper for shebang recipes needing mise tools
│   └── python/
│       └── configure_harbor_projects.py  # Harbor proxy cache project setup
├── base/                   # Deployed to EVERY cluster
│   ├── kubernetes/         # StorageClass, NVIDIA plugin, Harbor NS, perf tuning, image-cache-janitor, nodelocaldns, node-taint-remover, registry-mirror config, git-cache, algif-mitigation (CVE-2026-31431, temporary), dirtyfrag-mitigation (CVE-2026-43284 + CVE-2026-43500, temporary)
│   │   ├── eniconfigs/     # AZ-named ENIConfig CRs (one per AZ from private_subnets_by_az terraform output); inert until VPC CNI Custom Networking is enabled. Not used under IPv6-only mode (Custom Networking is IPv4-only); retained as IPv4 rollback safety net
│   │   ├── harbor/         # PDB template (sed-substituted, applied by _deploy-harbor)
│   │   ├── image-cache-janitor/ # Cleans up unused cached container images on nodes
│   │   ├── nodelocaldns/   # NodeLocal DNSCache — per-node CoreDNS DaemonSet (iptables-mode interception); deployed via own deploy.sh (resolves kube-dns ClusterIP at apply time)
│   │   ├── node-taint-remover/ # Operator that strips startup taints once their gating DaemonSets are Ready; ConfigMap registry generated by lib/taint_remover.py, consumed by cache-enforcer, registry-mirror-config, node-performance-tuning
│   │   └── tests/          # Base-level Kubernetes test fixtures
│   ├── helm/               # Harbor values
│   ├── docker/             # Container images (runner-base)
│   ├── scripts/            # Bootstrap (EKS node setup)
│   └── node-compactor/     # Node consolidation controller (taints underutilized nodes)
├── modules/                # Optional, per clusters.yaml
│   ├── eks/                # AWS infrastructure (VPC, EKS, Harbor S3/IAM, image mirroring)
│   ├── karpenter/          # Karpenter controller + AWS infra (IAM, SQS, EventBridge)
│   ├── keda/               # KEDA operator (provides ScaledObject/TriggerAuthentication CRDs used by buildkit autoscaling)
│   ├── arc/                # ARC controller (GitHub Actions)
│   ├── nodepools/          # Karpenter NodePools (pure compute provisioning)
│   ├── nodepools-b200/     # B200 (Blackwell) GPU nodepools
│   ├── nodepools-h100/     # H100 (Hopper) GPU nodepools
│   ├── arc-runners/        # ARC runner scale sets (requires arc + nodepools)
│   ├── arc-runners-b200/   # B200 GPU runner scale sets
│   ├── arc-runners-h100/   # H100 GPU runner scale sets
│   ├── buildkit/           # BuildKit build service (arm64 + amd64, HAProxy LB)
│   ├── cache-enforcer/     # DaemonSet blocking direct registry/PyPI access (forces Harbor + pypi-cache)
│   ├── harbor-cache-recovery/ # Recovers Harbor cache state after pod restarts/failures
│   ├── logging/            # Centralized log collection (Alloy DaemonSet for logs + Deployment for events -> Grafana Cloud Loki)
│   ├── monitoring/         # Metrics pipeline: kube-prometheus-stack CRDs/exporters + Alloy -> Grafana Cloud
│   ├── pypi-cache/         # Self-learning PyPI wheel cache (nginx + pypiserver)
│   └── zombie-cleanup/     # Periodic cleanup of orphaned/zombie pods and resources
├── integration-tests/      # Top-level integration suites (load-test, workflows, workload-test, scripts, docker)
├── tests/smoke/            # Top-level smoke tests
└── docs/                   # Architecture and operations docs
```

## Key Design Decisions

- **Harbor is always-on** — baked into base, not a module. Every cluster needs a pull-through cache.
- **NVIDIA device plugin in base** — foundational infrastructure, not optional.
- **Karpenter is a module** — allows clusters to opt-in to Karpenter. The controller, IAM, SQS, and EventBridge are all in `modules/karpenter/`. NodePools are separate in `modules/nodepools/`.
- **Compute split: nodepools + arc-runners** — `nodepools` deploys pure Karpenter NodePools (compute provisioning). `arc-runners` deploys ARC runner scale sets (requires both `arc` and `nodepools`). Non-ARC clusters can use `nodepools` alone for compute.
- **One terraform root, many clusters** — no code duplication per environment.
- **Modules are independent** — each has its own terraform state, k8s resources, deploy script. No cross-module imports.
- **Monitoring and logging are both modules** — metrics collection (Alloy Deployment -> Grafana Cloud Mimir) is opt-in via the `monitoring` module. Log collection (Alloy DaemonSet for pod/journal logs + Alloy Deployment for Kubernetes events -> Grafana Cloud Loki) is in `modules/logging/`. Both are secret-gated — no credentials, no Alloy.
- **Three Alloy installations** — separate Helm releases, namespaces, and RBAC for: monitoring metrics (Deployment, `modules/monitoring/helm/alloy-values.yaml`), logging pod/journal logs (DaemonSet, `modules/logging/helm/alloy-logging-values.yaml`), and logging Kubernetes events (Deployment, `modules/logging/helm/alloy-events-values.yaml`). This avoids config/permission conflicts and lets each scale independently. See `osdc-observability` skill for details.

## External Knowledge Base

The `actions-knowledge-base/` directory contains source code, documentation, and detailed reference material for many of the open-source projects and tools used by this project — including Actions Runner Controller, Harbor, the Harbor Helm chart, the GitHub Actions runner, runner images, cloud-provider credential actions, and more. Each project lives as a git submodule under `repos/` and the knowledge base's own `AGENTS.md` indexes every included repository with summaries and key paths.

**Finding it**: The directory is named `actions-knowledge-base` and lives somewhere above or beside this project. Walk upward from the current working directory, checking each ancestor and its children, until you find a directory named `actions-knowledge-base`. It is typically a sibling of the top-level repo (e.g., beside `ci-infra/`), but the exact location depends on the checkout layout. Do not hardcode a relative path — search for it.

The knowledge base has two key directories:
- **`actions-knowledge-base/repos/`** — Read-only upstream source code and docs (git submodules). Managed via `sync.py`.
- **`actions-knowledge-base/docs/`** — Our own findings, workarounds, and gotchas discovered during development and operations (e.g., BuildKit OTEL crash, Grafana Alloy setup, CRD ordering issues, deploy phase ordering).

**You should consult this knowledge base** whenever you need to:
- Understand how a dependency works (e.g., ARC Helm chart values, Harbor configuration, runner lifecycle)
- Clarify configuration quirks, edge cases, or undocumented behavior
- Look up default values, API surfaces, or internal implementation details
- Verify correct usage of upstream Helm charts, container images, or action inputs
- Review operational learnings and previously discovered issues (`docs/`)

In most cases, reading the relevant source or docs in the knowledge base first will produce more accurate results than guessing or relying on general knowledge alone. Treat it as a first-class resource.

## Key Files

| File | What it does |
|------|-------------|
| `clusters.yaml` | Defines all clusters, modules, and per-installation config (replicas, log levels, runner limits, etc.) |
| `justfile` | All operations (deploy, lint, test, setup, kubeconfig) |
| `mise.toml` | Tool versions (tofu, kubectl, helm, crane, ruff, shellcheck, etc.) + env vars |
| `pyproject.toml` | Python dependencies + dev deps, managed by uv |
| `scripts/cluster-config.py` | Reads clusters.yaml for justfile/shell consumption |
| `scripts/bootstrap-state.sh` | Creates S3 bucket + DynamoDB for tofu state |
| `scripts/destroy-cluster.sh` | Operator-facing helper for tearing down a cluster's tofu-managed infrastructure |
| `scripts/cleanup-git-cache.sh` | Transitional operator helper — removes orphaned in-cluster git-cache resources left over from #703 code removal; delete once all clusters are cleaned |
| `scripts/mise-activate.sh` | Sourceable helper — adds mise tools to PATH for shebang recipes |
| `scripts/deploy-log.sh`, `scripts/deploy-status.py` | Deploy logging + status reporting helpers (called by justfile) |
| `scripts/helm-upgrade.sh`, `scripts/kubectl-apply.sh`, `scripts/yaml-diff.py`, `scripts/kubeconfig-lock.sh` | Wrappers used by module deploy.sh scripts |
| `scripts/python/configure_harbor_projects.py` | Configures Harbor proxy cache projects via API |
| `scripts/python/` (analysis) | `analyze_node_utilization.py`, `simulate_cluster.py`, `instance_specs.py` (two dicts: `INSTANCE_SPECS` compute/memory/arch keyed by AWS instance type, `ENI_MAX_PODS` AWS-stock max-pods from `awslabs/amazon-eks-ami eni-max-pods.txt`, used by simulators and analyzers), `pytorch_workload_data.py`, `daemonset_overhead.py`, `collect_instance_memory.py`, `simulate_cluster_cli.py` — capacity / utilization analysis tooling |
| `scripts/python/cni_constants.py` | Single source of truth for the VPC CNI Custom Networking node label key (`ENI_CONFIG_LABEL = "ipam.osdc.internal/eni-config"`); imported by smoke tests and the nodepool generator, referenced as a string literal by bootstrap and tofu addon config |
| `scripts/state-config.sh` | Sourceable shell helper — exports `STATE_REGION` (AWS region of the shared tofu state bucket + DynamoDB lock table, independent of each cluster's own region). Sourced by every deploy script and module deploy.sh |
| `modules/eks/terraform/main.tf` | Parameterized infra (VPC, EKS, Harbor) |
| `modules/eks/terraform/variables.tf` | All variables driven from clusters.yaml |
| `modules/eks/images.yaml` | Harbor bootstrap images to mirror to ECR |
| `base/kubernetes/node-taint-remover/` | Operator that removes startup taints from nodes once their gating DaemonSets are Ready; `lib/taint_remover.py` generates the ConfigMap registry used by cache-enforcer, registry-mirror-config, and node-performance-tuning |
| `base/kubernetes/harbor/pdb.yaml.tpl` | Harbor PDB template (multi-doc, sed-substituted by `_deploy-harbor`) |
| `base/node-compactor/` | Node consolidation controller (taints underutilized Karpenter nodes) |
| `modules/karpenter/deploy.sh` | Karpenter controller + AWS infra (IAM, SQS, Helm install) |
| `modules/keda/deploy.sh` | KEDA operator install (provides ScaledObject CRDs used by buildkit autoscaling) |
| `modules/nodepools/defs/*.yaml` | NodePool definitions (instance type, arch, disk, gpu flag) |
| `modules/nodepools/deploy.sh` | Generate + apply Karpenter NodePools |
| `modules/arc-runners/defs/*.yaml` | Runner definitions (instance type, CPU, memory, GPU, max runners) |
| `modules/arc-runners/deploy.sh` | Generate + install ARC runner scale sets (requires arc module) |
| `modules/buildkit/deploy.sh` | Generate + deploy BuildKit (Deployments, HAProxy, NodePools) |
| `modules/monitoring/CLAUDE.md` | Per-module rules for monitoring (read first when editing this module) |
| `modules/monitoring/deploy.sh` | Deploy kube-prometheus-stack + monitors + conditionally Alloy |
| `modules/monitoring/helm/values.yaml` | kube-prometheus-stack Helm values (node placement, storage, auto-discovery) |
| `modules/monitoring/helm/alloy-values.yaml` | Grafana Alloy Deployment values for metrics (ServiceMonitor/PodMonitor discovery, remote_write to Mimir) |
| `modules/monitoring/kubernetes/monitors/` | ServiceMonitors + PodMonitors for OSDC components (ARC, Harbor, Karpenter, etc.) |
| `modules/monitoring/kubernetes/dcgm-exporter/` | DCGM GPU exporter DaemonSet + headless Service (GPU nodes only) |
| `modules/logging/deploy.sh` | Secret-gated Alloy install for log collection -> Grafana Cloud Loki (DaemonSet for journal logs, Deployment for Kubernetes events) |
| `modules/logging/pipelines/base.alloy` | Alloy DaemonSet config — journal source + `loki.write` (container stdout/stderr intentionally not shipped) |
| `modules/logging/helm/alloy-logging-values.yaml` | Alloy DaemonSet Helm values for journal logs (tolerates all taints, journal mount, positions hostPath) |
| `modules/logging/helm/alloy-events-values.yaml` | Alloy Deployment Helm values for Kubernetes events stream |
| `modules/logging/scripts/python/assemble_config.py` | Wraps `base.alloy` in a ConfigMap YAML |
| `integration-tests/`, `tests/smoke/` | Top-level integration suites and smoke tests (per-module `tests/` subdirs cover module-local checks) |

## Base Component Notes

### Node Taint Remover

`base/kubernetes/node-taint-remover/` runs an operator that strips startup taints once their gating DaemonSets are Ready on a node. The taint registry is generated by `lib/taint_remover.py` and consumed by base DaemonSets (cache-enforcer, registry-mirror-config, node-performance-tuning). When adding a new startup-taint-bearing DaemonSet to base or a module, register the taint here.

### Node Compactor

`base/node-compactor/` is a node consolidation controller that taints underutilized Karpenter nodes to compact workloads WITHOUT eviction. It relies on Karpenter's `WhenEmpty` consolidation to clean up empty nodes. See `osdc-runners-nodepools` skill for configuration details.

## Documentation (`docs/`)

**Keep this list in sync.** When adding, removing, or significantly editing a document in `docs/`, update this table (description, filename, or entry presence).

| Document | Description |
|----------|-------------|
| `arc-fork-build-deploy.md` | Building and deploying the ARC controller fork (image build, push, version pinning) |
| `architecture.md` | Modular platform architecture — base vs modules, deployment layers |
| `ci-pre-merge.md` | Pre-merge CI workflow and gating |
| `cluster-recreation.md` | Operator runbook for destroying and recreating an OSDC cluster (per-cluster cutover, accepted data losses for Harbor S3 + EFS pypi-cache, pre-flight checks) |
| `current_runner_load_distribution.md` | 30-day job counts and peak concurrency analysis by runner type |
| `loki_query.md` | CLI guide for querying Grafana Cloud Loki logs (credentials, LogQL patterns) |
| `mimir_query.md` | CLI guide for querying Grafana Cloud Mimir metrics (credentials, PromQL patterns) |
| `modules.md` | Module contract and deployment order (Terraform -> Kubernetes -> deploy.sh) |
| `node-utilization-optimization.md` | Runner-to-node packing efficiency analysis and instance type recommendations |
| `node-warmup-and-scheduling-gates.md` | Node startup flow — startup taints, gating DaemonSets (hooks, registry mirrors, perf tuning), init container gates, taint summary |
| `observability.md` | Three-Alloy observability architecture (metrics Deployment + logs DaemonSet + events Deployment -> Grafana Cloud) |
| `observability-estimates.md` | Per-unit cost estimates for metrics cardinality and log volume (Grafana Cloud billing) |
| `operations.md` | Operational procedures — bootstrap, deploy, and manage clusters |
| `prod-cluster-ha-us-west-1.md` | Production cluster HA design notes for us-west-1 |
| `pypi-package-cache.md` | PyPI wheel cache architecture and operational guide |
| `runner-image-autoresolve.md` | Deploy-time auto-resolution of the runner image to a digest-pinned reference; ConfigMap shape, rollback paths, failure modes |
| `runner_naming_convention.md` | Compact runner label naming scheme (42-char limit, abbreviation rules) |