Stacks

Overview

The stacks and stacks-instances repositories form the core of a GitOps-based system for provisioning Edge Developer Platforms (EDP). They implement a template-instance pattern that enables the deployment of reusable platform components across different environments. The concept of “stacks” originates from the CNOE.io project (Cloud Native Operational Excellence), which can be traced through the evolutionary development from edpbuilder (derived from CNOE.io’s EDPbuilder) to infra-deploy.

Key Features of the Everything-as-Code Stacks Approach

This declarative Stacks provisioning architecture is characterized by the following central properties:

Complete Code Declaration

Platform as Code: All Kubernetes resources, Helm charts, and application manifests are declaratively versioned as YAML files. The entire platform topology is traceable in Git.

Configuration as Code: Environment-specific configurations are generated through template hydration, not manually edited. Gomplate transforms generic templates into concrete configurations.

GitOps-Native Architecture

Single Source of Truth: Git is the sole source of truth for the desired state of all infrastructure and platform components.

Declarative State Management: ArgoCD continuously synchronizes the actual state with the desired state defined in Git. Deviations are automatically corrected.

Audit Trail: Every change to infrastructure or platform is documented through Git commits, with author, timestamp, and change description.

Pull-based Deployment: ArgoCD pulls changes from Git, rather than external systems requiring push access to the cluster. This significantly increases security.

Template-Instance Separation

DRY Principle (Don’t Repeat Yourself): Common platform components are defined once as templates and reused for all environments.

Environment Promotion: New environments can be quickly created through template hydration. Consistency across environments is guaranteed.

Centralized Maintainability: Updates to stack definitions can be made centrally in the stacks repository and then selectively rolled out to instances.

Customization Points: Despite reuse, environment-specific customizations remain possible through values files and manifest overlays.

Modular Composition

Stack-based Architecture: Platform capabilities are organized into independent, reusable stacks (core, otc, forgejo, observability).

Selective Deployment: Through the STACKS environment variable, only required components can be deployed selectively.

Mix-and-Match: Different stack combinations yield different platform profiles (Development, Production, Observability clusters).

Pluggable Components: New stacks can be added without modifying existing ones.

Environment Agnosticism

Cloud Provider Abstraction: Templates are formulated generically. Provider-specific details are introduced through hydration.

Multi-Cloud Ready: The architecture supports various cloud providers (currently OTC, historically KIND, extensible to AWS/Azure/GCP).

Environment Variables as Interface: All environment-specific aspects are controlled through clearly defined environment variables.

Portable Definitions: Stack definitions can be ported between environments and even cloud providers.

Self-Healing and Drift Detection

Automated Reconciliation: ArgoCD detects deviations from the desired state and corrects them automatically.

Continuous Monitoring: Permanent monitoring of cluster state compared to Git definition.

Declarative State Recovery: After failures or manual changes, the declared state is automatically restored.

Sync Policies: Configurable sync strategies (automated, manual, with pruning) per application.

Secrets Management

Secrets Outside Git: Sensitive data is not stored in Git but generated at runtime or injected from secret stores.

Generated Credentials: Passwords, tokens, and secrets are generated during deployment and directly created as Kubernetes Secrets.

Sealed Secrets Ready: The architecture is compatible with Sealed Secrets or External Secrets Operators for encrypted secret storage in Git.

Credential Rotation: Secrets can be regenerated through re-deployment.

Observability and Auditability

Declarative Monitoring: Observability stacks are part of the Platform-as-Code definition.

Deployment History: Complete history of all deployments and changes through Git log.

ArgoCD UI: Graphical representation of sync status and application topology.

Infrastructure Events: Terraform state changes and Terragrunt outputs document infrastructure changes.

Idempotence and Reproducibility

Idempotent Operations: Repeated execution of the same declaration leads to the same result without side effects.

Deterministic Builds: Same input parameters (Git commit + environment variables) produce identical environments.

Disaster Recovery: Complete environments can be rebuilt from code without restoring backups.

Testing in Production-Like Environments: Development and staging environments are code-identical to production, only with different parameter values.

Purpose in EDP

A ‘stack’ is the declarative description for the platform provisionning in an EDP installation.

Repository

Code:

• Stacks Templates Repo
• Stacks Instances Repo, used for ArgoCD Gitops
• EDP Stacks Deployment mechanism

Documentation: [Link to component-specific documentation]

• Outdated: The former ’edpbuilder’ as script, derived from CNOE’s ‘idpbuilder

The stacks Repository

Purpose and Structure

The stacks repository contains reusable template definitions for platform components. It serves as a central library of building blocks from which Edge Developer Platforms can be composed.

stacks/
└── template/
    ├── edfbuilder.yaml
    ├── registry/
    │   ├── core.yaml
    │   ├── otc.yaml
    │   ├── forgejo.yaml
    │   ├── observability.yaml
    │   └── observability-client.yaml
    └── stacks/
        ├── core/
        ├── otc/
        ├── forgejo/
        ├── observability/
        └── observability-client/

Components

edfbuilder.yaml: The central bootstrap definition. This is an ArgoCD Application that references the registry directory and serves as the entry point for the entire platform provisioning.

registry/: Contains ArgoCD ApplicationSets that function as a meta-layer. Each file defines a category of stacks (e.g., core, forgejo, observability) and references the corresponding subdirectory in stacks/.

stacks/: The actual platform components, organized into thematic categories:

• core: Fundamental components such as ArgoCD, CloudNative PostgreSQL, Dex (SSO)
• otc: Cloud-provider-specific components for Open Telekom Cloud (cert-manager, ingress-nginx, StorageClasses)
• forgejo: Git server and CI runners
• observability: Central observability components (Grafana, Victoria Metrics Stack)
• observability-client: Client-side metrics collection for non-observability clusters

Each stack consists of:

• YAML definitions (primarily ArgoCD Applications)
• values.yaml files for Helm charts
• manifests/ directories for additional Kubernetes resources

Templating Mechanism

The templates use Gomplate with delimiter syntax {{{ }}} for environment variables:

repoURL: "https://{{{ .Env.CLIENT_REPO_DOMAIN }}}/{{{ .Env.CLIENT_REPO_ORG_NAME }}}"
path: "{{{ .Env.CLIENT_REPO_ID }}}/{{{ .Env.DOMAIN }}}/stacks/core"

These placeholders are replaced with environment-specific values during the deployment phase.

The stacks-instances Repository

Purpose and Structure

The stacks-instances repository contains the materialized, environment-specific configurations. While stacks provides the blueprints, stacks-instances contains the actual deployment definitions for concrete environments.

stacks-instances/
└── otc/
    ├── osctest.t09.de/
    │   ├── edfbuilder.yaml
    │   ├── registry/
    │   └── stacks/
    ├── backup-test-manu.t09.de/
    │   ├── edfbuilder.yaml
    │   ├── registry/
    │   └── stacks/
    └── ...

Organizational Principle

The structure follows the schema {cloud-provider}/{domain}/:

• cloud-provider: Identifies the cloud environment (e.g., otc for Open Telekom Cloud)
• domain: The fully qualified domain name of the environment (e.g., osctest.t09.de)

Each environment replicates the structure of stacks/template, but with resolved template variables and environment-specific customizations.

Usage by ArgoCD

ArgoCD synchronizes directly from this repository. Applications reference paths such as:

source:
  path: "otc/osctest.t09.de/stacks/core"
  repoURL: "https://edp.buildth.ing/DevFW-CICD/stacks-instances"
  targetRevision: HEAD

This enables true GitOps: every change to the configurations is traceable through Git commits and automatically synchronized by ArgoCD in the target environment.

The infra-deploy Repository

Role in the Overall Architecture

The infra-deploy repository is the orchestration layer that coordinates both infrastructure and platform provisioning. It represents the evolution of edpbuilder, which was originally derived from the CNOE.io project’s EDPbuilder.

Two-Phase Provisioning

Phase 1: Infrastructure Provisioning

Uses Terragrunt Stacks (experimental feature) to provision cloud resources:

infra-deploy/
├── root.hcl
├── non-prod/
│   ├── tenant.hcl
│   ├── dns_zone/
│   │   ├── terragrunt.hcl
│   │   ├── terragrunt.stack.hcl
│   │   └── terragrunt.values.hcl
│   └── testing/
├── prod/
└── templates/
    └── forgejo/
        ├── terragrunt.hcl
        └── terragrunt.stack.hcl

Terragrunt Stacks provision:

• VPC and network segments
• Kubernetes clusters (CCE on OTC)
• Managed databases (RDS PostgreSQL)
• Load balancers and DNS entries
• Security groups and other cloud resources

Phase 2: Platform Provisioning

The script scripts/edp-install.sh executes the following steps:

1. Template Hydration:

   • Checkout of the stacks repository
   • Execution of Gomplate to resolve template variables
   • Generation of environment-specific manifests

2. Instance Management:

   • Checkout/update of the stacks-instances repository
   • During CI execution: commit and push of the new instance

3. Secrets Management:

   • Generation of credentials (database passwords, SSO secrets, API tokens)
   • Creation of Kubernetes Secrets

4. Bootstrap:

   • Helm-based installation of ArgoCD
   • Application of edfbuilder.yaml or selective registry entries

5. GitOps Handover:

   • ArgoCD takes over further synchronization from stacks-instances
   • Continuous monitoring and self-healing

GitHub Actions Workflows

The .github/workflows/ directory contains three central workflows:

deploy.yaml: Complete deployment pipeline with the following inputs:

• Cluster environment and tenant (prod/non-prod)
• Node flavor and availability zone
• Stack selection (core, otc, forgejo, observability, etc.)
• Infra-catalogue version

plan.yaml: Terraform/Terragrunt plan preview without execution

destroy.yaml: Controlled teardown of environments

Deployment Workflow

The complete provisioning process proceeds as follows:

1. Initiation: GitHub Actions workflow is triggered (manually or automatically)

2. Environment Preparation:

   export CLUSTER_ENVIRONMENT=qa-stage
   cd scripts
   ./new-otc-env.sh  # Creates Terragrunt configuration if new
   

3. Infrastructure Provisioning:

   ./ensure-cluster.sh otc
   # Internally executes:
   # - ./ensure-otc-cluster.sh
   # - terragrunt stack run apply
   

4. Platform Provisioning:

   ./edp-install.sh
   # Executes:
   # - Checkout of stacks
   # - Gomplate hydration
   # - Checkout/update of stacks-instances
   # - Secrets generation
   # - ArgoCD installation
   # - Bootstrap of stacks
   

5. ArgoCD Synchronization: ArgoCD continuously reads from stacks-instances and synchronizes the desired state

The CNOE.io Stacks Concept

The term “stacks” originates from the Cloud Native Operational Excellence (CNOE.io) project. The core idea is the composition of platform capabilities from modular, reusable building blocks.

Principles

Modularity: Each stack is a self-contained unit with clear dependencies

Composability: Stacks can be freely combined to create different platform profiles

Declarativeness: All configurations are declarative and GitOps-capable

Environment-agnostic: Templates are generic; environment specifics are introduced through hydration

Stack Selection and Combinations

The environment variable STACKS controls which components are deployed:

# Complete EDP with central observability
STACKS="core,otc,forgejo,observability"

# Application cluster with client-side observability
STACKS="core,otc,forgejo,observability-client"

# Minimal development environment
STACKS="core,forgejo"

Data Flow and Dependencies

┌─────────────────┐
│  GitHub Actions │
│  (deploy.yaml)  │
└────────┬────────┘
         │
         ├─> Phase 1: Infrastructure
         │   ┌──────────────────┐
         │   │  infra-deploy    │
         │   │  (Terragrunt)    │
         │   └────────┬─────────┘
         │            │
         │            v
         │   ┌──────────────────┐
         │   │  Cloud Provider  │
         │   │  (OTC)           │
         │   │  - VPC           │
         │   │  - K8s Cluster   │
         │   │  - RDS           │
         │   └──────────────────┘
         │
         └─> Phase 2: Platform
             ┌──────────────────┐
             │  edp-install.sh  │
             └────────┬─────────┘
                      │
                      ├─> Checkout: stacks (Templates)
                      │   └─> Gomplate Hydration
                      │
                      ├─> Checkout/Update: stacks-instances
                      │
                      ├─> Secrets Generation
                      │
                      ├─> ArgoCD Installation (Helm)
                      │
                      └─> Bootstrap (edfbuilder.yaml)
                          │
                          v
                 ┌────────────────┐
                 │    ArgoCD      │
                 └────────┬───────┘
                          │
                          └─> Continuous Synchronization
                              from stacks-instances
                              │
                              v
                       ┌──────────────┐
                       │  Kubernetes  │
                       │  Cluster     │
                       └──────────────┘

Historical Context: edpbuilder to infra-deploy

The evolution from edpbuilder to infra-deploy demonstrates the maturation of the architecture:

edpbuilder (Origin):

• Directly derived from CNOE.io’s EDPbuilder
• Focus on local KIND clusters
• Manual configuration
• Monolithic structure

infra-deploy (Current):

• Production-ready for cloud deployments (OTC)
• Terragrunt-based infrastructure orchestration
• CI/CD integration via GitHub Actions
• Clear separation between infrastructure and platform
• Template-instance separation through stacks/stacks-instances

Technical Particularities

Gomplate Templating

Gomplate is used with custom delimiters {{{ }}} to avoid conflicts with Helm templating ({{ }}):

gomplate --input-dir="stacks/template" \
         --output-dir="work" \
         --left-delim "{{{" \
         --right-delim "}}}"

Terragrunt Experimental Stacks

The use of Terragrunt Stacks requires the experimental flag:

export TG_EXPERIMENT_MODE=true
terragrunt stack run apply

This enables hierarchical organization of Terraform modules with dependency management.

ArgoCD ApplicationSets

The registry pattern uses ArgoCD Applications that reference directories:

source:
  path: "otc/osctest.t09.de/stacks/core"

ArgoCD automatically detects all YAML files in the path and synchronizes them as Applications.

Best Practices and Patterns

Immutable Infrastructure: Every environment is fully defined in Git

Secrets Outside Git: Sensitive data is generated at runtime or injected from secret stores

Progressive Rollouts: New environments start as template instances, then are individually customized

Version Pinning: Critical components (Helm charts, Terragrunt modules) are pinned to specific versions

Namespace Isolation: Each stack deploys into dedicated namespaces

Self-Healing: ArgoCD’s automated sync policy enables automatic drift correction

Usage Examples

Deployment by Pipeline

The platform deployment is the second part of the EDP installtaion. First there is the infrastructure setup, which ends with a created kubernetes cluster. Then the platform provisioning by the defined stacks is done. Both is runnable by the deploypipelien in infra-deploy:

The green pipeline looks liek this:

Local setup with ‘kind’

It’s also possible to just run the second part, the stcks provisionning. Then you need to have a kubernetes cluster already running, which is e.g. feasable by a local kind-cluster.

So imagine, you want to to the stacks ‘core,observability’ on your local machine. Then you can run the local entzr

# have kind insatlled
# in /infra-deploy

# provide a kind cluster
kind delete clusters --all
./scripts/ensure-kind-cluster.sh -r

# provide some emnv vars
export TERRAFORM=/bin/bash
export LOADBALANCER_ID=ABC
export DOMAIN=ABC
export DOMAIN_GITEA=ABC
export OS_ACCESS_KEY=ABC
export OS_SECRET_KEY=ABC
export STACKS=core,observability

# deploy
./scripts/edp-install.sh

Status

Maturity: [Production]

Additional Resources

• CNOE