Deployment

• 1: Basic Concepts
• 1.1: Platform Orchestration
• 1.2: Application Orchestration
• 2: Infrastructure as Code
• 2.1: Terraform-based deployment of EDP
• 2.2: Stacks
• 2.2.1: Core
• 2.2.2: OTC
• 2.2.3: Coder
• 2.2.4: Terralist
• 2.2.5: Forgejo
• 2.2.6: Observability
• 2.2.7: Observability Client
• 3: Deploying to OTC
• 3.1: EDP Environments in OTC
• 3.2: Managing Instances

# 1. Install idpbuilder
curl -fsSL https://cnoe.io/install.sh | bash

# 2. Bootstrap platform
idpbuilder create \
  --use-path-routing \
  --package-dir https://github.com/cnoe-io/stacks//ref-implementation

# 3. Wait for platform ready (ca. 10 minutes)
kubectl get applications -A

# Get platform secrets (credentials)
idpbuilder get secrets

# Check all ArgoCD applications
kubectl get applications -n argocd

# Expected: All applications "Synced" and "Healthy"

idpbuilder create \
  --use-path-routing \
  --package-dir https://github.com/cnoe-io/stacks//ref-implementation \
  --log-level debug

# Workflow:
# 1. Creates Kind cluster
# 2. Installs ingress-nginx
# 3. Clones and installs ArgoCD
# 4. Installs Forgejo
# 5. Waits for core services
# 6. Creates technical users
# 7. Configures Git repositories
# 8. Installs remaining stacks via ArgoCD

# Create ArgoCD Application for new component
cat <<EOF | kubectl apply -f -
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: external-secrets
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://charts.external-secrets.io
    targetRevision: 0.9.9
    chart: external-secrets
  destination:
    server: https://kubernetes.default.svc
    namespace: external-secrets-system
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
EOF

# 1. Update via Git (GitOps)
cd your-platform-config-repo
git pull

# 2. Update stack version
vim argocd/applications/component.yaml
# Change targetRevision to new version

# 3. Commit and push
git add .
git commit -m "Update component to v1.2.3"
git push

# 4. ArgoCD will automatically sync
# 5. Monitor the update
argocd app sync component --watch

┌─────────────────┐
│   idpbuilder    │  Bootstrap Tool
│  (Initial Run)  │
└────────┬────────┘
         │
         ▼
┌─────────────────────────────────────────────────────┐
│              Kubernetes Cluster                      │
│                                                      │
│  ┌──────────────┐         ┌──────────────┐         │
│  │   ArgoCD     │────────▶│   Forgejo    │         │
│  │  (GitOps)    │         │  (Git Repo)  │         │
│  └──────┬───────┘         └──────────────┘         │
│         │                                            │
│         │  Monitors & Syncs                         │
│         │                                            │
│         ▼                                            │
│  ┌──────────────────────────────────────┐          │
│  │     Platform Stacks                  │          │
│  │                                      │          │
│  │  ┌──────────┐  ┌──────────┐        │          │
│  │  │Forgejo   │  │Keycloak  │        │          │
│  │  └──────────┘  └──────────┘        │          │
│  │  ┌──────────┐  ┌──────────┐        │          │
│  │  │Observ-   │  │Ingress   │        │          │
│  │  │ability   │  │          │        │          │
│  │  └──────────┘  └──────────┘        │          │
│  └──────────────────────────────────────┘          │
└─────────────────────────────────────────────────────┘

# Path-based routing (recommended for local development)
idpbuilder create --use-path-routing

# Custom package directory
idpbuilder create --package-dir /path/to/custom/packages

# Custom Kind cluster config
idpbuilder create --kind-config custom-kind.yaml

# Enable debug logging
idpbuilder create --log-level debug

# argocd-cm ConfigMap
data:
  # Enable automatic sync
  application.instanceLabelKey: argocd.argoproj.io/instance

  # Repository credentials
  repositories: |
    - url: https://github.com/cnoe-io/stacks
      name: cnoe-stacks
      type: git

  # Resource exclusions
  resource.exclusions: |
    - apiGroups:
      - cilium.io
      kinds:
      - CiliumIdentity

# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: platform-system

resources:
  - argocd-app.yaml
  - forgejo-app.yaml
  - keycloak-app.yaml

patches:
  - target:
      kind: Application
    patch: |-
      - op: add
        path: /spec/syncPolicy
        value:
          automated:
            prune: true
            selfHeal: true

# 1. Check if all pods are running
kubectl get pods -A

# 2. Check ArgoCD application status
kubectl get applications -n argocd

# 3. Check ingress
kubectl get ingress -A

# 4. Verify DNS resolution
nslookup cnoe.localtest.me

# 5. Check idpbuilder logs
idpbuilder get logs

# 1. Check application details
argocd app get <app-name>

# 2. View sync status
argocd app sync <app-name> --dry-run

# 3. Force sync
argocd app sync <app-name> --force

# 4. Check for errors in ArgoCD logs
kubectl logs -n argocd deployment/argocd-application-controller

# 1. Verify repository configuration
argocd repo list

# 2. Test connection
argocd repo get https://your-git-repo

# 3. Check credentials
kubectl get secret -n argocd

# 4. Re-add repository with correct credentials
argocd repo add https://your-git-repo \
  --username <user> \
  --password <token>

1 - Basic Concepts

Overview

Platform Orchestration refers to the automation and management of the platform infrastructure itself. This includes the provisioning, configuration, and lifecycle management of all components that make up the Internal Developer Platform (IDP).

In the context of IPCEI-CIS, Platform Orchestration means:

• Platform Bootstrap: Initial setup of Kubernetes clusters and core services
• Platform Services Management: Deployment and management of ArgoCD, Forgejo, Keycloak, etc.
• Infrastructure-as-Code: Declarative management using Terraform and GitOps
• Multi-Cluster Orchestration: Coordination across different Kubernetes clusters
• Platform Stacks: Reusable bundles of platform components (CNOE concept)

Target Audience

Platform Orchestration is primarily aimed at:

• Platform Engineering Teams: Teams that build and operate the IDP
• Infrastructure Architects: Those responsible for the platform architecture
• SRE Teams: Teams responsible for reliability and operations

Key Features

Declarative Platform Definition

The entire platform is defined declaratively as code:

• GitOps-First: Everything is versioned in Git and traceable
• Reproducibility: The platform can be rebuilt at any time
• Environment Parity: Consistency between Dev, Test, and Production
• Auditability: Complete history of all changes

Self-Bootstrapping

The platform can bootstrap itself:

1. Initial Bootstrap: Minimal tool (like idpbuilder) starts the platform
2. Self-Management: After bootstrap, ArgoCD takes over management
3. Continuous Reconciliation: Platform is continuously reconciled with Git state
4. Self-Healing: Automatic recovery on deviations

Stack-based Composition

Platform components are organized as reusable stacks (CNOE concept):

• Modularity: Components can be updated individually
• Reusability: Stacks can be used across different environments
• Composability: Compose complex platforms from simple building blocks
• Versioning: Stacks can be versioned and tested

In IPCEI-CIS: The stacks concept from CNOE is the core organizational principle for platform components.

Multi-Cluster Support

Platform Orchestration supports different cluster topologies:

• Control Plane + Worker Clusters: Centralized control, distributed workloads
• Hub-and-Spoke: One management cluster manages multiple target clusters
• Federation: Coordination across multiple independent clusters

Purpose in EDP

Platform Orchestration is the foundation of the IPCEI-CIS Edge Developer Platform. It enables:

Foundation for Developer Self-Service

Platform Orchestration ensures all services are available that developers need for self-service:

• GitOps Engine (ArgoCD) for continuous deployment
• Source Control (Forgejo) for code and configuration management
• Identity Management (Keycloak) for authentication and authorization
• Observability (Grafana, Prometheus) for monitoring and logging
• CI/CD (Forgejo Actions/Pipelines) for automated build and test

Consistency Across Environments

Through declarative definition, consistency is guaranteed:

• Development, test, and production environments are identically configured
• No “configuration drift” between environments
• Predictable behavior across all stages

Platform as Code

The platform itself is treated like software:

• Version Control: All changes are versioned in Git
• Code Review: Platform changes go through review processes
• Testing: Platform configurations can be tested
• Rollback: Easy rollback on problems

Reduced Operational Overhead

Automation reduces manual effort:

• No manual installation steps
• Automatic updates and patching
• Self-healing on failures
• Standardized deployment processes

Repository

CNOE Reference Implementation: cnoe-io/stacks

CNOE idpbuilder: cnoe-io/idpbuilder

Documentation: CNOE.io Documentation

Getting Started

Prerequisites

• Docker: For local Kubernetes clusters (Kind)
• kubectl: Kubernetes CLI tool
• Git: For repository management
• idpbuilder: CNOE bootstrap tool

Quick Start

Platform Orchestration with CNOE Reference Implementation:

# 1. Install idpbuilder
curl -fsSL https://cnoe.io/install.sh | bash

# 2. Bootstrap platform
idpbuilder create \
  --use-path-routing \
  --package-dir https://github.com/cnoe-io/stacks//ref-implementation

# 3. Wait for platform ready (ca. 10 minutes)
kubectl get applications -A

Verification

Verify the platform is running correctly:

# Get platform secrets (credentials)
idpbuilder get secrets

# Check all ArgoCD applications
kubectl get applications -n argocd

# Expected: All applications "Synced" and "Healthy"

Access URLs (with path-routing):

• ArgoCD: https://cnoe.localtest.me:8443/argocd
• Forgejo: https://cnoe.localtest.me:8443/gitea
• Keycloak: https://cnoe.localtest.me:8443/keycloak

Usage Examples

Use Case 1: Platform Bootstrap

Initial bootstrapping of a new platform instance:

idpbuilder create \
  --use-path-routing \
  --package-dir https://github.com/cnoe-io/stacks//ref-implementation \
  --log-level debug

# Workflow:
# 1. Creates Kind cluster
# 2. Installs ingress-nginx
# 3. Clones and installs ArgoCD
# 4. Installs Forgejo
# 5. Waits for core services
# 6. Creates technical users
# 7. Configures Git repositories
# 8. Installs remaining stacks via ArgoCD

After approximately 10 minutes, the platform is fully deployed.

Use Case 2: Adding New Platform Components

Add new platform components via ArgoCD:

# Create ArgoCD Application for new component
cat <<EOF | kubectl apply -f -
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: external-secrets
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://charts.external-secrets.io
    targetRevision: 0.9.9
    chart: external-secrets
  destination:
    server: https://kubernetes.default.svc
    namespace: external-secrets-system
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
EOF

Use Case 3: Platform Updates

Update platform components:

# 1. Update via Git (GitOps)
cd your-platform-config-repo
git pull

# 2. Update stack version
vim argocd/applications/component.yaml
# Change targetRevision to new version

# 3. Commit and push
git add .
git commit -m "Update component to v1.2.3"
git push

# 4. ArgoCD will automatically sync
# 5. Monitor the update
argocd app sync component --watch

Integration Points

ArgoCD Integration

• Bootstrap: ArgoCD is initially installed via idpbuilder
• Self-Management: After bootstrap, ArgoCD manages itself via Application CRD
• Platform Coordination: ArgoCD orchestrates all other platform components
• Health Monitoring: ArgoCD monitors health status of all platform services

Forgejo Integration

• Source of Truth: Git repositories contain all platform definitions
• GitOps Workflow: Changes in Git trigger platform updates
• Backup: Git serves as backup of platform configuration
• Audit Trail: Git history documents all platform changes
• CI/CD: Forgejo Actions can automate platform operations

Terraform Integration

• Infrastructure Provisioning: Terraform provisions cloud resources for platform
• State Management: Terraform state tracks infrastructure
• Integration: Terraform can be triggered via Forgejo pipelines
• Multi-Cloud: Support for multiple cloud providers

Architecture

Platform Orchestration Flow

Platform Bootstrap Sequence

The idpbuilder executes the following workflow:

1. Create Kind Kubernetes cluster
2. Install ingress-nginx controller
3. Install ArgoCD
4. Install Forgejo Git server
5. Wait for services to be ready
6. Create technical users in Forgejo
7. Create repository for platform state in Forgejo
8. Push platform stacks to Forgejo
9. Create ArgoCD Applications for all stacks
10. ArgoCD takes over continuous synchronization

Deployment Architecture

The platform is deployed in different namespaces:

• argocd: ArgoCD and its components
• gitea: Forgejo Git server
• keycloak: Identity and access management
• observability: Prometheus, Grafana, etc.
• ingress-nginx: Ingress controller

Configuration

idpbuilder Configuration

Key configuration options for idpbuilder:

# Path-based routing (recommended for local development)
idpbuilder create --use-path-routing

# Custom package directory
idpbuilder create --package-dir /path/to/custom/packages

# Custom Kind cluster config
idpbuilder create --kind-config custom-kind.yaml

# Enable debug logging
idpbuilder create --log-level debug

ArgoCD Configuration

Important ArgoCD configurations for platform orchestration:

# argocd-cm ConfigMap
data:
  # Enable automatic sync
  application.instanceLabelKey: argocd.argoproj.io/instance

  # Repository credentials
  repositories: |
    - url: https://github.com/cnoe-io/stacks
      name: cnoe-stacks
      type: git

  # Resource exclusions
  resource.exclusions: |
    - apiGroups:
      - cilium.io
      kinds:
      - CiliumIdentity

Platform Stack Configuration

Configuration of platform stacks via Kustomize:

# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: platform-system

resources:
  - argocd-app.yaml
  - forgejo-app.yaml
  - keycloak-app.yaml

patches:
  - target:
      kind: Application
    patch: |-
      - op: add
        path: /spec/syncPolicy
        value:
          automated:
            prune: true
            selfHeal: true

Troubleshooting

Platform not reachable

Problem: After idpbuilder create, platform services are not reachable

Solution:

# 1. Check if all pods are running
kubectl get pods -A

# 2. Check ArgoCD application status
kubectl get applications -n argocd

# 3. Check ingress
kubectl get ingress -A

# 4. Verify DNS resolution
nslookup cnoe.localtest.me

# 5. Check idpbuilder logs
idpbuilder get logs

ArgoCD Applications not synchronized

Problem: ArgoCD Applications show status “OutOfSync”

Solution:

# 1. Check application details
argocd app get <app-name>

# 2. View sync status
argocd app sync <app-name> --dry-run

# 3. Force sync
argocd app sync <app-name> --force

# 4. Check for errors in ArgoCD logs
kubectl logs -n argocd deployment/argocd-application-controller

Git Repository Connection Issues

Problem: ArgoCD cannot access Git repository

Solution:

# 1. Verify repository configuration
argocd repo list

# 2. Test connection
argocd repo get https://your-git-repo

# 3. Check credentials
kubectl get secret -n argocd

# 4. Re-add repository with correct credentials
argocd repo add https://your-git-repo \
  --username <user> \
  --password <token>

Platform Orchestration Best Practices

Based on experience and CNCF Guidelines:

1. Start Simple: Begin with the CNOE reference stack, extend gradually
2. Automate Everything: Manual platform changes are anti-pattern
3. Monitor Continuously: Use observability tools for platform health
4. Document Well: Platform documentation is essential for adoption
5. Version Everything: All platform components should be versioned
6. Test Changes: Platform updates should be tested in non-prod
7. Plan for Disaster: Backup and disaster recovery strategies are important
8. Use Stacks: Organize platform components as reusable stacks

Status

Maturity: Production (for CNOE Reference Implementation)

Stability: Stable

Support: Community Support via CNOE Community

Additional Resources

CNOE Resources

• CNOE Official Website
• CNOE GitHub Organization
• CNOE Reference Implementation
• CNOE Community Slack

Platform Engineering

• CNCF Platforms White Paper
• Platform Engineering Maturity Model
• Team Topologies - Organizational patterns

GitOps

• GitOps Working Group
• ArgoCD Best Practices
• GitOps Principles

CNOE Stacks

• Understanding CNOE Stacks
• Creating Custom Stacks

1.1 - Platform Orchestration

Overview

Orchestration in the context of Platform Engineering refers to the coordinated automation and management of infrastructure, platform, and application components throughout their entire lifecycle. It is a fundamental concept that bridges the gap between declarative specifications (what should be deployed) and actual execution (how it is deployed).

The Role of Orchestration in Platform Engineering

Platform Engineering has emerged as a discipline to improve developer experience and reduce cognitive load on development teams (CNCF Platforms White Paper). Orchestration is the central mechanism that enables this vision:

1. Automation of Complex Workflows: Orchestration coordinates multiple steps and dependencies automatically
2. Consistency and Reproducibility: Guaranteed, repeatable deployments across different environments
3. Self-Service Capabilities: Developers can independently orchestrate resources and deployments
4. Governance and Compliance: Centralized control over policies and best practices

What Does Orchestration Do?

Orchestration systems perform the following tasks:

• Workflow Coordination: Coordination of complex, multi-step deployment processes
• Dependency Management: Resolution and management of dependencies between components
• State Management: Continuous monitoring and reconciliation between desired and actual state
• Resource Provisioning: Automatic provisioning of infrastructure and services
• Configuration Management: Management of configurations across different environments
• Health Monitoring: Monitoring the health of deployed resources

Three Layers of Orchestration

In modern Platform Engineering, we distinguish three fundamental layers of orchestration:

Infrastructure Orchestration

Infrastructure Orchestration deals with the lowest level - the physical and virtual infrastructure layer. This includes:

• Provisioning of compute, network, and storage resources
• Cloud resource management (VMs, networking, storage)
• Infrastructure-as-Code deployment (Terraform, etc.)
• Bare metal and hypervisor management

Target Audience: Infrastructure Engineers, Cloud Architects

Note: Detailed documentation for Infrastructure Orchestration is maintained separately.

More details: Infrastructure Orchestration →

Platform Orchestration

Platform Orchestration focuses on deploying and managing the platform itself - the services and tools that development teams use. This includes:

• Installation and configuration of Kubernetes clusters
• Deployment of platform services (GitOps tools, Observability, Security)
• Management of platform components via Stacks
• Multi-cluster orchestration

Target Audience: Platform Engineering Teams, SRE Teams

In IPCEI-CIS: Platform orchestration is realized using the CNOE stack concept with ArgoCD and Forgejo.

More details: Platform Orchestration →

Application Orchestration

Application Orchestration concentrates on the deployment and lifecycle management of applications running on the platform. This includes:

• Deployment of microservices and containerized applications
• CI/CD pipeline orchestration
• Configuration management and secrets handling
• Application health monitoring and auto-scaling

Target Audience: Application Developers, DevOps Engineers

In IPCEI-CIS: Application orchestration uses Forgejo pipelines for CI/CD and ArgoCD for GitOps-based deployment.

More details: Application Orchestration →

GitOps as Orchestration Paradigm

A central approach in modern platform orchestration solutions is GitOps. GitOps uses Git repositories as the single source of truth for declarative infrastructure and applications:

• Declarative Approach: The desired state is defined in Git
• Automatic Synchronization: Controllers monitor Git and reconcile the live state
• Audit Trail: All changes are traceable in Git history
• Rollback Capability: Easy rollback through Git revert

Continuous Reconciliation

An important concept is continuous reconciliation:

1. The orchestrator monitors both the source (Git) and the target (e.g., Kubernetes cluster)
2. Deviations trigger automatic corrective actions
3. Health checks validate that the desired state has been achieved
4. Drift detection warns of unexpected changes

Orchestration Tools in IPCEI-CIS

Within the IPCEI-CIS platform, we utilize the CNOE (Cloud Native Operational Excellence) stack concept with the following orchestration components:

ArgoCD

• Continuous Delivery for Kubernetes based on GitOps
• Synchronizes Kubernetes manifests from Git repositories
• Supports Helm Charts, Kustomize, Jsonnet, and plain YAML
• Multi-cluster deployment capabilities
• Application Sets for parameterized deployments

Role in IPCEI-CIS: ArgoCD is the central component for GitOps-based deployment management. After the initial bootstrapping phase, ArgoCD takes over the technical coordination of all components.

Forgejo

• Git Repository Management and source control
• CI/CD Pipelines via Forgejo Actions (GitHub Actions compatible)
• Developer Portal Capabilities (initially planned, project discontinued)
• Package registry and artifact management
• Integration with ArgoCD for GitOps workflows

Role in IPCEI-CIS: Forgejo serves as the Git repository host and CI/CD engine. It was initially planned as a developer portal (similar to Backstage’s role in other stacks) but this aspect was not fully realized before project completion.

Note on Backstage: In typical CNOE implementations, Backstage serves as the developer portal providing golden paths through software templates. IPCEI-CIS initially planned to use Forgejo for this purpose but the project concluded before full implementation.

Terraform

• Infrastructure-as-Code provisioning
• Multi-cloud resource management
• State management for infrastructure
• Integration with Forgejo pipelines for automated deployment

Role in IPCEI-CIS: Terraform handles infrastructure provisioning at the infrastructure orchestration layer, integrated into automated workflows via Forgejo pipelines.

CNOE Stacks Concept

• Modular Platform Components bundled as stacks
• Reusable, composable platform building blocks
• Version-controlled stack definitions
• GitOps-based stack deployment via ArgoCD

Role in IPCEI-CIS: The stacks concept from CNOE provides the structural foundation for platform orchestration, enabling modular deployment and management of platform components.

The Orchestration Workflow

A typical orchestration workflow in the IPCEI-CIS platform:

Workflow Steps:

1. Definition: Developer defines application/infrastructure as code
2. Commit: Changes are committed to Forgejo Git repository
3. CI Pipeline: Forgejo Actions build, test, and package the application
4. Sync: ArgoCD detects changes and triggers deployment
5. Provision: Terraform orchestrates required cloud resources (if needed)
6. Deploy: Application is deployed to Kubernetes
7. Monitor: Continuous monitoring and health checks
8. Reconcile: Automatic correction on drift detection

Benefits of Coordinated Orchestration

The integration of infrastructure, platform, and application orchestration provides crucial advantages:

• Reduced Complexity: Developers don’t need to know all infrastructure details
• Faster Time-to-Market: Automated workflows accelerate deployments
• Consistency: Standardized patterns across all teams
• Governance: Central policies are automatically enforced
• Scalability: Platform teams can support many application teams
• Self-Service: Developers can provision services independently
• Audit and Compliance: Complete traceability through Git history

Best Practices

Successful orchestration follows proven principles (Platform Engineering Principles):

1. Platform as a Product: Treat the platform as a product with focus on user experience
2. Self-Service First: Enable developers to use services autonomously
3. Documentation: Comprehensive documentation of golden paths
4. Feedback Loops: Continuous improvement through user feedback
5. Thin Platform Layer: Use managed services where possible instead of building everything
6. Progressive Disclosure: Offer different abstraction levels
7. Focus on Common Problems: Solve recurring problems centrally
8. Treat Glue as Valuable: Integration of different tools is valuable
9. Clear Mission: Define clear goals and responsibilities

Avoiding Anti-Patterns

Common mistakes in platform orchestration (How to fail at Platform Engineering):

• Product Misfit: Building platform without involving developers
• Overly Complex Design: Too many features and unnecessary complexity
• Swiss Knife Syndrome: Trying to solve all problems with one tool
• Insufficient Documentation: Missing or outdated documentation
• Siloed Development: Platform and development teams working in isolation
• Stagnant Platform: Platform not continuously evolved

Sub-Components

The orchestration component includes the following sub-areas:

• Infrastructure Orchestration: Low-level infrastructure deployment and provisioning
• Platform Orchestration: Platform-level component deployment via Stacks
• Application Orchestration: Application-level deployment and CI/CD
• Stacks: Reusable component bundles and compositions

Further Resources

Fundamentals

• CNCF Platforms White Paper - Comprehensive paper on Platform Engineering
• Platform Engineering Definition - What is Platform Engineering?
• Team Topologies - Organizational structures for modern teams

GitOps

• GitOps Principles - Official GitOps principles
• ArgoCD Documentation - ArgoCD documentation

Tools

• CNOE.io - Cloud Native Operational Excellence Framework
• Forgejo - Self-hosted Git service with CI/CD
• Terraform - Infrastructure as Code tool

1.2 - Application Orchestration

Overview

Application Orchestration deals with the automation of application deployment and lifecycle management. It encompasses the entire workflow from source code to running application in production.

In the context of IPCEI-CIS, Application Orchestration includes:

• CI/CD Pipelines: Automated build, test, and deployment pipelines
• GitOps Deployment: Declarative application deployment via ArgoCD
• Progressive Delivery: Canary deployments, blue-green deployments
• Application Configuration: Environment-specific configuration management
• Golden Paths: Standardized deployment templates and workflows

Target Audience

Application Orchestration is primarily for:

• Application Developers: Teams developing and deploying applications
• DevOps Teams: Teams responsible for deployment automation
• Product Teams: Teams responsible for application lifecycle

Key Features

Automated CI/CD Pipelines

Forgejo Actions provides GitHub Actions-compatible CI/CD:

• Build Automation: Automatic building of container images
• Test Automation: Automated unit, integration, and E2E tests
• Security Scanning: Vulnerability scanning of dependencies and images
• Artifact Publishing: Publishing to container registries
• Deployment Triggering: Automatic deployment after successful build

GitOps-based Deployment

ArgoCD enables declarative application deployment:

• Declarative Configuration: Applications defined as Kubernetes manifests
• Automated Sync: Automatic synchronization between Git and cluster
• Rollback Capability: Easy rollback to previous versions
• Multi-Environment: Consistent deployment across Dev/Test/Prod
• Health Monitoring: Continuous monitoring of application health

Progressive Delivery

Support for advanced deployment strategies:

• Canary Deployments: Gradual rollout to subset of users
• Blue-Green Deployments: Zero-downtime deployments with instant rollback
• A/B Testing: Traffic splitting for feature testing
• Feature Flags: Dynamic feature enablement without deployment

Configuration Management

Flexible configuration for different environments:

• Environment Variables: Configuration via environment variables
• ConfigMaps: Kubernetes-native configuration
• Secrets Management: Secure handling of sensitive data
• External Secrets: Integration with external secret stores (Vault, etc.)

Purpose in EDP

Application Orchestration is the core of developer experience in IPCEI-CIS Edge Developer Platform.

Developer Self-Service

Developers can deploy applications independently:

• Self-Service Deployment: No dependency on operations team
• Standardized Workflows: Clear, documented deployment processes
• Fast Feedback: Quick feedback through automated pipelines
• Environment Parity: Consistent behavior across all environments

Quality and Security

Automated checks ensure quality and security:

• Automated Testing: All changes are automatically tested
• Security Scans: Vulnerability scanning of dependencies and images
• Policy Enforcement: Automated policy checks (OPA, Kyverno)
• Compliance: Auditability of all deployments

Efficiency and Productivity

Automation increases team efficiency:

• Faster Time-to-Market: Faster deployment of new features
• Reduced Manual Work: Automation of repetitive tasks
• Fewer Errors: Fewer manual mistakes through automation
• Better Collaboration: Clear interfaces between Dev and Ops

Repository

Forgejo: forgejo.org

Forgejo Actions: Forgejo Actions Documentation

ArgoCD: argoproj.github.io/cd

Getting Started

Prerequisites

• Forgejo Account: Access to Forgejo instance
• Kubernetes Cluster: Target cluster for deployments
• ArgoCD Access: Access to ArgoCD instance
• Git: For repository management

Quick Start: Application Deployment

1. Create Application Repository

# Create new repository in Forgejo
git init my-application
cd my-application

# Add application code and Dockerfile
cat > Dockerfile <<EOF
FROM node:18-alpine
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production
COPY . .
EXPOSE 3000
CMD ["node", "server.js"]
EOF

2. Add CI/CD Pipeline

Create .forgejo/workflows/build.yaml:

name: Build and Push

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2
      
      - name: Login to Registry
        uses: docker/login-action@v2
        with:
          registry: registry.example.com
          username: ${{ secrets.REGISTRY_USER }}
          password: ${{ secrets.REGISTRY_PASSWORD }}
      
      - name: Build and push
        uses: docker/build-push-action@v4
        with:
          context: .
          push: ${{ github.event_name == 'push' }}
          tags: registry.example.com/my-app:${{ github.sha }}

3. Create Kubernetes Manifests

Create k8s/deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-application
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-application
  template:
    metadata:
      labels:
        app: my-application
    spec:
      containers:
      - name: app
        image: registry.example.com/my-app:latest
        ports:
        - containerPort: 3000
        env:
        - name: NODE_ENV
          value: "production"
---
apiVersion: v1
kind: Service
metadata:
  name: my-application
spec:
  selector:
    app: my-application
  ports:
  - port: 80
    targetPort: 3000

4. Configure ArgoCD Application

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-application
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://forgejo.example.com/myteam/my-application
    targetRevision: main
    path: k8s
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

5. Deploy

# Commit and push
git add .
git commit -m "Add application and deployment configuration"
git push origin main

# ArgoCD will automatically deploy the application
argocd app sync my-application --watch

Usage Examples

Use Case 1: Multi-Environment Deployment

Deploy application to multiple environments:

Repository Structure:

my-application/
├── .forgejo/
│   └── workflows/
│       └── build.yaml
├── base/
│   ├── deployment.yaml
│   ├── service.yaml
│   └── kustomization.yaml
├── overlays/
│   ├── dev/
│   │   ├── kustomization.yaml
│   │   └── patches.yaml
│   ├── staging/
│   │   ├── kustomization.yaml
│   │   └── patches.yaml
│   └── production/
│       ├── kustomization.yaml
│       └── patches.yaml

Kustomize Base (base/kustomization.yaml):

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - deployment.yaml
  - service.yaml

commonLabels:
  app: my-application

Environment Overlay (overlays/production/kustomization.yaml):

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

bases:
  - ../../base

namespace: production

replicas:
  - name: my-application
    count: 5

images:
  - name: my-app
    newTag: v1.2.3

patches:
  - patches.yaml

ArgoCD Applications for each environment:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-application-prod
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://forgejo.example.com/myteam/my-application
    targetRevision: main
    path: overlays/production
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Use Case 2: Canary Deployment

Progressive rollout with canary strategy:

Argo Rollouts Canary:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: my-application
spec:
  replicas: 10
  strategy:
    canary:
      steps:
      - setWeight: 10
      - pause: {duration: 5m}
      - setWeight: 30
      - pause: {duration: 5m}
      - setWeight: 60
      - pause: {duration: 5m}
      - setWeight: 100
  selector:
    matchLabels:
      app: my-application
  template:
    metadata:
      labels:
        app: my-application
    spec:
      containers:
      - name: app
        image: registry.example.com/my-app:v2.0.0

Use Case 3: Feature Flags

Dynamic feature control without deployment:

Application Code with Feature Flag:

const Unleash = require('unleash-client');

const unleash = new Unleash({
  url: 'http://unleash.platform/api/',
  appName: 'my-application',
  customHeaders: {
    Authorization: process.env.UNLEASH_API_TOKEN
  }
});

// Use feature flag
if (unleash.isEnabled('new-checkout-flow')) {
  // New checkout implementation
  renderNewCheckout();
} else {
  // Old checkout implementation
  renderOldCheckout();
}

Integration Points

Forgejo Integration

Forgejo serves as central source code management and CI/CD platform:

• Source Control: Git repositories for application code
• CI/CD Pipelines: Forgejo Actions for automated builds and tests
• Container Registry: Built-in container registry for images
• Webhook Integration: Triggers for external systems
• Pull Request Workflows: Code review and approval processes

ArgoCD Integration

ArgoCD handles declarative application deployment:

• GitOps Sync: Continuous synchronization with Git state
• Health Monitoring: Application health status monitoring
• Rollback Support: Easy rollback to previous versions
• Multi-Cluster: Deployment to multiple clusters
• UI and CLI: Web interface and command-line access

Observability Integration

Integration with monitoring and logging:

• Metrics: Prometheus metrics from applications
• Logs: Centralized log collection via Loki/ELK
• Tracing: Distributed tracing with Jaeger/Tempo
• Alerting: Alert rules for application issues

Architecture

Application Deployment Flow

CI/CD Pipeline Architecture

Typical Forgejo Actions pipeline stages:

1. Checkout: Clone source code
2. Build: Compile application and dependencies
3. Test: Run unit and integration tests
4. Security Scan: Scan dependencies and code for vulnerabilities
5. Build Image: Create container image
6. Push Image: Push to container registry
7. Update Manifests: Update Kubernetes manifests with new image tag
8. Notify: Send notifications on success/failure

Configuration

Forgejo Actions Configuration

Example for Node.js application:

name: CI/CD Pipeline

on:
  push:
    branches: [ main, develop ]
  pull_request:
    branches: [ main ]

env:
  REGISTRY: registry.example.com
  IMAGE_NAME: ${{ github.repository }}

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: '18'
          cache: 'npm'
      
      - name: Install dependencies
        run: npm ci
      
      - name: Run tests
        run: npm test
      
      - name: Run linter
        run: npm run lint

  security:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Run Trivy vulnerability scanner
        uses: aquasecurity/trivy-action@master
        with:
          scan-type: 'fs'
          scan-ref: '.'
          format: 'sarif'
          output: 'trivy-results.sarif'

  build-and-push:
    needs: [test, security]
    runs-on: ubuntu-latest
    if: github.event_name == 'push'
    steps:
      - uses: actions/checkout@v3
      
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2
      
      - name: Login to Registry
        uses: docker/login-action@v2
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ secrets.REGISTRY_USER }}
          password: ${{ secrets.REGISTRY_PASSWORD }}
      
      - name: Extract metadata
        id: meta
        uses: docker/metadata-action@v4
        with:
          images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
          tags: |
            type=ref,event=branch
            type=sha,prefix={{branch}}-
      
      - name: Build and push
        uses: docker/build-push-action@v4
        with:
          context: .
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

ArgoCD Application Configuration

Complete configuration example:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-application
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  
  source:
    repoURL: https://forgejo.example.com/myteam/my-application
    targetRevision: main
    path: k8s/overlays/production
    
    # Kustomize options
    kustomize:
      version: v5.0.0
      images:
        - my-app=registry.example.com/my-app:v1.2.3
  
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  
  # Sync policy
  syncPolicy:
    automated:
      prune: true        # Delete resources not in Git
      selfHeal: true     # Override manual changes
      allowEmpty: false  # Don't delete everything on empty repo
    
    syncOptions:
      - CreateNamespace=true
      - PruneLast=true
      - RespectIgnoreDifferences=true
    
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m
  
  # Ignore differences (avoid sync loops)
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas  # Ignore if HPA manages replicas

Troubleshooting

Pipeline Fails

Problem: Forgejo Actions pipeline fails

Solution:

# 1. Check pipeline logs in Forgejo UI
# Navigate to: Repository → Actions → Select failed run

# 2. Check runner status
# In Forgejo: Site Admin → Actions → Runners

# 3. Check runner logs
kubectl logs -n forgejo-runner deployment/act-runner

# 4. Test pipeline locally with act
act -l  # List available jobs
act -j build  # Run specific job

ArgoCD Application OutOfSync

Problem: Application shows “OutOfSync” status

Solution:

# 1. Check differences
argocd app diff my-application

# 2. View sync status details
argocd app get my-application

# 3. Manual sync
argocd app sync my-application

# 4. Hard refresh (ignore cache)
argocd app sync my-application --force

# 5. Check for ignored differences
argocd app get my-application --show-operation

Application Deployment Fails

Problem: Application pod crashes after deployment

Solution:

# 1. Check pod status
kubectl get pods -n production

# 2. View pod logs
kubectl logs -n production deployment/my-application

# 3. Describe pod for events
kubectl describe pod -n production <pod-name>

# 4. Check resource limits
kubectl top pod -n production

# 5. Rollback via ArgoCD
argocd app rollback my-application

Image Pull Errors

Problem: Kubernetes cannot pull container image

Solution:

# 1. Verify image exists
docker pull registry.example.com/my-app:v1.2.3

# 2. Check image pull secret
kubectl get secret -n production regcred

# 3. Create image pull secret if missing
kubectl create secret docker-registry regcred \
  --docker-server=registry.example.com \
  --docker-username=user \
  --docker-password=password \
  -n production

# 4. Reference secret in deployment
kubectl patch deployment my-application -n production \
  -p '{"spec":{"template":{"spec":{"imagePullSecrets":[{"name":"regcred"}]}}}}'

Best Practices

Golden Path Templates

Provide standardized templates for common use cases:

1. Web Application Template: Node.js, Python, Go web services
2. API Service Template: RESTful API with OpenAPI
3. Batch Job Template: Kubernetes CronJob configurations
4. Microservice Template: Service mesh integration

Example repository template structure:

application-template/
├── .forgejo/
│   └── workflows/
│       ├── build.yaml
│       ├── test.yaml
│       └── deploy.yaml
├── k8s/
│   ├── base/
│   └── overlays/
├── src/
│   └── ...
├── Dockerfile
├── README.md
└── .gitignore

Deployment Checklist

Before deploying to production:

• ✅ All tests passing
• ✅ Security scans completed
• ✅ Resource limits defined
• ✅ Health checks configured
• ✅ Monitoring and alerts set up
• ✅ Backup strategy defined
• ✅ Rollback plan documented
• ✅ Team notified about deployment

Configuration Management

• Use ConfigMaps for non-sensitive configuration
• Use Secrets for sensitive data
• Use External Secrets Operator for vault integration
• Never commit secrets to Git
• Use environment-specific overlays (Kustomize)
• Document all configuration options

Status

Maturity: Production

Stability: Stable

Support: Internal Platform Team

Additional Resources

Forgejo

• Forgejo Documentation
• Forgejo Actions Guide
• Forgejo API Reference

ArgoCD

• ArgoCD Documentation
• ArgoCD Best Practices
• ArgoCD Sync Waves

GitOps

• GitOps Principles
• GitOps Patterns
• Kubernetes Deployment Strategies

CI/CD

• GitHub Actions Documentation (Forgejo Actions compatible)
• Docker Best Practices
• Container Security Best Practices

2 - Infrastructure as Code

Overview

Infrastructure as Code (IaC) is the practice of managing and provisioning infrastructure through code rather than manual processes. Instead of clicking through web consoles or running one-off commands, infrastructure is defined in version-controlled files that can be executed repeatedly to produce identical environments.

This approach treats infrastructure with the same rigor as application code: it’s versioned, reviewed, tested, and deployed through automated pipelines.

Why Infrastructure as Code?

The problem with manual infrastructure

Traditional infrastructure management faces several challenges:

• Inconsistency: Manual steps vary between operators and environments
• Undocumented: Critical knowledge exists only in operators’ heads
• Error-Prone: Human mistakes during repetitive tasks
• Slow: Manual provisioning takes hours or days
• Untrackable: No audit trail of what changed, when, or why
• Irreproducible: Difficulty recreating environments exactly

The IaC solution

Infrastructure as Code addresses these challenges by making infrastructure:

Declarative - Describe the desired state, not the steps to achieve it. The IaC tool handles the implementation details.

Versioned - Every infrastructure change is committed to Git, providing complete history and the ability to rollback.

Automated - Infrastructure deploys through pipelines without human intervention, eliminating manual errors.

Testable - Infrastructure changes can be validated before production deployment.

Documented - The code itself is the documentation, always current and accurate.

Reproducible - The same code produces identical infrastructure every time, across all environments.

Core Concepts

Declarative vs imperative

Imperative approaches specify the exact steps: “Create a server, then install software, then configure networking.”

Declarative approaches specify the desired outcome: “I need a server with this software and network configuration.” The IaC tool determines the necessary steps.

Most modern IaC tools use the declarative approach, making them more maintainable and resilient.

State Management

IaC tools maintain a “state” - a record of what infrastructure currently exists. When you change your code and re-run the tool, it compares the desired state (your code) with the actual state (what exists) and makes only the necessary changes.

This enables:

• Drift detection - Identify manual changes made outside IaC
• Safe updates - Modify only what changed
• Dependency management - Update resources in the correct order

Idempotency

Running the same IaC code multiple times produces the same result. If infrastructure already matches the code, the tool makes no changes. This property is called idempotency and is essential for reliable automation.

Infrastructure as Code in EDP

The Edge Developer Platform uses IaC extensively:

Terraform and Terragrunt

Terraform is our primary IaC tool for provisioning cloud resources. We use Terragrunt as an orchestration layer to manage multiple Terraform modules and reduce code duplication.

Our implementation includes:

• infra-catalogue - Reusable infrastructure components (modules, units, and stacks)
• infra-deploy - Full environment definitions using catalogue components

Platform stacks

We organize infrastructure into stacks - coherent bundles of related components:

• Core Stack - Essential platform services
• Forgejo Stack - Source control and CI/CD
• Observability Stack - Monitoring and logging
• OTC Stack - Cloud provider resources
• Coder Stack - Development environments
• Terralist Stack - Terraform registry

Each stack is defined as code, versioned independently, and can be deployed across different environments.

GitOps integration

Our IaC integrates with GitOps principles:

1. All infrastructure definitions live in Git repositories
2. Changes go through code review processes
3. Automated pipelines deploy infrastructure
4. ArgoCD continuously reconciles Kubernetes resources with Git state

This creates an auditable, automated, and reliable deployment process.

Benefits realized

Consistency across environments

Development, testing, and production environments are deployed from the same code. This eliminates the “works on my machine” problem at the infrastructure level.

Rapid environment provisioning

A complete EDP environment can be provisioned in minutes rather than days. This enables:

• Quick disaster recovery
• Easy creation of test environments
• Fast onboarding for new team members

Reduced operational risk

Code review catches infrastructure errors before deployment. Automated testing validates changes. Version control enables instant rollback if problems occur.

Knowledge sharing

Infrastructure configuration is explicit and discoverable in code. New team members can understand the platform by reading the repository rather than shadowing experienced operators.

Compliance and auditability

Every infrastructure change is tracked in Git history with author, timestamp, and reason. This provides audit trails required for compliance and simplifies troubleshooting.

Getting started

To work with EDP’s Infrastructure as Code:

1. Understand Terraform basics - Review Terraform documentation
2. Explore infra-catalogue - Browse infra-catalogue to understand available components
3. Review existing deployments - Examine infra-deploy to see how components are composed
4. Follow the Terraform guide - See Terraform-based deployment for detailed instructions

Best Practices

Based on our experience building and operating IaC:

Version everything - All infrastructure code belongs in version control. No exceptions.

Keep it simple - Start with basic modules. Add abstraction only when duplication becomes painful.

Test before production - Deploy infrastructure changes to test environments first.

Use meaningful commit messages - Explain why changes were made, not just what changed.

Review all changes - Infrastructure changes should go through the same review process as application code.

Document assumptions - Use code comments to explain non-obvious decisions.

Manage secrets securely - Never commit credentials to version control. Use secret management tools.

Plan for drift - Regularly compare actual infrastructure with code state to detect manual changes.

Challenges and limitations

Infrastructure as Code is powerful but has challenges:

Learning curve - Teams need to learn IaC tools and practices. Initial productivity may decrease.

State management complexity - State files must be stored securely and accessed by multiple team members. State corruption can cause serious issues.

Provider limitations - Not all infrastructure can be managed as code. Some resources require manual configuration.

Breaking changes - Poorly written code can destroy infrastructure. Safeguards and testing are essential.

Tool lock-in - Switching IaC tools (e.g., Terraform to Pulumi) requires rewriting infrastructure code.

Despite these challenges, the benefits far outweigh the costs for any infrastructure of meaningful complexity.

Why we invest in IaC

The IPCEI-CIS Edge Developer Platform requires reliable, reproducible infrastructure. Manual provisioning cannot meet these requirements at scale.

By investing in Infrastructure as Code:

• We can deploy complete environments consistently
• Platform engineers can focus on improvement rather than repetitive tasks
• Infrastructure changes are transparent and auditable
• New team members can contribute confidently
• Disaster recovery becomes routine rather than heroic

Our IaC tools (infra-catalogue and infra-deploy) embody these principles and enable the platform’s reliability.

Additional Resources

Terraform Ecosystem

• Terraform Documentation
• OpenTofu - Community-driven Terraform fork
• Terragrunt - Terraform orchestration

Infrastructure as Code Concepts

• Infrastructure as Code book by Kief Morris
• Terraform Best Practices
• CNCF Platforms White Paper

EDP-Specific Resources

• Terraform-based deployment - Detailed deployment guide
• Infrastructure Stacks - Reusable component bundles
• Platform Orchestration - How IaC fits into overall deployment

2.1 - Terraform-based deployment of EDP

Overview

The infra-deploy and infra-catalogue repositories work together to provide a framework for deploying Edge Developer Platform instances.

infra-catalogue contains individual, atomic infrastructure components: terraform modules and terragrunt units and stacks, such as Kubernetes clusters and Postgres databases.

infra-deploy then contains full definitions of stacks built using these components - such as the production site at edp.buildth.ing. It also includes scripts with which to deploy these stacks.

Note that both repositories rely on the wide range of features available on OTC. Several of these features, such as S3-compatible storage and on-demand managed Postgres instances, are not yet available on more sovereign clouds such as Edge, so these are not currently supported.

Key Features

• ‘Catalogue’ of infrastructure stacks to be used in deployments
• Definition of deployment stacks for each environment in prod or dev
• Scripts to govern deployment, installation and drift-correction of EDP

Purpose in EDP

For our Edge Developer Platform to be reliable it must be deployable in a consistent manner. When errors occur, or after any manual alterations, the system can then be safely reset to a working state. This state should be provided in code to allow for automated validation and deployment, and to allow it to be deployed from an always-identical CI/CD pipeline rather than a variable local deployment environment.

Repositories

Infra-deploy: https://edp.buildth.ing/DevFW/infra-deploy

Infra-catalogue: https://edp.buildth.ing/DevFW/infra-catalogue

Getting Started

Prerequisites

• Docker
• Kubernetes management
• Access to OTC
• HashiCorp Terraform or its open-source equivalent, OpenTofu
• Terragrunt, an orchestrator for Terraform stacks

Quick Start

1. Set up OTC credentials per README section
2. Set cluster environment and run install script per README section

Alternatively, manually trigger automated deployment pipeline.

• You will be asked for essential information like the deployment name and tenant.
• Any fields marked INITIAL only need to be set when first creating an environment
• Thereafter, the cached values are used and the INITIAL values provided to the pipeline are ignored.
  • Specifically, they are cached in a terragrunt.values.hcl file within infra-deploy/<tenant>/<cluster-name>, where both variables are set in the pipeline
  • e.g. prod/edp or nonprod/garm-provider-test

Verification

After the deploymenet completes, and a short startup time, you should be able to access your Forgejo instance at <cluster-name>.buildth.ing (production tenant) or <cluster-name>.t09.de (non-prod tenant). <cluster-name> is the name you provided in the deployment pipeline, or the $CLUSTER_ENVIRONMENT variable when running manually.

For example, the primary production cluster is called edp and can be accessed at edp.buildth.ing.

Screens

Deployment using production pipeline:

…

Configuration

Configuration of clusters is done in two ways. The first, mentioned above, is to provide INITIAL configuration when creating a new cluster. Thereafter, configuration is done within the relevant infra-deploy/<tenant> directory (e.g. prod/edp). Variables may be changed within the terragrunt.values.hcl file, but equally the terragrunt.stack.hcl file contains references to the lower-level code set up in infra-catalogue.

These are organised in layers, according to Terragrunt’s natural structure. First is a stack, a high-level abstraction for a whole cluster. This in turn references terragrunt units, which in turn are wrappers around standard Terraform modules. When deployed, the Terraform modules require a provider.tf file which is automatically generated by Terragrunt using tenant-level and global configuration stored in infra-deploy.

When deploying manually (e.g. with install.sh), you can observe these layers as Terragrunt will cache them on your machine, within the .terragrunt-stack/ directory generated within /<tenant>/<cluster-name>/.

Troubleshooting

Version updates

Problem: Updates to infra-catalogue are not immediately reflected in deployed clusters, even after running deploy.

Solution: Versions must be updated. Each cluster deployment specifies a catalogue version in its terragrunt.values.hcl; this refers to a tag in infra-catalogue. Within infra-catalogue, stacks reference units and modules from the same tag.

Thus, to test a new change to infra-catalogue, first make a new tag, then update the relevant values file to point to it.

Status

Maturity: TRL-9

Additional Resources

• Terraform
• OpenTofu, the community-driven replacement for Terraform
• Terragrunt

2.2 - Stacks

Overview

The stacks and stacks-instances repositories form the core of a GitOps-based system for provisioning Edge Developer Platforms (EDP). They implement a template-instance pattern that enables the deployment of reusable platform components across different environments. The concept of “stacks” originates from the CNOE.io project (Cloud Native Operational Excellence), which can be traced through the evolutionary development from edpbuilder (derived from CNOE.io’s EDPbuilder) to infra-deploy.

Key Features of the Everything-as-Code Stacks Approach

This declarative Stacks provisioning architecture is characterized by the following central properties:

Complete Code Declaration

Platform as Code: All Kubernetes resources, Helm charts, and application manifests are declaratively versioned as YAML files. The entire platform topology is traceable in Git.

Configuration as Code: Environment-specific configurations are generated through template hydration, not manually edited. Gomplate transforms generic templates into concrete configurations.

GitOps-Native Architecture

Single Source of Truth: Git is the sole source of truth for the desired state of all infrastructure and platform components.

Declarative State Management: ArgoCD continuously synchronizes the actual state with the desired state defined in Git. Deviations are automatically corrected.

Audit Trail: Every change to infrastructure or platform is documented through Git commits, with author, timestamp, and change description.

Pull-based Deployment: ArgoCD pulls changes from Git, rather than external systems requiring push access to the cluster. This significantly increases security.

Template-Instance Separation

DRY Principle (Don’t Repeat Yourself): Common platform components are defined once as templates and reused for all environments.

Environment Promotion: New environments can be quickly created through template hydration. Consistency across environments is guaranteed.

Centralized Maintainability: Updates to stack definitions can be made centrally in the stacks repository and then selectively rolled out to instances.

Customization Points: Despite reuse, environment-specific customizations remain possible through values files and manifest overlays.

Modular Composition

Stack-based Architecture: Platform capabilities are organized into independent, reusable stacks (core, otc, forgejo, observability).

Selective Deployment: Through the STACKS environment variable, only required components can be deployed selectively.

Mix-and-Match: Different stack combinations yield different platform profiles (Development, Production, Observability clusters).

Pluggable Components: New stacks can be added without modifying existing ones.

Environment Agnosticism

Cloud Provider Abstraction: Templates are formulated generically. Provider-specific details are introduced through hydration.

Multi-Cloud Ready: The architecture supports various cloud providers (currently OTC, historically KIND, extensible to AWS/Azure/GCP).

Environment Variables as Interface: All environment-specific aspects are controlled through clearly defined environment variables.

Portable Definitions: Stack definitions can be ported between environments and even cloud providers.

Self-Healing and Drift Detection

Automated Reconciliation: ArgoCD detects deviations from the desired state and corrects them automatically.

Continuous Monitoring: Permanent monitoring of cluster state compared to Git definition.

Declarative State Recovery: After failures or manual changes, the declared state is automatically restored.

Sync Policies: Configurable sync strategies (automated, manual, with pruning) per application.

Secrets Management

Secrets Outside Git: Sensitive data is not stored in Git but generated at runtime or injected from secret stores.

Generated Credentials: Passwords, tokens, and secrets are generated during deployment and directly created as Kubernetes Secrets.

Sealed Secrets Ready: The architecture is compatible with Sealed Secrets or External Secrets Operators for encrypted secret storage in Git.

Credential Rotation: Secrets can be regenerated through re-deployment.

Observability and Auditability

Declarative Monitoring: Observability stacks are part of the Platform-as-Code definition.

Deployment History: Complete history of all deployments and changes through Git log.

ArgoCD UI: Graphical representation of sync status and application topology.

Infrastructure Events: Terraform state changes and Terragrunt outputs document infrastructure changes.

Idempotence and Reproducibility

Idempotent Operations: Repeated execution of the same declaration leads to the same result without side effects.

Deterministic Builds: Same input parameters (Git commit + environment variables) produce identical environments.

Disaster Recovery: Complete environments can be rebuilt from code without restoring backups.

Testing in Production-Like Environments: Development and staging environments are code-identical to production, only with different parameter values.

Purpose in EDP

A ‘stack’ is the declarative description for the platform provisionning in an EDP installation.

Repository

Code:

• Stacks Templates Repo
• Stacks Instances Repo, used for ArgoCD Gitops
• EDP Stacks Deployment mechanism

Documentation: [Link to component-specific documentation]

• Outdated: The former ’edpbuilder’ as script, derived from CNOE’s ‘idpbuilder

The stacks Repository

Purpose and Structure

The stacks repository contains reusable template definitions for platform components. It serves as a central library of building blocks from which Edge Developer Platforms can be composed.

stacks/
└── template/
    ├── edfbuilder.yaml
    ├── registry/
    │   ├── core.yaml
    │   ├── otc.yaml
    │   ├── forgejo.yaml
    │   ├── observability.yaml
    │   └── observability-client.yaml
    └── stacks/
        ├── core/
        ├── otc/
        ├── forgejo/
        ├── observability/
        └── observability-client/

Components

edfbuilder.yaml: The central bootstrap definition. This is an ArgoCD Application that references the registry directory and serves as the entry point for the entire platform provisioning.

registry/: Contains ArgoCD ApplicationSets that function as a meta-layer. Each file defines a category of stacks (e.g., core, forgejo, observability) and references the corresponding subdirectory in stacks/.

stacks/: The actual platform components, organized into thematic categories:

• core: Fundamental components such as ArgoCD, CloudNative PostgreSQL, Dex (SSO)
• otc: Cloud-provider-specific components for Open Telekom Cloud (cert-manager, ingress-nginx, StorageClasses)
• forgejo: Git server and CI runners
• observability: Central observability components (Grafana, Victoria Metrics Stack)
• observability-client: Client-side metrics collection for non-observability clusters

Each stack consists of:

• YAML definitions (primarily ArgoCD Applications)
• values.yaml files for Helm charts
• manifests/ directories for additional Kubernetes resources

Templating Mechanism

The templates use Gomplate with delimiter syntax {{{ }}} for environment variables:

repoURL: "https://{{{ .Env.CLIENT_REPO_DOMAIN }}}/{{{ .Env.CLIENT_REPO_ORG_NAME }}}"
path: "{{{ .Env.CLIENT_REPO_ID }}}/{{{ .Env.DOMAIN }}}/stacks/core"

These placeholders are replaced with environment-specific values during the deployment phase.

The stacks-instances Repository

Purpose and Structure

The stacks-instances repository contains the materialized, environment-specific configurations. While stacks provides the blueprints, stacks-instances contains the actual deployment definitions for concrete environments.

stacks-instances/
└── otc/
    ├── osctest.t09.de/
    │   ├── edfbuilder.yaml
    │   ├── registry/
    │   └── stacks/
    ├── backup-test-manu.t09.de/
    │   ├── edfbuilder.yaml
    │   ├── registry/
    │   └── stacks/
    └── ...

Organizational Principle

The structure follows the schema {cloud-provider}/{domain}/:

• cloud-provider: Identifies the cloud environment (e.g., otc for Open Telekom Cloud)
• domain: The fully qualified domain name of the environment (e.g., osctest.t09.de)

Each environment replicates the structure of stacks/template, but with resolved template variables and environment-specific customizations.

Usage by ArgoCD

ArgoCD synchronizes directly from this repository. Applications reference paths such as:

source:
  path: "otc/osctest.t09.de/stacks/core"
  repoURL: "https://edp.buildth.ing/DevFW-CICD/stacks-instances"
  targetRevision: HEAD

This enables true GitOps: every change to the configurations is traceable through Git commits and automatically synchronized by ArgoCD in the target environment.

The infra-deploy Repository

Role in the Overall Architecture

The infra-deploy repository is the orchestration layer that coordinates both infrastructure and platform provisioning. It represents the evolution of edpbuilder, which was originally derived from the CNOE.io project’s EDPbuilder.

Two-Phase Provisioning

Phase 1: Infrastructure Provisioning

Uses Terragrunt Stacks (experimental feature) to provision cloud resources:

infra-deploy/
├── root.hcl
├── non-prod/
│   ├── tenant.hcl
│   ├── dns_zone/
│   │   ├── terragrunt.hcl
│   │   ├── terragrunt.stack.hcl
│   │   └── terragrunt.values.hcl
│   └── testing/
├── prod/
└── templates/
    └── forgejo/
        ├── terragrunt.hcl
        └── terragrunt.stack.hcl

Terragrunt Stacks provision:

• VPC and network segments
• Kubernetes clusters (CCE on OTC)
• Managed databases (RDS PostgreSQL)
• Load balancers and DNS entries
• Security groups and other cloud resources

Phase 2: Platform Provisioning

The script scripts/edp-install.sh executes the following steps:

1. Template Hydration:

   • Checkout of the stacks repository
   • Execution of Gomplate to resolve template variables
   • Generation of environment-specific manifests

2. Instance Management:

   • Checkout/update of the stacks-instances repository
   • During CI execution: commit and push of the new instance

3. Secrets Management:

   • Generation of credentials (database passwords, SSO secrets, API tokens)
   • Creation of Kubernetes Secrets

4. Bootstrap:

   • Helm-based installation of ArgoCD
   • Application of edfbuilder.yaml or selective registry entries

5. GitOps Handover:

   • ArgoCD takes over further synchronization from stacks-instances
   • Continuous monitoring and self-healing

GitHub Actions Workflows

The .github/workflows/ directory contains three central workflows:

deploy.yaml: Complete deployment pipeline with the following inputs:

• Cluster environment and tenant (prod/non-prod)
• Node flavor and availability zone
• Stack selection (core, otc, forgejo, observability, etc.)
• Infra-catalogue version

plan.yaml: Terraform/Terragrunt plan preview without execution

destroy.yaml: Controlled teardown of environments

Deployment Workflow

The complete provisioning process proceeds as follows:

1. Initiation: GitHub Actions workflow is triggered (manually or automatically)

2. Environment Preparation:

   export CLUSTER_ENVIRONMENT=qa-stage
   cd scripts
   ./new-otc-env.sh  # Creates Terragrunt configuration if new
   

3. Infrastructure Provisioning:

   ./ensure-cluster.sh otc
   # Internally executes:
   # - ./ensure-otc-cluster.sh
   # - terragrunt stack run apply
   

4. Platform Provisioning:

   ./edp-install.sh
   # Executes:
   # - Checkout of stacks
   # - Gomplate hydration
   # - Checkout/update of stacks-instances
   # - Secrets generation
   # - ArgoCD installation
   # - Bootstrap of stacks
   

5. ArgoCD Synchronization: ArgoCD continuously reads from stacks-instances and synchronizes the desired state

The CNOE.io Stacks Concept

The term “stacks” originates from the Cloud Native Operational Excellence (CNOE.io) project. The core idea is the composition of platform capabilities from modular, reusable building blocks.

Principles

Modularity: Each stack is a self-contained unit with clear dependencies

Composability: Stacks can be freely combined to create different platform profiles

Declarativeness: All configurations are declarative and GitOps-capable

Environment-agnostic: Templates are generic; environment specifics are introduced through hydration

Stack Selection and Combinations

The environment variable STACKS controls which components are deployed:

# Complete EDP with central observability
STACKS="core,otc,forgejo,observability"

# Application cluster with client-side observability
STACKS="core,otc,forgejo,observability-client"

# Minimal development environment
STACKS="core,forgejo"

Data Flow and Dependencies

┌─────────────────┐
│  GitHub Actions │
│  (deploy.yaml)  │
└────────┬────────┘
         │
         ├─> Phase 1: Infrastructure
         │   ┌──────────────────┐
         │   │  infra-deploy    │
         │   │  (Terragrunt)    │
         │   └────────┬─────────┘
         │            │
         │            v
         │   ┌──────────────────┐
         │   │  Cloud Provider  │
         │   │  (OTC)           │
         │   │  - VPC           │
         │   │  - K8s Cluster   │
         │   │  - RDS           │
         │   └──────────────────┘
         │
         └─> Phase 2: Platform
             ┌──────────────────┐
             │  edp-install.sh  │
             └────────┬─────────┘
                      │
                      ├─> Checkout: stacks (Templates)
                      │   └─> Gomplate Hydration
                      │
                      ├─> Checkout/Update: stacks-instances
                      │
                      ├─> Secrets Generation
                      │
                      ├─> ArgoCD Installation (Helm)
                      │
                      └─> Bootstrap (edfbuilder.yaml)
                          │
                          v
                 ┌────────────────┐
                 │    ArgoCD      │
                 └────────┬───────┘
                          │
                          └─> Continuous Synchronization
                              from stacks-instances
                              │
                              v
                       ┌──────────────┐
                       │  Kubernetes  │
                       │  Cluster     │
                       └──────────────┘

Historical Context: edpbuilder to infra-deploy

The evolution from edpbuilder to infra-deploy demonstrates the maturation of the architecture:

edpbuilder (Origin):

• Directly derived from CNOE.io’s EDPbuilder
• Focus on local KIND clusters
• Manual configuration
• Monolithic structure

infra-deploy (Current):

• Production-ready for cloud deployments (OTC)
• Terragrunt-based infrastructure orchestration
• CI/CD integration via GitHub Actions
• Clear separation between infrastructure and platform
• Template-instance separation through stacks/stacks-instances

Technical Particularities

Gomplate Templating

Gomplate is used with custom delimiters {{{ }}} to avoid conflicts with Helm templating ({{ }}):

gomplate --input-dir="stacks/template" \
         --output-dir="work" \
         --left-delim "{{{" \
         --right-delim "}}}"

Terragrunt Experimental Stacks

The use of Terragrunt Stacks requires the experimental flag:

export TG_EXPERIMENT_MODE=true
terragrunt stack run apply

This enables hierarchical organization of Terraform modules with dependency management.

ArgoCD ApplicationSets

The registry pattern uses ArgoCD Applications that reference directories:

source:
  path: "otc/osctest.t09.de/stacks/core"

ArgoCD automatically detects all YAML files in the path and synchronizes them as Applications.

Best Practices and Patterns

Immutable Infrastructure: Every environment is fully defined in Git

Secrets Outside Git: Sensitive data is generated at runtime or injected from secret stores

Progressive Rollouts: New environments start as template instances, then are individually customized

Version Pinning: Critical components (Helm charts, Terragrunt modules) are pinned to specific versions

Namespace Isolation: Each stack deploys into dedicated namespaces

Self-Healing: ArgoCD’s automated sync policy enables automatic drift correction

Usage Examples

Deployment by Pipeline

The platform deployment is the second part of the EDP installtaion. First there is the infrastructure setup, which ends with a created kubernetes cluster. Then the platform provisioning by the defined stacks is done. Both is runnable by the deploypipelien in infra-deploy:

The green pipeline looks liek this:

Local setup with ‘kind’

It’s also possible to just run the second part, the stcks provisionning. Then you need to have a kubernetes cluster already running, which is e.g. feasable by a local kind-cluster.

So imagine, you want to to the stacks ‘core,observability’ on your local machine. Then you can run the local entzr

# have kind insatlled
# in /infra-deploy

# provide a kind cluster
kind delete clusters --all
./scripts/ensure-kind-cluster.sh -r

# provide some emnv vars
export TERRAFORM=/bin/bash
export LOADBALANCER_ID=ABC
export DOMAIN=ABC
export DOMAIN_GITEA=ABC
export OS_ACCESS_KEY=ABC
export OS_SECRET_KEY=ABC
export STACKS=core,observability

# deploy
./scripts/edp-install.sh

Status

Maturity: [Production]

Additional Resources

• CNOE

2.2.1 - Core

Overview

The Core stack provides foundational infrastructure components required by all other Edge Developer Platform stacks. It establishes the base layer for continuous deployment, database services, and centralized authentication, enabling a secure, scalable platform architecture.

The Core stack deploys ArgoCD for GitOps orchestration, CloudNativePG for PostgreSQL database management, and Dex for OpenID Connect single sign-on capabilities.

Key Features

• GitOps Continuous Deployment: ArgoCD manages declarative infrastructure and application deployments
• Database Operator: CloudNativePG provides enterprise-grade PostgreSQL clusters for platform services
• Single Sign-On: Dex offers centralized OIDC authentication across platform components
• Automated Synchronization: Self-healing deployments with automatic drift correction
• Role-Based Access Control: Integrated RBAC for secure platform administration
• TLS Certificate Management: Automated certificate provisioning and renewal

Repository

Code: Core Stack Templates

Documentation:

• ArgoCD Documentation
• CloudNativePG Documentation
• Dex Documentation

Getting Started

Prerequisites

• Kubernetes cluster (1.24+)
• kubectl configured with cluster access
• Ingress controller (nginx recommended)
• cert-manager for TLS certificate management
• Domain names configured for platform services

Quick Start

The Core stack is deployed as the foundation of the EDP installation:

1. Trigger Deploy Pipeline

   • Go to Infra Deploy Pipeline
   • Click on Run workflow
   • Enter a name in “Select environment directory to deploy”. This must be DNS Compatible. (if you enter test-me then domains will be argocd.test-me.t09.de, dex.test-me.t09.de)
   • Execute workflow

2. ArgoCD Bootstrap The deployment automatically provisions:

   • ArgoCD control plane in argocd namespace
   • CloudNativePG operator in cloudnative-pg namespace
   • Dex identity provider in dex namespace
   • Ingress configurations with TLS certificates
   • OIDC authentication integration

Verification

Verify the Core stack deployment:

# Check ArgoCD installation
kubectl get application -n argocd
kubectl get pods -n argocd

# Verify CloudNativePG operator
kubectl get pods -n cloudnative-pg
kubectl get crd | grep cnpg.io

# Check Dex deployment
kubectl get pods -n dex
kubectl get ingress -n dex

# Verify ingress configurations
kubectl get ingress -n argocd

Access ArgoCD at https://argocd.{DOMAIN} and authenticate via Dex SSO. Or use username admin and the secret inside of kubernetes argocd/argocd-initial-admin-secret as password kubectl get secret -n argocd argocd-initial-admin-secret -ojson | jq -r .data.password | base64 -d.

Architecture

Component Architecture

The Core stack establishes a three-tier foundation:

ArgoCD Control Plane:

• Application management and GitOps reconciliation
• Multi-repository tracking with automated sync
• Resource health monitoring and drift detection
• Integrated RBAC with SSO authentication

CloudNativePG Operator:

• PostgreSQL cluster lifecycle management
• Automated backup and recovery
• High availability and failover
• Storage provisioning via CSI drivers

Dex Identity Provider:

• OpenID Connect authentication service
• Multiple connector support (Forgejo/Gitea, LDAP, SAML)
• Static client registration for platform services
• Token issuance and validation

Networking

Ingress Architecture:

• nginx ingress controller for external access
• TLS termination with cert-manager integration
• Domain-based routing for platform services

Kubernetes Services:

• Internal service communication via ClusterIP
• DNS-based service discovery
• Network policies for security segmentation

Configuration

ArgoCD Configuration

Deployed via Helm chart v9.1.5 with custom values in stacks/core/argocd/values.yaml:

OIDC Authentication:

configs:
  cm:
    url: "https://{DOMAIN_ARGOCD}"
    oidc.config: |
      name: Forgejo
      issuer: https://{DOMAIN_DEX}
      clientID: controller-argocd-dex
      clientSecret: $dex-controller-argocd-dex:dex-controller-argocd-dex
      requestedScopes: ["openid", "profile", "email", "groups"]

RBAC Policy:

policy.csv: |
  g, DevFW, role:admin

Server Settings:

• Insecure mode enabled (TLS handled by ingress)
• Annotation-based resource tracking
• 60-second reconciliation timeout
• Resource exclusions for ProviderConfigUsage and CiliumIdentity

CloudNativePG Configuration

Deployed via Helm chart v0.26.1 with values in stacks/core/cloudnative-pg/values.yaml:

Operator Settings:

• Namespace: cloudnative-pg
• Automated database cluster provisioning
• Custom resource definitions for Cluster, Database, and Pooler resources

Storage Configuration:

• Uses csi-disk storage class by default
• PVC provisioning for PostgreSQL data
• Backup storage integration (S3-compatible)

Dex Configuration

Deployed via Helm chart v0.23.0 with values in stacks/core/dex/values.yaml:

Issuer Configuration:

config:
  issuer: https://{DOMAIN_DEX}
  storage:
    type: memory  # Use persistent storage for production
  oauth2:
    skipApprovalScreen: true
    alwaysShowLoginScreen: false

Forgejo Connector:

connectors:
  - type: gitea
    id: forgejo
    name: Forgejo
    config:
      clientID: $FORGEJO_CLIENT_ID
      clientSecret: $FORGEJO_CLIENT_SECRET
      redirectURI: https://{DOMAIN_DEX}/callback
      baseURL: https://edp.buildth.ing
      orgs:
        - name: DevFW

Static OAuth2 Clients:

• ArgoCD: controller-argocd-dex
• Grafana: controller-grafana-dex

Environment Variables

Core stack services use the following environment variables:

Domain Configuration:

• DOMAIN_ARGOCD: ArgoCD web interface URL
• DOMAIN_DEX: Dex authentication service URL
• DOMAIN_GITEA: Forgejo/Gitea repository URL
• DOMAIN_GRAFANA: Grafana observability dashboard URL

Repository Configuration:

• CLIENT_REPO_ID: Repository identifier for stack configurations
• CLIENT_REPO_DOMAIN: Git repository domain
• CLIENT_REPO_ORG_NAME: Organization name for stack instances

Usage Examples

Managing Applications with ArgoCD

Access and manage applications through ArgoCD:

# Login to ArgoCD CLI
argocd login argocd.${DOMAIN} --sso

# List all applications
argocd app list

# Get application status
argocd app get coder

# Sync application manually
argocd app sync coder

# View application logs
argocd app logs coder

# Diff application state
argocd app diff coder

Creating a PostgreSQL Database

Deploy a PostgreSQL cluster using CloudNativePG:

# database-cluster.yaml
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: app-db
  namespace: my-app
spec:
  instances: 3
  storage:
    size: 20Gi
    storageClass: csi-disk
  postgresql:
    parameters:
      max_connections: "100"
      shared_buffers: "256MB"
  bootstrap:
    initdb:
      database: appdb
      owner: appuser

Apply the configuration:

kubectl apply -f database-cluster.yaml

# Check cluster status
kubectl get cluster app-db -n my-app
kubectl get pods -n my-app -l cnpg.io/cluster=app-db

# Get connection credentials
kubectl get secret app-db-app -n my-app -o jsonpath='{.data.password}' | base64 -d

Configuring SSO for Applications

Add OAuth2 applications to Dex for SSO integration:

# Add to dex values.yaml
staticClients:
  - id: my-app-client
    redirectURIs:
      - 'https://myapp.{DOMAIN}/callback'
    name: 'My Application'
    secretEnv: MY_APP_CLIENT_SECRET

Configure the application to use Dex:

# Application OIDC configuration
OIDC_ISSUER=https://dex.${DOMAIN}
OIDC_CLIENT_ID=my-app-client
OIDC_CLIENT_SECRET=${MY_APP_CLIENT_SECRET}
OIDC_REDIRECT_URI=https://myapp.${DOMAIN}/callback

Deploying Applications via ArgoCD

Create an ArgoCD Application manifest:

# my-app.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
  namespace: argocd
spec:
  project: default
  source:
    repoURL: 'https://github.com/myorg/my-app'
    targetRevision: main
    path: k8s
  destination:
    server: 'https://kubernetes.default.svc'
    namespace: my-app
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Push it to stacks instances to be picked up by argo

Integration Points

• All Stacks: Core stack is a prerequisite for all other EDP stacks
• OTC Stack: Provides ingress-nginx and cert-manager dependencies
• Coder Stack: Uses CloudNativePG for workspace database management
• Forgejo Stack: Integrates with Dex for SSO and ArgoCD for deployment
• Observability Stack: Uses Dex for Grafana authentication and ArgoCD for deployment
• Provider Stack: Deploys Terraform providers via ArgoCD

Troubleshooting

ArgoCD Not Accessible

Problem: Cannot access ArgoCD web interface

Solution:

1. Verify ingress configuration:

   kubectl get ingress -n argocd
   kubectl describe ingress -n argocd
   

2. Check ArgoCD server status:

   kubectl get pods -n argocd
   kubectl logs -n argocd -l app.kubernetes.io/name=argocd-server
   

3. Verify TLS certificate:

   kubectl get certificate -n argocd
   kubectl describe certificate -n argocd
   

4. Test DNS resolution:

   nslookup argocd.${DOMAIN}
   

Dex Authentication Failing

Problem: SSO login fails or redirects incorrectly

Solution:

1. Check Dex logs:

   kubectl logs -n dex -l app.kubernetes.io/name=dex
   

2. Verify Forgejo connector configuration:

   kubectl get secret -n dex
   kubectl get configmap -n dex dex -o yaml
   

3. Test Dex issuer endpoint:

   curl https://dex.${DOMAIN}/.well-known/openid-configuration
   

4. Verify OAuth2 client credentials match in both Dex and consuming application

CloudNativePG Operator Not Running

Problem: PostgreSQL clusters fail to provision

Solution:

1. Check operator status:

   kubectl get pods -n cloudnative-pg
   kubectl logs -n cloudnative-pg -l app.kubernetes.io/name=cloudnative-pg
   

2. Verify CRDs are installed:

   kubectl get crd | grep cnpg.io
   kubectl describe crd clusters.postgresql.cnpg.io
   

3. Check operator logs for errors:

   kubectl logs -n cloudnative-pg -l app.kubernetes.io/name=cloudnative-pg --tail=100
   

Application Sync Failures

Problem: ArgoCD applications remain out of sync or fail to deploy

Solution:

1. Check application status:

   argocd app get <app-name>
   kubectl describe application <app-name> -n argocd
   

2. Review sync operation logs:

   argocd app logs <app-name>
   

3. Verify repository access:

   argocd repo list
   argocd repo get <repo-url>
   

4. Check for resource conflicts or missing dependencies:

   kubectl get events -n <app-namespace> --sort-by='.lastTimestamp'
   

Database Connection Issues

Problem: Applications cannot connect to CloudNativePG databases

Solution:

1. Verify cluster is ready:

   kubectl get cluster <cluster-name> -n <namespace>
   kubectl describe cluster <cluster-name> -n <namespace>
   

2. Check database credentials secret:

   kubectl get secret <cluster-name>-app -n <namespace>
   kubectl get secret <cluster-name>-app -n <namespace> -o yaml
   

3. Test connection from a pod:

   kubectl run -it --rm psql-test --image=postgres:16 --restart=Never -- \
     psql "$(kubectl get secret <cluster-name>-app -n <namespace> -o jsonpath='{.data.uri}' | base64 -d)"
   

4. Review PostgreSQL logs:

   kubectl logs -n <namespace> <cluster-name>-1
   

Additional Resources

• ArgoCD Documentation
• ArgoCD Best Practices
• CloudNativePG Documentation
• CloudNativePG Architecture
• Dex Documentation
• Dex Connectors
• OpenID Connect Specification

2.2.2 - OTC

Overview

The OTC (Open Telekom Cloud) stack provides essential infrastructure components for deploying applications on Open Telekom Cloud environments. It configures ingress routing, automated TLS certificate management, and cloud-native storage provisioning tailored specifically for OTC’s Kubernetes infrastructure.

This stack serves as a foundational layer that other platform stacks depend on for external access, secure communication, and persistent storage.

Key Features

• Automated TLS Certificate Management: Let’s Encrypt integration via cert-manager for automatic certificate provisioning and renewal
• Cloud Load Balancer Integration: Nginx ingress controller configured with OTC-specific Elastic Load Balancer (ELB) annotations
• Native Storage Provisioning: Default StorageClass using Huawei FlexVolume provisioner for block storage
• Prometheus Metrics: Built-in monitoring capabilities for ingress traffic and performance
• High Availability: Rolling update strategy with minimal downtime
• HTTP-01 Challenge Support: ACME validation through ingress for certificate issuance

Repository

Code: OTC Stack Templates

Documentation:

• cert-manager Documentation
• ingress-nginx Documentation
• Open Telekom Cloud Documentation

Getting Started

Prerequisites

• Kubernetes cluster running on Open Telekom Cloud
• ArgoCD installed (provided by core stack)
• Environment variables configured:
  • LOADBALANCER_ID: OTC Elastic Load Balancer ID
  • LOADBALANCER_IP: OTC Elastic Load Balancer IP address
  • CLIENT_REPO_DOMAIN: Git repository domain
  • CLIENT_REPO_ORG_NAME: Git repository organization
  • CLIENT_REPO_ID: Client repository identifier
  • DOMAIN: Domain name for the environment

Quick Start

The OTC stack is deployed as part of the EDP installation process:

1. Trigger Deploy Pipeline

   • Go to Infra Deploy Pipeline
   • Click on Run workflow
   • Enter a name in “Select environment directory to deploy”. This must be DNS Compatible.
   • Execute workflow

2. ArgoCD Synchronization ArgoCD automatically deploys:

   • cert-manager with ClusterIssuer for Let’s Encrypt
   • ingress-nginx controller with OTC load balancer integration
   • Default StorageClass for OTC block storage

Verification

Verify the OTC stack deployment:

# Check ArgoCD applications status
kubectl get application otc -n argocd
kubectl get application cert-manager -n argocd
kubectl get application ingress-nginx -n argocd
kubectl get application storageclass -n argocd

# Verify cert-manager pods
kubectl get pods -n cert-manager

# Check ingress-nginx controller
kubectl get pods -n ingress-nginx

# Verify ClusterIssuer status
kubectl get clusterissuer main

# Check StorageClass
kubectl get storageclass default

Architecture

Component Architecture

The OTC stack consists of three primary components:

cert-manager:

• Automates TLS certificate lifecycle management
• Integrates with Let’s Encrypt ACME server (production endpoint)
• Uses HTTP-01 challenge validation via ingress
• Creates and manages certificates as Kubernetes resources
• Single replica deployment

ingress-nginx:

• Kubernetes ingress controller based on Nginx
• Routes external traffic to internal services
• Integrated with OTC Elastic Load Balancer (ELB)
• Supports TLS termination with cert-manager issued certificates
• Rolling update strategy with max 1 unavailable pod
• Prometheus metrics exporter with ServiceMonitor

StorageClass:

• Default storage provisioner for persistent volumes
• Uses Huawei FlexVolume driver (flexvolume-huawei.com/fuxivol)
• SATA block storage type
• Immediate volume binding mode
• Supports dynamic volume expansion

Integration Flow

External Traffic → OTC ELB → ingress-nginx → Kubernetes Services
                                    ↓
                              cert-manager (TLS certificates)
                                    ↓
                              Let's Encrypt ACME

Configuration

cert-manager Configuration

Helm Values (stacks/otc/cert-manager/values.yaml):

crds:
  enabled: true
replicaCount: 1

ClusterIssuer (stacks/otc/cert-manager/manifests/clusterissuer.yaml):

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: main
spec:
  acme:
    email: admin@think-ahead.tech
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: cluster-issuer-account-key
    solvers:
    - http01:
        ingress:
          ingressClassName: nginx

Key Settings:

• CRDs installed automatically
• Production Let’s Encrypt ACME endpoint
• HTTP-01 validation through nginx ingress
• ClusterIssuer named main for cluster-wide certificate issuance

ingress-nginx Configuration

Helm Values (stacks/otc/ingress-nginx/values.yaml):

controller:
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
  service:
    annotations:
      kubernetes.io/elb.class: union
      kubernetes.io/elb.port: '80'
      kubernetes.io/elb.id: {{{ .Env.LOADBALANCER_ID }}}
      kubernetes.io/elb.ip: {{{ .Env.LOADBALANCER_IP }}}
  ingressClassResource:
    name: nginx
  allowSnippetAnnotations: true
  config:
    proxy-buffer-size: 32k
    use-forwarded-headers: "true"
  metrics:
    enabled: true
    serviceMonitor:
      additionalLabels:
        release: "ingress-nginx"
      enabled: true

Key Settings:

• OTC Load Balancer Integration: Annotations configure connection to OTC ELB
• Rolling Updates: Minimizes downtime with 1 pod unavailable during updates
• Snippet Annotations: Enabled for advanced ingress configuration (idpbuilder compatibility)
• Proxy Buffer: 32k buffer size for handling large headers
• Forwarded Headers: Preserves original client information through proxies
• Metrics: Prometheus ServiceMonitor for observability

StorageClass Configuration

StorageClass (stacks/otc/storageclass/storageclass.yaml):

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.beta.kubernetes.io/is-default-class: "true"
  name: default
parameters:
  kubernetes.io/hw:passthrough: "true"
  kubernetes.io/storagetype: BS
  kubernetes.io/volumetype: SATA
  kubernetes.io/zone: eu-de-02
provisioner: flexvolume-huawei.com/fuxivol
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true

Key Settings:

• Default StorageClass: Automatically used when no StorageClass specified
• OTC Zone: Provisioned in eu-de-02 availability zone
• SATA Volumes: Block storage (BS) with SATA performance tier
• Volume Expansion: Supports resizing persistent volumes dynamically
• Reclaim Policy: Volumes deleted when PersistentVolumeClaim is removed

ArgoCD Application Configuration

Registry Application (template/registry/otc.yaml):

• Name: otc
• Manages the OTC stack directory
• Automated sync with prune and self-heal enabled
• Creates namespaces automatically

Component Applications:

cert-manager (referenced in stack):

• Deploys cert-manager Helm chart
• Automated self-healing enabled
• Includes ClusterIssuer manifest for Let’s Encrypt

ingress-nginx (template/stacks/otc/ingress-nginx.yaml):

• Deploys from official Kubernetes ingress-nginx repository
• Chart version: helm-chart-4.12.1
• References environment-specific values from stacks-instances repository

storageclass (template/stacks/otc/storageclass.yaml):

• Deploys StorageClass manifest
• Managed as ArgoCD Application
• Automated sync with unlimited retries

Usage Examples

Creating an Ingress with Automatic TLS

Create an ingress resource that automatically provisions a TLS certificate:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-app
  namespace: my-namespace
  annotations:
    cert-manager.io/cluster-issuer: main
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - myapp.example.com
    secretName: myapp-tls
  rules:
  - host: myapp.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: my-app-service
            port:
              number: 80

cert-manager will automatically:

1. Detect the ingress with cert-manager.io/cluster-issuer annotation
2. Create a Certificate resource
3. Request certificate from Let’s Encrypt using HTTP-01 challenge
4. Store certificate in myapp-tls secret
5. Renew certificate before expiration

Creating a PersistentVolumeClaim

Use the default OTC StorageClass for persistent storage:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-data
  namespace: my-namespace
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: csi-disk

Expanding an Existing Volume

Resize a persistent volume by editing the PVC:

# Edit the PVC storage request
kubectl patch pvc my-data -n my-namespace -p '{"spec":{"resources":{"requests":{"storage":"20Gi"}}}}'

# Verify expansion
kubectl get pvc my-data -n my-namespace

The volume will expand automatically due to allowVolumeExpansion: true in the StorageClass.

Custom Ingress Configuration

Use nginx ingress snippets for advanced routing:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: advanced-app
  annotations:
    cert-manager.io/cluster-issuer: main
    nginx.ingress.kubernetes.io/configuration-snippet: |
      more_set_headers "X-Custom-Header: value";
      if ($http_user_agent ~* "bot") {
        return 403;
      }
spec:
  ingressClassName: nginx
  rules:
  - host: app.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: app-service
            port:
              number: 8080

Integration Points

• Core Stack: Requires ArgoCD for deployment orchestration
• All Application Stacks: Depends on OTC stack for:
  • External access via ingress-nginx
  • TLS certificates via cert-manager
  • Persistent storage via default StorageClass
• Observability Stack: ingress-nginx metrics exported to Prometheus
• Coder Stack: Uses ingress and cert-manager for workspace access
• Forgejo Stack: Requires ingress and TLS for Git repository access

Troubleshooting

Certificate Issuance Fails

Problem: Certificate remains in Pending state and is not issued

Solution:

1. Check Certificate status:

   kubectl get certificate -A
   kubectl describe certificate <cert-name> -n <namespace>
   

2. Verify ClusterIssuer is ready:

   kubectl get clusterissuer main
   kubectl describe clusterissuer main
   

3. Check cert-manager logs:

   kubectl logs -n cert-manager -l app=cert-manager
   

4. Verify HTTP-01 challenge can reach ingress:

   kubectl get challenges -A
   kubectl describe challenge <challenge-name> -n <namespace>
   

5. Common issues:

   • DNS not pointing to load balancer IP
   • Firewall blocking HTTP (port 80) traffic
   • Ingress class not set to nginx
   • Let’s Encrypt rate limits exceeded

Ingress Controller Not Ready

Problem: ingress-nginx pods are not running or LoadBalancer service has no external IP

Solution:

1. Check ingress controller status:

   kubectl get pods -n ingress-nginx
   kubectl logs -n ingress-nginx -l app.kubernetes.io/component=controller
   

2. Verify LoadBalancer service:

   kubectl get svc -n ingress-nginx
   kubectl describe svc ingress-nginx-controller -n ingress-nginx
   

3. Check OTC load balancer annotations:

   kubectl get svc ingress-nginx-controller -n ingress-nginx -o yaml
   

4. Verify environment variables are set correctly:

   • LOADBALANCER_ID matches OTC ELB ID
   • LOADBALANCER_IP matches ELB public IP

5. Check OTC console for ELB configuration and health checks

Storage Provisioning Fails

Problem: PersistentVolumeClaim remains in Pending state

Solution:

1. Check PVC status:

   kubectl get pvc -A
   kubectl describe pvc <pvc-name> -n <namespace>
   

2. Verify StorageClass exists and is default:

   kubectl get storageclass
   kubectl describe storageclass default
   

3. Check volume provisioner logs:

   kubectl logs -n kube-system -l app=csi-disk-plugin
   

4. Common issues:

   • Insufficient quota in OTC project
   • Invalid zone configuration (must be eu-de-02)
   • Requested storage size exceeds limits
   • Missing IAM permissions for volume creation

Ingress Returns 503 Service Unavailable

Problem: Ingress configured but returns 503 error

Solution:

1. Verify backend service exists:

   kubectl get svc <service-name> -n <namespace>
   kubectl get endpoints <service-name> -n <namespace>
   

2. Check if pods are ready:

   kubectl get pods -n <namespace> -l <service-selector>
   

3. Verify ingress configuration:

   kubectl describe ingress <ingress-name> -n <namespace>
   

4. Check nginx ingress logs:

   kubectl logs -n ingress-nginx -l app.kubernetes.io/component=controller --tail=100
   

5. Test service connectivity from ingress controller:

   kubectl exec -n ingress-nginx <controller-pod> -- curl http://<service-name>.<namespace>.svc.cluster.local:<port>
   

TLS Certificate Shows as Invalid