Edge Developer Platform

• 1: Forgejo
• 1.1: Forgejo Actions
• 1.1.1: Runners
• 1.1.1.1: Runner Resource Optimization
• 1.1.1.2: Orchestration with GARM
• 1.2: Project Management in Forgejo
• 2: Deployment
• 2.1: Basic Concepts
• 2.1.1: Platform Orchestration
• 2.1.2: Application Orchestration
• 2.2: Infrastructure as Code
• 2.2.1: Terraform-based deployment of EDP
• 2.2.2: Stacks
• 2.2.2.1: Core
• 2.2.2.2: OTC
• 2.2.2.3: Coder
• 2.2.2.4: Terralist
• 2.2.2.5: Forgejo
• 2.2.2.6: Observability
• 2.2.2.7: Observability Client
• 2.3: Deploying to OTC
• 2.3.1: EDP Environments in OTC
• 2.3.2: Managing Instances
• 3: CI Sizer
• 3.1: Configuration
• 3.2: Web Dashboard
• 3.3: Sizing Algorithm
• 3.4: OOM Detection & Confidence-Gated Sizing
• 3.5: Energy Estimation
• 3.6: API Reference
• 3.7: GitLab CI Integration
• 3.8: KPI Benchmark
• 4: Operations
• 5: StageX Container Images
• 6: Documentation System

1 - Forgejo

The internal service is officially designated as the Edge Developer Platform (EDP). It is hosted at edp.buildth.ing. The domain selection followed a democratic team process to establish a unique identity distinct from standard corporate naming conventions.

Technical Architecture & Deployment

Infrastructure Stack

The platform is hosted on the Open Telekom Cloud (OTC). The infrastructure adheres to Infrastructure-as-Code (IaC) principles.

• Deployment Method: The official Forgejo Helm Chart is deployed via ArgoCD.
• Infrastructure Provisioning: Terraform is used to provision all underlying OTC services, including:
  • Container Orchestration: CCE (Cloud Container Engine): Kubernetes
  • Database: RDS (Distributed Cache Service): PostgreSQL
  • Caching: DCS (Distributed Cache Service): Redis
  • Object Storage: OBS (Object Storage Service, S3-compatible): for user data (avatars, attachments).
  • Search: CSS (Cloud Search Service): Elasticsearch

The “Self-Replicating” Pipeline

A key architectural feature is the ability of the platform to maintain itself. A Forgejo Action can trigger the deployment script, which runs Terraform and syncs ArgoCD, effectively allowing “Forgejo to create/update Forgejo.”

graph TD
    subgraph "Open Telekom Cloud (OTC)"
        subgraph "Control Plane"
            Dev[DevOps Engineer] -->|Triggers| Pipeline[Deployment Pipeline]
            Pipeline -->|Executes| TF[Terraform]
        end

        subgraph "Provisioned Infrastructure"
            TF -->|Provisions| CCE[(CCE K8s Cluster)]
            TF -->|Provisions| RDS[(RDS PostgreSQL)]
            TF -->|Provisions| Redis[(DCS Redis)]
            TF -->|Provisions| S3[(OBS S3 Bucket)]
            TF -->|Provisions| CSS[(CSS Elasticsearch)]
        end

        subgraph "Application Layer (on CCE K8s)"
            Pipeline -->|Helm Chart| Argo[ArgoCD]
            Argo -->|Deploys| ForgejoApp[Forgejo]
        end

        CCE -- Runs --> Argo
        CCE -- Runs --> ForgejoApp
        ForgejoApp -->|Connects| RDS
        ForgejoApp -->|Connects| Redis
        ForgejoApp -->|Connects| S3
        ForgejoApp -->|Connects| CSS
    end

Migration History

The initial environment was a manual setup on the Open Sovereign Cloud (OSC). Once the automation stack (Terraform/ArgoCD) was matured, the platform was migrated to the current OTC environment.

Application Extensions

Core Functionality

Beyond standard Git versioning, the platform utilizes:

• Releases: Hosting binaries for software distribution (e.g., Edge Connect CLI).
• CI/CD: Extensive pipeline usage for build, test, and deployment automation.
• Note on Issues: While initially used, issue tracking was migrated to JIRA to align with the broader IPCEI program standards.

GARM (Git-based Actions Runner Manager)

The primary technical innovation was the integration of GARM to enable ephemeral, scalable runners. This required extending Forgejo’s capabilities to support GitHub-compatible runner registration and webhook events.

Development Methodology & Contributions

Workflow

• Branching Strategy: Trunk-based development was utilized to ensure rapid integration.
• Collaboration: The team adopted Mob Programming. This practice proved essential for knowledge sharing and onboarding junior developers, creating a resilient and high-intensity learning environment.
• Versions: The platform evolved from Forgejo v7/8 through v11.0.3-edp1 to the current v14 (upgraded Q1 2026, IPCEICIS-7848). The Forgejo 14 upgrade resolved outstanding version lag and enabled adoption of the latest upstream GARM integration features.

Open Source Contributions

We actively contributed our extensions back to the upstream Forgejo project in a list of Codeberg.org pull requests

Artifact Caching (Pull-Through Proxy)

We implemented a feature allowing Forgejo to act as a pull-through proxy for remote container registries, optimizing bandwidth and build speeds.

• Source Code Branch: refactor-remote-registry-client

Security & Platform Hardening (2026)

A security hardening initiative was completed in Q1 2026 across the EDP platform:

Multi-Factor Authentication

MFA is now enabled and enforced for all EDP platform users at edp.buildth.ing. Users are required to configure a TOTP-compatible authenticator on next login.

Forgejo Administration Cleanup

A review of Forgejo administration accounts and service accounts was carried out. Redundant admin and bot accounts were removed or scoped down, tightening the overall access surface of the platform.

Automated Vulnerability Scanning (Trivy)

Trivy vulnerability scanning is now automated in EDP CI/CD pipelines via a Forgejo Action. Scans cover container images, source code dependencies, and IaC configurations. Results are automatically uploaded to the Dependency-Track instance for tracking and triage.

Redis Reliability Fix

Redis (the Distributed Cache Service powering Forgejo on OTC) was prone to filling up under active crawling load, causing 500 errors across Forgejo operations. An automated remediation was implemented:

• An OTC Cloud Eye alarm monitors Redis memory usage
• A notification channel triggers a cloud function when the threshold is approached
• The cloud function automatically clears Redis data before it causes Forgejo to break

This eliminates the need for manual intervention to restore Forgejo availability after Redis saturation events.

Key Performance Indicators (KPIs)

These KPIs measure the effectiveness of the Forgejo setup and quantify our strategic commitment to the Forgejo community.

1.1 - Forgejo Actions

Overview

Forgejo Actions is a built-in CI/CD automation system that enables developers to define and execute workflows directly within their Forgejo repositories. As a continuous integration and continuous deployment platform, Forgejo Actions automates software development tasks such as building, testing, packaging, and deploying applications whenever specific events occur in your repository.

Forgejo Actions provides GitHub Actions similarity, allowing teams to easily adapt existing GitHub Actions workflows and marketplace actions with minimal or no modifications. This compatibility significantly reduces migration effort for teams transitioning from GitHub to Forgejo, while maintaining familiar syntax and workflow patterns.

Workflows are defined using YAML files stored in the .forgejo/workflows/ directory of your repository. Each workflow consists of one or more jobs that execute on action runners when triggered by repository events such as pushes, pull requests, tags, or manual dispatch. This enables automation of repetitive development tasks, ensuring consistent build and deployment processes across your software delivery pipeline.

By integrating CI/CD directly into the repository management platform, Forgejo Actions eliminates the need for external CI/CD systems, reducing infrastructure complexity and providing a unified development experience.

Key Features

• Automated Workflow Execution - Execute automated workflows triggered by repository events such as code pushes, pull requests, tag creation, or manual dispatch, enabling continuous integration and deployment without manual intervention
• GitHub Actions Similarity - Maintains similarity with GitHub Actions syntax and workflows, allowing reuse of existing actions from the GitHub marketplace and simplifying migration from GitHub-based CI/CD pipelines

Purpose in EDP

Forgejo Actions enables EDP customers to execute complete CI/CD pipelines directly on the platform for building, testing, packaging, and deploying software. This integrated automation capability is fundamental to the EDP value proposition.

Without native CI/CD automation, customers would face significant integration overhead connecting external CI/CD systems to their EDP workflows. This fragmentation would complicate pipeline management, increase operational complexity, and reduce the platform’s effectiveness as a unified development solution.

Since Forgejo Actions is natively integrated into Forgejo, EDP provides this critical CI/CD capability with minimal additional infrastructure. Customers benefit from seamless automation without requiring separate tool provisioning, authentication configuration, or cross-system integration maintenance.

Getting Started

Prerequisites

• Installed Forgejo
• Installed Forgejo runner (see Runner Installation Quick Start)

Quick Start

1. Create a repository
2. Create file /.forgejo/workflows/example.yaml

# example.yaml
name: example

on:
  workflow_dispatch:

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Hello World
        run: |
          echo "Hello World!"

1. Navigate to Actions > example.yaml > Run workflow

Verification

See the logs, there should appear a “Hello World!” in “Hello World” Step

Usage Examples

Use actions to deploy infrastructure

See infra-deploy repository as a example

Use goreleaser to build, test, package and release a project

This pipeline is triggered when a tag with the prefix v is pushed to the repository. Then, it fetches the current repository with all tags and checks out the version for the current run.

After that the application is being built.

# .github/workflows/release.yaml
name: ci

on:
  push:
    tags:
      - v*

jobs:
  goreleaser:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Set up Go
        uses: actions/setup-go@v6
        with:
          go-version: ">=1.25.1"
      - name: Test code
        run: make test
      - name: Import GPG key
        id: import_gpg
        uses: https://github.com/crazy-max/ghaction-import-gpg@v6
        with:
          gpg_private_key: ${{ secrets.GPG_PRIVATE_KEY }}
          passphrase: ${{ secrets.GPG_PASSPHRASE }}
      - name: Run GoReleaser
        uses: https://github.com/goreleaser/goreleaser-action@v6
        env:
          GITEA_TOKEN: ${{ secrets.PACKAGES_TOKEN }}
          GPG_FINGERPRINT: ${{ steps.import_gpg.outputs.fingerprint }}
        with:
          args: release --clean

Troubleshooting

The job is not being executed by a runner

Problem: The job is not being picked up by a runner

Solution: Probably, there is currently no runner available with the label defined in your job runs-on attribute. Check the available runner for your repository by navigating to the repository settings > Actions > Runners. Now you can see all available runners and their Labels. Choose on of them as your runs-on attribute.

Status

Maturity: Production

Additional Resources

• Forgejo Actions
• GitHub Actions
• GitHub Actions similarity

1.1.1 - Runners

Overview

Action runners are the execution environment for Forgejo Actions workflows. By design, runners execute remote code submitted through CI/CD pipelines, making their architecture highly dependent on the underlying infrastructure and security requirements.

The primary objective in any runner setup is the separation and isolation of individual runs. Since runners are specifically built to execute arbitrary code from repositories, proper isolation is critical to prevent data and secret leakage between different pipeline executions. Each runner must be thoroughly cleaned or recreated after every job to ensure no residual data persists that could compromise subsequent runs.

Beyond isolation concerns, action runners represent high-value targets for supply chain attacks. Runners frequently compile, build, and package software binaries that may be distributed to thousands or millions of end users. Compromising a runner could allow attackers to inject malicious code directly into the software supply chain, making runner security a critical consideration in any deployment.

This document explores different runner architectures, examining their security characteristics, operational trade-offs, and suitability for various infrastructure environments and showing off an example deployment using a Containerized Kubernetes environment.

Key Features

• Consistent environment for Forgejo Actions
• Primary location to execute code e.g. deployments
• Good security practices essential due to broad remit

Purpose in EDP

A actions runner are executing Forgejo actions, which can be used to build, test, package and deploy software. To ensure that EDP customers do not need to provision their own action runners with high efford, we provide globally registered actions runners to pick up jobs.

Repository

Code:

• Runner on edge connect using GARM
• Static runner

Documentation: Forgejo Runner installation guide

Runner Setups

Different runner deployment architectures offer varying levels of isolation, security, and operational complexity. The choice depends on your infrastructure capabilities, security requirements, and operational overhead tolerance.

On Bare Metal

Bare metal runners execute directly on physical hardware without virtualization layers.

Advantages:

• Maximum performance with direct hardware access
• Complete hardware isolation between different physical machines
• No hypervisor overhead or virtualization complexity

Disadvantages:

• Difficult to clean after each run, requiring manual intervention or full OS reinstallation
• Long provisioning time for individual runners
• Complex provisioning processes requiring physical access or remote management tools
• Limited scalability due to physical hardware constraints
• Higher risk of persistent contamination between runs

Use case: Best suited for specialized workloads requiring specific hardware, performance-critical builds, or environments where virtualization is not available.

On Virtual Machines

VM-based runners operate within virtualized environments managed by a hypervisor.

Advantages:

• Strong isolation through hypervisor and hardware memory mapping
• Virtual machine images enable faster provisioning compared to bare metal
• Easy to snapshot, clone, and restore to clean states
• Better resource utilization through multiple VMs per physical host
• Automated cleanup by destroying and recreating VMs after each run

Disadvantages:

• Requires hypervisor infrastructure and management
• Slower provisioning than containers
• Higher resource overhead compared to containerized solutions
• More complex orchestration for scaling runner fleets

Use case: Ideal for environments requiring strong isolation guarantees, multi-tenant scenarios, or when running untrusted code from external contributors.

In Containerized Environment

Container-based runners execute within isolated containers using OCI-compliant runtimes.

Advantages:

• Kernel-level isolation using Linux namespaces and cgroups
• Fast provisioning and startup times
• Easy deployment through standardized OCI container images
• Lightweight resource usage enabling high-density runner deployments
• Simple orchestration with Kubernetes or Docker Compose

Disadvantages:

• Weaker isolation than VMs since containers share the host kernel
• Requires elevated permissions or privileged access for certain workflows (e.g., Docker-in-Docker)
• Potential kernel-level vulnerabilities affect all containers on the host
• Container escape vulnerabilities pose security risks in multi-tenant environments

Use case: Best for high-volume CI/CD workloads, trusted code repositories, and environments prioritizing speed and efficiency over maximum isolation.

Getting Started

Prerequisites

• Forgejo instance
• Runner registration token has been generated for a given scope
  • Global runners in admin settings > actions > runner > Create new runner
  • Organization runners in organization settings > actions > runner > Create new runner
  • Repository runners in repository settings > actions > runner > Create new runner
• Kubernetes cluster

Quick Start

1. Download Kubernetes manifest
2. Replace ${RUNNER_SECRET} with the runner registration token
3. Replace ${RUNNER_NAME} with the name the runner should have
4. Replace ${FORGEJO_INSTANCE_URL} with the instance url
5. (if namespace does not exists) kubectl create ns gitea
6. Run kubectl apply -f <file>

Verification

Take a look at the runners page, where you generated the token. There should be 3 runners in idle state now.

Sequence Diagrams

---
title: Forgejo Runner executed in daemon mode
---
sequenceDiagram
    Runner->>Forgejo: Register runner
    loop Job Workflow
    Runner->>Forgejo: Fetch job
    Runner->>Runner: Work on job
    Runner->>Forgejo: Send result
    end

Deployment Architecture

[Add infrastructure and deployment diagrams showing how the component is deployed]

Configuration

There is a sophisticated configuration file, where finetuning can be done. The most important thing is done by using labels to define the execution environment.

The label ubuntu-latest:docker://ghcr.io/catthehacker/ubuntu:act-22.04 (as used in example runner). That a job that uses ubuntu-latest label will be executed as docker container inside the ghcr.io/catthehacker/ubuntu:act-22.04 image.

Alternatives to docker are lxc and host.

Troubleshooting

In containerized environments, I want to build container images

Problem: In containerized environment, containers usually do not have many privileges. To start or build containers additional privleges, usually root is required inside of the kernel, the container runtime needs to manage linux namespaces and cgroups.

Solution: A partial solution for this is buildkitd utilizing rootlesskit. This allows containers to be built (but not run) in a non root environment. Several examples can be found in the official buildkit repo.

Rootless vs User namespaces:

As of Kubernetes 1.33, uid mapping can be enabled for pods using pod.spec.hostUsers: false utilizing user namespaces to map user and group ids between the container ids (0-65535) to high host ids (0-65535 + n * 65536) where n is an arbitrary number of containers. This allows that the container runs with actual root permission in its user namespace without being root on the host system.

Rootless is considered the more secure version, as the executable is mapped to a privileged entitiy at all.

Status

Maturity: Beta

Additional Resources

• Forgejo Runner installation guide
• Static Runners on Kubernetes
• Runner Orchestartion using GARM on Edge Connect

1.1.1.1 - Runner Resource Optimization

Overview

Runner Resource Optimization is a PoC feature delivered in Q1 2026 (IPCEICIS-6887) that analyses historical CPU and memory data from CI/CD pipeline executions to recommend right-sized runner configurations. Alongside it, a runner sustainability dashboard (IPCEICIS-7421) was shipped into the Forgejo runner settings page, giving users visibility into historical runner usage and current runner statuses.

The motivation is straightforward: manually choosing a runner size is guesswork. Developers tend to over-provision to avoid failures, leaving compute unused and energy wasted. By using real utilization data, the system can suggest the smallest runner that still safely completes the job.

Key Features

• Historical utilization analysis: Collects CPU and memory metrics at 10-second intervals across pipeline runs and retains 30 days of data
• Right-sizing recommendations: Calculates peak and average resource consumption per pipeline/job type and recommends the smallest runner size with a 20% safety margin above peak usage
• Runner sustainability dashboard: Embedded in the Forgejo runner settings page — shows which runners were used in workflow jobs, historical usage trends, and current runner statuses
• Workflow execution metrics collection: Gathers structured per-job metrics to feed the recommendation algorithm (IPCEICIS-7413)

Purpose in EDP

CI/CD runners are the largest variable compute cost in the EDP. Most users default to a fixed runner size regardless of actual job requirements. This feature closes that gap by:

• Surfacing utilization data that would otherwise be invisible
• Giving teams actionable, evidence-based recommendations without requiring deep infrastructure knowledge
• Tracking runner usage per project, supporting sustainability reporting (“which runners powered my workflows?”)

How the Algorithm Works

The recommendation algorithm operates as follows for a given pipeline/job type:

1. Collect: Retrieve the last n runs’ CPU and memory utilization for the job
2. Analyse: Calculate peak and average resource consumption across those runs
3. Recommend: Identify the smallest runner size in the family (small → medium → large → xlarge) where peak usage fits within the available resources, plus a 20% safety margin
4. Output: Present current runner size vs. recommended size side-by-side

Example output:
  Job: build-and-test
  Current runner: large  (8 vCPU, 16 GB RAM)
  Peak CPU:  2.4 vCPU   Peak RAM: 5.8 GB
  Recommended: medium    (4 vCPU, 8 GB RAM)  [peak + 20% margin fits]

The recommendation is conservative by design: it does not auto-apply changes. Teams review and opt-in, avoiding surprise failures.

Runner Sustainability Dashboard

The dashboard is accessible from the Forgejo runner settings page (same permission scope as the runner list). It provides:

The data is surfaced without leaving Forgejo — no external dashboarding tool is required for basic usage. For deeper observability, metrics are also exported to the EDP Grafana instance at observability.buildth.ing.

Metrics Collection (IPCEICIS-7413)

Workflow execution metrics are gathered during pipeline runs with less than 5% overhead on pipeline execution time. The collected data includes:

• Job start/end timestamps
• Runner identity and size
• Peak and average CPU utilization (sampled at 10-second intervals)
• Peak and average memory utilization
• Job exit status (success/failure)

These metrics feed the recommendation algorithm and the dashboard simultaneously.

Status

Maturity: PoC — the recommendation algorithm and dashboard are functional and deployed on edp.buildth.ing. Auto-enforcement (automatic runner resizing) is explicitly out of scope for this iteration.

Additional Resources

• Runners overview
• GARM runner orchestration
• EDP Observability
• Jira: IPCEICIS-6887 (Epic), IPCEICIS-7421 (Dashboard), IPCEICIS-7413 (Metrics collection)

1.1.1.2 - Orchestration with GARM

Overview

GARM provides on-demand runner orchestration for Forgejo Actions through dynamic autoscaling. As Forgejo has similar API structure to Gitea (from which it was forked), GARM’s Gitea/GitHub compatibility makes it a natural fit for automated runner provisioning. GARM supports custom providers, enabling runner infrastructure deployment across multiple cloud and infrastructure platforms.

A custom edge-connect provider was implemented for GARM to enable infrastructure provisioning. Additionally, Forgejo was adapted to align more closely with Gitea’s API, ensuring seamless integration with GARM’s orchestration capabilities.

Key Features

• Autoscales Forgejo Actions runners dynamically based on workload demand
• Leverages edge-connect infrastructure for distributed runner provisioning

Purpose in EDP

• Provides CI/CD infrastructure for all software development projects
• Enhances the EDP platform capabilities through improved Forgejo automation
• Enables teams to focus on development by consuming platform-managed runners without capacity planning concerns

Repository

Code:

• GARM Provider for Edge Connect
• GARM deploy script
• GARM deploy manifests

Getting Started

Prerequisites

• Container Runtime installed (e.g. docker)
• Forgejo, Gitea or Github

Quick Start

1. Clone the GARM Provider repository
2. Build the Docker image: docker buildx build -t <your-image-tag> .
3. Push the image to your container registry
4. Deploy GARM using the deployment script from the infra-deploy repository, targeting your Kubernetes cluster: ./local-helm.sh --garm

Verification

• Verify the GARM pod is running: kubectl get pods -n garm
• Retrieve the GARM domain endpoint: kubectl get ing -n garm
• Get the GARM admin password: kubectl get secret -n garm garm-credentials -o json | jq .data.GARM_ADMIN_PASSWORD -r | base64 -d
• Configure endpoints, credentials, repositories, and runner pools in GARM as described in the garm-provider-test repository.

Integration Points

• Forgejo: Picks up pending action jobs, listen in Forgejo
• Edge Connect: Uses this infrastructure to deploy runners that can pick up open jobs in forgejo

Architecture

The primary technical innovation was the integration of GARM to enable ephemeral, scalable runners. This required extending Forgejo’s capabilities to support GitHub-compatible runner registration and webhook events.

Workflow Architecture:

1. Event: A workflow event occurs in Forgejo.
2. Trigger: A webhook notifies GARM.
3. Provisioning: GARM spins up a fresh, ephemeral runner.
4. Execution: The runner registers via the API, executes the job, and is terminated immediately after, ensuring a clean build environment.

sequenceDiagram
    participant User
    participant Forgejo
    participant GARM
    participant Runner as Ephemeral Runner

    User->>Forgejo: Push Code / Trigger Event
    Forgejo->>GARM: Webhook Event (Workflow Dispatch)
    GARM->>Forgejo: Register Runner (via API)
    GARM->>Runner: Spin up Instance
    Runner->>Forgejo: Request Job
    Forgejo->>Runner: Send Job Payload
    Runner->>Runner: Execute Steps
    Runner->>Forgejo: Report Status
    GARM->>Runner: Terminate (Ephemeral)

Sequence Diagrams

The diagram below shows how a trigger of an action results in deployment of a runner on edge-connect.

Deployment Architecture

Configuration

Provider Setup

The config below configures an external provder for garm. Especially important is the provider.external.config_file which refers to the configuration of the external provider (example below) and provider.external.provider_executable which needs to point to the provider executable.

# config.toml
...
[[provider]]
name = "edge-connect"
description = "edge connect provider"
provider_type = "external"

[provider.external]
config_file = "/etc/garm/edge-connect-provider-config.toml"
provider_executable = "/opt/garm/providers.d/garm-provider-edge-connect"
environment_variables = ["EDP_EDGE_CONNECT_"]

# edge-connect-provider-config.toml
log_file = "/garm/provider.log"
credentials_file = "/etc/garm-creds/credentials.toml" # to authenticate agains edge_connect.url

[edge_connect]
organization = "edp-developer-framework"
region = "EU"
url = "https://hub.apps.edge.platform.mg3.mdb.osc.live"
default_flavor = "EU.small"

[edge_connect.cloudlet]
name = "Munich"
organization = "TelekomOP"

# credentials.toml for edge connect platform
username = ""
password = ""

Runner Pool Configuration

Once the configuration is in place and garm has been deployed. You can connect garm to Forgejo/Gitea/Github, using the commands below. If you have a forgejo instance, you want to create a gitea endpoint.

# https://edp.buildth.ing/DevFW/garm-deploy/src/branch/master/helm/garm/templates/init-job.yaml#L39-L56
garm-cli init --name gitea --password ${GARM_ADMIN_PASSWORD} --username ${GARM_ADMIN_USERNAME} --email ${GARM_ADMIN_EMAIL} --url ${GARM_URL}
if [ $? -ne 0 ]; then
  echo "garm maybe already initialized"
  exit 0
fi

# API_GIT_URL=https://garm-provider-test.t09.de/api/v1
# GIT_URL=https://garm-provider-test.t09.de
garm-cli gitea endpoint create \
--api-base-url ${API_GIT_URL} \
--base-url ${GIT_URL} \
--description "My first Gitea endpoint" \
--name local-gitea

garm-cli gitea credentials add \
--endpoint local-gitea \
--auth-type pat \
--pat-oauth-token $GITEA_TOKEN \
--name autotoken \
--description "Gitea token"

Now, connect to the WebUI, use GARM_ADMIN_USERNAME and GARM_ADMIN_PASSWORD as credentials to authenticate. Click on repositories and

Status

Maturity: Beta

Additional Resources

• GARM repository
  • How to use

1.2 - Project Management in Forgejo

Overview

This was an attempt to extend Forgejo’s project and issue management capabilities beyond the repository level. The goal was to enable organizations and users to create projects and issues that could span multiple repositories or exist independently of any repository.

Problem Statement

Forgejo’s issue management is repository-centered. While this works well for code-specific issues, it creates challenges for broader project management:

• Cross-repository work: Tasks often span multiple repositories but must be artificially tied to one
• Non-code projects: Some projects don’t map cleanly to a repository (e.g., planning, documentation initiatives)
• Related repositories: Symbiotically related repos would benefit from shared issue tracking

Real-world examples:

• Upstream: forgejo-actions-feature-requests - arguably doesn’t need repository/code functionality
• EDP: infra-deploy and infra-catalogue - symbiotically related projects

Implementation Status

Status: Prototype level - basic operations work but not production-ready

What was built:

• Projects can be created at the organization/user level (not tied to repositories)
• Issues can be created within these organization-level projects
• Issues can be moved between columns within any projects
• Basic Create and View Issue pages function without errors

What was incomplete:

• Several features on Create/View pages disabled rather than adapted, e.g. due dates
• Repository-specific features (tags, code reviews, etc.) not resolved for org-level context
• Broader issue management features not yet functional

Discontinuation

Development was discontinued due to:

• Project priorities shifted to other platform features
• Scope of remaining work deemed too large for the anticipated value
• Concerns about maintaining a custom feature divergent from upstream Forgejo

Repository

Code: edp-forgejo (Remark: You must be logged into edp.buildth.ing as the repo is internal)

This is a fork of upstream Forgejo with the organization-level project management changes. The fork is based on Forgejo v11.x (upstream has progressed to at least v13.x).

Implementation: Changes to both UI (in TypeScript) and server-side (Golang) functionality.

Technical Approach

The implementation involved:

• Minimally modifying Forgejo’s data model to associate projects with organizations/users instead of repositories
• Adapting issue creation and display logic to work without repository context
• Addressing repository-specific settings (labels, milestones, code review integration) for org-level issues
• UI changes to support project creation and issue management at the organization level

Integration Points

This feature was developed as an isolated extension to Forgejo. Its code is within the edp-forgejo repository alongside other EDP updates - such as magenta colour scheme - but in terms of functionality has minimal overlap/links with other EDP components.

Lessons Learned

• Repository-centric design is deeply embedded in Forgejo’s architecture
• Maintaining custom features in a fork creates significant maintenance burden
• The scope of fully-functional cross-repository project management is substantial
  • This is related to Issues and Repositories being two of the most extensive features in Forgejo
• Alternative approaches (using dedicated project management tools, or simply ‘shell’ repositories) may be more sustainable
• Clear buy-in is needed for the long term in order to make a change like this viable

2 - Deployment

Overview

Platform Orchestration refers to the automation and management of the platform infrastructure itself. This includes the provisioning, configuration, and lifecycle management of all components that make up the Internal Developer Platform (IDP).

In the context of IPCEI-CIS, Platform Orchestration means:

• Platform Bootstrap: Initial setup of Kubernetes clusters and core services
• Platform Services Management: Deployment and management of ArgoCD, Forgejo, Keycloak, etc.
• Infrastructure-as-Code: Declarative management using Terraform and GitOps
• Multi-Cluster Orchestration: Coordination across different Kubernetes clusters
• Platform Stacks: Reusable bundles of platform components (CNOE concept)

Target Audience

Platform Orchestration is primarily aimed at:

• Platform Engineering Teams: Teams that build and operate the IDP
• Infrastructure Architects: Those responsible for the platform architecture
• SRE Teams: Teams responsible for reliability and operations

Key Features

Declarative Platform Definition

The entire platform is defined declaratively as code:

• GitOps-First: Everything is versioned in Git and traceable
• Reproducibility: The platform can be rebuilt at any time
• Environment Parity: Consistency between Dev, Test, and Production
• Auditability: Complete history of all changes

Self-Bootstrapping

The platform can bootstrap itself:

1. Initial Bootstrap: Minimal tool (like idpbuilder) starts the platform
2. Self-Management: After bootstrap, ArgoCD takes over management
3. Continuous Reconciliation: Platform is continuously reconciled with Git state
4. Self-Healing: Automatic recovery on deviations

Stack-based Composition

Platform components are organized as reusable stacks (CNOE concept):

• Modularity: Components can be updated individually
• Reusability: Stacks can be used across different environments
• Composability: Compose complex platforms from simple building blocks
• Versioning: Stacks can be versioned and tested

In IPCEI-CIS: The stacks concept from CNOE is the core organizational principle for platform components.

Multi-Cluster Support

Platform Orchestration supports different cluster topologies:

• Control Plane + Worker Clusters: Centralized control, distributed workloads
• Hub-and-Spoke: One management cluster manages multiple target clusters
• Federation: Coordination across multiple independent clusters

Purpose in EDP

Platform Orchestration is the foundation of the IPCEI-CIS Edge Developer Platform. It enables:

Foundation for Developer Self-Service

Platform Orchestration ensures all services are available that developers need for self-service:

• GitOps Engine (ArgoCD) for continuous deployment
• Source Control (Forgejo) for code and configuration management
• Identity Management (Keycloak) for authentication and authorization
• Observability (Grafana, Prometheus) for monitoring and logging
• CI/CD (Forgejo Actions/Pipelines) for automated build and test

Consistency Across Environments

Through declarative definition, consistency is guaranteed:

• Development, test, and production environments are identically configured
• No “configuration drift” between environments
• Predictable behavior across all stages

Platform as Code

The platform itself is treated like software:

• Version Control: All changes are versioned in Git
• Code Review: Platform changes go through review processes
• Testing: Platform configurations can be tested
• Rollback: Easy rollback on problems

Reduced Operational Overhead

Automation reduces manual effort:

• No manual installation steps
• Automatic updates and patching
• Self-healing on failures
• Standardized deployment processes

Repository

CNOE Reference Implementation: cnoe-io/stacks

CNOE idpbuilder: cnoe-io/idpbuilder

Documentation: CNOE.io Documentation

Getting Started

Prerequisites

• Docker: For local Kubernetes clusters (Kind)
• kubectl: Kubernetes CLI tool
• Git: For repository management
• idpbuilder: CNOE bootstrap tool

Quick Start

Platform Orchestration with CNOE Reference Implementation:

# 1. Install idpbuilder
curl -fsSL https://cnoe.io/install.sh | bash

# 2. Bootstrap platform
idpbuilder create \
  --use-path-routing \
  --package-dir https://github.com/cnoe-io/stacks//ref-implementation

# 3. Wait for platform ready (ca. 10 minutes)
kubectl get applications -A

Verification

Verify the platform is running correctly:

# Get platform secrets (credentials)
idpbuilder get secrets

# Check all ArgoCD applications
kubectl get applications -n argocd

# Expected: All applications "Synced" and "Healthy"

Access URLs (with path-routing):

• ArgoCD: https://cnoe.localtest.me:8443/argocd
• Forgejo: https://cnoe.localtest.me:8443/gitea
• Keycloak: https://cnoe.localtest.me:8443/keycloak

Usage Examples

Use Case 1: Platform Bootstrap

Initial bootstrapping of a new platform instance:

idpbuilder create \
  --use-path-routing \
  --package-dir https://github.com/cnoe-io/stacks//ref-implementation \
  --log-level debug

# Workflow:
# 1. Creates Kind cluster
# 2. Installs ingress-nginx
# 3. Clones and installs ArgoCD
# 4. Installs Forgejo
# 5. Waits for core services
# 6. Creates technical users
# 7. Configures Git repositories
# 8. Installs remaining stacks via ArgoCD

After approximately 10 minutes, the platform is fully deployed.

Use Case 2: Adding New Platform Components

Add new platform components via ArgoCD:

# Create ArgoCD Application for new component
cat <<EOF | kubectl apply -f -
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: external-secrets
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://charts.external-secrets.io
    targetRevision: 0.9.9
    chart: external-secrets
  destination:
    server: https://kubernetes.default.svc
    namespace: external-secrets-system
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
EOF

Use Case 3: Platform Updates

Update platform components:

# 1. Update via Git (GitOps)
cd your-platform-config-repo
git pull

# 2. Update stack version
vim argocd/applications/component.yaml
# Change targetRevision to new version

# 3. Commit and push
git add .
git commit -m "Update component to v1.2.3"
git push

# 4. ArgoCD will automatically sync
# 5. Monitor the update
argocd app sync component --watch

Integration Points

ArgoCD Integration

• Bootstrap: ArgoCD is initially installed via idpbuilder
• Self-Management: After bootstrap, ArgoCD manages itself via Application CRD
• Platform Coordination: ArgoCD orchestrates all other platform components
• Health Monitoring: ArgoCD monitors health status of all platform services

Forgejo Integration

• Source of Truth: Git repositories contain all platform definitions
• GitOps Workflow: Changes in Git trigger platform updates
• Backup: Git serves as backup of platform configuration
• Audit Trail: Git history documents all platform changes
• CI/CD: Forgejo Actions can automate platform operations

Terraform Integration

• Infrastructure Provisioning: Terraform provisions cloud resources for platform
• State Management: Terraform state tracks infrastructure
• Integration: Terraform can be triggered via Forgejo pipelines
• Multi-Cloud: Support for multiple cloud providers

Architecture

Platform Orchestration Flow

┌─────────────────┐
│   idpbuilder    │  Bootstrap Tool
│  (Initial Run)  │
└────────┬────────┘
         │
         ▼
┌─────────────────────────────────────────────────────┐
│              Kubernetes Cluster                      │
│                                                      │
│  ┌──────────────┐         ┌──────────────┐         │
│  │   ArgoCD     │────────▶│   Forgejo    │         │
│  │  (GitOps)    │         │  (Git Repo)  │         │
│  └──────┬───────┘         └──────────────┘         │
│         │                                            │
│         │  Monitors & Syncs                         │
│         │                                            │
│         ▼                                            │
│  ┌──────────────────────────────────────┐          │
│  │     Platform Stacks                  │          │
│  │                                      │          │
│  │  ┌──────────┐  ┌──────────┐        │          │
│  │  │Forgejo   │  │Keycloak  │        │          │
│  │  └──────────┘  └──────────┘        │          │
│  │  ┌──────────┐  ┌──────────┐        │          │
│  │  │Observ-   │  │Ingress   │        │          │
│  │  │ability   │  │          │        │          │
│  │  └──────────┘  └──────────┘        │          │
│  └──────────────────────────────────────┘          │
└─────────────────────────────────────────────────────┘

Platform Bootstrap Sequence

The idpbuilder executes the following workflow:

1. Create Kind Kubernetes cluster
2. Install ingress-nginx controller
3. Install ArgoCD
4. Install Forgejo Git server
5. Wait for services to be ready
6. Create technical users in Forgejo
7. Create repository for platform state in Forgejo
8. Push platform stacks to Forgejo
9. Create ArgoCD Applications for all stacks
10. ArgoCD takes over continuous synchronization

Deployment Architecture

The platform is deployed in different namespaces:

• argocd: ArgoCD and its components
• gitea: Forgejo Git server
• keycloak: Identity and access management
• observability: Prometheus, Grafana, etc.
• ingress-nginx: Ingress controller

Configuration

idpbuilder Configuration

Key configuration options for idpbuilder:

# Path-based routing (recommended for local development)
idpbuilder create --use-path-routing

# Custom package directory
idpbuilder create --package-dir /path/to/custom/packages

# Custom Kind cluster config
idpbuilder create --kind-config custom-kind.yaml

# Enable debug logging
idpbuilder create --log-level debug

ArgoCD Configuration

Important ArgoCD configurations for platform orchestration:

# argocd-cm ConfigMap
data:
  # Enable automatic sync
  application.instanceLabelKey: argocd.argoproj.io/instance

  # Repository credentials
  repositories: |
    - url: https://github.com/cnoe-io/stacks
      name: cnoe-stacks
      type: git

  # Resource exclusions
  resource.exclusions: |
    - apiGroups:
      - cilium.io
      kinds:
      - CiliumIdentity

Platform Stack Configuration

Configuration of platform stacks via Kustomize:

# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: platform-system

resources:
  - argocd-app.yaml
  - forgejo-app.yaml
  - keycloak-app.yaml

patches:
  - target:
      kind: Application
    patch: |-
      - op: add
        path: /spec/syncPolicy
        value:
          automated:
            prune: true
            selfHeal: true

Troubleshooting

Platform not reachable

Problem: After idpbuilder create, platform services are not reachable

Solution:

# 1. Check if all pods are running
kubectl get pods -A

# 2. Check ArgoCD application status
kubectl get applications -n argocd

# 3. Check ingress
kubectl get ingress -A

# 4. Verify DNS resolution
nslookup cnoe.localtest.me

# 5. Check idpbuilder logs
idpbuilder get logs

ArgoCD Applications not synchronized

Problem: ArgoCD Applications show status “OutOfSync”

Solution:

# 1. Check application details
argocd app get <app-name>

# 2. View sync status
argocd app sync <app-name> --dry-run

# 3. Force sync
argocd app sync <app-name> --force

# 4. Check for errors in ArgoCD logs
kubectl logs -n argocd deployment/argocd-application-controller

Git Repository Connection Issues

Problem: ArgoCD cannot access Git repository

Solution:

# 1. Verify repository configuration
argocd repo list

# 2. Test connection
argocd repo get https://your-git-repo

# 3. Check credentials
kubectl get secret -n argocd

# 4. Re-add repository with correct credentials
argocd repo add https://your-git-repo \
  --username <user> \
  --password <token>

Platform Orchestration Best Practices

Based on experience and CNCF Guidelines:

1. Start Simple: Begin with the CNOE reference stack, extend gradually
2. Automate Everything: Manual platform changes are anti-pattern
3. Monitor Continuously: Use observability tools for platform health
4. Document Well: Platform documentation is essential for adoption
5. Version Everything: All platform components should be versioned
6. Test Changes: Platform updates should be tested in non-prod
7. Plan for Disaster: Backup and disaster recovery strategies are important
8. Use Stacks: Organize platform components as reusable stacks

Status

Maturity: Production (for CNOE Reference Implementation)

Stability: Stable

Support: Community Support via CNOE Community

Additional Resources

CNOE Resources

• CNOE Official Website
• CNOE GitHub Organization
• CNOE Reference Implementation
• CNOE Community Slack

Platform Engineering

• CNCF Platforms White Paper
• Platform Engineering Maturity Model
• Team Topologies - Organizational patterns

GitOps

• GitOps Working Group
• ArgoCD Best Practices
• GitOps Principles

CNOE Stacks

• Understanding CNOE Stacks
• Creating Custom Stacks

2.1 - Basic Concepts

Overview

Platform Orchestration refers to the automation and management of the platform infrastructure itself. This includes the provisioning, configuration, and lifecycle management of all components that make up the Internal Developer Platform (IDP).

In the context of IPCEI-CIS, Platform Orchestration means:

• Platform Bootstrap: Initial setup of Kubernetes clusters and core services
• Platform Services Management: Deployment and management of ArgoCD, Forgejo, Keycloak, etc.
• Infrastructure-as-Code: Declarative management using Terraform and GitOps
• Multi-Cluster Orchestration: Coordination across different Kubernetes clusters
• Platform Stacks: Reusable bundles of platform components (CNOE concept)

Target Audience

Platform Orchestration is primarily aimed at:

• Platform Engineering Teams: Teams that build and operate the IDP
• Infrastructure Architects: Those responsible for the platform architecture
• SRE Teams: Teams responsible for reliability and operations

Key Features

Declarative Platform Definition

The entire platform is defined declaratively as code:

• GitOps-First: Everything is versioned in Git and traceable
• Reproducibility: The platform can be rebuilt at any time
• Environment Parity: Consistency between Dev, Test, and Production
• Auditability: Complete history of all changes

Self-Bootstrapping

The platform can bootstrap itself:

1. Initial Bootstrap: Minimal tool (like idpbuilder) starts the platform
2. Self-Management: After bootstrap, ArgoCD takes over management
3. Continuous Reconciliation: Platform is continuously reconciled with Git state
4. Self-Healing: Automatic recovery on deviations

Stack-based Composition

Platform components are organized as reusable stacks (CNOE concept):

• Modularity: Components can be updated individually
• Reusability: Stacks can be used across different environments
• Composability: Compose complex platforms from simple building blocks
• Versioning: Stacks can be versioned and tested

In IPCEI-CIS: The stacks concept from CNOE is the core organizational principle for platform components.

Multi-Cluster Support

Platform Orchestration supports different cluster topologies:

• Control Plane + Worker Clusters: Centralized control, distributed workloads
• Hub-and-Spoke: One management cluster manages multiple target clusters
• Federation: Coordination across multiple independent clusters

Purpose in EDP

Platform Orchestration is the foundation of the IPCEI-CIS Edge Developer Platform. It enables:

Foundation for Developer Self-Service

Platform Orchestration ensures all services are available that developers need for self-service:

• GitOps Engine (ArgoCD) for continuous deployment
• Source Control (Forgejo) for code and configuration management
• Identity Management (Keycloak) for authentication and authorization
• Observability (Grafana, Prometheus) for monitoring and logging
• CI/CD (Forgejo Actions/Pipelines) for automated build and test

Consistency Across Environments

Through declarative definition, consistency is guaranteed:

• Development, test, and production environments are identically configured
• No “configuration drift” between environments
• Predictable behavior across all stages

Platform as Code

The platform itself is treated like software:

• Version Control: All changes are versioned in Git
• Code Review: Platform changes go through review processes
• Testing: Platform configurations can be tested
• Rollback: Easy rollback on problems

Reduced Operational Overhead

Automation reduces manual effort:

• No manual installation steps
• Automatic updates and patching
• Self-healing on failures
• Standardized deployment processes

Repository

CNOE Reference Implementation: cnoe-io/stacks

CNOE idpbuilder: cnoe-io/idpbuilder

Documentation: CNOE.io Documentation

Getting Started

Prerequisites

• Docker: For local Kubernetes clusters (Kind)
• kubectl: Kubernetes CLI tool
• Git: For repository management
• idpbuilder: CNOE bootstrap tool

Quick Start

Platform Orchestration with CNOE Reference Implementation:

# 1. Install idpbuilder
curl -fsSL https://cnoe.io/install.sh | bash

# 2. Bootstrap platform
idpbuilder create \
  --use-path-routing \
  --package-dir https://github.com/cnoe-io/stacks//ref-implementation

# 3. Wait for platform ready (ca. 10 minutes)
kubectl get applications -A

Verification

Verify the platform is running correctly:

# Get platform secrets (credentials)
idpbuilder get secrets

# Check all ArgoCD applications
kubectl get applications -n argocd

# Expected: All applications "Synced" and "Healthy"

Access URLs (with path-routing):

• ArgoCD: https://cnoe.localtest.me:8443/argocd
• Forgejo: https://cnoe.localtest.me:8443/gitea
• Keycloak: https://cnoe.localtest.me:8443/keycloak

Usage Examples

Use Case 1: Platform Bootstrap

Initial bootstrapping of a new platform instance:

idpbuilder create \
  --use-path-routing \
  --package-dir https://github.com/cnoe-io/stacks//ref-implementation \
  --log-level debug

# Workflow:
# 1. Creates Kind cluster
# 2. Installs ingress-nginx
# 3. Clones and installs ArgoCD
# 4. Installs Forgejo
# 5. Waits for core services
# 6. Creates technical users
# 7. Configures Git repositories
# 8. Installs remaining stacks via ArgoCD

After approximately 10 minutes, the platform is fully deployed.

Use Case 2: Adding New Platform Components

Add new platform components via ArgoCD:

# Create ArgoCD Application for new component
cat <<EOF | kubectl apply -f -
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: external-secrets
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://charts.external-secrets.io
    targetRevision: 0.9.9
    chart: external-secrets
  destination:
    server: https://kubernetes.default.svc
    namespace: external-secrets-system
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
EOF

Use Case 3: Platform Updates

Update platform components:

# 1. Update via Git (GitOps)
cd your-platform-config-repo
git pull

# 2. Update stack version
vim argocd/applications/component.yaml
# Change targetRevision to new version

# 3. Commit and push
git add .
git commit -m "Update component to v1.2.3"
git push

# 4. ArgoCD will automatically sync
# 5. Monitor the update
argocd app sync component --watch

Integration Points

ArgoCD Integration

• Bootstrap: ArgoCD is initially installed via idpbuilder
• Self-Management: After bootstrap, ArgoCD manages itself via Application CRD
• Platform Coordination: ArgoCD orchestrates all other platform components
• Health Monitoring: ArgoCD monitors health status of all platform services

Forgejo Integration

• Source of Truth: Git repositories contain all platform definitions
• GitOps Workflow: Changes in Git trigger platform updates
• Backup: Git serves as backup of platform configuration
• Audit Trail: Git history documents all platform changes
• CI/CD: Forgejo Actions can automate platform operations

Terraform Integration

• Infrastructure Provisioning: Terraform provisions cloud resources for platform
• State Management: Terraform state tracks infrastructure
• Integration: Terraform can be triggered via Forgejo pipelines
• Multi-Cloud: Support for multiple cloud providers

Architecture

Platform Orchestration Flow

Platform Bootstrap Sequence

The idpbuilder executes the following workflow:

1. Create Kind Kubernetes cluster
2. Install ingress-nginx controller
3. Install ArgoCD
4. Install Forgejo Git server
5. Wait for services to be ready
6. Create technical users in Forgejo
7. Create repository for platform state in Forgejo
8. Push platform stacks to Forgejo
9. Create ArgoCD Applications for all stacks
10. ArgoCD takes over continuous synchronization

Deployment Architecture

The platform is deployed in different namespaces:

• argocd: ArgoCD and its components
• gitea: Forgejo Git server
• keycloak: Identity and access management
• observability: Prometheus, Grafana, etc.
• ingress-nginx: Ingress controller

Configuration

idpbuilder Configuration

Key configuration options for idpbuilder:

# Path-based routing (recommended for local development)
idpbuilder create --use-path-routing

# Custom package directory
idpbuilder create --package-dir /path/to/custom/packages

# Custom Kind cluster config
idpbuilder create --kind-config custom-kind.yaml

# Enable debug logging
idpbuilder create --log-level debug

ArgoCD Configuration

Important ArgoCD configurations for platform orchestration:

# argocd-cm ConfigMap
data:
  # Enable automatic sync
  application.instanceLabelKey: argocd.argoproj.io/instance

  # Repository credentials
  repositories: |
    - url: https://github.com/cnoe-io/stacks
      name: cnoe-stacks
      type: git

  # Resource exclusions
  resource.exclusions: |
    - apiGroups:
      - cilium.io
      kinds:
      - CiliumIdentity

Platform Stack Configuration

Configuration of platform stacks via Kustomize:

# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: platform-system

resources:
  - argocd-app.yaml
  - forgejo-app.yaml
  - keycloak-app.yaml

patches:
  - target:
      kind: Application
    patch: |-
      - op: add
        path: /spec/syncPolicy
        value:
          automated:
            prune: true
            selfHeal: true

Troubleshooting

Platform not reachable

Problem: After idpbuilder create, platform services are not reachable

Solution:

# 1. Check if all pods are running
kubectl get pods -A

# 2. Check ArgoCD application status
kubectl get applications -n argocd

# 3. Check ingress
kubectl get ingress -A

# 4. Verify DNS resolution
nslookup cnoe.localtest.me

# 5. Check idpbuilder logs
idpbuilder get logs

ArgoCD Applications not synchronized

Problem: ArgoCD Applications show status “OutOfSync”

Solution:

# 1. Check application details
argocd app get <app-name>

# 2. View sync status
argocd app sync <app-name> --dry-run

# 3. Force sync
argocd app sync <app-name> --force

# 4. Check for errors in ArgoCD logs
kubectl logs -n argocd deployment/argocd-application-controller

Git Repository Connection Issues

Problem: ArgoCD cannot access Git repository

Solution:

# 1. Verify repository configuration
argocd repo list

# 2. Test connection
argocd repo get https://your-git-repo

# 3. Check credentials
kubectl get secret -n argocd

# 4. Re-add repository with correct credentials
argocd repo add https://your-git-repo \
  --username <user> \
  --password <token>

Platform Orchestration Best Practices

Based on experience and CNCF Guidelines:

1. Start Simple: Begin with the CNOE reference stack, extend gradually
2. Automate Everything: Manual platform changes are anti-pattern
3. Monitor Continuously: Use observability tools for platform health
4. Document Well: Platform documentation is essential for adoption
5. Version Everything: All platform components should be versioned
6. Test Changes: Platform updates should be tested in non-prod
7. Plan for Disaster: Backup and disaster recovery strategies are important
8. Use Stacks: Organize platform components as reusable stacks

Status

Maturity: Production (for CNOE Reference Implementation)

Stability: Stable

Support: Community Support via CNOE Community

Additional Resources

CNOE Resources

• CNOE Official Website
• CNOE GitHub Organization
• CNOE Reference Implementation
• CNOE Community Slack

Platform Engineering

• CNCF Platforms White Paper
• Platform Engineering Maturity Model
• Team Topologies - Organizational patterns

GitOps

• GitOps Working Group
• ArgoCD Best Practices
• GitOps Principles

CNOE Stacks

• Understanding CNOE Stacks
• Creating Custom Stacks

2.1.1 - Platform Orchestration

Overview

Orchestration in the context of Platform Engineering refers to the coordinated automation and management of infrastructure, platform, and application components throughout their entire lifecycle. It is a fundamental concept that bridges the gap between declarative specifications (what should be deployed) and actual execution (how it is deployed).

The Role of Orchestration in Platform Engineering

Platform Engineering has emerged as a discipline to improve developer experience and reduce cognitive load on development teams (CNCF Platforms White Paper). Orchestration is the central mechanism that enables this vision:

1. Automation of Complex Workflows: Orchestration coordinates multiple steps and dependencies automatically
2. Consistency and Reproducibility: Guaranteed, repeatable deployments across different environments
3. Self-Service Capabilities: Developers can independently orchestrate resources and deployments
4. Governance and Compliance: Centralized control over policies and best practices

What Does Orchestration Do?

Orchestration systems perform the following tasks:

• Workflow Coordination: Coordination of complex, multi-step deployment processes
• Dependency Management: Resolution and management of dependencies between components
• State Management: Continuous monitoring and reconciliation between desired and actual state
• Resource Provisioning: Automatic provisioning of infrastructure and services
• Configuration Management: Management of configurations across different environments
• Health Monitoring: Monitoring the health of deployed resources

Three Layers of Orchestration

In modern Platform Engineering, we distinguish three fundamental layers of orchestration:

Infrastructure Orchestration

Infrastructure Orchestration deals with the lowest level - the physical and virtual infrastructure layer. This includes:

• Provisioning of compute, network, and storage resources
• Cloud resource management (VMs, networking, storage)
• Infrastructure-as-Code deployment (Terraform, etc.)
• Bare metal and hypervisor management

Target Audience: Infrastructure Engineers, Cloud Architects

Note: Detailed documentation for Infrastructure Orchestration is maintained separately.

More details: Infrastructure Orchestration →

Platform Orchestration

Platform Orchestration focuses on deploying and managing the platform itself - the services and tools that development teams use. This includes:

• Installation and configuration of Kubernetes clusters
• Deployment of platform services (GitOps tools, Observability, Security)
• Management of platform components via Stacks
• Multi-cluster orchestration

Target Audience: Platform Engineering Teams, SRE Teams

In IPCEI-CIS: Platform orchestration is realized using the CNOE stack concept with ArgoCD and Forgejo.

More details: Platform Orchestration →

Application Orchestration

Application Orchestration concentrates on the deployment and lifecycle management of applications running on the platform. This includes:

• Deployment of microservices and containerized applications
• CI/CD pipeline orchestration
• Configuration management and secrets handling
• Application health monitoring and auto-scaling

Target Audience: Application Developers, DevOps Engineers

In IPCEI-CIS: Application orchestration uses Forgejo pipelines for CI/CD and ArgoCD for GitOps-based deployment.

More details: Application Orchestration →

GitOps as Orchestration Paradigm

A central approach in modern platform orchestration solutions is GitOps. GitOps uses Git repositories as the single source of truth for declarative infrastructure and applications:

• Declarative Approach: The desired state is defined in Git
• Automatic Synchronization: Controllers monitor Git and reconcile the live state
• Audit Trail: All changes are traceable in Git history
• Rollback Capability: Easy rollback through Git revert

Continuous Reconciliation

An important concept is continuous reconciliation:

1. The orchestrator monitors both the source (Git) and the target (e.g., Kubernetes cluster)
2. Deviations trigger automatic corrective actions
3. Health checks validate that the desired state has been achieved
4. Drift detection warns of unexpected changes

Orchestration Tools in IPCEI-CIS

Within the IPCEI-CIS platform, we utilize the CNOE (Cloud Native Operational Excellence) stack concept with the following orchestration components:

ArgoCD

• Continuous Delivery for Kubernetes based on GitOps
• Synchronizes Kubernetes manifests from Git repositories
• Supports Helm Charts, Kustomize, Jsonnet, and plain YAML
• Multi-cluster deployment capabilities
• Application Sets for parameterized deployments

Role in IPCEI-CIS: ArgoCD is the central component for GitOps-based deployment management. After the initial bootstrapping phase, ArgoCD takes over the technical coordination of all components.

Forgejo

• Git Repository Management and source control
• CI/CD Pipelines via Forgejo Actions (GitHub Actions compatible)
• Developer Portal Capabilities (initially planned, project discontinued)
• Package registry and artifact management
• Integration with ArgoCD for GitOps workflows

Role in IPCEI-CIS: Forgejo serves as the Git repository host and CI/CD engine. It was initially planned as a developer portal (similar to Backstage’s role in other stacks) but this aspect was not fully realized before project completion.

Note on Backstage: In typical CNOE implementations, Backstage serves as the developer portal providing golden paths through software templates. IPCEI-CIS initially planned to use Forgejo for this purpose but the project concluded before full implementation.

Terraform

• Infrastructure-as-Code provisioning
• Multi-cloud resource management
• State management for infrastructure
• Integration with Forgejo pipelines for automated deployment

Role in IPCEI-CIS: Terraform handles infrastructure provisioning at the infrastructure orchestration layer, integrated into automated workflows via Forgejo pipelines.

CNOE Stacks Concept

• Modular Platform Components bundled as stacks
• Reusable, composable platform building blocks
• Version-controlled stack definitions
• GitOps-based stack deployment via ArgoCD

Role in IPCEI-CIS: The stacks concept from CNOE provides the structural foundation for platform orchestration, enabling modular deployment and management of platform components.

The Orchestration Workflow

A typical orchestration workflow in the IPCEI-CIS platform:

Workflow Steps:

1. Definition: Developer defines application/infrastructure as code
2. Commit: Changes are committed to Forgejo Git repository
3. CI Pipeline: Forgejo Actions build, test, and package the application
4. Sync: ArgoCD detects changes and triggers deployment
5. Provision: Terraform orchestrates required cloud resources (if needed)
6. Deploy: Application is deployed to Kubernetes
7. Monitor: Continuous monitoring and health checks
8. Reconcile: Automatic correction on drift detection

Benefits of Coordinated Orchestration

The integration of infrastructure, platform, and application orchestration provides crucial advantages:

• Reduced Complexity: Developers don’t need to know all infrastructure details
• Faster Time-to-Market: Automated workflows accelerate deployments
• Consistency: Standardized patterns across all teams
• Governance: Central policies are automatically enforced
• Scalability: Platform teams can support many application teams
• Self-Service: Developers can provision services independently
• Audit and Compliance: Complete traceability through Git history

Best Practices

Successful orchestration follows proven principles (Platform Engineering Principles):

1. Platform as a Product: Treat the platform as a product with focus on user experience
2. Self-Service First: Enable developers to use services autonomously
3. Documentation: Comprehensive documentation of golden paths
4. Feedback Loops: Continuous improvement through user feedback
5. Thin Platform Layer: Use managed services where possible instead of building everything
6. Progressive Disclosure: Offer different abstraction levels
7. Focus on Common Problems: Solve recurring problems centrally
8. Treat Glue as Valuable: Integration of different tools is valuable
9. Clear Mission: Define clear goals and responsibilities

Avoiding Anti-Patterns

Common mistakes in platform orchestration (How to fail at Platform Engineering):

• Product Misfit: Building platform without involving developers
• Overly Complex Design: Too many features and unnecessary complexity
• Swiss Knife Syndrome: Trying to solve all problems with one tool
• Insufficient Documentation: Missing or outdated documentation
• Siloed Development: Platform and development teams working in isolation
• Stagnant Platform: Platform not continuously evolved

Sub-Components

The orchestration component includes the following sub-areas:

• Infrastructure Orchestration: Low-level infrastructure deployment and provisioning
• Platform Orchestration: Platform-level component deployment via Stacks
• Application Orchestration: Application-level deployment and CI/CD
• Stacks: Reusable component bundles and compositions

Further Resources

Fundamentals

• CNCF Platforms White Paper - Comprehensive paper on Platform Engineering
• Platform Engineering Definition - What is Platform Engineering?
• Team Topologies - Organizational structures for modern teams

GitOps

• GitOps Principles - Official GitOps principles
• ArgoCD Documentation - ArgoCD documentation

Tools

• CNOE.io - Cloud Native Operational Excellence Framework
• Forgejo - Self-hosted Git service with CI/CD
• Terraform - Infrastructure as Code tool

2.1.2 - Application Orchestration

Overview

Application Orchestration deals with the automation of application deployment and lifecycle management. It encompasses the entire workflow from source code to running application in production.

In the context of IPCEI-CIS, Application Orchestration includes:

• CI/CD Pipelines: Automated build, test, and deployment pipelines
• GitOps Deployment: Declarative application deployment via ArgoCD
• Progressive Delivery: Canary deployments, blue-green deployments
• Application Configuration: Environment-specific configuration management
• Golden Paths: Standardized deployment templates and workflows

Target Audience

Application Orchestration is primarily for:

• Application Developers: Teams developing and deploying applications
• DevOps Teams: Teams responsible for deployment automation
• Product Teams: Teams responsible for application lifecycle

Key Features

Automated CI/CD Pipelines

Forgejo Actions provides GitHub Actions-compatible CI/CD:

• Build Automation: Automatic building of container images
• Test Automation: Automated unit, integration, and E2E tests
• Security Scanning: Vulnerability scanning of dependencies and images
• Artifact Publishing: Publishing to container registries
• Deployment Triggering: Automatic deployment after successful build

GitOps-based Deployment

ArgoCD enables declarative application deployment:

• Declarative Configuration: Applications defined as Kubernetes manifests
• Automated Sync: Automatic synchronization between Git and cluster
• Rollback Capability: Easy rollback to previous versions
• Multi-Environment: Consistent deployment across Dev/Test/Prod
• Health Monitoring: Continuous monitoring of application health

Progressive Delivery

Support for advanced deployment strategies:

• Canary Deployments: Gradual rollout to subset of users
• Blue-Green Deployments: Zero-downtime deployments with instant rollback
• A/B Testing: Traffic splitting for feature testing
• Feature Flags: Dynamic feature enablement without deployment

Configuration Management

Flexible configuration for different environments:

• Environment Variables: Configuration via environment variables
• ConfigMaps: Kubernetes-native configuration
• Secrets Management: Secure handling of sensitive data
• External Secrets: Integration with external secret stores (Vault, etc.)

Purpose in EDP

Application Orchestration is the core of developer experience in IPCEI-CIS Edge Developer Platform.

Developer Self-Service

Developers can deploy applications independently:

• Self-Service Deployment: No dependency on operations team
• Standardized Workflows: Clear, documented deployment processes
• Fast Feedback: Quick feedback through automated pipelines
• Environment Parity: Consistent behavior across all environments

Quality and Security

Automated checks ensure quality and security:

• Automated Testing: All changes are automatically tested
• Security Scans: Vulnerability scanning of dependencies and images
• Policy Enforcement: Automated policy checks (OPA, Kyverno)
• Compliance: Auditability of all deployments

Efficiency and Productivity

Automation increases team efficiency:

• Faster Time-to-Market: Faster deployment of new features
• Reduced Manual Work: Automation of repetitive tasks
• Fewer Errors: Fewer manual mistakes through automation
• Better Collaboration: Clear interfaces between Dev and Ops

Repository

Forgejo: forgejo.org

Forgejo Actions: Forgejo Actions Documentation

ArgoCD: argoproj.github.io/cd

Getting Started

Prerequisites

• Forgejo Account: Access to Forgejo instance
• Kubernetes Cluster: Target cluster for deployments
• ArgoCD Access: Access to ArgoCD instance
• Git: For repository management

Quick Start: Application Deployment

1. Create Application Repository

# Create new repository in Forgejo
git init my-application
cd my-application

# Add application code and Dockerfile
cat > Dockerfile <<EOF
FROM node:18-alpine
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production
COPY . .
EXPOSE 3000
CMD ["node", "server.js"]
EOF

2. Add CI/CD Pipeline

Create .forgejo/workflows/build.yaml:

name: Build and Push

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2
      
      - name: Login to Registry
        uses: docker/login-action@v2
        with:
          registry: registry.example.com
          username: ${{ secrets.REGISTRY_USER }}
          password: ${{ secrets.REGISTRY_PASSWORD }}
      
      - name: Build and push
        uses: docker/build-push-action@v4
        with:
          context: .
          push: ${{ github.event_name == 'push' }}
          tags: registry.example.com/my-app:${{ github.sha }}

3. Create Kubernetes Manifests

Create k8s/deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-application
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-application
  template:
    metadata:
      labels:
        app: my-application
    spec:
      containers:
      - name: app
        image: registry.example.com/my-app:latest
        ports:
        - containerPort: 3000
        env:
        - name: NODE_ENV
          value: "production"
---
apiVersion: v1
kind: Service
metadata:
  name: my-application
spec:
  selector:
    app: my-application
  ports:
  - port: 80
    targetPort: 3000

4. Configure ArgoCD Application

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-application
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://forgejo.example.com/myteam/my-application
    targetRevision: main
    path: k8s
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

5. Deploy

# Commit and push
git add .
git commit -m "Add application and deployment configuration"
git push origin main

# ArgoCD will automatically deploy the application
argocd app sync my-application --watch

Usage Examples

Use Case 1: Multi-Environment Deployment

Deploy application to multiple environments:

Repository Structure:

my-application/
├── .forgejo/
│   └── workflows/
│       └── build.yaml
├── base/
│   ├── deployment.yaml
│   ├── service.yaml
│   └── kustomization.yaml
├── overlays/
│   ├── dev/
│   │   ├── kustomization.yaml
│   │   └── patches.yaml
│   ├── staging/
│   │   ├── kustomization.yaml
│   │   └── patches.yaml
│   └── production/
│       ├── kustomization.yaml
│       └── patches.yaml

Kustomize Base (base/kustomization.yaml):

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - deployment.yaml
  - service.yaml

commonLabels:
  app: my-application

Environment Overlay (overlays/production/kustomization.yaml):

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

bases:
  - ../../base

namespace: production

replicas:
  - name: my-application
    count: 5

images:
  - name: my-app
    newTag: v1.2.3

patches:
  - patches.yaml

ArgoCD Applications for each environment:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-application-prod
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://forgejo.example.com/myteam/my-application
    targetRevision: main
    path: overlays/production
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Use Case 2: Canary Deployment

Progressive rollout with canary strategy:

Argo Rollouts Canary:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: my-application
spec:
  replicas: 10
  strategy:
    canary:
      steps:
      - setWeight: 10
      - pause: {duration: 5m}
      - setWeight: 30
      - pause: {duration: 5m}
      - setWeight: 60
      - pause: {duration: 5m}
      - setWeight: 100
  selector:
    matchLabels:
      app: my-application
  template:
    metadata:
      labels:
        app: my-application
    spec:
      containers:
      - name: app
        image: registry.example.com/my-app:v2.0.0

Use Case 3: Feature Flags

Dynamic feature control without deployment:

Application Code with Feature Flag:

const Unleash = require('unleash-client');

const unleash = new Unleash({
  url: 'http://unleash.platform/api/',
  appName: 'my-application',
  customHeaders: {
    Authorization: process.env.UNLEASH_API_TOKEN
  }
});

// Use feature flag
if (unleash.isEnabled('new-checkout-flow')) {
  // New checkout implementation
  renderNewCheckout();
} else {
  // Old checkout implementation
  renderOldCheckout();
}

Integration Points

Forgejo Integration

Forgejo serves as central source code management and CI/CD platform:

• Source Control: Git repositories for application code
• CI/CD Pipelines: Forgejo Actions for automated builds and tests
• Container Registry: Built-in container registry for images
• Webhook Integration: Triggers for external systems
• Pull Request Workflows: Code review and approval processes

ArgoCD Integration

ArgoCD handles declarative application deployment:

• GitOps Sync: Continuous synchronization with Git state
• Health Monitoring: Application health status monitoring
• Rollback Support: Easy rollback to previous versions
• Multi-Cluster: Deployment to multiple clusters
• UI and CLI: Web interface and command-line access

Observability Integration

Integration with monitoring and logging:

• Metrics: Prometheus metrics from applications
• Logs: Centralized log collection via Loki/ELK
• Tracing: Distributed tracing with Jaeger/Tempo
• Alerting: Alert rules for application issues

Architecture

Application Deployment Flow

CI/CD Pipeline Architecture

Typical Forgejo Actions pipeline stages:

1. Checkout: Clone source code
2. Build: Compile application and dependencies
3. Test: Run unit and integration tests
4. Security Scan: Scan dependencies and code for vulnerabilities
5. Build Image: Create container image
6. Push Image: Push to container registry
7. Update Manifests: Update Kubernetes manifests with new image tag
8. Notify: Send notifications on success/failure

Configuration

Forgejo Actions Configuration

Example for Node.js application:

name: CI/CD Pipeline

on:
  push:
    branches: [ main, develop ]
  pull_request:
    branches: [ main ]

env:
  REGISTRY: registry.example.com
  IMAGE_NAME: ${{ github.repository }}

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: '18'
          cache: 'npm'
      
      - name: Install dependencies
        run: npm ci
      
      - name: Run tests
        run: npm test
      
      - name: Run linter
        run: npm run lint

  security:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Run Trivy vulnerability scanner
        uses: aquasecurity/trivy-action@master
        with:
          scan-type: 'fs'
          scan-ref: '.'
          format: 'sarif'
          output: 'trivy-results.sarif'

  build-and-push:
    needs: [test, security]
    runs-on: ubuntu-latest
    if: github.event_name == 'push'
    steps:
      - uses: actions/checkout@v3
      
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2
      
      - name: Login to Registry
        uses: docker/login-action@v2
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ secrets.REGISTRY_USER }}
          password: ${{ secrets.REGISTRY_PASSWORD }}
      
      - name: Extract metadata
        id: meta
        uses: docker/metadata-action@v4
        with:
          images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
          tags: |
            type=ref,event=branch
            type=sha,prefix={{branch}}-
      
      - name: Build and push
        uses: docker/build-push-action@v4
        with:
          context: .
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

ArgoCD Application Configuration

Complete configuration example:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-application
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  
  source:
    repoURL: https://forgejo.example.com/myteam/my-application
    targetRevision: main
    path: k8s/overlays/production
    
    # Kustomize options
    kustomize:
      version: v5.0.0
      images:
        - my-app=registry.example.com/my-app:v1.2.3
  
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  
  # Sync policy
  syncPolicy:
    automated:
      prune: true        # Delete resources not in Git
      selfHeal: true     # Override manual changes
      allowEmpty: false  # Don't delete everything on empty repo
    
    syncOptions:
      - CreateNamespace=true
      - PruneLast=true
      - RespectIgnoreDifferences=true
    
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m
  
  # Ignore differences (avoid sync loops)
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas  # Ignore if HPA manages replicas

Troubleshooting

Pipeline Fails

Problem: Forgejo Actions pipeline fails

Solution:

# 1. Check pipeline logs in Forgejo UI
# Navigate to: Repository → Actions → Select failed run

# 2. Check runner status
# In Forgejo: Site Admin → Actions → Runners

# 3. Check runner logs
kubectl logs -n forgejo-runner deployment/act-runner

# 4. Test pipeline locally with act
act -l  # List available jobs
act -j build  # Run specific job

ArgoCD Application OutOfSync

Problem: Application shows “OutOfSync” status

Solution:

# 1. Check differences
argocd app diff my-application

# 2. View sync status details
argocd app get my-application

# 3. Manual sync
argocd app sync my-application

# 4. Hard refresh (ignore cache)
argocd app sync my-application --force

# 5. Check for ignored differences
argocd app get my-application --show-operation

Application Deployment Fails

Problem: Application pod crashes after deployment

Solution:

# 1. Check pod status
kubectl get pods -n production

# 2. View pod logs
kubectl logs -n production deployment/my-application

# 3. Describe pod for events
kubectl describe pod -n production <pod-name>

# 4. Check resource limits
kubectl top pod -n production

# 5. Rollback via ArgoCD
argocd app rollback my-application

Image Pull Errors

Problem: Kubernetes cannot pull container image

Solution:

# 1. Verify image exists
docker pull registry.example.com/my-app:v1.2.3

# 2. Check image pull secret
kubectl get secret -n production regcred

# 3. Create image pull secret if missing
kubectl create secret docker-registry regcred \
  --docker-server=registry.example.com \
  --docker-username=user \
  --docker-password=password \
  -n production

# 4. Reference secret in deployment
kubectl patch deployment my-application -n production \
  -p '{"spec":{"template":{"spec":{"imagePullSecrets":[{"name":"regcred"}]}}}}'

Best Practices

Golden Path Templates

Provide standardized templates for common use cases:

1. Web Application Template: Node.js, Python, Go web services
2. API Service Template: RESTful API with OpenAPI
3. Batch Job Template: Kubernetes CronJob configurations
4. Microservice Template: Service mesh integration

Example repository template structure:

application-template/
├── .forgejo/
│   └── workflows/
│       ├── build.yaml
│       ├── test.yaml
│       └── deploy.yaml
├── k8s/
│   ├── base/
│   └── overlays/
├── src/
│   └── ...
├── Dockerfile
├── README.md
└── .gitignore

Deployment Checklist

Before deploying to production:

• ✅ All tests passing
• ✅ Security scans completed
• ✅ Resource limits defined
• ✅ Health checks configured
• ✅ Monitoring and alerts set up
• ✅ Backup strategy defined
• ✅ Rollback plan documented
• ✅ Team notified about deployment

Configuration Management

• Use ConfigMaps for non-sensitive configuration
• Use Secrets for sensitive data
• Use External Secrets Operator for vault integration
• Never commit secrets to Git
• Use environment-specific overlays (Kustomize)
• Document all configuration options

Status

Maturity: Production

Stability: Stable

Support: Internal Platform Team

Additional Resources

Forgejo

• Forgejo Documentation
• Forgejo Actions Guide
• Forgejo API Reference

ArgoCD

• ArgoCD Documentation
• ArgoCD Best Practices
• ArgoCD Sync Waves

GitOps

• GitOps Principles
• GitOps Patterns
• Kubernetes Deployment Strategies

CI/CD

• GitHub Actions Documentation (Forgejo Actions compatible)
• Docker Best Practices
• Container Security Best Practices

2.2 - Infrastructure as Code

Overview

Infrastructure as Code (IaC) is the practice of managing and provisioning infrastructure through code rather than manual processes. Instead of clicking through web consoles or running one-off commands, infrastructure is defined in version-controlled files that can be executed repeatedly to produce identical environments.

This approach treats infrastructure with the same rigor as application code: it’s versioned, reviewed, tested, and deployed through automated pipelines.

Why Infrastructure as Code?

The problem with manual infrastructure

Traditional infrastructure management faces several challenges:

• Inconsistency: Manual steps vary between operators and environments
• Undocumented: Critical knowledge exists only in operators’ heads
• Error-Prone: Human mistakes during repetitive tasks
• Slow: Manual provisioning takes hours or days
• Untrackable: No audit trail of what changed, when, or why
• Irreproducible: Difficulty recreating environments exactly

The IaC solution

Infrastructure as Code addresses these challenges by making infrastructure:

Declarative - Describe the desired state, not the steps to achieve it. The IaC tool handles the implementation details.

Versioned - Every infrastructure change is committed to Git, providing complete history and the ability to rollback.

Automated - Infrastructure deploys through pipelines without human intervention, eliminating manual errors.

Testable - Infrastructure changes can be validated before production deployment.

Documented - The code itself is the documentation, always current and accurate.

Reproducible - The same code produces identical infrastructure every time, across all environments.

Core Concepts

Declarative vs imperative

Imperative approaches specify the exact steps: “Create a server, then install software, then configure networking.”

Declarative approaches specify the desired outcome: “I need a server with this software and network configuration.” The IaC tool determines the necessary steps.

Most modern IaC tools use the declarative approach, making them more maintainable and resilient.

State Management

IaC tools maintain a “state” - a record of what infrastructure currently exists. When you change your code and re-run the tool, it compares the desired state (your code) with the actual state (what exists) and makes only the necessary changes.

This enables:

• Drift detection - Identify manual changes made outside IaC
• Safe updates - Modify only what changed
• Dependency management - Update resources in the correct order

Idempotency

Running the same IaC code multiple times produces the same result. If infrastructure already matches the code, the tool makes no changes. This property is called idempotency and is essential for reliable automation.

Infrastructure as Code in EDP

The Edge Developer Platform uses IaC extensively:

Terraform and Terragrunt

Terraform is our primary IaC tool for provisioning cloud resources. We use Terragrunt as an orchestration layer to manage multiple Terraform modules and reduce code duplication.

Our implementation includes:

• infra-catalogue - Reusable infrastructure components (modules, units, and stacks)
• infra-deploy - Full environment definitions using catalogue components

Platform stacks

We organize infrastructure into stacks - coherent bundles of related components:

• Core Stack - Essential platform services
• Forgejo Stack - Source control and CI/CD
• Observability Stack - Monitoring and logging
• OTC Stack - Cloud provider resources
• Coder Stack - Development environments
• Terralist Stack - Terraform registry

Each stack is defined as code, versioned independently, and can be deployed across different environments.

GitOps integration

Our IaC integrates with GitOps principles:

1. All infrastructure definitions live in Git repositories
2. Changes go through code review processes
3. Automated pipelines deploy infrastructure
4. ArgoCD continuously reconciles Kubernetes resources with Git state

This creates an auditable, automated, and reliable deployment process.

Benefits realized

Consistency across environments

Development, testing, and production environments are deployed from the same code. This eliminates the “works on my machine” problem at the infrastructure level.

Rapid environment provisioning

A complete EDP environment can be provisioned in minutes rather than days. This enables:

• Quick disaster recovery
• Easy creation of test environments
• Fast onboarding for new team members

Reduced operational risk

Code review catches infrastructure errors before deployment. Automated testing validates changes. Version control enables instant rollback if problems occur.

Knowledge sharing

Infrastructure configuration is explicit and discoverable in code. New team members can understand the platform by reading the repository rather than shadowing experienced operators.

Compliance and auditability

Every infrastructure change is tracked in Git history with author, timestamp, and reason. This provides audit trails required for compliance and simplifies troubleshooting.

Getting started

To work with EDP’s Infrastructure as Code:

1. Understand Terraform basics - Review Terraform documentation
2. Explore infra-catalogue - Browse infra-catalogue to understand available components
3. Review existing deployments - Examine infra-deploy to see how components are composed
4. Follow the Terraform guide - See Terraform-based deployment for detailed instructions

Best Practices

Based on our experience building and operating IaC:

Version everything - All infrastructure code belongs in version control. No exceptions.

Keep it simple - Start with basic modules. Add abstraction only when duplication becomes painful.

Test before production - Deploy infrastructure changes to test environments first.

Use meaningful commit messages - Explain why changes were made, not just what changed.

Review all changes - Infrastructure changes should go through the same review process as application code.

Document assumptions - Use code comments to explain non-obvious decisions.

Manage secrets securely - Never commit credentials to version control. Use secret management tools.

Plan for drift - Regularly compare actual infrastructure with code state to detect manual changes.

Challenges and limitations

Infrastructure as Code is powerful but has challenges:

Learning curve - Teams need to learn IaC tools and practices. Initial productivity may decrease.

State management complexity - State files must be stored securely and accessed by multiple team members. State corruption can cause serious issues.

Provider limitations - Not all infrastructure can be managed as code. Some resources require manual configuration.

Breaking changes - Poorly written code can destroy infrastructure. Safeguards and testing are essential.

Tool lock-in - Switching IaC tools (e.g., Terraform to Pulumi) requires rewriting infrastructure code.

Despite these challenges, the benefits far outweigh the costs for any infrastructure of meaningful complexity.

Why we invest in IaC

The IPCEI-CIS Edge Developer Platform requires reliable, reproducible infrastructure. Manual provisioning cannot meet these requirements at scale.

By investing in Infrastructure as Code:

• We can deploy complete environments consistently
• Platform engineers can focus on improvement rather than repetitive tasks
• Infrastructure changes are transparent and auditable
• New team members can contribute confidently
• Disaster recovery becomes routine rather than heroic

Our IaC tools (infra-catalogue and infra-deploy) embody these principles and enable the platform’s reliability.

Additional Resources

Terraform Ecosystem

• Terraform Documentation
• OpenTofu - Community-driven Terraform fork
• Terragrunt - Terraform orchestration

Infrastructure as Code Concepts

• Infrastructure as Code book by Kief Morris
• Terraform Best Practices
• CNCF Platforms White Paper

EDP-Specific Resources

• Terraform-based deployment - Detailed deployment guide
• Infrastructure Stacks - Reusable component bundles
• Platform Orchestration - How IaC fits into overall deployment

2.2.1 - Terraform-based deployment of EDP

Overview

The infra-deploy and infra-catalogue repositories work together to provide a framework for deploying Edge Developer Platform instances.

infra-catalogue contains individual, atomic infrastructure components: terraform modules and terragrunt units and stacks, such as Kubernetes clusters and Postgres databases.

infra-deploy then contains full definitions of stacks built using these components - such as the production site at edp.buildth.ing. It also includes scripts with which to deploy these stacks.

Note that both repositories rely on the wide range of features available on OTC. Several of these features, such as S3-compatible storage and on-demand managed Postgres instances, are not yet available on more sovereign clouds such as Edge, so these are not currently supported.

Key Features

• ‘Catalogue’ of infrastructure stacks to be used in deployments
• Definition of deployment stacks for each environment in prod or dev
• Scripts to govern deployment, installation and drift-correction of EDP

Purpose in EDP

For our Edge Developer Platform to be reliable it must be deployable in a consistent manner. When errors occur, or after any manual alterations, the system can then be safely reset to a working state. This state should be provided in code to allow for automated validation and deployment, and to allow it to be deployed from an always-identical CI/CD pipeline rather than a variable local deployment environment.

Repositories

Infra-deploy: https://edp.buildth.ing/DevFW/infra-deploy

Infra-catalogue: https://edp.buildth.ing/DevFW/infra-catalogue

Getting Started

Prerequisites

• Docker
• Kubernetes management
• Access to OTC
• HashiCorp Terraform or its open-source equivalent, OpenTofu
• Terragrunt, an orchestrator for Terraform stacks

Quick Start

1. Set up OTC credentials per README section
2. Set cluster environment and run install script per README section

Alternatively, manually trigger automated deployment pipeline.

• You will be asked for essential information like the deployment name and tenant.
• Any fields marked INITIAL only need to be set when first creating an environment
• Thereafter, the cached values are used and the INITIAL values provided to the pipeline are ignored.
  • Specifically, they are cached in a terragrunt.values.hcl file within infra-deploy/<tenant>/<cluster-name>, where both variables are set in the pipeline
  • e.g. prod/edp or nonprod/garm-provider-test

Verification

After the deploymenet completes, and a short startup time, you should be able to access your Forgejo instance at <cluster-name>.buildth.ing (production tenant) or <cluster-name>.t09.de (non-prod tenant). <cluster-name> is the name you provided in the deployment pipeline, or the $CLUSTER_ENVIRONMENT variable when running manually.

For example, the primary production cluster is called edp and can be accessed at edp.buildth.ing.

Screens

Deployment using production pipeline:

…

Configuration

Configuration of clusters is done in two ways. The first, mentioned above, is to provide INITIAL configuration when creating a new cluster. Thereafter, configuration is done within the relevant infra-deploy/<tenant> directory (e.g. prod/edp). Variables may be changed within the terragrunt.values.hcl file, but equally the terragrunt.stack.hcl file contains references to the lower-level code set up in infra-catalogue.

These are organised in layers, according to Terragrunt’s natural structure. First is a stack, a high-level abstraction for a whole cluster. This in turn references terragrunt units, which in turn are wrappers around standard Terraform modules. When deployed, the Terraform modules require a provider.tf file which is automatically generated by Terragrunt using tenant-level and global configuration stored in infra-deploy.

When deploying manually (e.g. with install.sh), you can observe these layers as Terragrunt will cache them on your machine, within the .terragrunt-stack/ directory generated within /<tenant>/<cluster-name>/.

Troubleshooting

Version updates

Problem: Updates to infra-catalogue are not immediately reflected in deployed clusters, even after running deploy.

Solution: Versions must be updated. Each cluster deployment specifies a catalogue version in its terragrunt.values.hcl; this refers to a tag in infra-catalogue. Within infra-catalogue, stacks reference units and modules from the same tag.

Thus, to test a new change to infra-catalogue, first make a new tag, then update the relevant values file to point to it.

Status

Maturity: TRL-9

Additional Resources

• Terraform
• OpenTofu, the community-driven replacement for Terraform
• Terragrunt

2.2.2 - Stacks

Overview

The stacks and stacks-instances repositories form the core of a GitOps-based system for provisioning Edge Developer Platforms (EDP). They implement a template-instance pattern that enables the deployment of reusable platform components across different environments. The concept of “stacks” originates from the CNOE.io project (Cloud Native Operational Excellence), which can be traced through the evolutionary development from edpbuilder (derived from CNOE.io’s EDPbuilder) to infra-deploy.

Key Features of the Everything-as-Code Stacks Approach

This declarative Stacks provisioning architecture is characterized by the following central properties:

Complete Code Declaration

Platform as Code: All Kubernetes resources, Helm charts, and application manifests are declaratively versioned as YAML files. The entire platform topology is traceable in Git.

Configuration as Code: Environment-specific configurations are generated through template hydration, not manually edited. Gomplate transforms generic templates into concrete configurations.

GitOps-Native Architecture

Single Source of Truth: Git is the sole source of truth for the desired state of all infrastructure and platform components.

Declarative State Management: ArgoCD continuously synchronizes the actual state with the desired state defined in Git. Deviations are automatically corrected.

Audit Trail: Every change to infrastructure or platform is documented through Git commits, with author, timestamp, and change description.

Pull-based Deployment: ArgoCD pulls changes from Git, rather than external systems requiring push access to the cluster. This significantly increases security.

Template-Instance Separation

DRY Principle (Don’t Repeat Yourself): Common platform components are defined once as templates and reused for all environments.

Environment Promotion: New environments can be quickly created through template hydration. Consistency across environments is guaranteed.

Centralized Maintainability: Updates to stack definitions can be made centrally in the stacks repository and then selectively rolled out to instances.

Customization Points: Despite reuse, environment-specific customizations remain possible through values files and manifest overlays.

Modular Composition

Stack-based Architecture: Platform capabilities are organized into independent, reusable stacks (core, otc, forgejo, observability).

Selective Deployment: Through the STACKS environment variable, only required components can be deployed selectively.

Mix-and-Match: Different stack combinations yield different platform profiles (Development, Production, Observability clusters).

Pluggable Components: New stacks can be added without modifying existing ones.

Environment Agnosticism

Cloud Provider Abstraction: Templates are formulated generically. Provider-specific details are introduced through hydration.

Multi-Cloud Ready: The architecture supports various cloud providers (currently OTC, historically KIND, extensible to AWS/Azure/GCP).

Environment Variables as Interface: All environment-specific aspects are controlled through clearly defined environment variables.

Portable Definitions: Stack definitions can be ported between environments and even cloud providers.

Self-Healing and Drift Detection

Automated Reconciliation: ArgoCD detects deviations from the desired state and corrects them automatically.

Continuous Monitoring: Permanent monitoring of cluster state compared to Git definition.

Declarative State Recovery: After failures or manual changes, the declared state is automatically restored.

Sync Policies: Configurable sync strategies (automated, manual, with pruning) per application.

Secrets Management

Secrets Outside Git: Sensitive data is not stored in Git but generated at runtime or injected from secret stores.

Generated Credentials: Passwords, tokens, and secrets are generated during deployment and directly created as Kubernetes Secrets.

Sealed Secrets Ready: The architecture is compatible with Sealed Secrets or External Secrets Operators for encrypted secret storage in Git.

Credential Rotation: Secrets can be regenerated through re-deployment.

Observability and Auditability

Declarative Monitoring: Observability stacks are part of the Platform-as-Code definition.

Deployment History: Complete history of all deployments and changes through Git log.

ArgoCD UI: Graphical representation of sync status and application topology.

Infrastructure Events: Terraform state changes and Terragrunt outputs document infrastructure changes.

Idempotence and Reproducibility

Idempotent Operations: Repeated execution of the same declaration leads to the same result without side effects.

Deterministic Builds: Same input parameters (Git commit + environment variables) produce identical environments.

Disaster Recovery: Complete environments can be rebuilt from code without restoring backups.

Testing in Production-Like Environments: Development and staging environments are code-identical to production, only with different parameter values.

Purpose in EDP

A ‘stack’ is the declarative description for the platform provisionning in an EDP installation.

Repository

Code:

• Stacks Templates Repo
• Stacks Instances Repo, used for ArgoCD Gitops
• EDP Stacks Deployment mechanism

Documentation: [Link to component-specific documentation]

• Outdated: The former ’edpbuilder’ as script, derived from CNOE’s ‘idpbuilder

The stacks Repository

Purpose and Structure

The stacks repository contains reusable template definitions for platform components. It serves as a central library of building blocks from which Edge Developer Platforms can be composed.

stacks/
└── template/
    ├── edfbuilder.yaml
    ├── registry/
    │   ├── core.yaml
    │   ├── otc.yaml
    │   ├── forgejo.yaml
    │   ├── observability.yaml
    │   └── observability-client.yaml
    └── stacks/
        ├── core/
        ├── otc/
        ├── forgejo/
        ├── observability/
        └── observability-client/

Components

edfbuilder.yaml: The central bootstrap definition. This is an ArgoCD Application that references the registry directory and serves as the entry point for the entire platform provisioning.

registry/: Contains ArgoCD ApplicationSets that function as a meta-layer. Each file defines a category of stacks (e.g., core, forgejo, observability) and references the corresponding subdirectory in stacks/.

stacks/: The actual platform components, organized into thematic categories:

• core: Fundamental components such as ArgoCD, CloudNative PostgreSQL, Dex (SSO)
• otc: Cloud-provider-specific components for Open Telekom Cloud (cert-manager, ingress-nginx, StorageClasses)
• forgejo: Git server and CI runners
• observability: Central observability components (Grafana, Victoria Metrics Stack)
• observability-client: Client-side metrics collection for non-observability clusters

Each stack consists of:

• YAML definitions (primarily ArgoCD Applications)
• values.yaml files for Helm charts
• manifests/ directories for additional Kubernetes resources

Templating Mechanism

The templates use Gomplate with delimiter syntax {{{ }}} for environment variables:

repoURL: "https://{{{ .Env.CLIENT_REPO_DOMAIN }}}/{{{ .Env.CLIENT_REPO_ORG_NAME }}}"
path: "{{{ .Env.CLIENT_REPO_ID }}}/{{{ .Env.DOMAIN }}}/stacks/core"

These placeholders are replaced with environment-specific values during the deployment phase.

The stacks-instances Repository

Purpose and Structure

The stacks-instances repository contains the materialized, environment-specific configurations. While stacks provides the blueprints, stacks-instances contains the actual deployment definitions for concrete environments.

stacks-instances/
└── otc/
    ├── osctest.t09.de/
    │   ├── edfbuilder.yaml
    │   ├── registry/
    │   └── stacks/
    ├── backup-test-manu.t09.de/
    │   ├── edfbuilder.yaml
    │   ├── registry/
    │   └── stacks/
    └── ...

Organizational Principle

The structure follows the schema {cloud-provider}/{domain}/:

• cloud-provider: Identifies the cloud environment (e.g., otc for Open Telekom Cloud)
• domain: The fully qualified domain name of the environment (e.g., osctest.t09.de)

Each environment replicates the structure of stacks/template, but with resolved template variables and environment-specific customizations.

Usage by ArgoCD

ArgoCD synchronizes directly from this repository. Applications reference paths such as:

source:
  path: "otc/osctest.t09.de/stacks/core"
  repoURL: "https://edp.buildth.ing/DevFW-CICD/stacks-instances"
  targetRevision: HEAD

This enables true GitOps: every change to the configurations is traceable through Git commits and automatically synchronized by ArgoCD in the target environment.

The infra-deploy Repository

Role in the Overall Architecture

The infra-deploy repository is the orchestration layer that coordinates both infrastructure and platform provisioning. It represents the evolution of edpbuilder, which was originally derived from the CNOE.io project’s EDPbuilder.

Two-Phase Provisioning

Phase 1: Infrastructure Provisioning

Uses Terragrunt Stacks (experimental feature) to provision cloud resources:

infra-deploy/
├── root.hcl
├── non-prod/
│   ├── tenant.hcl
│   ├── dns_zone/
│   │   ├── terragrunt.hcl
│   │   ├── terragrunt.stack.hcl
│   │   └── terragrunt.values.hcl
│   └── testing/
├── prod/
└── templates/
    └── forgejo/
        ├── terragrunt.hcl
        └── terragrunt.stack.hcl

Terragrunt Stacks provision:

• VPC and network segments
• Kubernetes clusters (CCE on OTC)
• Managed databases (RDS PostgreSQL)
• Load balancers and DNS entries
• Security groups and other cloud resources

Phase 2: Platform Provisioning

The script scripts/edp-install.sh executes the following steps:

1. Template Hydration:

   • Checkout of the stacks repository
   • Execution of Gomplate to resolve template variables
   • Generation of environment-specific manifests

2. Instance Management:

   • Checkout/update of the stacks-instances repository
   • During CI execution: commit and push of the new instance

3. Secrets Management:

   • Generation of credentials (database passwords, SSO secrets, API tokens)
   • Creation of Kubernetes Secrets

4. Bootstrap:

   • Helm-based installation of ArgoCD
   • Application of edfbuilder.yaml or selective registry entries

5. GitOps Handover:

   • ArgoCD takes over further synchronization from stacks-instances
   • Continuous monitoring and self-healing

GitHub Actions Workflows

The .github/workflows/ directory contains three central workflows:

deploy.yaml: Complete deployment pipeline with the following inputs:

• Cluster environment and tenant (prod/non-prod)
• Node flavor and availability zone
• Stack selection (core, otc, forgejo, observability, etc.)
• Infra-catalogue version

plan.yaml: Terraform/Terragrunt plan preview without execution

destroy.yaml: Controlled teardown of environments

Deployment Workflow

The complete provisioning process proceeds as follows:

1. Initiation: GitHub Actions workflow is triggered (manually or automatically)

2. Environment Preparation:

   export CLUSTER_ENVIRONMENT=qa-stage
   cd scripts
   ./new-otc-env.sh  # Creates Terragrunt configuration if new
   

3. Infrastructure Provisioning:

   ./ensure-cluster.sh otc
   # Internally executes:
   # - ./ensure-otc-cluster.sh
   # - terragrunt stack run apply
   

4. Platform Provisioning:

   ./edp-install.sh
   # Executes:
   # - Checkout of stacks
   # - Gomplate hydration
   # - Checkout/update of stacks-instances
   # - Secrets generation
   # - ArgoCD installation
   # - Bootstrap of stacks
   

5. ArgoCD Synchronization: ArgoCD continuously reads from stacks-instances and synchronizes the desired state

The CNOE.io Stacks Concept

The term “stacks” originates from the Cloud Native Operational Excellence (CNOE.io) project. The core idea is the composition of platform capabilities from modular, reusable building blocks.

Principles

Modularity: Each stack is a self-contained unit with clear dependencies

Composability: Stacks can be freely combined to create different platform profiles

Declarativeness: All configurations are declarative and GitOps-capable

Environment-agnostic: Templates are generic; environment specifics are introduced through hydration

Stack Selection and Combinations

The environment variable STACKS controls which components are deployed:

# Complete EDP with central observability
STACKS="core,otc,forgejo,observability"

# Application cluster with client-side observability
STACKS="core,otc,forgejo,observability-client"

# Minimal development environment
STACKS="core,forgejo"

Data Flow and Dependencies

┌─────────────────┐
│  GitHub Actions │
│  (deploy.yaml)  │
└────────┬────────┘
         │
         ├─> Phase 1: Infrastructure
         │   ┌──────────────────┐
         │   │  infra-deploy    │
         │   │  (Terragrunt)    │
         │   └────────┬─────────┘
         │            │
         │            v
         │   ┌──────────────────┐
         │   │  Cloud Provider  │
         │   │  (OTC)           │
         │   │  - VPC           │
         │   │  - K8s Cluster   │
         │   │  - RDS           │
         │   └──────────────────┘
         │
         └─> Phase 2: Platform
             ┌──────────────────┐
             │  edp-install.sh  │
             └────────┬─────────┘
                      │
                      ├─> Checkout: stacks (Templates)
                      │   └─> Gomplate Hydration
                      │
                      ├─> Checkout/Update: stacks-instances
                      │
                      ├─> Secrets Generation
                      │
                      ├─> ArgoCD Installation (Helm)
                      │
                      └─> Bootstrap (edfbuilder.yaml)
                          │
                          v
                 ┌────────────────┐
                 │    ArgoCD      │
                 └────────┬───────┘
                          │
                          └─> Continuous Synchronization
                              from stacks-instances
                              │
                              v
                       ┌──────────────┐
                       │  Kubernetes  │
                       │  Cluster     │
                       └──────────────┘

Historical Context: edpbuilder to infra-deploy

The evolution from edpbuilder to infra-deploy demonstrates the maturation of the architecture:

edpbuilder (Origin):

• Directly derived from CNOE.io’s EDPbuilder
• Focus on local KIND clusters
• Manual configuration
• Monolithic structure

infra-deploy (Current):

• Production-ready for cloud deployments (OTC)
• Terragrunt-based infrastructure orchestration
• CI/CD integration via GitHub Actions
• Clear separation between infrastructure and platform
• Template-instance separation through stacks/stacks-instances

Technical Particularities

Gomplate Templating

Gomplate is used with custom delimiters {{{ }}} to avoid conflicts with Helm templating ({{ }}):

gomplate --input-dir="stacks/template" \
         --output-dir="work" \
         --left-delim "{{{" \
         --right-delim "}}}"

Terragrunt Experimental Stacks

The use of Terragrunt Stacks requires the experimental flag:

export TG_EXPERIMENT_MODE=true
terragrunt stack run apply

This enables hierarchical organization of Terraform modules with dependency management.

ArgoCD ApplicationSets

The registry pattern uses ArgoCD Applications that reference directories:

source:
  path: "otc/osctest.t09.de/stacks/core"

ArgoCD automatically detects all YAML files in the path and synchronizes them as Applications.

Best Practices and Patterns

Immutable Infrastructure: Every environment is fully defined in Git

Secrets Outside Git: Sensitive data is generated at runtime or injected from secret stores

Progressive Rollouts: New environments start as template instances, then are individually customized

Version Pinning: Critical components (Helm charts, Terragrunt modules) are pinned to specific versions

Namespace Isolation: Each stack deploys into dedicated namespaces

Self-Healing: ArgoCD’s automated sync policy enables automatic drift correction

Usage Examples

Deployment by Pipeline

The platform deployment is the second part of the EDP installtaion. First there is the infrastructure setup, which ends with a created kubernetes cluster. Then the platform provisioning by the defined stacks is done. Both is runnable by the deploypipelien in infra-deploy:

The green pipeline looks liek this:

Local setup with ‘kind’

It’s also possible to just run the second part, the stcks provisionning. Then you need to have a kubernetes cluster already running, which is e.g. feasable by a local kind-cluster.

So imagine, you want to to the stacks ‘core,observability’ on your local machine. Then you can run the local entzr

# have kind insatlled
# in /infra-deploy

# provide a kind cluster
kind delete clusters --all
./scripts/ensure-kind-cluster.sh -r

# provide some emnv vars
export TERRAFORM=/bin/bash
export LOADBALANCER_ID=ABC
export DOMAIN=ABC
export DOMAIN_GITEA=ABC
export OS_ACCESS_KEY=ABC
export OS_SECRET_KEY=ABC
export STACKS=core,observability

# deploy
./scripts/edp-install.sh

Status

Maturity: [Production]

Additional Resources

• CNOE

2.2.2.1 - Core

Overview

The Core stack provides foundational infrastructure components required by all other Edge Developer Platform stacks. It establishes the base layer for continuous deployment, database services, and centralized authentication, enabling a secure, scalable platform architecture.

The Core stack deploys ArgoCD for GitOps orchestration, CloudNativePG for PostgreSQL database management, and Dex for OpenID Connect single sign-on capabilities.

Key Features

• GitOps Continuous Deployment: ArgoCD manages declarative infrastructure and application deployments
• Database Operator: CloudNativePG provides enterprise-grade PostgreSQL clusters for platform services
• Single Sign-On: Dex offers centralized OIDC authentication across platform components
• Automated Synchronization: Self-healing deployments with automatic drift correction
• Role-Based Access Control: Integrated RBAC for secure platform administration
• TLS Certificate Management: Automated certificate provisioning and renewal

Repository

Code: Core Stack Templates

Documentation:

• ArgoCD Documentation
• CloudNativePG Documentation
• Dex Documentation

Getting Started

Prerequisites

• Kubernetes cluster (1.24+)
• kubectl configured with cluster access
• Ingress controller (nginx recommended)
• cert-manager for TLS certificate management
• Domain names configured for platform services

Quick Start

The Core stack is deployed as the foundation of the EDP installation:

1. Trigger Deploy Pipeline

   • Go to Infra Deploy Pipeline
   • Click on Run workflow
   • Enter a name in “Select environment directory to deploy”. This must be DNS Compatible. (if you enter test-me then domains will be argocd.test-me.t09.de, dex.test-me.t09.de)
   • Execute workflow

2. ArgoCD Bootstrap The deployment automatically provisions:

   • ArgoCD control plane in argocd namespace
   • CloudNativePG operator in cloudnative-pg namespace
   • Dex identity provider in dex namespace
   • Ingress configurations with TLS certificates
   • OIDC authentication integration

Verification

Verify the Core stack deployment:

# Check ArgoCD installation
kubectl get application -n argocd
kubectl get pods -n argocd

# Verify CloudNativePG operator
kubectl get pods -n cloudnative-pg
kubectl get crd | grep cnpg.io

# Check Dex deployment
kubectl get pods -n dex
kubectl get ingress -n dex

# Verify ingress configurations
kubectl get ingress -n argocd

Access ArgoCD at https://argocd.{DOMAIN} and authenticate via Dex SSO. Or use username admin and the secret inside of kubernetes argocd/argocd-initial-admin-secret as password kubectl get secret -n argocd argocd-initial-admin-secret -ojson | jq -r .data.password | base64 -d.

Architecture

Component Architecture

The Core stack establishes a three-tier foundation:

ArgoCD Control Plane:

• Application management and GitOps reconciliation
• Multi-repository tracking with automated sync
• Resource health monitoring and drift detection
• Integrated RBAC with SSO authentication

CloudNativePG Operator:

• PostgreSQL cluster lifecycle management
• Automated backup and recovery
• High availability and failover
• Storage provisioning via CSI drivers

Dex Identity Provider:

• OpenID Connect authentication service
• Multiple connector support (Forgejo/Gitea, LDAP, SAML)
• Static client registration for platform services
• Token issuance and validation

Networking

Ingress Architecture:

• nginx ingress controller for external access
• TLS termination with cert-manager integration
• Domain-based routing for platform services

Kubernetes Services:

• Internal service communication via ClusterIP
• DNS-based service discovery
• Network policies for security segmentation

Configuration

ArgoCD Configuration

Deployed via Helm chart v9.1.5 with custom values in stacks/core/argocd/values.yaml:

OIDC Authentication:

configs:
  cm:
    url: "https://{DOMAIN_ARGOCD}"
    oidc.config: |
      name: Forgejo
      issuer: https://{DOMAIN_DEX}
      clientID: controller-argocd-dex
      clientSecret: $dex-controller-argocd-dex:dex-controller-argocd-dex
      requestedScopes: ["openid", "profile", "email", "groups"]

RBAC Policy:

policy.csv: |
  g, DevFW, role:admin

Server Settings:

• Insecure mode enabled (TLS handled by ingress)
• Annotation-based resource tracking
• 60-second reconciliation timeout
• Resource exclusions for ProviderConfigUsage and CiliumIdentity

CloudNativePG Configuration

Deployed via Helm chart v0.26.1 with values in stacks/core/cloudnative-pg/values.yaml:

Operator Settings:

• Namespace: cloudnative-pg
• Automated database cluster provisioning
• Custom resource definitions for Cluster, Database, and Pooler resources

Storage Configuration:

• Uses csi-disk storage class by default
• PVC provisioning for PostgreSQL data
• Backup storage integration (S3-compatible)

Dex Configuration

Deployed via Helm chart v0.23.0 with values in stacks/core/dex/values.yaml:

Issuer Configuration:

config:
  issuer: https://{DOMAIN_DEX}
  storage:
    type: memory  # Use persistent storage for production
  oauth2:
    skipApprovalScreen: true
    alwaysShowLoginScreen: false

Forgejo Connector:

connectors:
  - type: gitea
    id: forgejo
    name: Forgejo
    config:
      clientID: $FORGEJO_CLIENT_ID
      clientSecret: $FORGEJO_CLIENT_SECRET
      redirectURI: https://{DOMAIN_DEX}/callback
      baseURL: https://edp.buildth.ing
      orgs:
        - name: DevFW

Static OAuth2 Clients:

• ArgoCD: controller-argocd-dex
• Grafana: controller-grafana-dex

Environment Variables

Core stack services use the following environment variables:

Domain Configuration:

• DOMAIN_ARGOCD: ArgoCD web interface URL
• DOMAIN_DEX: Dex authentication service URL
• DOMAIN_GITEA: Forgejo/Gitea repository URL
• DOMAIN_GRAFANA: Grafana observability dashboard URL

Repository Configuration:

• CLIENT_REPO_ID: Repository identifier for stack configurations
• CLIENT_REPO_DOMAIN: Git repository domain
• CLIENT_REPO_ORG_NAME: Organization name for stack instances

Usage Examples

Managing Applications with ArgoCD

Access and manage applications through ArgoCD:

# Login to ArgoCD CLI
argocd login argocd.${DOMAIN} --sso

# List all applications
argocd app list

# Get application status
argocd app get coder

# Sync application manually
argocd app sync coder

# View application logs
argocd app logs coder

# Diff application state
argocd app diff coder

Creating a PostgreSQL Database

Deploy a PostgreSQL cluster using CloudNativePG:

# database-cluster.yaml
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: app-db
  namespace: my-app
spec:
  instances: 3
  storage:
    size: 20Gi
    storageClass: csi-disk
  postgresql:
    parameters:
      max_connections: "100"
      shared_buffers: "256MB"
  bootstrap:
    initdb:
      database: appdb
      owner: appuser

Apply the configuration:

kubectl apply -f database-cluster.yaml

# Check cluster status
kubectl get cluster app-db -n my-app
kubectl get pods -n my-app -l cnpg.io/cluster=app-db

# Get connection credentials
kubectl get secret app-db-app -n my-app -o jsonpath='{.data.password}' | base64 -d

Configuring SSO for Applications

Add OAuth2 applications to Dex for SSO integration:

# Add to dex values.yaml
staticClients:
  - id: my-app-client
    redirectURIs:
      - 'https://myapp.{DOMAIN}/callback'
    name: 'My Application'
    secretEnv: MY_APP_CLIENT_SECRET

Configure the application to use Dex:

# Application OIDC configuration
OIDC_ISSUER=https://dex.${DOMAIN}
OIDC_CLIENT_ID=my-app-client
OIDC_CLIENT_SECRET=${MY_APP_CLIENT_SECRET}
OIDC_REDIRECT_URI=https://myapp.${DOMAIN}/callback

Deploying Applications via ArgoCD

Create an ArgoCD Application manifest:

# my-app.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
  namespace: argocd
spec:
  project: default
  source:
    repoURL: 'https://github.com/myorg/my-app'
    targetRevision: main
    path: k8s
  destination:
    server: 'https://kubernetes.default.svc'
    namespace: my-app
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Push it to stacks instances to be picked up by argo

Integration Points

• All Stacks: Core stack is a prerequisite for all other EDP stacks
• OTC Stack: Provides ingress-nginx and cert-manager dependencies
• Coder Stack: Uses CloudNativePG for workspace database management
• Forgejo Stack: Integrates with Dex for SSO and ArgoCD for deployment
• Observability Stack: Uses Dex for Grafana authentication and ArgoCD for deployment
• Provider Stack: Deploys Terraform providers via ArgoCD

Troubleshooting

ArgoCD Not Accessible

Problem: Cannot access ArgoCD web interface

Solution:

1. Verify ingress configuration:

   kubectl get ingress -n argocd
   kubectl describe ingress -n argocd
   

2. Check ArgoCD server status:

   kubectl get pods -n argocd
   kubectl logs -n argocd -l app.kubernetes.io/name=argocd-server
   

3. Verify TLS certificate:

   kubectl get certificate -n argocd
   kubectl describe certificate -n argocd
   

4. Test DNS resolution:

   nslookup argocd.${DOMAIN}
   

Dex Authentication Failing

Problem: SSO login fails or redirects incorrectly

Solution:

1. Check Dex logs:

   kubectl logs -n dex -l app.kubernetes.io/name=dex
   

2. Verify Forgejo connector configuration:

   kubectl get secret -n dex
   kubectl get configmap -n dex dex -o yaml
   

3. Test Dex issuer endpoint:

   curl https://dex.${DOMAIN}/.well-known/openid-configuration
   

4. Verify OAuth2 client credentials match in both Dex and consuming application

CloudNativePG Operator Not Running

Problem: PostgreSQL clusters fail to provision

Solution:

1. Check operator status:

   kubectl get pods -n cloudnative-pg
   kubectl logs -n cloudnative-pg -l app.kubernetes.io/name=cloudnative-pg
   

2. Verify CRDs are installed:

   kubectl get crd | grep cnpg.io
   kubectl describe crd clusters.postgresql.cnpg.io
   

3. Check operator logs for errors:

   kubectl logs -n cloudnative-pg -l app.kubernetes.io/name=cloudnative-pg --tail=100
   

Application Sync Failures

Problem: ArgoCD applications remain out of sync or fail to deploy

Solution:

1. Check application status:

   argocd app get <app-name>
   kubectl describe application <app-name> -n argocd
   

2. Review sync operation logs:

   argocd app logs <app-name>
   

3. Verify repository access:

   argocd repo list
   argocd repo get <repo-url>
   

4. Check for resource conflicts or missing dependencies:

   kubectl get events -n <app-namespace> --sort-by='.lastTimestamp'
   

Database Connection Issues

Problem: Applications cannot connect to CloudNativePG databases

Solution:

1. Verify cluster is ready:

   kubectl get cluster <cluster-name> -n <namespace>
   kubectl describe cluster <cluster-name> -n <namespace>
   

2. Check database credentials secret:

   kubectl get secret <cluster-name>-app -n <namespace>
   kubectl get secret <cluster-name>-app -n <namespace> -o yaml
   

3. Test connection from a pod:

   kubectl run -it --rm psql-test --image=postgres:16 --restart=Never -- \
     psql "$(kubectl get secret <cluster-name>-app -n <namespace> -o jsonpath='{.data.uri}' | base64 -d)"
   

4. Review PostgreSQL logs:

   kubectl logs -n <namespace> <cluster-name>-1
   

Additional Resources

• ArgoCD Documentation
• ArgoCD Best Practices
• CloudNativePG Documentation
• CloudNativePG Architecture
• Dex Documentation
• Dex Connectors
• OpenID Connect Specification

2.2.2.2 - OTC

Overview

The OTC (Open Telekom Cloud) stack provides essential infrastructure components for deploying applications on Open Telekom Cloud environments. It configures ingress routing, automated TLS certificate management, and cloud-native storage provisioning tailored specifically for OTC’s Kubernetes infrastructure.

This stack serves as a foundational layer that other platform stacks depend on for external access, secure communication, and persistent storage.

Key Features

• Automated TLS Certificate Management: Let’s Encrypt integration via cert-manager for automatic certificate provisioning and renewal
• Cloud Load Balancer Integration: Nginx ingress controller configured with OTC-specific Elastic Load Balancer (ELB) annotations
• Native Storage Provisioning: Default StorageClass using Huawei FlexVolume provisioner for block storage
• Prometheus Metrics: Built-in monitoring capabilities for ingress traffic and performance
• High Availability: Rolling update strategy with minimal downtime
• HTTP-01 Challenge Support: ACME validation through ingress for certificate issuance

Repository

Code: OTC Stack Templates

Documentation:

• cert-manager Documentation
• ingress-nginx Documentation
• Open Telekom Cloud Documentation