Basic Concepts

• 1: Platform Orchestration
• 2: Application Orchestration

# 1. Install idpbuilder
curl -fsSL https://cnoe.io/install.sh | bash

# 2. Bootstrap platform
idpbuilder create \
  --use-path-routing \
  --package-dir https://github.com/cnoe-io/stacks//ref-implementation

# 3. Wait for platform ready (ca. 10 minutes)
kubectl get applications -A

# Get platform secrets (credentials)
idpbuilder get secrets

# Check all ArgoCD applications
kubectl get applications -n argocd

# Expected: All applications "Synced" and "Healthy"

idpbuilder create \
  --use-path-routing \
  --package-dir https://github.com/cnoe-io/stacks//ref-implementation \
  --log-level debug

# Workflow:
# 1. Creates Kind cluster
# 2. Installs ingress-nginx
# 3. Clones and installs ArgoCD
# 4. Installs Forgejo
# 5. Waits for core services
# 6. Creates technical users
# 7. Configures Git repositories
# 8. Installs remaining stacks via ArgoCD

# Create ArgoCD Application for new component
cat <<EOF | kubectl apply -f -
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: external-secrets
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://charts.external-secrets.io
    targetRevision: 0.9.9
    chart: external-secrets
  destination:
    server: https://kubernetes.default.svc
    namespace: external-secrets-system
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
EOF

# 1. Update via Git (GitOps)
cd your-platform-config-repo
git pull

# 2. Update stack version
vim argocd/applications/component.yaml
# Change targetRevision to new version

# 3. Commit and push
git add .
git commit -m "Update component to v1.2.3"
git push

# 4. ArgoCD will automatically sync
# 5. Monitor the update
argocd app sync component --watch

# Path-based routing (recommended for local development)
idpbuilder create --use-path-routing

# Custom package directory
idpbuilder create --package-dir /path/to/custom/packages

# Custom Kind cluster config
idpbuilder create --kind-config custom-kind.yaml

# Enable debug logging
idpbuilder create --log-level debug

# argocd-cm ConfigMap
data:
  # Enable automatic sync
  application.instanceLabelKey: argocd.argoproj.io/instance

  # Repository credentials
  repositories: |
    - url: https://github.com/cnoe-io/stacks
      name: cnoe-stacks
      type: git

  # Resource exclusions
  resource.exclusions: |
    - apiGroups:
      - cilium.io
      kinds:
      - CiliumIdentity

# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: platform-system

resources:
  - argocd-app.yaml
  - forgejo-app.yaml
  - keycloak-app.yaml

patches:
  - target:
      kind: Application
    patch: |-
      - op: add
        path: /spec/syncPolicy
        value:
          automated:
            prune: true
            selfHeal: true

# 1. Check if all pods are running
kubectl get pods -A

# 2. Check ArgoCD application status
kubectl get applications -n argocd

# 3. Check ingress
kubectl get ingress -A

# 4. Verify DNS resolution
nslookup cnoe.localtest.me

# 5. Check idpbuilder logs
idpbuilder get logs

# 1. Check application details
argocd app get <app-name>

# 2. View sync status
argocd app sync <app-name> --dry-run

# 3. Force sync
argocd app sync <app-name> --force

# 4. Check for errors in ArgoCD logs
kubectl logs -n argocd deployment/argocd-application-controller

# 1. Verify repository configuration
argocd repo list

# 2. Test connection
argocd repo get https://your-git-repo

# 3. Check credentials
kubectl get secret -n argocd

# 4. Re-add repository with correct credentials
argocd repo add https://your-git-repo \
  --username <user> \
  --password <token>

1 - Platform Orchestration

Overview

Orchestration in the context of Platform Engineering refers to the coordinated automation and management of infrastructure, platform, and application components throughout their entire lifecycle. It is a fundamental concept that bridges the gap between declarative specifications (what should be deployed) and actual execution (how it is deployed).

The Role of Orchestration in Platform Engineering

Platform Engineering has emerged as a discipline to improve developer experience and reduce cognitive load on development teams (CNCF Platforms White Paper). Orchestration is the central mechanism that enables this vision:

1. Automation of Complex Workflows: Orchestration coordinates multiple steps and dependencies automatically
2. Consistency and Reproducibility: Guaranteed, repeatable deployments across different environments
3. Self-Service Capabilities: Developers can independently orchestrate resources and deployments
4. Governance and Compliance: Centralized control over policies and best practices

What Does Orchestration Do?

Orchestration systems perform the following tasks:

• Workflow Coordination: Coordination of complex, multi-step deployment processes
• Dependency Management: Resolution and management of dependencies between components
• State Management: Continuous monitoring and reconciliation between desired and actual state
• Resource Provisioning: Automatic provisioning of infrastructure and services
• Configuration Management: Management of configurations across different environments
• Health Monitoring: Monitoring the health of deployed resources

Three Layers of Orchestration

In modern Platform Engineering, we distinguish three fundamental layers of orchestration:

Infrastructure Orchestration

Infrastructure Orchestration deals with the lowest level - the physical and virtual infrastructure layer. This includes:

• Provisioning of compute, network, and storage resources
• Cloud resource management (VMs, networking, storage)
• Infrastructure-as-Code deployment (Terraform, etc.)
• Bare metal and hypervisor management

Target Audience: Infrastructure Engineers, Cloud Architects

Note: Detailed documentation for Infrastructure Orchestration is maintained separately.

More details: Infrastructure Orchestration →

Platform Orchestration

Platform Orchestration focuses on deploying and managing the platform itself - the services and tools that development teams use. This includes:

• Installation and configuration of Kubernetes clusters
• Deployment of platform services (GitOps tools, Observability, Security)
• Management of platform components via Stacks
• Multi-cluster orchestration

Target Audience: Platform Engineering Teams, SRE Teams

In IPCEI-CIS: Platform orchestration is realized using the CNOE stack concept with ArgoCD and Forgejo.

More details: Platform Orchestration →

Application Orchestration

Application Orchestration concentrates on the deployment and lifecycle management of applications running on the platform. This includes:

• Deployment of microservices and containerized applications
• CI/CD pipeline orchestration
• Configuration management and secrets handling
• Application health monitoring and auto-scaling

Target Audience: Application Developers, DevOps Engineers

In IPCEI-CIS: Application orchestration uses Forgejo pipelines for CI/CD and ArgoCD for GitOps-based deployment.

More details: Application Orchestration →

GitOps as Orchestration Paradigm

A central approach in modern platform orchestration solutions is GitOps. GitOps uses Git repositories as the single source of truth for declarative infrastructure and applications:

• Declarative Approach: The desired state is defined in Git
• Automatic Synchronization: Controllers monitor Git and reconcile the live state
• Audit Trail: All changes are traceable in Git history
• Rollback Capability: Easy rollback through Git revert

Continuous Reconciliation

An important concept is continuous reconciliation:

1. The orchestrator monitors both the source (Git) and the target (e.g., Kubernetes cluster)
2. Deviations trigger automatic corrective actions
3. Health checks validate that the desired state has been achieved
4. Drift detection warns of unexpected changes

Orchestration Tools in IPCEI-CIS

Within the IPCEI-CIS platform, we utilize the CNOE (Cloud Native Operational Excellence) stack concept with the following orchestration components:

ArgoCD

• Continuous Delivery for Kubernetes based on GitOps
• Synchronizes Kubernetes manifests from Git repositories
• Supports Helm Charts, Kustomize, Jsonnet, and plain YAML
• Multi-cluster deployment capabilities
• Application Sets for parameterized deployments

Role in IPCEI-CIS: ArgoCD is the central component for GitOps-based deployment management. After the initial bootstrapping phase, ArgoCD takes over the technical coordination of all components.

Forgejo

• Git Repository Management and source control
• CI/CD Pipelines via Forgejo Actions (GitHub Actions compatible)
• Developer Portal Capabilities (initially planned, project discontinued)
• Package registry and artifact management
• Integration with ArgoCD for GitOps workflows

Role in IPCEI-CIS: Forgejo serves as the Git repository host and CI/CD engine. It was initially planned as a developer portal (similar to Backstage’s role in other stacks) but this aspect was not fully realized before project completion.

Note on Backstage: In typical CNOE implementations, Backstage serves as the developer portal providing golden paths through software templates. IPCEI-CIS initially planned to use Forgejo for this purpose but the project concluded before full implementation.

Terraform

• Infrastructure-as-Code provisioning
• Multi-cloud resource management
• State management for infrastructure
• Integration with Forgejo pipelines for automated deployment

Role in IPCEI-CIS: Terraform handles infrastructure provisioning at the infrastructure orchestration layer, integrated into automated workflows via Forgejo pipelines.

CNOE Stacks Concept

• Modular Platform Components bundled as stacks
• Reusable, composable platform building blocks
• Version-controlled stack definitions
• GitOps-based stack deployment via ArgoCD

Role in IPCEI-CIS: The stacks concept from CNOE provides the structural foundation for platform orchestration, enabling modular deployment and management of platform components.

The Orchestration Workflow

A typical orchestration workflow in the IPCEI-CIS platform:

Workflow Steps:

1. Definition: Developer defines application/infrastructure as code
2. Commit: Changes are committed to Forgejo Git repository
3. CI Pipeline: Forgejo Actions build, test, and package the application
4. Sync: ArgoCD detects changes and triggers deployment
5. Provision: Terraform orchestrates required cloud resources (if needed)
6. Deploy: Application is deployed to Kubernetes
7. Monitor: Continuous monitoring and health checks
8. Reconcile: Automatic correction on drift detection

Benefits of Coordinated Orchestration

The integration of infrastructure, platform, and application orchestration provides crucial advantages:

• Reduced Complexity: Developers don’t need to know all infrastructure details
• Faster Time-to-Market: Automated workflows accelerate deployments
• Consistency: Standardized patterns across all teams
• Governance: Central policies are automatically enforced
• Scalability: Platform teams can support many application teams
• Self-Service: Developers can provision services independently
• Audit and Compliance: Complete traceability through Git history

Best Practices

Successful orchestration follows proven principles (Platform Engineering Principles):

1. Platform as a Product: Treat the platform as a product with focus on user experience
2. Self-Service First: Enable developers to use services autonomously
3. Documentation: Comprehensive documentation of golden paths
4. Feedback Loops: Continuous improvement through user feedback
5. Thin Platform Layer: Use managed services where possible instead of building everything
6. Progressive Disclosure: Offer different abstraction levels
7. Focus on Common Problems: Solve recurring problems centrally
8. Treat Glue as Valuable: Integration of different tools is valuable
9. Clear Mission: Define clear goals and responsibilities

Avoiding Anti-Patterns

Common mistakes in platform orchestration (How to fail at Platform Engineering):

• Product Misfit: Building platform without involving developers
• Overly Complex Design: Too many features and unnecessary complexity
• Swiss Knife Syndrome: Trying to solve all problems with one tool
• Insufficient Documentation: Missing or outdated documentation
• Siloed Development: Platform and development teams working in isolation
• Stagnant Platform: Platform not continuously evolved

Sub-Components

The orchestration component includes the following sub-areas:

• Infrastructure Orchestration: Low-level infrastructure deployment and provisioning
• Platform Orchestration: Platform-level component deployment via Stacks
• Application Orchestration: Application-level deployment and CI/CD
• Stacks: Reusable component bundles and compositions

Further Resources

Fundamentals

• CNCF Platforms White Paper - Comprehensive paper on Platform Engineering
• Platform Engineering Definition - What is Platform Engineering?
• Team Topologies - Organizational structures for modern teams

GitOps

• GitOps Principles - Official GitOps principles
• ArgoCD Documentation - ArgoCD documentation

Tools

• CNOE.io - Cloud Native Operational Excellence Framework
• Forgejo - Self-hosted Git service with CI/CD
• Terraform - Infrastructure as Code tool

2 - Application Orchestration

Overview

Application Orchestration deals with the automation of application deployment and lifecycle management. It encompasses the entire workflow from source code to running application in production.

In the context of IPCEI-CIS, Application Orchestration includes:

• CI/CD Pipelines: Automated build, test, and deployment pipelines
• GitOps Deployment: Declarative application deployment via ArgoCD
• Progressive Delivery: Canary deployments, blue-green deployments
• Application Configuration: Environment-specific configuration management
• Golden Paths: Standardized deployment templates and workflows

Target Audience

Application Orchestration is primarily for:

• Application Developers: Teams developing and deploying applications
• DevOps Teams: Teams responsible for deployment automation
• Product Teams: Teams responsible for application lifecycle

Key Features

Automated CI/CD Pipelines

Forgejo Actions provides GitHub Actions-compatible CI/CD:

• Build Automation: Automatic building of container images
• Test Automation: Automated unit, integration, and E2E tests
• Security Scanning: Vulnerability scanning of dependencies and images
• Artifact Publishing: Publishing to container registries
• Deployment Triggering: Automatic deployment after successful build

GitOps-based Deployment

ArgoCD enables declarative application deployment:

• Declarative Configuration: Applications defined as Kubernetes manifests
• Automated Sync: Automatic synchronization between Git and cluster
• Rollback Capability: Easy rollback to previous versions
• Multi-Environment: Consistent deployment across Dev/Test/Prod
• Health Monitoring: Continuous monitoring of application health

Progressive Delivery

Support for advanced deployment strategies:

• Canary Deployments: Gradual rollout to subset of users
• Blue-Green Deployments: Zero-downtime deployments with instant rollback
• A/B Testing: Traffic splitting for feature testing
• Feature Flags: Dynamic feature enablement without deployment

Configuration Management

Flexible configuration for different environments:

• Environment Variables: Configuration via environment variables
• ConfigMaps: Kubernetes-native configuration
• Secrets Management: Secure handling of sensitive data
• External Secrets: Integration with external secret stores (Vault, etc.)

Purpose in EDP

Application Orchestration is the core of developer experience in IPCEI-CIS Edge Developer Platform.

Developer Self-Service

Developers can deploy applications independently:

• Self-Service Deployment: No dependency on operations team
• Standardized Workflows: Clear, documented deployment processes
• Fast Feedback: Quick feedback through automated pipelines
• Environment Parity: Consistent behavior across all environments

Quality and Security

Automated checks ensure quality and security:

• Automated Testing: All changes are automatically tested
• Security Scans: Vulnerability scanning of dependencies and images
• Policy Enforcement: Automated policy checks (OPA, Kyverno)
• Compliance: Auditability of all deployments

Efficiency and Productivity

Automation increases team efficiency:

• Faster Time-to-Market: Faster deployment of new features
• Reduced Manual Work: Automation of repetitive tasks
• Fewer Errors: Fewer manual mistakes through automation
• Better Collaboration: Clear interfaces between Dev and Ops

Repository

Forgejo: forgejo.org

Forgejo Actions: Forgejo Actions Documentation

ArgoCD: argoproj.github.io/cd

Getting Started

Prerequisites

• Forgejo Account: Access to Forgejo instance
• Kubernetes Cluster: Target cluster for deployments
• ArgoCD Access: Access to ArgoCD instance
• Git: For repository management

Quick Start: Application Deployment

1. Create Application Repository

# Create new repository in Forgejo
git init my-application
cd my-application

# Add application code and Dockerfile
cat > Dockerfile <<EOF
FROM node:18-alpine
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production
COPY . .
EXPOSE 3000
CMD ["node", "server.js"]
EOF

2. Add CI/CD Pipeline

Create .forgejo/workflows/build.yaml:

name: Build and Push

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2
      
      - name: Login to Registry
        uses: docker/login-action@v2
        with:
          registry: registry.example.com
          username: ${{ secrets.REGISTRY_USER }}
          password: ${{ secrets.REGISTRY_PASSWORD }}
      
      - name: Build and push
        uses: docker/build-push-action@v4
        with:
          context: .
          push: ${{ github.event_name == 'push' }}
          tags: registry.example.com/my-app:${{ github.sha }}

3. Create Kubernetes Manifests

Create k8s/deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-application
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-application
  template:
    metadata:
      labels:
        app: my-application
    spec:
      containers:
      - name: app
        image: registry.example.com/my-app:latest
        ports:
        - containerPort: 3000
        env:
        - name: NODE_ENV
          value: "production"
---
apiVersion: v1
kind: Service
metadata:
  name: my-application
spec:
  selector:
    app: my-application
  ports:
  - port: 80
    targetPort: 3000

4. Configure ArgoCD Application

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-application
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://forgejo.example.com/myteam/my-application
    targetRevision: main
    path: k8s
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

5. Deploy

# Commit and push
git add .
git commit -m "Add application and deployment configuration"
git push origin main

# ArgoCD will automatically deploy the application
argocd app sync my-application --watch

Usage Examples

Use Case 1: Multi-Environment Deployment

Deploy application to multiple environments:

Repository Structure:

my-application/
├── .forgejo/
│   └── workflows/
│       └── build.yaml
├── base/
│   ├── deployment.yaml
│   ├── service.yaml
│   └── kustomization.yaml
├── overlays/
│   ├── dev/
│   │   ├── kustomization.yaml
│   │   └── patches.yaml
│   ├── staging/
│   │   ├── kustomization.yaml
│   │   └── patches.yaml
│   └── production/
│       ├── kustomization.yaml
│       └── patches.yaml

Kustomize Base (base/kustomization.yaml):

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - deployment.yaml
  - service.yaml

commonLabels:
  app: my-application

Environment Overlay (overlays/production/kustomization.yaml):

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

bases:
  - ../../base

namespace: production

replicas:
  - name: my-application
    count: 5

images:
  - name: my-app
    newTag: v1.2.3

patches:
  - patches.yaml

ArgoCD Applications for each environment:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-application-prod
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://forgejo.example.com/myteam/my-application
    targetRevision: main
    path: overlays/production
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Use Case 2: Canary Deployment

Progressive rollout with canary strategy:

Argo Rollouts Canary:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: my-application
spec:
  replicas: 10
  strategy:
    canary:
      steps:
      - setWeight: 10
      - pause: {duration: 5m}
      - setWeight: 30
      - pause: {duration: 5m}
      - setWeight: 60
      - pause: {duration: 5m}
      - setWeight: 100
  selector:
    matchLabels:
      app: my-application
  template:
    metadata:
      labels:
        app: my-application
    spec:
      containers:
      - name: app
        image: registry.example.com/my-app:v2.0.0

Use Case 3: Feature Flags

Dynamic feature control without deployment:

Application Code with Feature Flag:

const Unleash = require('unleash-client');

const unleash = new Unleash({
  url: 'http://unleash.platform/api/',
  appName: 'my-application',
  customHeaders: {
    Authorization: process.env.UNLEASH_API_TOKEN
  }
});

// Use feature flag
if (unleash.isEnabled('new-checkout-flow')) {
  // New checkout implementation
  renderNewCheckout();
} else {
  // Old checkout implementation
  renderOldCheckout();
}

Integration Points

Forgejo Integration

Forgejo serves as central source code management and CI/CD platform:

• Source Control: Git repositories for application code
• CI/CD Pipelines: Forgejo Actions for automated builds and tests
• Container Registry: Built-in container registry for images
• Webhook Integration: Triggers for external systems
• Pull Request Workflows: Code review and approval processes

ArgoCD Integration

ArgoCD handles declarative application deployment:

• GitOps Sync: Continuous synchronization with Git state
• Health Monitoring: Application health status monitoring
• Rollback Support: Easy rollback to previous versions
• Multi-Cluster: Deployment to multiple clusters
• UI and CLI: Web interface and command-line access

Observability Integration

Integration with monitoring and logging:

• Metrics: Prometheus metrics from applications
• Logs: Centralized log collection via Loki/ELK
• Tracing: Distributed tracing with Jaeger/Tempo
• Alerting: Alert rules for application issues

Architecture

Application Deployment Flow

CI/CD Pipeline Architecture

Typical Forgejo Actions pipeline stages:

1. Checkout: Clone source code
2. Build: Compile application and dependencies
3. Test: Run unit and integration tests
4. Security Scan: Scan dependencies and code for vulnerabilities
5. Build Image: Create container image
6. Push Image: Push to container registry
7. Update Manifests: Update Kubernetes manifests with new image tag
8. Notify: Send notifications on success/failure

Configuration

Forgejo Actions Configuration

Example for Node.js application:

name: CI/CD Pipeline

on:
  push:
    branches: [ main, develop ]
  pull_request:
    branches: [ main ]

env:
  REGISTRY: registry.example.com
  IMAGE_NAME: ${{ github.repository }}

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: '18'
          cache: 'npm'
      
      - name: Install dependencies
        run: npm ci
      
      - name: Run tests
        run: npm test
      
      - name: Run linter
        run: npm run lint

  security:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Run Trivy vulnerability scanner
        uses: aquasecurity/trivy-action@master
        with:
          scan-type: 'fs'
          scan-ref: '.'
          format: 'sarif'
          output: 'trivy-results.sarif'

  build-and-push:
    needs: [test, security]
    runs-on: ubuntu-latest
    if: github.event_name == 'push'
    steps:
      - uses: actions/checkout@v3
      
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2
      
      - name: Login to Registry
        uses: docker/login-action@v2
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ secrets.REGISTRY_USER }}
          password: ${{ secrets.REGISTRY_PASSWORD }}
      
      - name: Extract metadata
        id: meta
        uses: docker/metadata-action@v4
        with:
          images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
          tags: |
            type=ref,event=branch
            type=sha,prefix={{branch}}-
      
      - name: Build and push
        uses: docker/build-push-action@v4
        with:
          context: .
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

ArgoCD Application Configuration

Complete configuration example:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-application
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  
  source:
    repoURL: https://forgejo.example.com/myteam/my-application
    targetRevision: main
    path: k8s/overlays/production
    
    # Kustomize options
    kustomize:
      version: v5.0.0
      images:
        - my-app=registry.example.com/my-app:v1.2.3
  
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  
  # Sync policy
  syncPolicy:
    automated:
      prune: true        # Delete resources not in Git
      selfHeal: true     # Override manual changes
      allowEmpty: false  # Don't delete everything on empty repo
    
    syncOptions:
      - CreateNamespace=true
      - PruneLast=true
      - RespectIgnoreDifferences=true
    
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m
  
  # Ignore differences (avoid sync loops)
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas  # Ignore if HPA manages replicas

Troubleshooting

Pipeline Fails

Problem: Forgejo Actions pipeline fails

Solution:

# 1. Check pipeline logs in Forgejo UI
# Navigate to: Repository → Actions → Select failed run

# 2. Check runner status
# In Forgejo: Site Admin → Actions → Runners

# 3. Check runner logs
kubectl logs -n forgejo-runner deployment/act-runner

# 4. Test pipeline locally with act
act -l  # List available jobs
act -j build  # Run specific job

ArgoCD Application OutOfSync

Problem: Application shows “OutOfSync” status

Solution:

# 1. Check differences
argocd app diff my-application

# 2. View sync status details
argocd app get my-application

# 3. Manual sync
argocd app sync my-application

# 4. Hard refresh (ignore cache)
argocd app sync my-application --force

# 5. Check for ignored differences
argocd app get my-application --show-operation

Application Deployment Fails

Problem: Application pod crashes after deployment

Solution:

# 1. Check pod status
kubectl get pods -n production

# 2. View pod logs
kubectl logs -n production deployment/my-application

# 3. Describe pod for events
kubectl describe pod -n production <pod-name>

# 4. Check resource limits
kubectl top pod -n production

# 5. Rollback via ArgoCD
argocd app rollback my-application

Image Pull Errors

Problem: Kubernetes cannot pull container image

Solution:

# 1. Verify image exists
docker pull registry.example.com/my-app:v1.2.3

# 2. Check image pull secret
kubectl get secret -n production regcred

# 3. Create image pull secret if missing
kubectl create secret docker-registry regcred \
  --docker-server=registry.example.com \
  --docker-username=user \
  --docker-password=password \
  -n production

# 4. Reference secret in deployment
kubectl patch deployment my-application -n production \
  -p '{"spec":{"template":{"spec":{"imagePullSecrets":[{"name":"regcred"}]}}}}'

Best Practices

Golden Path Templates

Provide standardized templates for common use cases:

1. Web Application Template: Node.js, Python, Go web services
2. API Service Template: RESTful API with OpenAPI
3. Batch Job Template: Kubernetes CronJob configurations
4. Microservice Template: Service mesh integration

Example repository template structure:

application-template/
├── .forgejo/
│   └── workflows/
│       ├── build.yaml
│       ├── test.yaml
│       └── deploy.yaml
├── k8s/
│   ├── base/
│   └── overlays/
├── src/
│   └── ...
├── Dockerfile
├── README.md
└── .gitignore

Deployment Checklist

Before deploying to production:

• ✅ All tests passing
• ✅ Security scans completed
• ✅ Resource limits defined
• ✅ Health checks configured
• ✅ Monitoring and alerts set up
• ✅ Backup strategy defined
• ✅ Rollback plan documented
• ✅ Team notified about deployment

Configuration Management

• Use ConfigMaps for non-sensitive configuration
• Use Secrets for sensitive data
• Use External Secrets Operator for vault integration
• Never commit secrets to Git
• Use environment-specific overlays (Kustomize)
• Document all configuration options

Status

Maturity: Production

Stability: Stable

Support: Internal Platform Team

Additional Resources

Forgejo

• Forgejo Documentation
• Forgejo Actions Guide
• Forgejo API Reference

ArgoCD

• ArgoCD Documentation
• ArgoCD Best Practices
• ArgoCD Sync Waves

GitOps

• GitOps Principles
• GitOps Patterns
• Kubernetes Deployment Strategies

CI/CD

• GitHub Actions Documentation (Forgejo Actions compatible)
• Docker Best Practices
• Container Security Best Practices