Flux CD Migration Guide¶

This document describes the migration from direct kubectl deployments to Flux CD GitOps-based deployments.

Overview¶

Previously, this project used GitHub Actions to build Docker images and directly deploy them to Kubernetes clusters using kubectl rollout restart. Now, it uses Flux CD for GitOps-based deployments with automatic image updates.

What Changed¶

1. GitHub Actions Workflow Updates¶

Image Tags: Modified Docker image tagging to include timestamp-based tags for Flux CD image automation
Deployment Method: Removed direct kubectl deployments, replaced with GitOps notifications
Tag Format: Now generates tags like main-abc123-1672531200 for automatic detection by Flux CD

2. Kubernetes Deployment Updates¶

Image Policy Annotation: Added # {"$imagepolicy": "flux-system:portfolio-image-policy"} to deployment.yaml
Automated Updates: Flux CD will automatically update this image reference when new images are detected

3. New Flux CD Components¶

GitRepository: Monitors the GitHub repository for changes
ImageRepository: Monitors the container registry for new images
ImagePolicy: Defines which images to select (timestamp-based sorting)
ImageUpdateAutomation: Automatically commits image updates back to Git
Kustomizations: Manages deployments to dev, staging, and prod environments

Installation¶

Step 1: Install Flux CD CLI (Local Development)¶

# macOS
brew install fluxcd/tap/flux

# Linux
curl -s https://fluxcd.io/install.sh | sudo bash

# Verify installation
flux version

Step 2: Install Flux CD in Cluster¶

# Bootstrap Flux CD (this installs the controllers)
flux install --version=latest

# Verify installation
kubectl get pods -n flux-system

Step 3: Create Required Secrets¶

GitHub Token Secret¶

Create a Personal Access Token with repo permissions and add it as a secret:

kubectl create secret generic github-token \
  --from-literal=username=your-github-username \
  --from-literal=password=your-github-token \
  --namespace=flux-system

GCP Registry Secret¶

Create a service account key for accessing Google Artifact Registry:

# Create service account key (JSON format)
gcloud iam service-accounts keys create key.json \
  --iam-account=your-service-account@project-id.iam.gserviceaccount.com

# Create the secret
kubectl create secret docker-registry gcp-gcr-token \
  --docker-server=us-central1-docker.pkg.dev \
  --docker-username=_json_key \
  --docker-password="$(cat key.json)" \
  --namespace=flux-system

# Clean up the key file
rm key.json

Step 4: Apply Flux CD Manifests¶

# Apply the Flux CD components
kubectl apply -f k8s/flux-system/gotk-components.yaml
kubectl apply -f k8s/flux-system/kustomizations.yaml

# Verify resources are created
kubectl get gitrepository,imagerepository,imagepolicy,imageupdateautomation -n flux-system
kubectl get kustomization -n flux-system

How It Works Now¶

Image Build Process¶

GitHub Actions: Builds and pushes Docker images with timestamp-based tags
Flux ImageRepository: Monitors the registry every 1 minute for new images
Flux ImagePolicy: Selects the latest image based on timestamp extraction from tag
Flux ImageUpdateAutomation: Updates the deployment.yaml file with the new image tag
Flux GitRepository: Detects the updated manifests and syncs them to the cluster

Deployment Pipeline¶

Dev Environment: Deploys immediately when changes are detected
Staging Environment: Deploys after dev is healthy (depends on portfolio-dev)
Prod Environment: Deploys after staging is healthy (depends on portfolio-staging)

Branch-Based Deployments¶

develop branch → dev namespace
staging branch → staging namespace
main branch → prod namespace

Monitoring and Troubleshooting¶

Check Flux CD Status¶

# Overview of all Flux resources
flux get all

# Check specific resources
flux get sources git
flux get sources helm
flux get kustomizations
flux get images all

# Logs for troubleshooting
kubectl logs -n flux-system -l app=source-controller
kubectl logs -n flux-system -l app=kustomize-controller
kubectl logs -n flux-system -l app=image-automation-controller
kubectl logs -n flux-system -l app=image-reflector-controller

Manual Reconciliation¶

# Force reconciliation
flux reconcile source git portfolio-source
flux reconcile kustomization portfolio-dev
flux reconcile image repository portfolio-image-repo
flux reconcile image update portfolio-image-updates

Check Image Updates¶

# See what images Flux has detected
kubectl describe imagerepository portfolio-image-repo -n flux-system
kubectl describe imagepolicy portfolio-image-policy -n flux-system

Configuration Details¶

Image Policy Pattern¶

The image policy uses this pattern to extract timestamps from tags:

filterTags:
  pattern: "^main-[a-f0-9]+-(?P<ts>[0-9]+)$"
  extract: "$ts"

This matches tags like: main-abc123-1672531200 and extracts 1672531200 for comparison.

Reconciliation Intervals¶

GitRepository: 1 minute (detects Git changes)
ImageRepository: 1 minute (detects new images)
ImageUpdateAutomation: 1 minute (updates manifests)
Dev Kustomization: 5 minutes (deploys to dev)
Staging Kustomization: 5 minutes (deploys to staging)
Prod Kustomization: 10 minutes (deploys to prod)

Health Checks¶

Each environment waits for the portfolio deployment to be ready before marking as successful.

Migration Benefits¶

GitOps Approach: All changes are tracked in Git
Automatic Image Updates: No manual intervention needed
Progressive Deployment: Automatic promotion through environments
Rollback Capability: Easy to revert through Git
Observability: Clear status of all deployments
Security: No cluster credentials needed in CI/CD

Rollback Process¶

To rollback to a previous image:

Find the previous image tag in Git history
Update the deployment.yaml manually with the desired image
Commit the change - Flux will deploy it automatically

Or use Flux CLI:

# Suspend automation temporarily
flux suspend image update portfolio-image-updates

# Update to specific image
kubectl set image deployment/portfolio portfolio=us-central1-docker.pkg.dev/kame-457417/kame-house-images/portfolio:main-abc123-1672531100 -n prod

# Resume automation when ready
flux resume image update portfolio-image-updates

Next Steps¶

After migration is complete:

Monitor Flux CD dashboards and logs
Test the deployment pipeline with a small change
Update runbooks to reference Flux CD commands instead of kubectl
Consider adding Slack/email notifications for deployment status
Set up Flux CD monitoring with Prometheus/Grafana if desired