This repository provides declarative, native Kubernetes manifests for deploying the CKAN data platform alongside its critical infrastructure dependencies. It acts as a Remote Centralized Base Engine using Kustomize, allowing any external project or GitOps repository to consume, configure, and toggle resources dynamically without modifying core code.
.
├── .gitignore # Excludes hydrated YAML files and secrets from Git
├── README.md # Repository documentation
├── base/ # Core Application Blueprint (100% Environment Agnostic)
│ ├── kustomization.yaml # Core application orchestrator
│ ├── ckan-deployment.yaml # Vanilla CKAN workload
│ ├── ckan-service.yaml # Internal ClusterIP routing
│ ├── ckan-configmap.yaml # Static application settings
│ ├── ingress-internal.yaml # Generic Ingress layout with domain placeholder
│ ├── serviceaccount.yaml # Dedicated runtime identity
│ └── init-scripts-configmap.yaml # Database entrypoint setup scripts
├── infrastructure/ # Pluggable Infrastructure Modules (Vendored/Static)
│ ├── cert-manager/ # X.509 Certificate Automation Engine
│ ├── nginx/ # F5 NGINX Ingress Controller Base
│ ├── redis/ # Standalone Redis Caching Engine
│ └── solr/ # Apache Solr Search (Standalone)
└── overlays/ # Environment & Architecture Workspaces
├── cluster-infra/ # Cluster-Scoped Resources (LoadBalancers, CRDs, Issuers)
│ ├── cluster-issuer.yaml.tmpl # Let's Encrypt ACME configuration
│ ├── kustomization.yaml.tmpl # Infra orchestrator
│ └── nginx-azure-patch.yaml.tmpl # Binds NGINX to an Azure Static Public IP
└── production/ # Namespace-Scoped Resources (App, Certs, Routing)
├── add-init-container-patch.yaml # Attaches Database init workflow to CKAN
├── certificate.yaml.tmpl # Domain-specific TLS certificate request
├── ingress-patch.yaml.tmpl # Injects target domain into NGINX routing
└── kustomization.yaml.tmpl # CI/CD secret generation & App configuration
Downstream GitOps repositories can implement this platform using two distinct patterns depending on internal engineering structures:
Downstream projects do not copy files. They call this public repository URL directly within their own local kustomization.yaml via HTTPS.
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
# 1. Automatically prepend your unique environment token to all resources
namePrefix: project-alpha-prod-
# 2. Reference this repository remotely (Pin to a specific tag or branch)
resources:
- https://github.com/datopian/ckan-kustomize-base//base?ref=v1.0.0
- https://github.com/datopian/ckan-kustomize-base//infrastructure/redis?ref=v1.0.0
- https://github.com/datopian/ckan-kustomize-base//infrastructure/solr?ref=v1.0.0
# 3. Supply your specific environment configurations
secretGenerator:
- name: ckan-envvars
type: Opaque
literals:
- CKAN_SITE_URL="https://ckan.alpha.com"
If you prefer keeping infrastructure and configurations under a single repository:
- Clone this entire project into a subdirectory of your deployment infrastructure.
- Link your environment overlays natively using standard filesystem paths (as demonstrated in the
overlays/directory of this repo).
Kustomize does not use dynamic if/else programmatic scripting. Features are handled via Composition. To toggle any piece of infrastructure on or off, modify your target overlay's kustomization.yaml resources: array.
Infrastructure components are completely decoupled from the core application. To enable or disable dependencies depending on your target architecture (e.g., using a managed Cloud Redis instead of a pod), simply comment them out:
resources:
- ../../base # Required Core App
- ../../infrastructure/nginx # ENABLED: Drops in Ingress Controller
# - ../../infrastructure/redis # DISABLED: Bypassed for cloud-managed Redis
- ../../infrastructure/solr # ENABLED: Standalone search pod
Because all sensitive values and environment specific data are handled via .tmpl files, adding new environments to this repository is incredibly simple.
If you want to add a staging environment:
- Duplicate the overlay: Copy the entire
overlays/productionfolder and name itoverlays/staging. - Configure your CI/CD Pipeline: You do not need to change the template files. Simply configure your CI/CD runner to inject the staging environment variables (e.g.,
ENVIRONMENT_NAMESPACE="myckanproject-staging",CKAN_SITE_DOMAIN="staging.ckan.datopian.com") during theenvsubsthydration step. - Deploy: The pipeline compiles the staging overlay independently of production.
To maintain a secure and DRY (Don't Repeat Yourself) GitOps workflow, environment-specific configurations and secrets are never hardcoded into this repository. Instead, we use .tmpl files.
During the CI/CD pipeline deployment, a tool like envsubst dynamically reads the environment variables stored in our GitHub Secrets/Azure KeyVault and injects them into the templates to generate the final deployment manifests.
These templates govern the actual CKAN application, its credentials, and its routing.
| Template File | Purpose | Required Variables |
|---|---|---|
kustomization.yaml.tmpl |
The master blueprint for the environment. Defines namespaces, container registry targets, and all plaintext/sensitive configuration variables for CKAN, Postgres, and Redis. | ENVIRONMENT_NAMESPACE, ENVIRONMENT_PREFIX, CKAN_SITE_URL, CKAN_SECRET_KEY, POSTGRES_PASSWORD, REDIS_PASSWORD, ACR_DOCKERCONFIG_JSON, etc. |
ingress-patch.yaml.tmpl |
Injects the environment's specific domain name into the NGINX routing rules. | CKAN_SITE_DOMAIN |
certificate.yaml.tmpl |
Instructs Cert-Manager to fetch a Let's Encrypt TLS certificate specifically for this environment's URL. | CKAN_SITE_DOMAIN, ENVIRONMENT_NAMESPACE |
These templates govern cluster-wide services that span multiple namespaces.
| Template File | Purpose | Required Variables |
|---|---|---|
cluster-issuer.yaml.tmpl |
Configures the ACME protocol for Let's Encrypt to verify domain ownership. | TLS_EMAIL |
nginx-azure-patch.yaml.tmpl |
Binds the NGINX Ingress Controller to a specific static Public IP address provisioned in Azure. | AZURE_LB_IP, AZURE_RESOURCE_GROUP |
kustomization.yaml.tmpl |
The entrypoint for infrastructure deployment. | None required |
To ensure secure workflows, the active .yaml files containing raw database strings and registry credentials are excluded from version control via .gitignore.
To execute a validated manual dry-run compile locally without touching the live cluster:
# 1. Export your local testing variables
export ENVIRONMENT_NAMESPACE="myckanproject-test"
export ENVIRONMENT_PREFIX="myckanproject-test"
export CKAN_SITE_DOMAIN="ckan-local.test.com"
# ... export other required variables ...
# 2. Hydrate the templates
envsubst < overlays/production/kustomization.yaml.tmpl > overlays/production/kustomization.yaml
envsubst < overlays/production/ingress-patch.yaml.tmpl > overlays/production/ingress-patch.yaml
envsubst < overlays/production/certificate.yaml.tmpl > overlays/production/certificate.yaml
# 3. Compile and review raw output manifests safely
kubectl kustomize overlays/production
# 4. Test API schema compliance against the live cluster safely
kubectl kustomize overlays/production | kubectl apply --server-side --dry-run=server -f -
# 5. Execute deployment (If authorized)
kubectl apply -k overlays/production
The infrastructure components are vendored (hydrated into pure, static text files) to assure environment immutability, security audits, and air-gapped performance.
| Component | Upstream Source Registry | Original Version | Hydration Command Reference |
|---|---|---|---|
| Cert Manager | https://github.com/cert-manager/cert-manager |
v1.20.2 |
curl -sL https://github.com/cert-manager/cert-manager/releases/download/v1.20.2/cert-manager.yaml > cert-manager.yaml |
| NGINX | oci://ghcr.io/nginx/charts/nginx-ingress |
2.5.3 |
helm template nginx-ingress <source> --version 2.5.3 > nginx-ingress.yaml |
| Redis | registry-1.docker.io/bitnamicharts/redis |
23.1.1 |
helm template redis <source> --version 23.1.1 > redis.yaml |
| Solr | ckan/ckan-solr (Docker Hub) |
2.11-solr9.9 | N/A (Manually managed native StatefulSet & Service) |
- For Cert Manager (if using), replace the cert-manager.yaml with newer version.
- For Helm-based components (NGINX, Redis):
- Re-run the corresponding hydration helm template (or curl) command using your newly targeted upstream software version.
- Route the output stream to overwrite the old file within infrastructure//.
- Clear out metadata leftovers via: sed -i '/heritage:[[:space:]]Helm/d' infrastructure//.yaml.
- For Solr:
- Simply update the Kustomize images: array in your overlay, or manually bump the image tag directly in infrastructure/solr/statefulset.yaml.
When publishing infrastructure enhancements, structural optimization, or security updates:
- Keep the
base/Pure: Never commit hardcoded environment prefixes, target load balancer IPs, or plaintext organizational credentials to the root base files. - Tag Releases (Semantic Versioning): Use explicit Git tags for structural alterations so downstream clusters don't break unexpectedly.
git tag -a v1.0.0 -m "release: baseline clean architecture with modular infrastructure"
git push origin v1.0.0
- Application Tag Updates: To bump application tags across deployments cleanly without touching core files, utilize Kustomize's standard
images:array inside the environment overlay layer:
images:
- name: ckan/ckan
newName: datopian/ckan
newTag: v4.1.2 # Updates target image tags safely during pipeline compilation