How to migrate an application to Kubernetes and OpenShift
I've moved more than 10 applications from on-prem servers to OpenShift, and 7 of them are in production now. I also manage the nginx and the routing for them. Almost none of them broke because of Kubernetes itself. They broke because of things the app assumed about the server it used to live on.
Here are the steps I follow and what to expect at each one.
Before you touch the cluster
Write down what the app depends on today: the ports it listens on, the files it writes, where its config and secrets come from, which hostnames and IPs it calls, and how it gets started. Most surprises later are something on that list that nobody wrote down.
The steps, in order
- Set up the foothold permissions. The service account that deploys needs permission to manage the namespace. I do this before anything else.
- Onboard the namespace. In my setup this step is automated, and it creates the rest of the permissions and secrets the app needs. Doing it by hand for every app is where the days go.
- Split what needs splitting. If the app is a monolith, decide which parts become their own services. Not everything has to, but the parts that scale or deploy on a different schedule usually should.
- Set up the routes and nginx. More on this below.
- Write the Dockerfiles so the build runs in the pipeline and not on someone's laptop.
- Register the app and its components wherever the team deploys from. For my teams that's the developer platform I built.
- Wire up the pipeline. A GitHub Actions workflow with the right inputs builds the image, and a GitOps tool such as Argo CD syncs the Helm chart to the cluster.
- Run the build and troubleshoot. Plan time for this. It's never zero.
What breaks the first time it runs on OpenShift
- It expects to run as root. OpenShift runs containers as a random non-root user by default. An app that writes to its own install directory, or a base image that assumes root, fails at start-up. Make the directories it writes to owned by group 0 and group-writable, and stop writing into the image at runtime.
- It listens on port 80 or 443. Ports below 1024 need privileges the pod doesn't have. Listen on 8080 or 8443, and let the service and the route handle the outside port.
- It writes files it expects to keep. Anything written inside the container is gone when the pod restarts. Uploads and data go to a persistent volume or, better, a service outside the app.
- Config lives in a file on the server. Move config to ConfigMaps and secrets to a secrets manager. Neither belongs in the image or in Git. How I handle secrets with Vault.
- Hardcoded hostnames and IPs. Inside the cluster, services find each other by service name. Anything that pointed at a server's IP has to change.
- Network policies block it. With a default-deny network policy, nothing talks to anything until you allow it. The app starts fine and then times out calling its database.
The root and port fixes in a Dockerfile look like this. The base image is a placeholder for whatever your app runs on.
FROM registry.example.com/base/nodejs:20
WORKDIR /opt/app
COPY --chown=1001:0 . .
RUN chmod -R g=u /opt/app
USER 1001
EXPOSE 8080
CMD ["node", "server.js"]
Add a .dockerignore that lists .git and .env, so COPY . . doesn't copy your Git history or local secrets into the image.
And a default-deny policy with one allow, so the app can reach its database and nothing else can. On OpenShift you'll also need an allow for the router, or the route stops working. Red Hat's network policy docs show the label for your version.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-ingress
namespace: my-namespace
spec:
podSelector: {}
policyTypes:
- Ingress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-my-app-to-my-db
namespace: my-namespace
spec:
podSelector:
matchLabels:
app: my-db
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchLabels:
app: my-app
ports:
- protocol: TCP
port: 5432
Routes, TLS and nginx
An OpenShift route is how traffic from outside reaches the service. The decision that matters is where TLS ends.
- Edge: TLS ends at the router, and traffic to the pod is plain HTTP. The simplest option, and fine for many internal apps.
- Passthrough: the router passes encrypted traffic straight to the pod, and the app holds the certificate. Use it when the app has to see the TLS connection itself.
- Re-encrypt: TLS ends at the router, and a new TLS connection goes to the pod. Use it when traffic has to stay encrypted inside the cluster.
An edge route that also sends plain HTTP to HTTPS:
apiVersion: route.openshift.io/v1
kind: Route
metadata:
name: my-app
namespace: my-namespace
spec:
host: my-app.apps.example.com
to:
kind: Service
name: my-app
port:
targetPort: http
tls:
termination: edge
insecureEdgeTerminationPolicy: Redirect
If the app sits behind nginx, check two things. Stock nginx images often run as root and listen on port 80, so use an unprivileged image or change the config. And the app now sits behind two proxies, so anything that builds URLs or checks the client's address has to trust the forwarded headers, or redirects start pointing at the wrong place.
A friendly hostname usually means a CNAME that points at the route. That's a DNS change somebody has to request, so ask for it early.
When it makes sense to get help
One simple, stateless app is a good first migration to do yourself, and you'll learn the platform doing it. Bring in help when there are many apps and a deadline, when the apps are old and the people who wrote them are gone, or when the platform itself isn't ready yet: the permissions, the namespace onboarding, the pipelines and the policies above. That foundation is what lets the next app onboard in hours instead of days.
Related: the case study on moving 10+ apps to OpenShift and the security gates in the pipeline.