Pushing my OpenShift 4 cluster to the limit

When I was involved in the design of the Red Hat OpenShift 4 platform for customer deployment, the common question coming from the customer was how much my cluster can take the load, how many object my cluster can take, and ultimately how I can push my cluster to the limit.

There is no magic number to this, each cluster in each specific environment will give different behavior and result of the performance due to multiple factors (such as network performance, CPU capacity, capability, etc).

There are many ways of load testing (or soak testing) the cluster. Here we are going to talk about the cluster loader.

Cluster loader works by creating an object defined by the user and the same time measuring performance metrics (from the OpenShift monitoring Grafana dashboard).

The cluster loader can be executed by running an origin-tests container image:

#> podman pull quay.io/openshift/origin-tests:4.9

There are built-in test suites, as example:

#> podman run -v ${LOCAL_KUBECONFIG}:/root/.kube/config:z -i quay.io/openshift/origin-tests:4.9 /bin/bash -c 'export KUBECONFIG=/root/.kube/config && openshift-tests run-test "[sig-scalability][Feature:Performance] Load cluster should populate the cluster [Slow][Serial] [Suite:openshift]"'

We also can set cluster loader configuration YAML file to suit our need, as an example to set the tuning set (stepping, delay, etc):

tuningsets: 
    - name: default
      pods:
        stepping: 
          stepsize: 5
          pause: 0 s
        rate_limit: 
          delay: 0 ms

Once we have the test result, we now can have a better view of how much our cluster can take. This also gives us some hint if we need to give more resources to the cluster.

It is recommended to extend this with application load testing as well. It is peace of mind to know what is the maximum load the cluster can take with the given resource with the maximum object deployed while the application is running at maximum capacity.

Therefore, we can do well-informed capacity planning and expansion when required.