Verrazzano Enterprise Container Platform – High Availability

Docs: Node Failure Guide

Mon, 01 Jan 0001 00:00:00 +0000

A Node failure can occur for many reasons, including hardware failures and network outages. This guide provides information about what to expect when a Node failure occurs and how to recover from a Node failure. Recovery depends on the Storage Provisioner and the type of storage that you use.

NOTE

This guide assumes that the storage provided in the cluster is physically separate from the Node and is recoverable. It does not apply to a local storage on the Node.

What to expect

By default, when a Node fails:

It may take up to a minute for the failure to reflect in the Kubernetes API server and update the Node status to NotReady.
After about five minutes of the Node status being NotReady, the status of the Pods on that Node will be changed to Unknown or NodeLost.
The status of the Pods with controllers, like Daemonsets, Statefulsets, and Deployments, will be changed to Terminating.

NOTE: Pods without a controller, started with a PodSpec, will not be terminated. They must be manually deleted and recreated.
New Pods will start on the Nodes that remain with Ready status.

NOTE: Statefulsets are a special case. The Statefulset controller maintains an ordinal list of Pods, one each for a given name. The Statefulset controller will not start a new Pod with the name of an existing Pod.
The Pods that have associated Persistent Volumes of type ReadWriteOnce, do not become Ready. This is because the Pods try to attach to the existing volumes that are still attached to the old Pod, which is still Terminating. This happens because, at a given point, the Persistent Volumes of type ReadWriteOnce, can be associated only with a single Node, and the new Pod resides on another Node.
If multiple Availability Domains are used in the Kubernetes cluster and the failed Node is the last in that Availability Domain, then the existing volumes will no longer be reachable by new Pods in a separate Availability Domain.

About recovery

After a Node fails, if the Node can be recovered within five minutes, then the Pods will return to a Running state. If the Node is not recovered after five minutes, then the Pods will complete termination and are deleted from the Kubernetes API server. New Pods that have Persistent Volumes of type ReadWriteOnce, will now be able to mount the Persistent Volumes and change to Running.

If a Node cannot be recovered and is replaced, then deleting the Node from the Kubernetes API server will terminate the old Pods and release the Persistent Volumes of type ReadWriteOnce, to be mounted by any new Pods.

If multiple Availability Domains are used in the Kubernetes cluster, then the replacement Node should be added to the same Availability Domain that the deleted Node occupied. This allows the Pods to be scheduled on the replacement Node that can reach the Persistent Volumes in that Availability Domain, and then the Pod status is changed to Running.

Do not forcefully delete Pods or Persistent Volumes in a failed Node that you plan to recover or replace. If you force delete Pods or Persistent Volumes in a failed Node, it may lead to loss of data and, in the case of Statefulsets, it may lead to split-brain scenarios. For more information about Statefulsets, see Force Delete StatefulSet Pods in the Kubernetes documentation.

You can force delete Pods and Persistent Volumes when a failed Node cannot be recovered or replaced in the same Availability Domain as the original Node.

Docs: Upgrade prod Installation for High Availability

Mon, 01 Jan 0001 00:00:00 +0000

The exact steps required to upgrade a Verrazzano environment to achieve high availability will vary based on the configuration of each environment.

Assess whether your Kubernetes configuration must be updated to support the level of high availability that you want to achieve. See Configure High Availability.
Upgrade Verrazzano to v1.5.0 or later. See Upgrade Verrazzano.
The examples/ha directory contains examples of highly available Verrazzano installations. The following example uses the ha.yaml file as an example of how to upgrade a default prod installation to a highly available Verrazzano environment.

a. Create a patch file:

$ cat > patch.yaml <<EOF
spec:
  components:
    authProxy:
      overrides:
      - values:
          replicas: 2
    certManager:
      overrides:
      - values:
          replicaCount: 2
          cainjector:
            replicaCount: 2
          webhook:
            replicaCount: 2
    console:
      overrides:
      - values:
          replicas: 2
    ingress:
      overrides:
      - values:
          controller:
            autoscaling:
              enabled: true
              minReplicas: 2
          defaultBackend:
            replicaCount: 2
    istio:
      overrides:
      - values:
          apiVersion: install.istio.io/v1alpha1
          kind: IstioOperator
          spec:
            components:
              pilot:
                k8s:
                  replicaCount: 2
              ingressGateways:
                - enabled: true
                  k8s:
                    affinity:
                      podAntiAffinity:
                        preferredDuringSchedulingIgnoredDuringExecution:
                        - podAffinityTerm:
                            labelSelector:
                              matchExpressions:
                              - key: app
                                operator: In
                                values:
                                - istio-ingressgateway
                            topologyKey: kubernetes.io/hostname
                          weight: 100
                    replicaCount: 2
                    service:
                      type: LoadBalancer
                  name: istio-ingressgateway
              egressGateways:
                - enabled: true
                  k8s:
                    affinity:
                      podAntiAffinity:
                        preferredDuringSchedulingIgnoredDuringExecution:
                        - podAffinityTerm:
                            labelSelector:
                              matchExpressions:
                              - key: app
                                operator: In
                                values:
                                - istio-egressgateway
                            topologyKey: kubernetes.io/hostname
                          weight: 100
                    replicaCount: 2
                  name: istio-egressgateway
    keycloak:
      overrides:
      - values:
          replicas: 2
      mysql:
        overrides:
        - values:
            serverInstances: 3
            routerInstances: 2
    opensearchDashboards:
      replicas: 2
    kiali:
      overrides:
      - values:
          deployment:
            replicas: 2
    prometheusOperator:
      overrides:
      - values:
          prometheus:
            prometheusSpec:
              replicas: 2
    opensearch:
      nodes:
      - name: es-ingest
        replicas: 2
EOF

b. Apply the patch:

$ kubectl patch verrazzano verrazzano --patch-file=patch.yaml --type=merge

c. Wait for the patch to be installed:

$ kubectl wait --timeout=30m --for=jsonpath='{.status.state}'=Ready verrazzano/verrazzano

Docs: Customize High Availability

Mon, 01 Jan 0001 00:00:00 +0000

High availability designs follow three main principles:

Elimination of single points of failure
Fault detection
Reliable failover points

Verrazzano provides a means to eliminate single points of failure among critical Verrazzano components. This is accomplished by increasing replica counts, anti-affinity rules, and implementing replicated data for components that rely on MySQL and OpenSearch.

The ha.yaml file illustrates how the prod profile can be extended to configure a highly available Verrazzano installation. The increased replica counts, along with the anti-affinity rules inherited from the prod profile, ensure that the pods of each component are distributed across the Kubernetes cluster nodes. MySQL and OpenSearch are configured to replicate data among replicas to avoid data loss.

The ha-oci-ccm.yaml file configures a highly available Verrazzano installation on OCNE.

The ha-ext-lb.yaml file configures a highly available Verrazzano installation using external load balancers.

Fault detection is managed natively by using Kubernetes Services and Istio VirtualServices that detect failed pods and route traffic to the remaining replicas.

MySQL and OpenSearch provide reliable failover points for the replicated data.

The result of these measures would be no loss of service if a cluster node became unavailable. For more information regarding node failure and recovery, read the Node Failure Guide.

When using the ha.yaml file, consider the following:

It does not ensure a fault-tolerant environment. Your applications still must be designed and implemented as highly available.
Running additional replicas of components will increase resource requirements. At least four CPUs, 100 GB disk storage, and 64 GB RAM available on the Kubernetes worker nodes is required.
Additional customizations may be required for your environment, including other customizations described in individual sections.

For the expected behavior of the MySQL Component in a highly available environment, see Customize Keycloak and MySQL.

Follow these best practices for a highly available Verrazzano installation:

Size your Kubernetes cluster according to your node failure tolerance and workload requirements.
Set the default Storage Class to one with a VolumeBindingMode of WaitForFirstConsumer. This is important for being able to recover from an Availability Domain or zone failure.
Set the replica counts to values that correspond to your node failure tolerance.

To install the example high availability configuration using the Verrazzano CLI:

$ vz install -f https://raw.githubusercontent.com/verrazzano/verrazzano/master/examples/ha/ha.yaml

Using the Verrazzano CLI, install the example high availability configuration on OCNE as follows:

 $ vz install -f https://raw.githubusercontent.com/verrazzano/verrazzano/master/examples/ha/ha-oci-ccm.yaml

Using the Verrazzano CLI, install the example high availability configuration with external load balancers as follows:

$ vz install -f https://raw.githubusercontent.com/verrazzano/verrazzano/master/examples/ha/ha-ext-lb.yaml

Upgrade recommendations

An OKE in-place upgrade scales the cluster to N-1 nodes, where N is the original number of nodes, so you must scale the cluster back to N. This will start a new replacement node using the new node pool. For more information on in-place upgrades, see Performing an In-Place Worker Node Kubernetes Upgrade by Updating an Existing Node Pool.