Verrazzano Enterprise Container Platform – Troubleshooting

Docs: Application Deployment

Mon, 01 Jan 0001 00:00:00 +0000

During application deployment, the oam-kubernetes-runtime and verrazzano-application-operator cooperate through the generation and update of Kubernetes resources. The oam-kubernetes-runtime processes the ApplicationConfiguration and Component resources provided by the user and generates workload and Trait resources. The verrazzano-application-operator processes Verrazzano specific workload and Trait resources. These are then used to generate additional child and related resources.

Troubleshooting application deployments should follow three general steps:

Review the status of the oam-kubernetes-runtime and verrazzano-application-operator operator pods.
Review the logs of the oam-kubernetes-runtime and verrazzano-application-operator operator pods.
Review the resources generated by the oam-kubernetes-runtime and the verrazzano-application-operator.

Review oam-kubernetes-runtime operator status

For application deployment to succeed, the oam-kubernetes-runtime pod must have a status of Running.

Use the following command to get the pod status:

$ kubectl get pods \
    -n verrazzano-system \
    -l app.kubernetes.io/name=oam-kubernetes-runtime

If the pod status is not Running, then see the instructions for reviewing the oam-kubernetes-runtime pod logs.

Review verrazzano-application-operator operator status

For application deployment to succeed, the verrazzano-application-operator pod must have a status of Running.

Use the following command to get the pod status:

$ kubectl get pods \
    -n verrazzano-system \
    -l app=verrazzano-application-operator

If the pod status is not Running, then see the instructions for reviewing the verrazzano-application-operator logs.

Review oam-kubernetes-runtime operator logs

Review the oam-kubernetes-runtime pod logs for any indication that pod startup or the generation of workloads or traits has failed.

Use the following command to get the logs:

$ kubectl logs \
    -n verrazzano-system \
    -l app.kubernetes.io/name=oam-kubernetes-runtime

Review verrazzano-application-operator logs

Review the verrazzano-application-operator logs for any indication that pod startup or resource generation has failed.

Use the following command to get the logs:

$ kubectl logs \
    -n verrazzano-system \
    -l app=verrazzano-application-operator

Review generated workload resources

The processing of a Component reference within an ApplicationConfiguration results in the generation of workloads. For example, a referenced Component might result in the generation of a VerrazzanoHelidonWorkload workload resource. In turn, the VerrazzanoHelidonWorkload workload resource will be processed and result in the generation of related Deployment and Service resources.

If the expected workload resource, for example VerrazzanoHelidonWorkload, is missing, then review the oam-kubernetes-runtime logs. If the expected related resources, for example Deployment or Service, are missing, then review the verrazzano-application-operator logs.

The following commands are examples of checking for the resources related to a VerrazzanoHelidonWorkload deployment:

$ kubectl get -n hello-helidon verrazzanohelidonworkload hello-helidon-workload
$ kubectl get -n hello-helidon deployment hello-helidon-deployment
$ kubectl get -n hello-helidon service hello-helidon-deployment

Review generated Trait resources

The processing of traits embedded with an ApplicationConfiguration results in the generation of Trait resources. For example, an IngressTrait embedded within an ApplicationConfiguration will result in the generation of an IngressTrait resource. In turn, the IngressTrait resource will be processed and result in the generation of related Certificate, Gateway, and VirtualService resources.

If the expected Trait resource, for example IngressTrait, is missing, then review the oam-kubernetes-runtime logs. If the expected related resources, for example Certificate, Gateway, and VirtualService, are missing, then review the verrazzano-application-operator logs.

The following commands are examples of checking for the resources related to an IngressTrait:

$ kubectl get -n hello-helidon ingresstrait hello-helidon-ingress
$ kubectl get -n istio-system Certificate hello-helidon-hello-helidon-appconf-cert
$ kubectl get -n hello-helidon gateway hello-helidon-hello-helidon-gw
$ kubectl get -n hello-helidon virtualservice hello-helidon-ingress-rule-0-vs

Check for RBAC privilege issues

The use of generic Kubernetes resources as workloads and traits can result in deployment failures if privileges are insufficient. In this case, the oam-kubernetes-runtime logs will contain errors containing the term forbidden.

The following command shows how to query for this type of failure message:

$ kubectl logs \
    -n verrazzano-system \
    -l app.kubernetes.io/name=oam-kubernetes-runtime | grep forbidden

Check resource owners

Kubernetes maintains the child to parent relationship within metadata fields.

The following example returns the parent of the IngressTrait, named hello-helidon-ingress, in the hello-helidon namespace:

$ kubectl get IngressTrait \
    -n hello-helidon hello-helidon-ingress \
    -o jsonpath='{range .metadata.ownerReferences[*]}{.name}{"\n"}{end}'

The results of this command can help identify the lineage of a given resource.

Some resources also record the related resources affected during their processing. For example, when processed, an IngressTrait will create related Gateway, VirtualService, and Certificate resources.

The following command is an example of how to obtain the related resources of an IngressTraits:

$ kubectl get IngressTrait \
    -n hello-helidon hello-helidon-ingress \
    -o jsonpath='{range .status.resources[*]}{.kind}: {.name}{"\n"}{end}'

The results of this command can help identify which other resources, the given resource affected.

Docs: Diagnostic Tools

Mon, 01 Jan 0001 00:00:00 +0000

Docs: Fluentd

Mon, 01 Jan 0001 00:00:00 +0000

Issue: Fluentd’s inability to push logs to OpenSearch due to a permission issue when SELinux is in enforcing mode.

If you are unable to see logs in OpenSearch, it may be due to a permission issue in Fluentd when SELinux is in enforcing mode, which prevents Fluentd from pushing the logs to OpenSearch.

To troubleshoot the Fluentd permission issue, follow these steps:

Check Fluentd pod logs.
- Identify the Fluentd pod related to the Verrazzano installation.
- View the logs of the Fluentd pod using the following command:
```
$ kubectl logs <fluentd-pod-name> -n verrazzano-system
```

Check for an error description.

Look for an error message in the Fluentd pod logs.
If you see an error with the following description, then follow the instructions in Step 3. Resolve permission issue.

unexpected error error_class=Errno::EACCES error="Permission denied @ rb_sysopen - /var/log/vz-fluentd-containers.log.pos", it indicates a permission issue. Fluentd doesn't have enough privilege to write `.pos` file.

Resolve permission issue.
- The issue occurs when SELinux is in enforcing mode on the worker nodes and Fluentd does not have the appropriate SELinux context to have read/write access to the logs directory.
- Check if SELinux is in enforcing mode by running this command on the worker nodes:
```
$ sudo getenforce
```
- If SELinux is in enforcing mode, then follow these steps:
  - To fix this issue, you need to override the default SELinux option in the Verrazzano Custom Resource.
  - Edit the Verrazzano CR and add the necessary SELinux options to provide read/write access to the logs directory in the Fluentd section. For example:
    spec: components: fluentd: overrides: - values: seLinuxOptions: type: spc_t
    NOTE
    The spc_t SELinux context is very permissive in that it gives the pod full access to the node on which it is running. If you don’t want the Fluentd pod to have the spc_t context, consider creating a custom SELinux context type with only the required privileges on all the worker nodes instead of using spc_t.
Verification.
- Verify that Fluentd is able to read and push the logs to OpenSearch by reviewing the Fluentd logs.
- Verify that the logs are visible in OpenSearch by accessing the OpenSearch dashboard or using relevant search queries.

Docs: Multicluster Verrazzano

Mon, 01 Jan 0001 00:00:00 +0000

This document describes some common problems you might encounter when using multicluster Verrazzano, and how to troubleshoot them.

If you created multicluster resources in the admin cluster, and specified a placement value in a managed cluster, then those resources will get created in that managed cluster. If they do not get created in the managed cluster, then use the following steps to troubleshoot:

Verify that the managed cluster is registered correctly and can connect to the admin cluster.
Verify that the VerrazzanoProject for the resource’s namespace, also has a placement in that managed cluster.
Check the multicluster resource’s status field on the admin cluster to know what the status of that resource is on each managed cluster to which it is targeted.

If you update the DNS of the admin cluster and notice that the managed cluster status is unavailable in the Rancher console, along with the error x509: certificate is valid for <rancher new url>, not <rancher old url> seen in the cattle-cluster-agent (Rancher Agent) logs on the managed cluster, then re-register the managed cluster, as described here.

Verify managed cluster registration and connectivity

You can verify that a managed cluster was successfully registered with an admin cluster by viewing the corresponding VerrazzanoManagedCluster (VMC) resource on the admin cluster. For example, to verify that a managed cluster named managed1 was successfully registered:

# on the admin cluster
$ kubectl get verrazzanomanagedcluster managed1 \
    -n verrazzano-mc \
    -o yaml

Partial sample output from the previous command.

  status:
    conditions:
    - lastTransitionTime: "2021-06-22T21:03:27Z"
      message: Ready
      status: "True"
      type: Ready
    lastAgentConnectTime: "2021-06-22T21:06:04Z"
    ... other fields ...

Check the lastAgentConnectTime in the status of the VMC resource. This is the last time at which the managed cluster connected to the admin cluster. If this value is not present, or is not recent (within the last three minutes), then the managed cluster named managed1 cannot successfully connect to the admin cluster. This could be due to several reasons:

The managed cluster registration process step of applying the registration YAML on the managed cluster, was not completed. For the complete setup instructions, see here.
The managed cluster does not have network connectivity to the admin cluster. The managed cluster will attempt to connect to the admin cluster at regular intervals, and any errors will be reported in the verrazzano-application-operator pod’s log on the managed cluster. View the logs using the following command:
```
# on the managed cluster
$ kubectl logs \
    -n verrazzano-system \
    -l app=verrazzano-application-operator
```

If these logs reveal that there is a connectivity issue, then in the case of an installation that includes Rancher on the admin cluster, there may have been a problem with Verrazzano pushing registration details or updates to the managed cluster. Try exporting and applying the registration manifest to the managed cluster as shown:

# on the admin cluster
     kubectl get secret \
     -n verrazzano-mc verrazzano-cluster-managed1-manifest \
     -o jsonpath={.data.yaml} | base64 --decode > register.yaml

# on the managed cluster
     kubectl apply -f register.yaml

NOTE: If your installation disabled Rancher on the admin cluster, then check the admin cluster Kubernetes server address that you provided during registration and ensure that it is correct, and that it is reachable from the managed cluster. If it is incorrect, then you will need to repeat the managed cluster registration process described in the setup instructions here.

Verify VerrazzanoProject placement

For Verrazzano to create an application namespace in a managed cluster, that namespace must be part of a VerrazzanoProject that:

Includes that namespace.
Has a placement value that includes that managed cluster.

View the details of the project that corresponds to your application’s namespace. In the example command that follows, the project name is assumed to be myproject. All projects are expected to be created in the verrazzano-mc namespace.

# on the admin cluster
$ kubectl get verrazzanoproject myproject \
    -n verrazzano-mc \
    -o yaml

The following partial sample output is for a project that will result in the namespace mynamespace being created on the managed cluster managed1.

spec:
  placement:
    clusters:
    - name: managed1
  template:
    namespaces:
    - metadata:
        name: mynamespace
....other fields....

Check the multicluster resource status

On the admin cluster, each multicluster resource’s status field is updated with the status of the underlying resource on each managed cluster in which it is placed.

The following example command shows how to view the status of a MultiClusterApplicationConfiguration named myapp, in the namespace mynamespace, that has a placement value that includes the managed cluster managed1.

$ kubectl get multiclusterapplicationconfiguration myapp \
    -n mynamespace \
    -o yaml

The status of the underlying resource in each cluster specified in the placement is shown in the following partial sample output.

  status:
    clusters:
    - lastUpdateTime: "2021-06-22T21:05:04Z"
      message: OAM Application Configuration created
      name: managed1
      state: Succeeded
    conditions:
    - lastTransitionTime: "2021-06-22T21:03:58Z"
      message: OAM Application Configuration created
      status: "True"
      type: DeployComplete
    state: Succeeded

The status message contains additional information on the operation’s success or failure.

Re-register the managed cluster

Perform the following steps to re-register the managed cluster with the admin cluster. The cluster against which to run the command is indicated in each code block.

On the admin cluster, export the register YAML file newly created on the admin cluster to re-register the managed cluster.
```
# On the admin cluster
$ kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
    get secret verrazzano-cluster-managed1-manifest \
    -n verrazzano-mc \
    -o jsonpath={.data.yaml} | base64 --decode > register_new.yaml
```
On the managed cluster, apply the registration file exported in the previous step.
```
# On the managed cluster
$ kubectl --kubeconfig $KUBECONFIG_MANAGED1 --context $KUBECONTEXT_MANAGED1 \
    apply -f register_new.yaml

# After the command succeeds, you may delete the register_new.yaml file
$ rm register_new.yaml
```
On the admin cluster, run kubectl patch clusters.management.cattle.io to trigger redeployment of the Rancher agent on the managed cluster.
```
# On the admin cluster
$ kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
    get clusters.management.cattle.io

# Sample output
NAME      AGE
c-mzb2h   4h48m
local     4h56m

$ kubectl --kubeconfig $KUBECONFIG_ADMIN --context $KUBECONTEXT_ADMIN \
    patch clusters.management.cattle.io <the managed cluster name from the above output> \
    -p '{"status":{"agentImage":"dummy"}}' --type merge

# Sample output
cluster.management.cattle.io/c-mzb2h patched
```

Docs: MySQL

Mon, 01 Jan 0001 00:00:00 +0000

There are known issues that can occur with MySQL. The Verrazzano platform operator will automatically detect each of the described issues and perform actions to repair them. The operator initiates a repair within a few minutes of detecting an issue.

The following sections are provided in the event that a manual repair of an issue is required.

MySQL pod stuck Terminating

A MySQL pod may get stuck in a terminating state. This may occur while upgrading the nodes of a Kubernetes cluster.

Here is an example of what this condition looks like. All the pod containers are terminated, but the pod never finishes terminating.

$ kubectl get pods -n keycloak -l component=mysqld
NAME      READY   STATUS        RESTARTS   AGE
mysql-0   0/3     Terminating   0          60m

You can repair this issue by restarting the mysql-operator pod.

$ kubectl delete pod -l name=mysql-operator -n mysql-operator

MySQL pod waiting for readiness gates

The mysql StatefulSet may get stuck while waiting to reach the ready state. This will occur when one or more MySQL pods not meeting its set of ReadinessGates.

Here is an example of what this condition looks like.

$ kubectl describe pods -n keycloak -l component=mysqld

# Excerpt from the command output
Readiness Gates:
  Type                          Status
  mysql.oracle.com/configured   False
  mysql.oracle.com/ready        True

You can repair this issue by restarting the mysql-operator pod.

$ kubectl delete pod -l name=mysql-operator -n mysql-operator

MySQL router pod in CrashLoopBackOff state

Here is an example of what this condition looks like.

$ kubectl get pods -n keycloak -l component=mysqlrouter
NAME                            READY   STATUS             RESTARTS   AGE
mysql-router-757595f6c5-pdgxj   1/2     CrashLoopBackOff   0          109m

You can repair this issue by deleting the pod that is in the CrashLoopBackOff state.

$ kubectl delete pod -n keycloak mysql-router-757595f6c5-pdgxj

InnoDBCluster object stuck Terminating

This condition has been observed to occur on an uninstallation of Verrazzano.

Here is an example of what this condition looks like.

$ kubectl get InnoDBCluster -n keycloak
NAME    STATUS    ONLINE   INSTANCES   ROUTERS   AGE
mysql   OFFLINE   0        1           1         7m51s

You can repair this issue by restarting the mysql-operator pod.

$ kubectl delete pod -l name=mysql-operator -n mysql-operator

Verrazzano Enterprise Container Platform – Troubleshooting

Docs: Application Deployment

Review oam-kubernetes-runtime operator status

Review verrazzano-application-operator operator status

Review oam-kubernetes-runtime operator logs

Review verrazzano-application-operator logs

Review generated workload resources

Review generated Trait resources

Check for RBAC privilege issues

Check resource owners

Check related resources

Docs: Diagnostic Tools

Docs: Fluentd

NOTE

Docs: Multicluster Verrazzano

Verify managed cluster registration and connectivity

Verify VerrazzanoProject placement

Check the multicluster resource status

Re-register the managed cluster

Docs: MySQL

MySQL pod stuck Terminating

MySQL pod waiting for readiness gates

MySQL router pod in CrashLoopBackOff state

InnoDBCluster object stuck Terminating