Verrazzano Enterprise Container Platform – Troubleshoot and FAQ

Docs: Diagnostic Tools

Mon, 01 Jan 0001 00:00:00 +0000

Docs: Fluentd Issues

Mon, 01 Jan 0001 00:00:00 +0000

Issue: Fluentd’s inability to push logs to OpenSearch due to a permission issue when SELinux is in enforcing mode.

If you are unable to see logs in OpenSearch, it may be due to a permission issue in Fluentd when SELinux is in enforcing mode, which prevents Fluentd from pushing the logs to OpenSearch.

To troubleshoot the Fluentd permission issue, follow these steps:

Check Fluentd pod logs.
- Identify the Fluentd pod related to the Verrazzano installation.
- View the logs of the Fluentd pod using the following command:
```
$ kubectl logs <fluentd-pod-name> -n verrazzano-system
```

Check for an error description.

Look for an error message in the Fluentd pod logs.
If you see an error with the following description, then follow the instructions in Step 3. Resolve permission issue.

unexpected error error_class=Errno::EACCES error="Permission denied @ rb_sysopen - /var/log/vz-fluentd-containers.log.pos", it indicates a permission issue. Fluentd doesn't have enough privilege to write `.pos` file.

Resolve permission issue.
- The issue occurs when SELinux is in enforcing mode on the worker nodes and Fluentd does not have the appropriate SELinux context to have read/write access to the logs (/var/log/) directory.
- Check if SELinux is in enforcing mode by running this command on the worker nodes:
```
$ sudo getenforce
```
  If SELinux is in enforcing mode, then follow the advice found here.
Verification.
- Verify that Fluentd is able to read and push the logs to OpenSearch by reviewing the Fluentd logs.
- Verify that the logs are visible in OpenSearch by accessing the OpenSearch dashboard or using relevant search queries.

Docs: MySQL Issues

Mon, 01 Jan 0001 00:00:00 +0000

There are known issues that can occur with MySQL. The Verrazzano platform operator will automatically detect each of the described issues and perform actions to repair them. The operator initiates a repair within a few minutes of detecting an issue.

The following sections are provided in the event that a manual repair of an issue is required.

MySQL pod stuck Terminating

A MySQL pod may get stuck in a terminating state. This may occur while upgrading the nodes of a Kubernetes cluster.

Here is an example of what this condition looks like. All the pod containers are terminated, but the pod never finishes terminating.

$ kubectl get pods -n keycloak -l component=mysqld
NAME      READY   STATUS        RESTARTS   AGE
mysql-0   0/3     Terminating   0          60m

You can repair this issue by restarting the mysql-operator pod.

$ kubectl delete pod -l name=mysql-operator -n mysql-operator

MySQL pod waiting for readiness gates

The mysql StatefulSet may get stuck while waiting to reach the ready state. This will occur when one or more MySQL pods not meeting its set of ReadinessGates.

Here is an example of what this condition looks like.

$ kubectl describe pods -n keycloak -l component=mysqld

# Excerpt from the command output
Readiness Gates:
  Type                          Status
  mysql.oracle.com/configured   False
  mysql.oracle.com/ready        True

You can repair this issue by restarting the mysql-operator pod.

$ kubectl delete pod -l name=mysql-operator -n mysql-operator

MySQL router pod in CrashLoopBackOff state

Here is an example of what this condition looks like.

$ kubectl get pods -n keycloak -l component=mysqlrouter
NAME                            READY   STATUS             RESTARTS   AGE
mysql-router-757595f6c5-pdgxj   1/2     CrashLoopBackOff   0          109m

You can repair this issue by deleting the pod that is in the CrashLoopBackOff state.

$ kubectl delete pod -n keycloak mysql-router-757595f6c5-pdgxj

InnoDBCluster object stuck Terminating

This condition has been observed to occur on an uninstallation of Verrazzano.

Here is an example of what this condition looks like.

$ kubectl get InnoDBCluster -n keycloak
NAME    STATUS    ONLINE   INSTANCES   ROUTERS   AGE
mysql   OFFLINE   0        1           1         7m51s

You can repair this issue by restarting the mysql-operator pod.

$ kubectl delete pod -l name=mysql-operator -n mysql-operator

Docs: OCNE Cluster Creation Issues

Mon, 01 Jan 0001 00:00:00 +0000

If you encounter a problem with the OCNE clusters created in the console (using the clusterAPI component), here are some techniques you can use to diagnose and solve the issue.

Assess the condition of the clusterAPI component

You can gather information about the clusterAPI component by reviewing its log files and by checking the status of clusterAPI custom resources on the admin cluster. In the console, the admin cluster is the local cluster.

Review clusterAPI log files

The clusterAPI component creates four clusterAPI pods within the verrazzano-capi namespace and each pod generates a separate log file. Review the log files to determine the cause of the issue.

Where nnnn is the unique ID for the pod.

To view the clusterAPI controller log file, use the following command:

$ kubectl logs -n verrazzano-capi capi-controller-manager-<nnnn>

To view the clusterAPI OCI provider log file, use the following command:

$ kubectl logs -n verrazzano-capi capoci-controller-manager-<nnnn>

To view the clusterAPI control plane provider log file, use the following command:

$ kubectl logs -n verrazzano-capi capi-ocne-control-plane-controller-manager-<nnnn>

To view the clusterAPI bootstrap provider log file, use the following command:

$ kubectl logs -n verrazzano-capi capi-ocne-bootstrap-controller-manager-<nnnn>

Review clusterAPI Kubernetes custom resources

On the admin cluster, use kubectl to check the status of the clusterAPI custom resource.

To see the status of the clusterAPI Kubernetes custom resource cluster, use the following command:

$ kubectl get clusters.cluster.x-k8s.io -A

The cluster status should be Provisioned.

To see the status of the clusterAPI Kubernetes custom resource machine, use the following command:

$ kubectl get machines -A

The status of each machine should be Running.

Verify the status of OCI resources

When creating clusters, Verrazzano creates the following resources in the associated OCI compartment:

If one of these resources was not created, or was created but is now experiencing issues, it can affect the rest of the cluster.

You can check the status of the required OCI resources in the OCI console or by reviewing the log files for the clusterAPI controller.

Causes may include:

OCI resource limits were reached: Cluster creation will fail if it needs to create more resources than allowed by your OCI tenancy. Check your OCI service limits to see if you exceeded its limits. If you need to request a service limit increase, see Requesting a Service Limit Increase.
Connectivity issues: Make sure your virtual cloud network is configured properly.
The network load balancer is in a critical state: The network load balancer may temporarily enter a critical state during initial cluster creation until the Kubernetes API server is up. If it remains in a critical state, then one of the following issues may have occurred:
- Traffic between the network load balancer and the OCNE control plane node is blocked over port 6443.
- The OCNE API server did not start, possibly because OCNE dependencies failed to install.
Node creation failed:
- The OCI credentials are invalid: Check for errors under Kubernetes events in the namespace where cluster objects are present or in the OCI provider log file.
- The Image ID used to deploy templates is invalid.
- Worker nodes only: Worker nodes began provisioning after the control node plane entered the Running state.
  - If OCNE dependencies failed to install on the control plane node, worker nodes remain in the Pending state and do not get created. Check the cloud-init log files on the control plane nodes to determine the cause.

Also, you should confirm that pods are running on the workload cluster. From the console, download the kubeconfig file for the workload cluster and run the following command:

$ kubectl --kubeconfig <workload-cluster-kubeconfig> get pods -A

If the status of any of the pods is not Running, run the following command to identify the error:

$ kubectl describe pod <pod-name>

NOTE

You can also refer to the Cluster API documentation for generic information on troubleshooting and known issues. Be aware that because the clusterAPI component is specific to Verrazzano, certain issues or solutions may not apply.

Docs: FAQ

Mon, 01 Jan 0001 00:00:00 +0000

Enable Google Chrome to accept self-signed Verrazzano certificates

There are some installation scenarios where Verrazzano generates SSL certificates that are not trusted by browsers:

The development (dev) profile installation, which uses its own self-signed CA to issue certificates.
Using the Let’s Encrypt Staging authority, which uses untrusted CAs to sign certificates.

These are typical development or testing scenarios, not recommended for production. When accessing Verrazzano sites using these certificates, some browsers like Firefox let you manually accept these certificates. However, Google Chrome now prevents users from being able to accept self-signed certificates by default. This will prevent you from accessing Verrazzano consoles that are using untrusted certificates.

When this occurs, while trying to access Verrazzano services, you will see an error message like the following:

opensearch.vmi.system.default.129.153.98.156.nip.io normally uses encryption to protect your information. When Chrome tried to connect to opensearch.vmi.system.default.129.153.98.156.nip.io this time, the website sent back unusual and incorrect credentials

You can choose to import the certificate into your local trust chain, but this will have to be done for each Verrazzano instance. From a security perspective, this is not recommended.

As an alternative, you can enter a secret passphrase in Chrome to enable it to prompt you to accept these certificates, by doing the following:

When you see an error such as the one shown previously, when the browser window has the keyboard focus, enter the phrase thisisunsafe.
Reload the site.
Chrome will prompt you to accept the certificate.

NOTE: This should be used only when accessing sites that are known to be safe, such as in this situation.

Verrazzano Enterprise Container Platform – Troubleshoot and FAQ

Docs: Diagnostic Tools

Docs: Fluentd Issues

Docs: MySQL Issues

MySQL pod stuck Terminating

MySQL pod waiting for readiness gates

MySQL router pod in CrashLoopBackOff state

InnoDBCluster object stuck Terminating

Docs: OCNE Cluster Creation Issues

Assess the condition of the clusterAPI component

Verify the status of OCI resources

NOTE

Docs: FAQ

Enable Google Chrome to accept self-signed Verrazzano certificates

Related articles