Verrazzano Enterprise Container Platform – Analyze Tool Findings and Reports

Docs: Cluster API Resource Not Ready

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that a Cluster API cluster.cluster.x-k8s.io resource was not in a ready state. A ready cluster resource will have a status with condition types all set to True.

Steps

Review the logs in the verrazzano-capi namespace for additional details as to why the cluster resource is not ready.

Docs: External DNS Configuration

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that there was an issue with the DNS configuration that is related to credentials. It could be an authorization issue or incorrectly supplied credentials.

Steps

Review the OCI credentials provided during DNS configuration.
Check the permission for OCI credentials or instance_principal.

OCI DNS

Docs: Image Pull Back Off

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that there were pods that had issues due to failures to pull an image or images.

The analysis was not able to identify a specific root cause, however, it might have supplied data that is related to the failures.

Steps

Review the analysis data. At a minimum, it will indicate which pods are being impacted and might give other clues on the root cause.
If the service is experiencing an outage, then consult the specific service status page. For common service status pages, see Related information.

Docs: Image Pull Not Found

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that there were pods which had issues due to failures to pull an image or images where the root cause was that the image was not found.

Steps

Review the analysis data; it enumerates the pods and related messages regarding which images had this issue.
Confirm that the image name, digest, and tag are correctly specified.

Kubernetes Troubleshooting

Docs: Image Pull Rate Limit

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that there were pods which had issues due to failures to pull an image or images.

The root cause was rate limit exceeded errors while pulling images.

Steps

Review the analysis data; it enumerates the pods and related messages regarding which images had this issue.
The detailed messages might provide specific instructions for the registry that is involved. For example, it might provide a link to instructions on how to increase the limit.

Increase Rate Limits

Docs: Image Pull Service Issue

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that there were pods which had issues due to failures to pull an image or images where the root cause was that the service was not available.

The service might be unreachable or might be incorrect.

Steps

Review the analysis data; it enumerates the pods and related messages about which images had this issue.
Confirm that the registry for the image is correct.
The messages might identify a connectivity issue.
If the service is experiencing an outage, then consult the specific service status page. For common service status pages, see Related information.

Docs: Ingress Controller Invalid Shape

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that the Verrazzano installation failed while installing the NGINX Ingress Controller.

The root cause appears to be that Verrazzano custom resource provided an invalid shape for Oracle Cloud Infrastructure load balancer.

Steps

Review the messages from the supporting details for the allowed shape for Oracle Cloud Infrastructure load balancer.
Refer to the Oracle Cloud Infrastructure documentation, Load Balancer Management.

Customize Load Balancers on OKE

Docs: Ingress Controller Load Balancer Service Limit Reached

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that the Verrazzano installation failed while installing the NGINX Ingress Controller.

The root cause appears to be that the load balancer service limit has been reached.

Steps

Review the messages from the supporting details for the exact limits, and delete unused load balancers.
If available, use a different load balancer shape. See Customizing Ingress.
Refer to the Oracle Cloud Infrastructure documentation on Service Limits.

Docs: Ingress Controller No Load Balancer IP

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that the Verrazzano installation failed while installing the NGINX Ingress Controller.

The root cause appears to be that the load balancer is either missing or unable to set the ingress IP address on the NGINX Ingress service.

Steps

Refer to the platform-specific environment setup for your platform here.

Docs: Ingress Controller Oracle Cloud Infrastructure IP Limit Exceeded

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that the Verrazzano installation failed while installing the NGINX Ingress Controller.

The root cause appears to be that an Oracle Cloud Infrastructure IP non-ephemeral address limit has been reached.

Steps

Review the messages from the supporting details for the exact limit.
Refer to the Oracle Cloud Infrastructure documentation related to managing IP Addresses.

Public IP Addresses

Docs: Install Ingress Controller Failure

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that the Verrazzano installation has failed related to the NGINX Ingress Controller, however, it was unable to isolate the specific root cause.

Steps

Review the analysis data, which might help identify the issue.

Docs: Install, Upgrade, or Uninstall Failure

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected a failure while installing, upgrading, or uninstalling Verrazzano; however, it did not isolate the exact reason for the failure.

Steps

Review the analysis data, which can help identify the issue. Upon failure, a bug-report archive file will be generated automatically and the filepath will be printed to the console.

Docs: Insufficient CPU

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that there were nodes reporting insufficient CPU.

Steps

Review the detailed analysis data to identify the specific nodes involved.
Review the nodes to determine why they do not have sufficient CPU.

a. Are the nodes sized correctly for the workload?
- For the minimum resources required for installing Verrazzano, see the Prerequisites.
- Make sure to take into account the resource guidelines of all the applications that you are deploying.
b. Is something unexpected running on the nodes or consuming more CPU than expected?

Docs: Insufficient Memory

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that there were nodes reporting insufficient memory.

Steps

Review the detailed analysis data to identify the specific nodes involved.
Review the nodes to determine why they do not have sufficient memory.

a. Are the nodes sized correctly for the workload?
- For the minimum resources required for installing Verrazzano, see the Prerequisites.
- Make sure to take into account the resource guidelines of all the applications that you are deploying.
b. Is something unexpected running on the nodes or consuming more memory than expected?

Docs: Istio Ingress Gateway No External IP

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that the Verrazzano installation failed while installing the Istio ingress gateway.

The root cause appears to be that the load balancer is either missing or unable to set the external IP address on the Istio ingress gateway service.

Steps

Refer to the platform-specific environment setup for your platform here.

Docs: Istio Ingress Load Balancer Creation Issue

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that the Verrazzano installation failed while creating the load balancer for the Istio ingress gateway.

The root cause was that, while creating the public load balancer, there were no public subnets available.

Steps

Refer to the platform-specific environment setup for your platform here.

Docs: Keycloak Data Migration Failure

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that the Verrazzano upgrade failed when migrating Keycloak data from the existing legacyDB to the new InnoDB.

Steps

Verify whether the dump-claim PVC exists and is bound to the PV.
```
$ k get pvc -n keycloak dump-claim

# Sample Output
NAME         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
dump-claim   Bound    pvc-c246d7c4-3041-4164-8c1e-744dda805686   2Gi        RWO            standard       12m
```
```
$ kubectl get pv

# Sample Output
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                 STORAGECLASS   REASON   AGE
pvc-c246d7c4-3041-4164-8c1e-744dda805686   2Gi        RWO            Retain           Bound    keycloak/dump-claim   standard                49m
```
Get the mysql-root password from the mysql or mysql-cluster-secret secret in the keycloak namespace. The password is required to access the MySQL server as a root user in the following steps.
```
$ kubectl get secret -n keycloak mysql -o jsonpath='{.data.mysql-root-password}' | base64 --decode

# Sample Output
lvYwPJjwFB
```
Create a load-dump pod to help in migrating the previous Keycloak database data into the new database instance.
```
$ kubectl apply -f - <<EOF
  apiVersion: v1
  kind: Pod
  metadata:
    name: my-load-dump
    namespace: keycloak
    labels:
      job-name: load-dump
  spec:
    containers:
    - name: mysqlsh-load-dump
      image: ghcr.io/verrazzano/mysql-server:8.0.32
      volumeMounts:
      - mountPath: /var/lib/dump
        name: keycloak-dump
    volumes:
    - name: keycloak-dump
      persistentVolumeClaim:
        claimName: dump-claim
EOF
```
Start a shell session inside the pod, and then run these commands:
```
$ kubectl exec -n keycloak my-load-dump -it -- /bin/bash
```
a. Fix the dump directory permission.
```
$ chown -R 27:27 /var/lib/dump
```
b. Check if the MySQL server is running. If it is not running, check if the MySQL pods in the keycloak namespace are ready, and then check again.
```
$ mysqladmin ping -h"mysql.keycloak.svc.cluster.local" -p{{ .RootPassword }}

# Sample Output
mysqld is alive
```
c. Migrate the Keycloak data.
```
$ mysqlsh -u root -p{{ .RootPassword }} -h mysql.keycloak.svc.cluster.local -e 'util.loadDump("/var/lib/dump/dump", {includeSchemas: ["keycloak"], includeUsers: ["keycloak"], loadUsers: true})'
```
```
# Sample Output
Loading DDL, Data and Users from '/var/lib/dump/dump' using 4 threads.
Opening dump...
Target is MySQL 8.0.32. Dump was produced from MySQL 8.0.29
Scanning metadata - done       
Checking for pre-existing objects...
Executing common preamble SQL
Executing DDL - done         
Executing user accounts SQL...
NOTE: Skipping CREATE/ALTER USER statements for user 'root'@'%'
NOTE: Skipping CREATE/ALTER USER statements for user 'root'@'localhost'
NOTE: Skipping GRANT statements for user 'root'@'%'
NOTE: Skipping GRANT statements for user 'root'@'localhost'
Executing view DDL - done       
Starting data load
4 thds loading \ 100% (159.79 KB / 159.79 KB), 0.00 B/s, 33 / 93 tables done
Recreating indexes - done       
Executing common postamble SQL                                              
93 chunks (1.31K rows, 159.79 KB) for 93 tables in 1 schemas were loaded in 3 sec (avg throughput 159.79 KB/s)
0 warnings were reported during the load.
```
In the db-migration secret, add a db-migrated field and set its value to true (base64 encoded). This will notify the Verrazzano platform operator that the Keycloak data was migrated manually.
```
$ k edit secret -n keycloak db-migration

# Sample Output
data:
  db-migrated: dHJ1ZQ==        # true base64 encoded is dHJ1ZQ==
  database-dumped: dHJ1ZQ==
  deployment-found: dHJ1ZQ==
  ...
```
Delete the load-dump pod.
```
$ kubectl delete pod -n keycloak my-load-dump
```

Docs: NGINX Ingress Controller Load Balancer Creation Issue

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that the Verrazzano installation failed while creating the load balancer for the NGINX Ingress Controller.

The root cause was that, while creating the public load balancer, there were no public subnets available.

Steps

Refer to the platform-specific environment setup for your platform here.

Docs: Pending Pods

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that there were pods which were in a pending state without detecting other specific issues related to them.

Steps

Review the analysis data. At a minimum, this should indicate which pods are being impacted and it might give other clues on the root cause.

Kubernetes Troubleshooting

Docs: Problem Pods

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that there were pods which were not in a running, succeeded, or pending state.

The analysis was not able to determine a specific root cause, however, it might have supplied data that is related to the pods in question. The root cause might be obvious from the supporting data, but the analysis tool isn’t isolating the specific scenario yet.

Steps

Review the analysis data. At a minimum, it should indicate which pods are being impacted and it might give other clues on the root cause.

Kubernetes Troubleshooting

Docs: Rancher Issues

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that one or more Rancher resources are in a failure state. See the following sections for details:

Applications

app.catalog.cattle.io
Analysis detected an issue with the deployment of a Helm chart.

Catalogs

catalog.management.cattle.io
Analysis detected an issue with the configuration of a catalog of application templates.

Cluster Drivers

kontainerdriver.management.cattle.io
Analysis detected that a KontainerDriver resource was not in a ready state. A ready KontainerDriver resource will have a status with condition types Active, Downloaded, and Installed set to True.

Fleet Bundle

bundle.fleet.cattle.io
Analysis detected an issue with the status of a Fleet bundle. The Fleet bundle resources are automatically created when a Fleet Git repository resource is created.

Fleet Bundled Deployment

bundledeployment.fleet.cattle.io
Analysis detected an issue with the status of a Fleet bundled deployment.

Fleet Clusters

cluster.fleet.cattle.io
Analysis detected an issue with the status of a cluster managed with Fleet.

Fleet Cluster Groups

clustergroup.fleet.cattle.io
Analysis detected an issue with the status of a cluster group managed with Fleet.

Fleet Cluster Registration

clusterregistration.fleet.cattle.io
Analysis detected an issue with the registration status of a cluster managed with Fleet.

Fleet Git Repository

gitrepo.fleet.cattle.io
Analysis detected an issue with the configuration of a Git repository for continuous delivery.

Git Jobs

gitjob.gitjob.cattle.io
Analysis detected an issue with the status of a Kubernetes job that is configured to be launched based on a Git event.

Helm Repositories

clusterrepo.catalog.cattle.io
Analysis detected that a Helm repository has not successfully downloaded.

Managed Charts

managedchart.management.cattle.io
Analysis detected an issue with the status of a managed chart.

Managed Clusters

cluster.management.cattle.io
Analysis detected that a cluster managed by Rancher is not ready. The state of the cluster will display Active on the home screen when it is available to be managed with Rancher.

There are interim states, such as Provisioning and Waiting, that may be displayed before a cluster becomes Active. The interim states typically show additional information, such as Waiting for cluster to be ready.

Nodes

node.management.cattle.io
Analysis detected that a node within the cluster is not ready.

There are interim states, such as Provisioning and Updating, that may be displayed before a node becomes Active.

Provisioning Clusters

cluster.provisioning.cattle.io
Analysis detected that a cluster being provisioned by Rancher is not ready. Clusters are provisioned by cluster drivers. The state of the cluster will display Active on the home screen when it is available to be managed with Rancher.

Steps

Review the Rancher logs in the cattle-system namespace for additional details about why there is a Rancher issue.

Verrazzano Enterprise Container Platform – Analyze Tool Findings and Reports

Docs: Cluster API Resource Not Ready

Summary

Steps

Docs: External DNS Configuration

Summary

Steps

Related information

Docs: Image Pull Back Off

Summary

Steps

Related information

Docs: Image Pull Not Found

Summary

Steps

Related information

Docs: Image Pull Rate Limit

Summary

Steps

Related information

Docs: Image Pull Service Issue

Summary

Steps

Related information

Docs: Ingress Controller Invalid Shape

Summary

Steps

Related information

Docs: Ingress Controller Load Balancer Service Limit Reached

Summary

Steps

Related information

Docs: Ingress Controller No Load Balancer IP

Summary

Steps

Related information

Docs: Ingress Controller Oracle Cloud Infrastructure IP Limit Exceeded

Summary

Steps

Related information

Docs: Install Ingress Controller Failure

Summary

Steps

Related information

Docs: Install, Upgrade, or Uninstall Failure

Summary

Steps

Related information

Docs: Insufficient CPU

Summary

Steps

Related information

Docs: Insufficient Memory

Summary

Steps

Related information

Docs: Istio Ingress Gateway No External IP

Summary

Steps

Related information

Docs: Istio Ingress Load Balancer Creation Issue

Summary

Steps

Related information

Docs: Keycloak Data Migration Failure

Summary

Steps

Related information

Docs: NGINX Ingress Controller Load Balancer Creation Issue

Summary

Steps

Related information

Docs: Pending Pods

Summary

Steps

Related information

Docs: Problem Pods

Summary

Steps

Related information