Verrazzano Enterprise Container Platform – Analysis Advice

Docs: External DNS Configuration

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that there was an issue with the DNS configuration that is related to credentials. It could be an authorization issue or incorrectly supplied credentials.

Steps

Review the OCI credentials provided during DNS configuration.
Check the permission for OCI credentials or instance_principal.

OCI DNS

Docs: Image Pull Back Off

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that there were pods that had issues due to failures to pull an image or images.

The analysis was not able to identify a specific root cause, however, it might have supplied data that is related to the failures.

Steps

Review the analysis data. At a minimum, it will indicate which pods are being impacted and might give other clues on the root cause.
If the service is experiencing an outage, then consult the specific service status page. For common service status pages, see Related information.

Docs: Image Pull Not Found

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that there were pods which had issues due to failures to pull an image or images where the root cause was that the image was not found.

Steps

Review the analysis data; it enumerates the pods and related messages regarding which images had this issue.
Confirm that the image name, digest, and tag are correctly specified.

Kubernetes Troubleshooting

Docs: Image Pull Rate Limit

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that there were pods which had issues due to failures to pull an image or images.

The root cause was rate limit exceeded errors while pulling images.

Steps

Review the analysis data; it enumerates the pods and related messages regarding which images had this issue.
The detailed messages might provide specific instructions for the registry that is involved. For example, it might provide a link to instructions on how to increase the limit.

Increase Rate Limits

Docs: Image Pull Service Issue

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that there were pods which had issues due to failures to pull an image or images where the root cause was that the service was not available.

The service might be unreachable or might be incorrect.

Steps

Review the analysis data; it enumerates the pods and related messages about which images had this issue.
Confirm that the registry for the image is correct.
The messages might identify a connectivity issue.
If the service is experiencing an outage, then consult the specific service status page. For common service status pages, see Related information.

Docs: Ingress Controller Invalid Shape

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that the Verrazzano installation failed while installing the NGINX Ingress Controller.

The root cause appears to be that Verrazzano custom resource provided an invalid shape for Oracle Cloud Infrastructure load balancer.

Steps

Review the messages from the supporting details for the allowed shape for Oracle Cloud Infrastructure load balancer.
Refer to the Oracle Cloud Infrastructure documentation, Load Balancer Management.

Customize Load Balancers on OKE

Docs: Ingress Controller Load Balancer Service Limit Reached

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that the Verrazzano installation failed while installing the NGINX Ingress Controller.

The root cause appears to be that the load balancer service limit has been reached.

Steps

Review the messages from the supporting details for the exact limits, and delete unused load balancers.
If available, use a different load balancer shape. See Customizing Ingress.
Refer to the Oracle Cloud Infrastructure documentation on Service Limits.

Docs: Ingress Controller No Load Balancer IP

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that the Verrazzano installation failed while installing the NGINX Ingress Controller.

The root cause appears to be that the load balancer is either missing or unable to set the ingress IP address on the NGINX Ingress service.

Steps

Refer to the platform-specific environment setup for your platform here.

Docs: Ingress Controller Oracle Cloud Infrastructure IP Limit Exceeded

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that the Verrazzano installation failed while installing the NGINX Ingress Controller.

The root cause appears to be that an Oracle Cloud Infrastructure IP non-ephemeral address limit has been reached.

Steps

Review the messages from the supporting details for the exact limit.
Refer to the Oracle Cloud Infrastructure documentation related to managing IP Addresses.

Public IP Addresses

Docs: Install Ingress Controller Failure

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that the Verrazzano installation has failed related to the NGINX Ingress Controller, however, it was unable to isolate the specific root cause.

Steps

Review the analysis data, which might help identify the issue.

Docs: Install, Upgrade, or Uninstall Failure

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected a failure while installing, upgrading, or uninstalling Verrazzano; however, it did not isolate the exact reason for the failure.

Steps

Review the analysis data, which can help identify the issue. Upon failure, a bug-report archive file will be generated automatically and the filepath will be printed to the console.

Docs: Insufficient CPU

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that there were nodes reporting insufficient CPU.

Steps

Review the detailed analysis data to identify the specific nodes involved.
Review the nodes to determine why they do not have sufficient CPU.

a. Are the nodes sized correctly for the workload?
- For the minimum resources required for installing Verrazzano, see the Prerequisites.
- Make sure to take into account the resource guidelines of all the applications that you are deploying.
b. Is something unexpected running on the nodes or consuming more CPU than expected?

Docs: Insufficient Memory

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that there were nodes reporting insufficient memory.

Steps

Review the detailed analysis data to identify the specific nodes involved.
Review the nodes to determine why they do not have sufficient memory.

a. Are the nodes sized correctly for the workload?
- For the minimum resources required for installing Verrazzano, see the Prerequisites.
- Make sure to take into account the resource guidelines of all the applications that you are deploying.
b. Is something unexpected running on the nodes or consuming more memory than expected?

Docs: Istio Ingress Gateway No External IP

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that the Verrazzano installation failed while installing the Istio ingress gateway.

The root cause appears to be that the load balancer is either missing or unable to set the external IP address on the Istio ingress gateway service.

Steps

Refer to the platform-specific environment setup for your platform here.

Docs: Istio Ingress Load Balancer Creation Issue

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that the Verrazzano installation failed while creating the load balancer for the Istio ingress gateway.

The root cause was that, while creating the public load balancer, there were no public subnets available.

Steps

Refer to the platform-specific environment setup for your platform here.

Docs: Keycloak Data Migration Failure

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that the Verrazzano upgrade failed when migrating Keycloak data from the existing legacyDB to the new InnoDB.

Steps

Verify whether the dump-claim PVC exists and is bound to the PV.
```
$ k get pvc -n keycloak dump-claim

# Sample Output
NAME         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
dump-claim   Bound    pvc-c246d7c4-3041-4164-8c1e-744dda805686   2Gi        RWO            standard       12m
```
```
$ kubectl get pv

# Sample Output
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                 STORAGECLASS   REASON   AGE
pvc-c246d7c4-3041-4164-8c1e-744dda805686   2Gi        RWO            Retain           Bound    keycloak/dump-claim   standard                49m
```
Get the mysql-root password from the mysql or mysql-cluster-secret secret in the keycloak namespace. The password is required to access the MySQL server as a root user in the following steps.
```
$ kubectl get secret -n keycloak mysql -o jsonpath='{.data.mysql-root-password}' | base64 --decode

# Sample Output
lvYwPJjwFB
```
Create a load-dump pod to help in migrating the previous Keycloak database data into the new database instance.
```
$ kubectl apply -f - <<EOF
  apiVersion: v1
  kind: Pod
  metadata:
    name: my-load-dump
    namespace: keycloak
    labels:
      job-name: load-dump
  spec:
    containers:
    - name: mysqlsh-load-dump
      image: ghcr.io/verrazzano/mysql-server:8.0.32
      volumeMounts:
      - mountPath: /var/lib/dump
        name: keycloak-dump
    volumes:
    - name: keycloak-dump
      persistentVolumeClaim:
        claimName: dump-claim
EOF
```
Start a shell session inside the pod, and then run these commands:
```
$ kubectl exec -n keycloak my-load-dump -it -- /bin/bash
```
a. Fix the dump directory permission.
```
$ chown -R 27:27 /var/lib/dump
```
b. Check if the MySQL server is running. If it is not running, check if the MySQL pods in the keycloak namespace are ready, and then check again.
```
$ mysqladmin ping -h"mysql.keycloak.svc.cluster.local" -p{{ .RootPassword }}

# Sample Output
mysqld is alive
```
c. Migrate the Keycloak data.
```
$ mysqlsh -u root -p{{ .RootPassword }} -h mysql.keycloak.svc.cluster.local -e 'util.loadDump("/var/lib/dump/dump", {includeSchemas: ["keycloak"], includeUsers: ["keycloak"], loadUsers: true})'
```
```
# Sample Output
Loading DDL, Data and Users from '/var/lib/dump/dump' using 4 threads.
Opening dump...
Target is MySQL 8.0.32. Dump was produced from MySQL 8.0.29
Scanning metadata - done       
Checking for pre-existing objects...
Executing common preamble SQL
Executing DDL - done         
Executing user accounts SQL...
NOTE: Skipping CREATE/ALTER USER statements for user 'root'@'%'
NOTE: Skipping CREATE/ALTER USER statements for user 'root'@'localhost'
NOTE: Skipping GRANT statements for user 'root'@'%'
NOTE: Skipping GRANT statements for user 'root'@'localhost'
Executing view DDL - done       
Starting data load
4 thds loading \ 100% (159.79 KB / 159.79 KB), 0.00 B/s, 33 / 93 tables done
Recreating indexes - done       
Executing common postamble SQL                                              
93 chunks (1.31K rows, 159.79 KB) for 93 tables in 1 schemas were loaded in 3 sec (avg throughput 159.79 KB/s)
0 warnings were reported during the load.
```
In the db-migration secret, add a db-migrated field and set its value to true (base64 encoded). This will notify the Verrazzano platform operator that the Keycloak data was migrated manually.
```
$ k edit secret -n keycloak db-migration

# Sample Output
data:
  db-migrated: dHJ1ZQ==        # true base64 encoded is dHJ1ZQ==
  database-dumped: dHJ1ZQ==
  deployment-found: dHJ1ZQ==
  ...
```
Delete the load-dump pod.
```
$ kubectl delete pod -n keycloak my-load-dump
```

Docs: NGINX Ingress Controller Load Balancer Creation Issue

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that the Verrazzano installation failed while creating the load balancer for the NGINX Ingress Controller.

The root cause was that, while creating the public load balancer, there were no public subnets available.

Steps

Refer to the platform-specific environment setup for your platform here.

Docs: Pending Pods

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that there were pods which were in a pending state without detecting other specific issues related to them.

Steps

Review the analysis data. At a minimum, this should indicate which pods are being impacted and it might give other clues on the root cause.

Kubernetes Troubleshooting

Docs: Problem Pods

Mon, 01 Jan 0001 00:00:00 +0000

Summary

Analysis detected that there were pods which were not in a running, succeeded, or pending state.

The analysis was not able to determine a specific root cause, however, it might have supplied data that is related to the pods in question. The root cause might be obvious from the supporting data, but the analysis tool isn’t isolating the specific scenario yet.

Steps

Review the analysis data. At a minimum, it should indicate which pods are being impacted and it might give other clues on the root cause.

Kubernetes Troubleshooting

Verrazzano Enterprise Container Platform – Analysis Advice

Docs: External DNS Configuration

Summary

Steps

Related information

Docs: Image Pull Back Off

Summary

Steps

Related information

Docs: Image Pull Not Found

Summary

Steps

Related information

Docs: Image Pull Rate Limit

Summary

Steps

Related information

Docs: Image Pull Service Issue

Summary

Steps

Related information

Docs: Ingress Controller Invalid Shape

Summary

Steps

Related information

Docs: Ingress Controller Load Balancer Service Limit Reached

Summary

Steps

Related information

Docs: Ingress Controller No Load Balancer IP

Summary

Steps

Related information

Docs: Ingress Controller Oracle Cloud Infrastructure IP Limit Exceeded

Summary

Steps

Related information

Docs: Install Ingress Controller Failure

Summary

Steps

Related information

Docs: Install, Upgrade, or Uninstall Failure

Summary

Steps

Related information

Docs: Insufficient CPU

Summary

Steps

Related information

Docs: Insufficient Memory

Summary

Steps

Related information

Docs: Istio Ingress Gateway No External IP

Summary

Steps

Related information

Docs: Istio Ingress Load Balancer Creation Issue

Summary

Steps

Related information

Docs: Keycloak Data Migration Failure

Summary

Steps

Related information

Docs: NGINX Ingress Controller Load Balancer Creation Issue

Summary

Steps

Related information

Docs: Pending Pods

Summary

Steps

Related information

Docs: Problem Pods

Summary

Steps

Related information