Verrazzano Enterprise Container Platform – Customize Verrazzano

Docs: AuthProxy

Mon, 01 Jan 0001 00:00:00 +0000

The Verrazzano AuthProxy component enables authentication and authorization for Keycloak users accessing Verrazzano resources. You can customize the AuthProxy component using settings in the Verrazzano custom resource.

The following table describes the fields in the Verrazzano custom resource pertaining to the AuthProxy component.

Path to Field Description

spec.components.authProxy.kubernetes.replicas The number of pods to replicate. The default is 2 for the prod profile and 1 for all other profiles.

Path to Field	Description
`spec.components.authProxy.kubernetes.replicas`	The number of pods to replicate. The default is `2` for the `prod` profile and `1` for all other profiles.
`spec.components.authProxy.kubernetes.affinity`	The pod affinity definition expressed as a standard Kubernetes affinity definition. The default configuration spreads the AuthProxy pods across the available nodes. spec: components: authProxy: kubernetes: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchExpressions: - key: app operator: In values: - verrazzano-authproxy topologyKey: kubernetes.io/hostname

spec.components.authProxy.kubernetes.affinity

The pod affinity definition expressed as a standard Kubernetes affinity definition. The default configuration spreads the AuthProxy pods across the available nodes.

spec:
  components:
    authProxy:
      kubernetes:
        affinity:
          podAntiAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
              - weight: 100
                podAffinityTerm:
                  labelSelector:
                    matchExpressions:
                      - key: app
                        operator: In
                        values:
                          - verrazzano-authproxy
                  topologyKey: kubernetes.io/hostname

The following example customizes a Verrazzano prod profile as follows:

Increases the replicas count to 3
Changes the podAffinity configuration to use requiredDuringSchedulingIgnoredDuringExecution

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: prod
  components:
    authProxy:
      overrides:
      - values:
          replicas: 3
          affinity:
            podAntiAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                - labelSelector:
                    matchExpressions:
                      - key: app
                        operator: In
                        values:
                          - verrazzano-authproxy
                  topologyKey: kubernetes.io/hostname

Docs: Certificates

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano issues certificates to secure access from external clients to secure system endpoints.
A certificate from a certificate authority (CA) must be configured to issue the endpoint certificates in one of the following ways:

Let Verrazzano generate a self-signed CA (the default).
Configure a CA that you provide.
Configure LetsEncrypt as the certificate issuer (requires Oracle Cloud Infrastructure DNS).

In all cases, Verrazzano uses cert-manager to manage the creation of certificates.

NOTE

Self-signed certificate authorities generate certificates that are NOT signed by a trusted authority; typically, they are not used in production environments.

Use the Verrazzano self-signed CA

By default, Verrazzano creates its own self-signed CA. No configuration is required.

Use a custom CA

If you want to provide your own CA, you must:

(Optional) Create your own signing key pair and CA certificate.

For example, you can use the openssl CLI to create a key pair for the nip.io domain.
```
# Generate a CA private key
$ openssl genrsa -out tls.key 2048

# Create a self-signed certificate, valid for 10yrs with the 'signing' option set
$ openssl req -x509 -new -nodes -key tls.key -subj "/CN=*.nip.io" -days 3650 -reqexts v3_req -extensions v3_ca -out tls.crt
```
The output of these commands will be two files, tls.key and tls.crt, the key and certificate for your signing key pair. These files must be named in that manner for the next step.

If you already have generated your own key pair, you must name the private key and certificate, tls.key and tls.crt, respectively. If your issuer represents an intermediate, ensure that tls.crt contains the issuer’s full chain in the correct order.

You can find more details on providing your own CA, in the cert-manager CA documentation.
Save your signing key pair as a Kubernetes secret.
```
$ kubectl create ns mynamespace
$ kubectl create secret tls myca --namespace=mynamespace --cert=tls.crt --key=tls.key
```
Specify the secret name and namespace location in the Verrazzano custom resource.

The custom CA secret must be provided to cert-manager using the following fields in spec.components.certManager.certificate.ca in the Verrazzano custom resource:
- spec.components.certManager.certificate.ca.secretName
- spec.components.certManager.certificate.ca.clusterResourceNamespace

For example, if you created a CA secret named myca in the namespace mynamespace, you would configure it as shown:

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: custom-ca-example
spec:
  profile: dev
  components:
    certManager:
      certificate:
        ca:
          secretName: myca
          clusterResourceNamespace: mynamespace

Use LetsEncrypt certificates

You can configure Verrazzano to use certificates generated by LetsEncrypt. LetsEncrypt implements the ACME protocol, which provides a standard protocol for the automated issuance of certificates signed by a trusted authority. This is managed through the spec.components.certManager.certificate.acme field in the Verrazzano custom resource.

NOTE

Using LetsEncrypt for certificates also requires using Oracle Cloud Infrastructure DNS for DNS management. For details, see the Customize DNS page.

To configure cert-manager to use LetsEncrypt as the certificates provider, you must configure a cert-manager ACME provider with the following values in the Verrazzano custom resource:

Set the spec.components.certManager.certificate.acme.provider field to letsEncrypt.
Set the spec.components.certManager.certificate.acme.emailAddress field to a valid email address for the letsEncrypt account.
(Optional) Set the spec.components.certManager.certificate.acme.environment field to either staging or production (the default).

The following example configures Verrazzano to use the LetsEncrypt production environment by default, with Oracle Cloud Infrastructure DNS for DNS record management.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: letsencrypt-certs-example
spec:
  profile: dev
  components:
    certManager:
      certificate:
        acme:
          provider: letsEncrypt
          emailAddress: jane.doe@mycompany.com
    dns:
      oci:
        ociConfigSecret: oci
        dnsZoneCompartmentOCID: ocid1.compartment.oc1.....
        dnsZoneOCID: ocid1.dns-zone.oc1.....
        dnsZoneName: example.com

The following example configures Verrazzano to use the LetsEncrypt staging environment with Oracle Cloud Infrastructure DNS.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: letsencrypt-certs-example
spec:
  profile: dev
  components:
    certManager:
      certificate:
        acme:
          provider: letsEncrypt
          emailAddress: jane.doe@mycompany.com
          environment: staging
    dns:
      oci:
        ociConfigSecret: oci
        dnsZoneCompartmentOCID: ocid1.compartment.oc1.....
        dnsZoneOCID: ocid1.dns-zone.oc1.....
        dnsZoneName: example.com

NOTE

Certificates issued by the LetsEncrypt staging environment are signed by untrusted authorities, similar to self-signed certificates. They are typically not used in production environments.

LetsEncrypt staging versus production

LetsEncrypt provides rate limits on generated certificates to ensure fair usage across all clients. The production environment limits can be exceeded more frequently in environments where Verrazzano may be installed or reinstalled frequently (like a test environment). This can result in failed installations due to rate limit exceptions on certificate generation.

In such environments, it is better to use the LetsEncrypt staging environment, which has much higher limits than the production environment. For test environments, the self-signed CA also may be more appropriate to completely avoid LetsEncrypt rate limits.

Docs: DNS

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano supports three DNS choices for Verrazzano services and applications:

Free wildcard DNS services (nip.io and sslip.io)
Oracle Cloud Infrastructure DNS managed by Verrazzano
Custom (user-managed) DNS

How Verrazzano constructs a DNS domain

Regardless of which DNS management you use, the value in the spec.environmentName field in your installation will be prepended to the configured domain in the spec.components.dns section of the custom resource, to form the full DNS domain name used to access Verrazzano endpoints.

For example, if spec.environmentName is set to sales and the domain is configured in spec.components.dns as us.example.com, Verrazzano will create sales.us.example.com as the DNS domain for the installation.

Wildcard DNS Services

Verrazzano can be configured to use either the nip.io or sslip.io free wildcard DNS services. When queried with a host name with an embedded IP address, wildcard DNS services return that IP address.

For example, using the nip.io service, the following DNS names all map to the IP address 10.0.0.1:

10.0.0.1.nip.io
app.10.0.0.1.nip.io
customer1.app.10.0.0.1.nip.io

To configure Verrazzano to use one of these services, set the spec.wildcard.domain field in the Verrazzano custom resource to either nip.io or sslip.io; the default is nip.io.

For example, the following configuration uses sslip.io, instead of nip.io, for wildcard DNS with a dev installation profile.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: dev
  components:
    dns:
      wildcard:
        domain: sslip.io

Oracle Cloud Infrastructure DNS

Verrazzano can directly manage records in Oracle Oracle Cloud Infrastructure DNS when configured to use the spec.components.dns.oci field. This is achieved through the External DNS Service, which is a component that is conditionally installed when Oracle Cloud Infrastructure DNS is configured for DNS management in Verrazzano.

Prerequisites

The following prerequisites must be met before using Oracle Cloud Infrastructure DNS with Verrazzano:

You must have control of a DNS domain.
You must have an Oracle Cloud Infrastructure DNS Service Zone that is configured to manage records for that domain. Verrazzano also supports the use of both GLOBAL and PRIVATE Oracle Cloud Infrastructure DNS zones.

A DNS Service Zone is a distinct portion of a domain namespace. You must ensure that the zone is appropriately associated with a parent domain. For example, an appropriate zone name for parent domain example.com is us.example.com.

To create an Oracle Cloud Infrastructure DNS zone using the Oracle Cloud Infrastructure CLI:
```
$ oci dns zone create \
    -c <compartment ocid> \
    --name <zone-name-prefix>.example.com \
    --zone-type PRIMARY
```
To create an Oracle Cloud Infrastructure DNS zone using the Oracle Cloud Infrastructure Console, see Managing DNS Service Zones.
You must have a valid Oracle Cloud Infrastructure API signing key that can be used to communicate with Oracle Cloud Infrastructure DNS in your tenancy.

For example, you can create an API signing key using the Oracle Cloud Infrastructure CLI.
```
  $ oci setup keys --key-name myapikey
  Enter a passphrase for your private key (empty for no passphrase):
  Public key written to: /Users/jdoe/.oci/myapikey_public.pem
  Private key written to: /Users/jdoe/.oci/myapikey.pem
  Public key fingerprint: 39:08:44:69:9f:f5:73:86:7a:46:d8:ad:34:4f:95:29
```
If you haven’t already uploaded your API signing public key through the console, follow the instructions in this section, How to upload the public key.

After the key pair has been created, you must upload the public key to your account in your Oracle Cloud Infrastructure tenancy. For details, see the Oracle Cloud Infrastructure documentation, Required Keys and OCIDs.

Create an Oracle Cloud Infrastructure API secret in the target cluster

To communicate with Oracle Cloud Infrastructure DNS to manage DNS records, Verrazzano needs to be made aware of the necessary API credentials.
A generic Kubernetes secret must be created in the cluster’s verrazzano-install namespace with the required credentials. That secret must then be referenced by the custom resource that is used to install Verrazzano.

After you have an Oracle Cloud Infrastructure API key ready for use, create a YAML file, oci.yaml, with the API credentials in the form:

auth:
  region: <oci-region>
  tenancy: <oci-tenancy-ocid>
  user: <oci-user-ocid>
  key: |
    <oci-api-private-key-file-contents>
  fingerprint: <oci-api-private-key-fingerprint>

This information typically can be found in your Oracle Cloud Infrastructure CLI config file or in the Oracle Cloud Infrastructure Console. The <oci-api-private-key-file-contents> contents are the PEM-encoded contents of the key_file value within the Oracle Cloud Infrastructure CLI configuration profile.

For example, your oci.yaml file will look similar to the following:

auth:
  region: us-ashburn-1
  tenancy: ocid1.tenancy.oc1.....
  user: ocid1.user.oc1.....
  key: |
    -----BEGIN RSA PRIVATE KEY-----
    ...
    -----END RSA PRIVATE KEY-----
  fingerprint: 12:d3:4c:gh:fd:9e:27:g8:b9:0d:9f:00:22:33:c3:gg

Verrazzano also supports the use of instance principals to communicate with Oracle Cloud Infrastructure in order to create or update Oracle Cloud Infrastructure DNS records. Instance principals require some prerequisites that can be found here.

When using instance principals, your oci.yaml file will look as follows:

auth:
  authtype: instance_principal

Then, you can create a generic Kubernetes secret in the cluster’s verrazzano-install namespace using kubectl.

$ kubectl create secret generic -n verrazzano-install <secret-name> --from-file=<path-to-oci-yaml-file>

For example, to create a secret named oci from a file oci.yaml, do the following:

$ kubectl create secret generic -n verrazzano-install oci --from-file=oci.yaml

This secret will later be referenced from the Verrazzano custom resource used during installation.

Use a Verrazzano helper script to create an Oracle Cloud Infrastructure secret

Verrazzano also provides a helper script to create the necessary Kubernetes secret based on your Oracle Cloud Infrastructure CLI configuration file, assuming that you have the Oracle Cloud Infrastructure CLI installed and a valid Oracle Cloud Infrastructure CLI profile with the required API key information. The script create_oci_config_secret.sh reads your Oracle Cloud Infrastructure CLI configuration file to create the secret.

First, download the create_oci_config_secret.sh script.

$ curl \
    -o ./create_oci_config_secret.sh \
    https://raw.githubusercontent.com/verrazzano/verrazzano/master/platform-operator/scripts/install/create_oci_config_secret.sh

Next, set your KUBECONFIG environment variable to point to your cluster and run create_oci_config_secret.sh -h to display the script options.

$ chmod +x create_oci_config_secret.sh
$ export KUBECONFIG=<kubeconfig-file>
$ ./create_oci_config_secret.sh  -h
usage: ./create_oci_config_secret.sh [-o oci_config_file] [-s config_file_section]
  -o oci_config_file         The full path to the Oracle Cloud Infrastructure configuration file (default ~/.oci/config)
  -s config_file_section     The properties section within the Oracle Cloud Infrastructure configuration file.  Default is DEFAULT
  -k secret_name             The secret name containing the Oracle Cloud Infrastructure configuration.  Default is oci
  -c context_name            The kubectl context to use
  -a auth_type               The auth_type to be used to access Oracle Cloud Infrastructure. Valid values are user_principal/instance_principal. Default is user_principal.
  -h                         Help

For example, to have the script create the YAML file using your [DEFAULT] Oracle Cloud Infrastructure CLI profile and then create a Kubernetes secret named oci, you can run the script with no arguments, as follows:

$ ./create_oci_config_secret.sh
secret/oci created

The following example creates a secret myoci using an Oracle Cloud Infrastructure CLI profile named [dev].

$ ./create_oci_config_secret.sh -s dev -k myoci
secret/myoci created

When using instance principals, all other parameters will be ignored automatically. The following example creates a secret myoci using Oracle Cloud Infrastructure instance principal.

$ ./create_oci_config_secret.sh -a instance_principal
secret/myoci created

Installation

After the Oracle Cloud Infrastructure API secret is created, create a Verrazzano custom resource for the installation that is configured to use Oracle Cloud Infrastructure DNS and reference the secret you created.

As a starting point, download the sample Verrazzano custom resource install-oci.yaml file for Oracle Cloud Infrastructure DNS.

$ curl \
    -o ./install-oci.yaml \
    https://raw.githubusercontent.com/verrazzano/verrazzano/master/platform-operator/config/samples/install-oci.yaml

Edit the install-oci.yaml file to provide values for the following configuration settings in the custom resource spec:

spec.environmentName
spec.components.dns.oci.ociConfigSecret
spec.components.dns.oci.dnsZoneCompartmentOCID
spec.components.dns.oci.dnsZoneOCID
spec.components.dns.oci.dnsZoneName
spec.components.dns.oci.dnsScope

The field spec.components.dns.oci.ociConfigSecret should reference the secret created earlier. For details on the Oracle Cloud Infrastructure DNS configuration settings, see spec.components.dns.oci.

For example, a custom resource for a prod installation profile using Oracle Cloud Infrastructure DNS might look as follows, yielding a domain of myenv.example.com (Oracle Cloud Infrastructure identifiers redacted):

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: prod
  environmentName: myenv
  components:
    dns:
      oci:
        ociConfigSecret: oci
        dnsZoneCompartmentOCID: ocid1.compartment.oc1..compartment-ocid
        dnsZoneOCID: ocid1.dns-zone.oc1..zone-ocid
        dnsZoneName: example.com

If using a private DNS zone, then the same prod installation profile using Oracle Cloud Infrastructure DNS will look as follows:

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: my-verrazzano
spec:
  profile: prod
  environmentName: myenv
  components:
    dns:
      oci:
        ociConfigSecret: oci
        dnsZoneCompartmentOCID: ocid1.compartment.oc1..compartment-ocid
        dnsZoneOCID: ocid1.dns-zone.oc1..zone-ocid
        dnsZoneName: example.com
        dnsScope: PRIVATE

After the custom resource is ready, apply it using kubectl apply -f <path-to-custom-resource-file>.

Custom DNS

You can specify your own externally managed, custom DNS domain. In this scenario, you manage your own DNS domain and all DNS records in that domain.

An externally managed DNS domain is specified in the spec.components.dns.external.suffix field of the Verrazzano custom resource.

When using an externally managed DNS domain, you are responsible for:

Configuring A records for Verrazzano ingress points (load balancers)
Configuring CNAME records for host names in the domain that point to the A records, as needed

The Verrazzano installer searches the DNS zone you provide for two specific A records.
These are used to configure the cluster and should refer to external addresses of the load balancers provisioned by the user.

The A records need to be created manually.

Record	Use
`ingress-mgmt`	Set as the `.spec.externalIPs` value of the `ingress-controller-nginx-ingress-controller` service.
`ingress-verrazzano`	Set as the `.spec.externalIPs` value of the `istio-ingressgateway` service.

For example, if spec.environmentName is set to myenv, and spec.components.dns.external.suffix is set to example.com, the A records would need to be set up as follows:

198.51.100.10                                   A       ingress-mgmt.myenv.example.com.
203.0.113.10                                    A       ingress-verrazzano.myenv.example.com.

This example assumes that load balancers exist for ingress-mgmt on 198.51.100.10 and for ingress-verrazzano on 203.0.113.10.

For a more complete example, see the documentation for setting up Verrazzano on the Oracle Cloud Native Environment Platform.

Docs: External Load Balancers

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano requires the following load balancers at installation:

Load balancer for NGINX ingress
Load balancer for Istio ingress

By default, Verrazzano automatically creates them as Kubernetes-managed load balancers, however, you have the option to use your own external load balancers. You can choose to replace either or both load balancers.

The following is an example of using external load balancers for both management and application ingress.

Prepare the external load balancers

External load balancer for management ingress:
- This load balancer must have a listener set up on port 443 with TCP protocol.
- The back end set for this listener needs to include the Kubernetes cluster node IP addresses on a port you pick, for example, 31443.
External load balancer for application ingress:
- This load balancer must have a listener set up on port 443 with TCP protocol.
- The back end set for this listener needs to include the Kubernetes cluster node IP addresses on a port you pick, for example, 32443.

Verrazzano installation options

External load balancer for management ingress:
- Set NodePort as the ingress type in the Ingress Component.
- Set controller.service.externalIPs with the IP address for the external management load balancer in the Ingress NGINX Overrides.
  
  NOTE: If the ingress type is NodePort, then a valid and accessible IP address must be specified using the controller.service.externalIPs key in NGINXInstallArgs.
- Set ports in the Ingress Component with a PortConfig that has 443 as port, 31443 as nodePort, https as targetPort, and TCP as protocol.
External load balancer for application ingress using the Istio ingress gateway overrides:
- Set service Type to NodePort.
- Set service externalIPs to the external application load balancer IP address.
- Set service ports with a https named entry, 443 as port, 32443 as nodePort, 8443 as targetPort, and TCP as protocol.

Example Custom Resource with management and application external load balancers

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: myvz
spec:
  components:
    ingressNGINX:
      overrides:
      - values:
          controller:
            service:
              externalIPs:
              - 11.22.33.44
      type: NodePort
      ports:
      - name: https
        port: 443
        nodePort: 31443
        protocol: TCP
        targetPort: https
    istio:
      overrides:
      - values:
          apiVersion: install.istio.io/v1alpha1
          kind: IstioOperator
          spec:
            components:
              ingressGateways:
                - enabled: true
                  name: istio-ingressgateway
                  k8s:
                    service:
                      type: NodePort
                      ports:
                      - name: https
                        port: 443
                        nodePort: 32443
                        protocol: TCP
                        targetPort: 8443
                      externalIPs:
                      - 11.22.33.55

Docs: Network File System

Mon, 01 Jan 0001 00:00:00 +0000

Complete the following steps to configure NFS storage in an Oracle Cloud Native Environment.

Create an OLCNE cluster. See OLCNE cluster.

The cluster must have at least 3 worker nodes.
Create an NFS server. For an example that uses an NFS server on Oracle Linux, see Create an NFS server on Oracle Linux.

a. Install the NFS utility package on the server and client instances.
```
$ sudo dnf install -y nfs-utils
```
b. Create a directory for your shared files. Make sure that the server does not have root ownership.

c. Define the shared directory in /etc/exports with the correct permissions. Make sure to disable root squashing.
```
$ <path to directory> <ip-address/subnet-mask>(rw,sync,no_root_squash,no_subtree_check)
```
d. Set the firewall to allow NFS traffic.
```
$ sudo firewall-cmd --permanent --zone=public --add-service=nfs
$ sudo firewall-cmd --reload
$ sudo firewall-cmd --list-all
```
e. Enable and start the NFS service.
```
$ sudo systemctl enable --now nfs-server
```
Deploy an NFS provisioner to your cluster.

a. Install an NFS client provisioner of your choice. For an example, see Kubernetes NFS Subdir External Provisioner.

b. Add the required Helm repo.
```
$ helm repo add nfs-subdir-external-provisioner https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner/
```
c. Install the provisioner. Set your storage class as a default and create a service account.
```
$ helm install nfs-test \
   --set nfs.server=<server ip address> \
   --set nfs.path=<path> \
   --set storageClass.name=<name> \
   --set storageClass.defaultClass=true,rbac.create=true \
   --set storageClass.provisionerName=nfsclientprov/nfs \
   --set serviceAccount.create=true \
   --set serviceAccount.name=nfs-svc-acc-nfs nfs-subdir-external-provisioner/nfs-subdir-external-provisioner
```
d. Only one storage class should be listed as the default. If required, edit the other storage classes and delete the following annotation:
```
$ storageclass.kubernetes.io/is-default-class: "true"
```

Docs: Grafana

Mon, 01 Jan 0001 00:00:00 +0000

Configure a Grafana database

By default, Verrazzano automatically installs and configures a Grafana database. However, you can use your own external database.

If you prefer to use your own Grafana database, complete the following steps:

Create a secret named grafana-db in the verrazzano-install namespace which contains the login credentials. For example:
```
$ ROOT_SECRET=$(echo <database root user secret> | base64)
$ USER=$(echo <database user> | base64)
$ USER_SECRET=$(echo <database user secret> | base64)
$ kubectl apply -f - <<-EOF
apiVersion: v1
kind: Secret
metadata:
  name: grafana-db
  namespace: verrazzano-install
type: Opaque
data:
  root-password: $ROOT_SECRET
  username: $USER
  password: $USER_SECRET
EOF
```
Configure the Grafana component of the Verrazzano custom resource. For example:
```
apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: grafana-db-example
spec:
  profile: dev
  components:
    grafana:
      database:
        host: mysql.verrazzano-install.svc.cluster.local
        name: grafana
```

Configure an SMTP server

To configure Grafana to send SMTP notifications, complete the following steps:

Create a secret named smtp-secret in the verrazzano-system namespace which contains the SMTP server credentials. For example:
```
$ kubectl -n verrazzano-system create secret generic smtp-secret \
 --from-literal=username="<smtp server username>" \
 --from-literal=password="<smtp server password>" \
 --from-file=cert=<path to file containing certificate> \
 --from-file=key=<path to file containing certificate key>
```
Configure the Grafana component of the Verrazzano custom resource. For example:
```
apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: grafana-smtp-example
spec:
  profile: dev
  components:
    grafana:
      smtp:
        certFileKey: "cert"
        ehloIdentity: "<Name to be used as client identity for EHLO in SMTP dialog>"
        enabled: true
        existingSecret: "smtp-secret"
        fromAddress: "<Address used when sending out emails>"
        fromName: "<Name to be used when sending out emails>"
        host: "<host or host:port for the smtp server>"
        passwordKey: "password"
        skipVerify: true
        startTLSPolicy: ""
        userKey: "username"
        keyFileKey: "key"
```

For more information about Grafana SMTP configurations, see the Grafana Documentation.

For more information about the component definition, see Grafana component in the Verrazzano custom resource.

Docs: Configure High Availability

Mon, 01 Jan 0001 00:00:00 +0000

High availability designs follow three main principles:

Elimination of single points of failure
Fault detection
Reliable failover points

Verrazzano provides a means to eliminate single points of failure among critical Verrazzano components. This is accomplished by increasing replica counts, anti-affinity rules, and implementing replicated data for components that rely on MySQL and OpenSearch.

The ha.yaml file illustrates how the prod profile can be extended to configure a highly available Verrazzano installation. The increased replica counts, along with the anti-affinity rules inherited from the prod profile, ensure that the pods of each component are distributed across the Kubernetes cluster nodes. MySQL and OpenSearch are configured to replicate data among replicas to avoid data loss.

The ha-oci-ccm.yaml file configures a highly available Verrazzano installation on OCNE.

The ha-ext-lb.yaml file configures a highly available Verrazzano installation using external load balancers.

Fault detection is managed natively by using Kubernetes Services and Istio VirtualServices that detect failed pods and route traffic to the remaining replicas.

MySQL and OpenSearch provide reliable failover points for the replicated data.

The result of these measures would be no loss of service if a cluster node became unavailable. For more information regarding node failure and recovery, read the Node Failure Guide.

When using the ha.yaml file, consider the following:

It does not ensure a fault-tolerant environment. Your applications still must be designed and implemented as highly available.
Running additional replicas of components will increase resource requirements. At least four CPUs, 100 GB disk storage, and 64 GB RAM available on the Kubernetes worker nodes is required.
Additional customizations may be required for your environment, including other customizations described in Customize Verrazzano.

For the expected behavior of the MySQL Component in a highly available environment, see Customize Keycloak and MySQL.

Follow these best practices for a highly available Verrazzano installation:

Size your Kubernetes cluster according to your node failure tolerance and workload requirements.
Set the default Storage Class to one with a VolumeBindingMode of WaitForFirstConsumer. This is important for being able to recover from an Availability Domain or zone failure.
Set the replica counts to values that correspond to your node failure tolerance.

To install the example high availability configuration using the Verrazzano CLI:

$ vz install -f https://raw.githubusercontent.com/verrazzano/verrazzano/master/examples/ha/ha.yaml

Using the Verrazzano CLI, install the example high availability configuration on OCNE as follows:

 $ vz install -f https://raw.githubusercontent.com/verrazzano/verrazzano/master/examples/ha/ha-oci-ccm.yaml

Using the Verrazzano CLI, install the example high availability configuration with external load balancers as follows:

$ vz install -f https://raw.githubusercontent.com/verrazzano/verrazzano/master/examples/ha/ha-ext-lb.yaml

Upgrade recommendations

An OKE in-place upgrade scales the cluster to N-1 nodes, where N is the original number of nodes, so you must scale the cluster back to N. This will start a new replacement node using the new node pool. For more information on in-place upgrades, see Performing an In-Place Worker Node Kubernetes Upgrade by Updating an Existing Node Pool.

Docs: Ingress

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano uses NGINX for ingress to Verrazzano system components and Istio for application ingress. You can customize the NGINX and Istio ingress installation configurations using Helm overrides specified in the Verrazzano custom resource. For example, the following Verrazzano custom resource overrides the shape of an Oracle Cloud Infrastructure load balancer for both NGINX and Istio ingresses.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: custom-lb-settings
spec:
  profile: prod
  components:
    ingressNGINX:
      type: LoadBalancer
      overrides:
      - values:
          contoller:
            service:
              annotations:
                service.beta.kubernetes.io/oci-load-balancer-shape: flexible
                service.beta.kubernetes.io/oci-load-balancer-shape-flex-max: "100"
                service.beta.kubernetes.io/oci-load-balancer-shape-flex-min: "100"
    istio:
      overrides:
      - values:
          apiVersion: install.istio.io/v1alpha1
          kind: IstioOperator
          spec:
            components:
              ingressGateways:
                - enabled: true
                  name: istio-ingressgateway
                  k8s:
                    serviceAnnotations:
                      service.beta.kubernetes.io/oci-load-balancer-shape: flexible
                      service.beta.kubernetes.io/oci-load-balancer-shape-flex-max: "100"
                      service.beta.kubernetes.io/oci-load-balancer-shape-flex-min: "100"

For more information about setting component overrides, see Customizing the Chart Before Installing.

Docs: Installation Overrides

Mon, 01 Jan 0001 00:00:00 +0000

Installation overrides let you supply custom values to the underlying Helm charts or operator for a given component. You can supply Verrazzano installation overrides by using a configMapRef, secretRef, or raw values.

The following tables have examples of the Istio component InstallOverrides ConfigMap, Secret, and Values, where the external IP addresses are specified instead of using the defaults. For the default values, see the IstioOperatorSpec.

In the examples, the ConfigMap and Secret overrides are applied before applying the Verrazzano resource installation YAML file.

ConfigMap

Note that the value of the metadata name in the configMap.yaml file must match the configMapRef name in the verrazzanoResourceWithConfigMapRef.yaml file. Also, the values of the key in the configMapRef and the key in the data section of the configMap must match.

ConfigMap
configMap.yaml ConfigMapRef
verrazzanoResourceWithConfigMapRef.yaml

ConfigMap `configMap.yaml`	ConfigMapRef `verrazzanoResourceWithConfigMapRef.yaml`
`apiVersion: v1 kind: ConfigMap metadata: name: istio-cm data: istio-override: \| apiVersion: install.istio.io/v1alpha1 kind: IstioOperator spec: components: ingressGateways: - k8s: service: externalIPs: - 11.22.33.44 type: NodePort name: istio-ingressgateway`	`apiVersion: install.verrazzano.io/v1beta1 kind: Verrazzano metadata: name: vz-with-cm spec: profile: dev components: rancher: enabled: false istio: overrides: - configMapRef: name: istio-cm key: istio-override`

apiVersion: v1
kind: ConfigMap
metadata:
  name: istio-cm
data:
  istio-override: |
    apiVersion: install.istio.io/v1alpha1
    kind: IstioOperator
    spec:
      components:
        ingressGateways:
        - k8s:
            service:
              externalIPs:
              - 11.22.33.44
              type: NodePort
          name: istio-ingressgateway

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: vz-with-cm
spec:
  profile: dev
  components:
    rancher:
      enabled: false
    istio:
      overrides:
      -  configMapRef:
           name: istio-cm
           key: istio-override

Secret

Secret
secret.yaml SecretRef
verrazzanoResourceWithSecretRef.yaml

Secret `secret.yaml`	SecretRef `verrazzanoResourceWithSecretRef.yaml`
`apiVersion: v1 kind: Secret metadata: name: istio-s stringData: istio-override: \| apiVersion: install.istio.io/v1alpha1 kind: IstioOperator spec: components: ingressGateways: - k8s: service: externalIPs: - 11.22.33.44 type: NodePort name: istio-ingressgateway`	`apiVersion: install.verrazzano.io/v1beta1 kind: Verrazzano metadata: name: vz-with-secret namespace: default spec: profile: dev components: rancher: enabled: false istio: overrides: - secretRef: name: istio-s key: istio-override`

apiVersion: v1
kind: Secret
metadata:
  name: istio-s
stringData:
  istio-override: |
    apiVersion: install.istio.io/v1alpha1
    kind: IstioOperator
    spec:
      components:
        ingressGateways:
        - k8s:
            service:
              externalIPs:
              - 11.22.33.44
              type: NodePort
          name: istio-ingressgateway

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: vz-with-secret
  namespace: default
spec:
  profile: dev
  components:
    rancher:
      enabled: false
    istio:
      overrides:
      -  secretRef:
           name: istio-s
           key: istio-override

Values

Values `verrazzanoResourceWithValues.yaml`
`apiVersion: install.verrazzano.io/v1beta1 kind: Verrazzano metadata: name: vz-with-values spec: components: istio: overrides: - values: apiVersion: install.istio.io/v1alpha1 kind: IstioOperator spec: components: ingressGateways: - enabled: true name: istio-ingressgateway k8s: service: type: NodePort ports: - name: https port: 443 nodePort: 32443 protocol: TCP targetPort: 8443 externalIPs: - 11.22.33.44`

Values
verrazzanoResourceWithValues.yaml

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: vz-with-values
spec:
  components:
    istio:
      overrides:
      - values:
          apiVersion: install.istio.io/v1alpha1
          kind: IstioOperator
          spec:
            components:
              ingressGateways:
                - enabled: true
                  name: istio-ingressgateway
                  k8s:
                    service:
                      type: NodePort
                      ports:
                      - name: https
                        port: 443
                        nodePort: 32443
                        protocol: TCP
                        targetPort: 8443
                      externalIPs:
                      - 11.22.33.44

Docs: Istio

Mon, 01 Jan 0001 00:00:00 +0000

You can customize Verrazzano Istio component using settings in the Verrazzano custom resource.

The following table describes the fields in the Verrazzano custom resource pertaining to the Istio component.

Path to Field	Description
`spec.components.istio.egress.kubernetes.replicas`	The number of pods to replicate. The default is `2` for the `prod` profile and `1` for all other profiles.
`spec.components.istio.egress.kubernetes.affinity`	The pod affinity definition expressed as a standard Kubernetes affinity definition. The default configuration spreads the Istio gateway pods across the available nodes. spec: components: istio: egress: kubernetes: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchExpressions: - key: app operator: In values: - istio-egressgateway topologyKey: kubernetes.io/hostname
`spec.components.istio.ingress.kubernetes.replicas`	The number of pods to replicate. The default is `2` for the `prod` profile and `1` for all other profiles.
`spec.components.istio.ingress.kubernetes.affinity`	The pod affinity definition expressed as a standard Kubernetes affinity definition. The default configuration spreads the Istio gateway pods across the available nodes. spec: components: istio: ingress: kubernetes: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchExpressions: - key: app operator: In values: - istio-ingressgateway topologyKey: kubernetes.io/hostname

The following example customizes a Verrazzano prod profile as follows:

Increases the replicas count to 3 for istio-ingressgateway and istio-egressgateway
Changes the podAffinity configuration to use requiredDuringSchedulingIgnoredDuringExecution for istio-ingressgateway and istio-egressgateway
```
apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: prod
  components:
    istio:
      overrides:
      - values:
          apiVersion: install.istio.io/v1alpha1
          kind: IstioOperator
          spec:
            components:
              egressGateways:
                - enabled: true
                  k8s:
                    affinity:
                      podAntiAffinity:
                        requiredDuringSchedulingIgnoredDuringExecution:
                          - podAffinityTerm:
                              labelSelector:
                                matchExpressions:
                                  - key: app
                                    operator: In
                                    values:
                                      - istio-egressgateway
                              topologyKey: kubernetes.io/hostname
                            weight: 100
                    replicaCount: 3
                  name: istio-egressgateway
              ingressGateways:
                - enabled: true
                  k8s:
                    affinity:
                      podAntiAffinity:
                        requiredDuringSchedulingIgnoredDuringExecution:
                          - podAffinityTerm:
                              labelSelector:
                                matchExpressions:
                                  - key: app
                                    operator: In
                                    values:
                                      - istio-ingressgateway
                              topologyKey: kubernetes.io/hostname
                            weight: 100
                    replicaCount: 3
                    service:
                      type: LoadBalancer
                  name: istio-ingressgateway
```

Docs: Keycloak and MySQL

Mon, 01 Jan 0001 00:00:00 +0000

To scale the number of MySQL instances use the Verrazzano Custom Resource, not the MySQL StatefulSet. Directly modifying the StatefulSet may change the status of the cluster to ONLINE_PARTIAL.
You must have at least one running mysql-router to access the MySQL InnoDB Cluster. Scaling the number of mysql-router instances to zero may result in the MySQL Operator permanently losing communication with the cluster and Keycloak being unable to communicate with MySQL.
There are limitations to MySQL group replication policy to provide distributed coordination between servers. See MySQL Fault-tolerance.

Docs: Load Balancers on OCI

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano sets up the following load balancers on Kubernetes at installation:

Load balancer for NGINX ingress
Load balancer for Istio ingress

Verrazzano allows customizing the load balancers allocated by Oracle Cloud Infrastructure (OCI) using annotations defined by the OCI Cloud Controller Manager (OCI-CCM). For a detailed description of different load balancer customization annotations, see the documentation here.

This document describes how to use these annotations to customize the following settings for Verrazzano load balancers:

Load balancer shape
Load balancer min/max bandwidth
Private IP address and subnet placement

Customize the load balancer shape

At installation, Verrazzano lets you customize the shape and size of the load balancers created. The shape of an OCI load balancer specifies its maximum total bandwidth. By default, load balancers are created with a shape of 100Mbps. Other shapes are available, including 400Mbps and 8000Mbps.

For more details on service limits and shape, see here.

For example, you can set up a flexible NGINX load balancer with a min and max bandwidth as follows:

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: dev
  components:
    ingressNGINX:
      type: LoadBalancer
      overrides:
        - values:
            controller:
              service:
                annotations:
                  service.beta.kubernetes.io/oci-load-balancer-shape: flexible
                  service.beta.kubernetes.io/oci-load-balancer-shape-flex-max: "100"
                  service.beta.kubernetes.io/oci-load-balancer-shape-flex-min: "100"

For example, you can set up a flexible Istio load balancer with a min and max bandwidth as follows:

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: dev
  components:
    istio:
      overrides:
        - values:
            apiVersion: install.istio.io/v1alpha1
            kind: IstioOperator
            spec:
              components:
                ingressGateways:
                  - enabled: true
                    name: istio-ingressgateway
                    k8s:
                      serviceAnnotations:
                        service.beta.kubernetes.io/oci-load-balancer-shape: flexible
                        service.beta.kubernetes.io/oci-load-balancer-shape-flex-max: "100"
                        service.beta.kubernetes.io/oci-load-balancer-shape-flex-min: "100"

Use private IP addresses with a load balancer

At installation, Verrazzano lets you customize the IP address and subnet of the load balancers created. This is achieved using OCI-CCM annotations on the NGINX and Istio load balancer services, as documented here.

The following example configures the NGINX load balancer service to have a private load balancer IP address on the private subnet identified by the OCID ocid1.subnet.oc1.phx.aaaa..sdjxa, and uses the default (public) load balancer configuration for Istio.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: dev
  components:
    ingressNGINX:
      type: LoadBalancer
      overrides:
        - values:
            controller:
              service:
                annotations:
                  service.beta.kubernetes.io/oci-load-balancer-internal: "true"
                  service.beta.kubernetes.io/oci-load-balancer-subnet1: "ocid1.subnet.oc1.phx.aaaa..sdjxa"

The following example configures the Istio ingress gateway service to have a private load balancer IP address on the private subnet identified by the OCID ocid1.subnet.oc1.phx.aaaa..sdjxa, and uses the default (public) load balancer configuration for NGINX.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: dev
  components:  
    istio:
      overrides:
        - values:
            apiVersion: install.istio.io/v1alpha1
            kind: IstioOperator
            spec:
              components:
                ingressGateways:
                  - enabled: true
                    name: istio-ingressgateway
                    k8s:
                      serviceAnnotations:
                        service.beta.kubernetes.io/oci-load-balancer-internal: "true"
                        service.beta.kubernetes.io/oci-load-balancer-subnet1: "ocid1.subnet.oc1.phx.aaaa..sdjxa"

The following example configures both NGINX and Istio to have a private load balancer IP address on the private subnet identified by the OCID ocid1.subnet.oc1.phx.aaaa..sdjxa.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: dev
  components:
    ingressNGINX:
      type: LoadBalancer
      overrides:
        - values:
            controller:
              service:
                annotations:
                  service.beta.kubernetes.io/oci-load-balancer-internal: "true"
                  service.beta.kubernetes.io/oci-load-balancer-subnet1: "ocid1.subnet.oc1.phx.aaaa..sdjxa"
    istio:
      overrides:
        - values:
            apiVersion: install.istio.io/v1alpha1
            kind: IstioOperator
            spec:
              components:
                ingressGateways:
                  - enabled: true
                    name: istio-ingressgateway
                    k8s:
                      serviceAnnotations:
                        service.beta.kubernetes.io/oci-load-balancer-internal: "true"
                        service.beta.kubernetes.io/oci-load-balancer-subnet1: "ocid1.subnet.oc1.phx.aaaa..sdjxa"

Docs: OpenSearch

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano supports two cluster topologies for an OpenSearch cluster:

A single-node cluster: master, ingest, and data roles performed by a single node.
A multi-node cluster configuration with separate master, data, and ingest nodes.

For information about the default OpenSearch cluster configurations provided by Verrazzano, see Installation Profiles.

Plan cluster topology

Start with an initial estimate of your hardware needs. The following recommendations will provide you with initial, educated estimates, but for ideal sizing, you will need to test them with representative workloads, monitor their performance, and then reiterate.

Storage Requirements

Input	Description	Value
$s$	Stored data size in GiB (log size per day * days to retain).	User defined
$sr$	Shard replica count per index.	User defined
$o$	Overall overhead, which is a constant.	1.45

Minimum storage requirement = $ ( s * ( 1 + sr ) ) * o $

Example

If you have $s$ = 66 GiB (6 GiB of log size per day * 11 days to retain) and, if you choose one shard replica per index, which makes $sr$ = 1

Then, minimum storage requirement = $(66 * (1 + 1) ) * 1.45$ = 192 GiB

Overhead, which is defined in the previous table, can be further explained as follows.

Input	Description	Value
$io$	Indexing overhead: Extra space used other than the actual data, which is generally 10% ( 0.1 ) of the index size.	1 + 0.1 = 1.1
$lrs$	Linux reserved space: Linux reserves 5% of the file system for the root user for some OS operations.	1- 0.05 = .95
$oo$	OpenSearch overhead: OpenSearch keeps a maximum 20% of the instance for segment merges, logs, and other internal operations.	1- 0.2 = 0.8

Overall overhead $o$ = $ io / lrs / oo $ = 1.45

Memory

For every 100 GiB of your storage requirement, you should have 8 GiB of memory.

With reference to the Example:

For 192 GiB of storage requirement, you need 16 GiB of memory.

Number of Data Nodes

Input	Description	Value
$ts$	Total storage in GiB.	User defined
$mem$	Memory per data node in GiB.	User defined
$md$	Memory:data ratio (1:30 ratio means that you have 30 times more storage on the node than you have RAM; the value used would be 30).	User defined
$fc$	One data node for failover capacity, which is a constant.	1

ROUNDUP $ts / mem / md + fc$

With reference to the Example:

$ts$ = 192 GiB , $mem$ = 8 GiB , $md$ = 1:10 and $fc$ = 1

Then, number of data nodes = ROUNDUP $ 192 / 8 / 10 + 1 $ = 3

JVM heap memory

The heap size is the amount of RAM allocated to the JVM of an OpenSearch node. The OpenSearch process is very memory intensive and close to 50% of the memory available on a node should be allocated to the JVM. The JVM machine uses memory for indexing and search operations. The other 50% is required for the file system cache, which keeps data that is regularly accessed in memory. As a general rule, you should set -Xms and -Xmx to the same value, which should be 50% of your total available RAM, subject to a maximum of (approximately) 31 GiB.

CPU

Hardware requirements vary dramatically by workload, but, typically, two vCPU cores for every 100 GiB of your storage requirement is sufficient.

With reference to the Example:

For 192 GiB of storage, the vCPU cores required are 4.

Shard Size

For logging, shard sizes between 10 GiB and 50 GiB typically perform well. For search-intensive operations, 10-25 GiB typically is a good shard size. Overall, it is a best practice that, for a single shard, the OpenSearch shard size should not go above 50GiB. When the shards exceed 50 GiB, you will have to reindex your data.

Primary shards count

Input	Description	Value
$s$	Stored data size in GiB (log size per day * days to retain).	User defined
$sh$	Desired shard size in GiB.	User defined
$io$	Indexing overhead: Extra space used other than the actual data which is generally 10% of the index size.	0.1

Primary shards = $ ( s * (1 + io) ) / sh $

With reference to the Example:

$s$ = 66 GiB and if you choose shard size $sh$ = 30 GiB

Then, primary shards count = $ ( 66 * 1.1 )/ 30 $ = 2

Recommended alarms

You can customize Prometheus to enable Alertmanager and configure recommended alarms (add alert rules) to get insight into your OpenSearch cluster and take some actions proactively.

Use the OSDataNodeFilesystemSpaceFillingUp alert to indicate that the OpenSearch average disk usage has exceeded the specified threshold. Adjust the alert thresholds according to your needs.

kubectl apply -f - <<EOF
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  labels:
    release: prometheus-operator
  name: prometheus-operator-os
  namespace: verrazzano-monitoring
spec:
  groups:
    - name: os
      rules:
        - alert: OSDataNodeFilesystemSpaceFillingUp
          annotations:
            runbook_url: <link to runbook>
            summary: Opensearch average disk usage exceeded 75%.
          expr: |-
            1 - (es_fs_total_available_bytes{node=~".*data.*"}/ es_fs_total_total_bytes) > .75            
          for: 30m
          labels:
            severity: warning
  EOF

Configure cluster topology

You can customize the node characteristics of your OpenSearch cluster by using the spec.components.opensearch.nodes field in the Verrazzano custom resource. When installing or upgrading Verrazzano, you can use this field to define an OpenSearch cluster using node groups.

The following example overrides the dev installation profile, OpenSearch configuration (a single-node cluster with 1Gi of memory and ephemeral storage) to use a multi-node cluster (three master nodes, and three combination data/ingest nodes) with persistent storage.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: custom-opensearch-example
spec:
  profile: dev
  components:
    opensearch:
      nodes:
        - name: master
          replicas: 3
          roles:
            - master
          storage:
            size: 50Gi
          resources:
            requests:
              memory: 1.5Gi
        - name: data-ingest
          replicas: 3
          roles:
            - data
            - ingest
          storage:
            size: 100Gi
          resources:
            requests:
              memory: 1Gi
        # Override the default node groups because we are providing our own topology.
        - name: os-master
          replicas: 0
        - name: os-data
          replicas: 0
        - name: os-ingest
          replicas: 0

Listing the pods and persistent volumes in the verrazzano-system namespace for the previous configuration shows the expected nodes are running with the appropriate data volumes.

$ kubectl get pvc,pod -l verrazzano-component=opensearch -n verrazzano-system

# Sample output
NAME                                                             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/opensearch-master-vmi-system-master-0      Bound    pvc-9ace042a-dd68-4975-816d-f2ca0dc4d9d8   50Gi       RWO            standard       5m22s
persistentvolumeclaim/opensearch-master-vmi-system-master-1      Bound    pvc-8bf68c2c-235e-4bd5-8741-5a5cd3453934   50Gi       RWO            standard       5m21s
persistentvolumeclaim/opensearch-master-vmi-system-master-2      Bound    pvc-da8a48b1-5762-4669-98f0-8479f30043fc   50Gi       RWO            standard       5m21s
persistentvolumeclaim/vmi-system-data-ingest                     Bound    pvc-7ad9f275-632b-4aac-b7bf-c5115215937c   100Gi      RWO            standard       5m23s
persistentvolumeclaim/vmi-system-data-ingest-1                   Bound    pvc-8a293e51-2c20-4cae-916b-1ce46a780403   100Gi      RWO            standard       5m23s
persistentvolumeclaim/vmi-system-data-ingest-2                   Bound    pvc-0025fcef-1d8c-4307-977c-3921545c6730   100Gi      RWO            standard       5m22s

NAME                                                   READY   STATUS     RESTARTS   AGE
pod/coherence-operator-6ffb6bbd4d-bpssc                1/1     Running    1          8m2s
pod/fluentd-ndshl                                      2/2     Running    0          5m51s
pod/oam-kubernetes-runtime-85cfd899d8-z9gv6            1/1     Running    0          8m14s
pod/verrazzano-application-operator-5fbcdf6655-72tw9   1/1     Running    0          7m49s
pod/verrazzano-authproxy-5f9d479455-5bvvt              2/2     Running    0          7m43s
pod/verrazzano-console-5b857d7b47-djbrk                2/2     Running    0          5m51s
pod/verrazzano-monitoring-operator-b4b446567-pgnfw     2/2     Running    0          5m51s
pod/vmi-system-data-ingest-0-5485dcd95d-rkhvk          2/2     Running    0          5m21s
pod/vmi-system-data-ingest-1-8d7db6489-kdhbv           2/2     Running    1          5m21s
pod/vmi-system-data-ingest-2-699d6bdd9c-z7nzx          2/2     Running    0          5m21s
pod/vmi-system-grafana-7947cdd84b-b7mks                2/2     Running    0          5m21s
pod/vmi-system-kiali-6c7bd6658b-d2zq9                  2/2     Running    0          5m37s
pod/vmi-system-opensearchDashboards-7d47f65dfc-zhjxp   2/2     Running    0          5m21s
pod/vmi-system-master-0                                2/2     Running    0          5m21s
pod/vmi-system-master-1                                2/2     Running    0          5m21s
pod/vmi-system-master-2                                2/2     Running    0          5m21s
pod/weblogic-operator-666b548749-lj66t                 2/2     Running    0          7m48s

Running the command kubectl describe pod -n verrazzano-system vmi-system-data-ingest-0-5485dcd95d-rkhvk shows the requested amount of memory.

Containers:
  os-data:
    ...
    Requests:
      memory:   1Gi

Default Index State Management policies

Index State Management (ISM) policies configure OpenSearch to manage the data in your indices. You can use policies to automatically rollover and prune old data, preventing your OpenSearch cluster from running out of disk space.

To help you manage issues, such as low disk space, the following two ISM policies are created by default:

vz-system: Manages the data in the Verrazzano system index.
vz-application: Manages the data in the application-related indices having the pattern, verrazzano-application*.

Both ISM policies have three states:

Hot: This is the default state. If the primary shard size is greater than the defined size (5 GB for vz-system and 1 GB for vz-application) or the index age is greater than the defined number of days (30 days for vz-system and 7 days for vz-application), then the index will be rolled over.
Cold: In this state, the index will be closed if the index age is greater than the defined number of days (30 days for vz-system and 7 days for vz-application). A closed index is blocked for read or write operations and does not allow any operations that the opened indices allow.
Delete: In this state, the index will be deleted if the index age is greater than the defined number of days (35 days for vz-system and 12 days for vz-application).

Override default ISM policies

The vz-system and vz-application policies are immutable and any change to these policies will be reverted immediately. However, the following two methods will override this behavior:

Disable default policies: You can disable the use of these default policies by setting the flag spec.components.opensearch.disableDefaultPolicy to true in the Verrazzano CR. This will delete the default ISM policies.
Override default policies: Both these default policies have a zero (0) priority. You can override the default policies by creating policies with policy.ism_template.priority greater than 0. Check Configure ISM Policies in order to configure/create your own policies.

NOTE

Avoid creating policies with policy IDs vz-system or vz-application because they are reserved for Verrazzano default policies names. In the Verrazzano CR, by default, if the flag spec.components.opensearch.disableDefaultPolicy is set to false, then policies that are created with these names will be overridden with the default ISM policies, .
The default policy will be applied only to the newly created indices. To manually attach the new policies to the older indices, see Step 2: Attach policies to indexes.

Configure ISM policies

Verrazzano lets you configure OpenSearch ISM policies using the Verrazzano custom resource. The ISM policy created by Verrazzano will contain two states: ingest and delete. The ingest state can be configured only for the rollover action. The rollover action for the ingest state will be configured based on the rollover configuration provided in the Verrazzano custom resource.

The following policy example configures OpenSearch to manage indices matching the pattern my-app-*. The data in these indices will be automatically pruned every 14 days, and will be rolled over if an index meets at least one of the following criteria:

Is three or more days old
Contains 1,000 documents or more
Is 10 GB in size or larger

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: custom-opensearch-example
spec:
  profile: dev
  components:
    opensearch:
      policies:
        - policyName: my-app
          indexPattern: my-app-*
          minIndexAge: 14d
          rollover:
            minIndexAge: 3d
            minDocCount: 1000
            minSize: 10gb

The previous Verrazzano custom resource will generate the following ISM policy.

{
  "_id" : "my-app",
  "_version" : 17,
  "_seq_no" : 16,
  "_primary_term" : 1,
  "policy" : {
    "policy_id" : "my-app",
    "description" : "__vmi-managed__",
    "last_updated_time" : 1671096525963,
    "schema_version" : 12,
    "error_notification" : null,
    "default_state" : "ingest",
    "states" : [
      {
        "name" : "ingest",
        "actions" : [
          {
            "rollover" : {
              "min_size" : "10gb",
              "min_doc_count" : 1000,
              "min_index_age" : "3d"
            }
          }
        ],
        "transitions" : [
          {
            "state_name" : "delete",
            "conditions" : {
              "min_index_age" : "14d"
            }
          }
        ]
      },
      {
        "name" : "delete",
        "actions" : [
          {
            "delete" : { }
          }
        ],
        "transitions" : [ ]
      }
    ],
    "ism_template" : [
      {
        "index_patterns" : [
          "my-app-*"
        ],
        "priority" : 1,
        "last_updated_time" : 1671096525963
      }
    ]
  }
}

NOTE: The ISM policy created using the Verrazzano custom resource contains a minimal set of configurations. To create a more detailed ISM policy, you can also use the OpenSearch REST API. To create a policy using the OpenSearch API, do the following:

$ PASS=$(kubectl get secret \
    --namespace verrazzano-system verrazzano \
    -o jsonpath={.data.password} | base64 \
    --decode; echo)

$ HOST=$(kubectl get ingress \
    -n verrazzano-system vmi-system-os-ingest \
    -o jsonpath={.spec.rules[0].host})

$ curl -ik -X PUT --user verrazzano:$PASS https://$HOST/_plugins/_ism/policies/policy_3 \
    -H 'Content-Type: application/json' \
    --data-binary @- << EOF
{
  "policy": {
    "description": "ingesting logs",
    "default_state": "ingest",
    "states": [
      {
        "name": "ingest",
        "actions": [
          {
            "rollover": {
              "min_doc_count": 5
            }
          }
        ],
        "transitions": [
          {
            "state_name": "search"
          }
        ]
      },
      {
        "name": "search",
        "actions": [],
        "transitions": [
          {
            "state_name": "delete",
            "conditions": {
              "min_index_age": "5m"
            }
          }
        ]
      },
      {
        "name": "delete",
        "actions": [
          {
            "delete": {}
          }
        ],
        "transitions": []
      }
    ]
  }
}
EOF

To view existing policies, do the following:

$ curl -ik \
    --user verrazzano:$PASS https://$HOST/_plugins/_ism/policies

Override default number of shards and replicas

Verrazzano provides a default index template, verrazzano-data-stream. In initial Verrazzano v1.5 installations (not upgrades), the default index template creates one shard and one replica for each index. (In previous and upgrade installations, it creates five shards and one replica.) You can override the default number of shards or replicas by overriding the default index template.

To do that, you need to get the default index template, copy the contents and change the number of shards, replicas, and index pattern, and then create your own index template with a higher priority so that the new template will override the default one.

You can use the OpenSearch Dev Tools Console to send given queries to OpenSearch. To open the console, select Dev Tools on the main OpenSearch Dashboards page and write your queries in the editor pane on the left side of the console.

To get the existing, default template:

$ GET /_index_template/verrazzano-data-stream

Here is an example to create a new index template, which changes the number of shards to 3 and replicas to 2.

$ PUT _index_template/my-template
    {
        "index_patterns" : [
          "verrazzano-application-myapp*"
        ],
        "template" : {
          "settings" : {
            "index" : {
              "mapping" : {
                "total_fields" : {
                  "limit" : "2000"
                }
              },
              "refresh_interval" : "5s",
              "number_of_shards" : "3",
              "auto_expand_replicas" : "0-1",
              "number_of_replicas" : "2"
            }
          },
          "mappings" : {
            "dynamic_templates" : [
              {
                "message_field" : {
                  "path_match" : "message",
                  "mapping" : {
                    "norms" : false,
                    "type" : "text"
                  },
                  "match_mapping_type" : "string"
                }
              },
              {
                "object_fields" : {
                  "mapping" : {
                    "type" : "object"
                  },
                  "match_mapping_type" : "object",
                  "match" : "*"
                }
              },
              {
                "all_non_object_fields" : {
                  "mapping" : {
                    "norms" : false,
                    "type" : "text",
                    "fields" : {
                      "keyword" : {
                        "ignore_above" : 256,
                        "type" : "keyword"
                      }
                    }
                  },
                  "match" : "*"
                }
              }
            ],
            "properties" : {
              "@timestamp" : {
                "format" : "strict_date_time||strict_date_optional_time||epoch_millis",
                "type" : "date"
              }
            }
          }
        },
        "priority" : 201,
        "data_stream" : {
          "timestamp_field" : {
            "name" : "@timestamp"
          }
        }
}

With this example, new indices that match the verrazzano-application-myapp* index pattern will be created with three shards and two replicas, and other indices that don’t match will continue to be created with the default number of shards and replicas. For more information, see Index templates in the OpenSearch documentation.

Install OpenSearch and OpenSearch Dashboards plug-ins

Verrazzano supports OpenSearch and OpenSearch Dashboard plug-in installation by providing plug-ins in the Verrazzano custom resource. To install plug-ins for OpenSearch, you define the field spec.components.opensearch.plugins in the Verrazzano custom resource.

The following Verrazzano custom resource example installs the analysis-stempel and opensearch-anomaly-detection plug-ins for OpenSearch:

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: custom-opensearch-example
spec:
  profile: dev
  components:
    opensearch:
      plugins:
        enabled: true
        installList:
          - analysis-stempel
          - https://repo1.maven.org/maven2/org/opensearch/plugin/opensearch-anomaly-detection/2.2.0.0/opensearch-anomaly-detection-2.2.0.0.zip

Pre-built plug-ins for OpenSearch

Here are some pre-built plug-ins that are bundled with the OpenSearch image:

analysis-icu
analysis-kuromoji
analysis-phonetic
analysis-smartcn
ingest-attachment
mapper-murmur3
mapper-size
opensearch-index-management
opensearch-job-scheduler
prometheus-exporter
repository-s3

There are three ways to specify a plug-in in the plugins.installList:

Specify a plug-in by name:

There are some pre-built additional plug-ins that are the only plug-ins you can install by name.
```
installList:
        - analysis-icu
```
Specify a plug-in from a remote ZIP file:

Provide the URL to a remote ZIP file that contains the required plug-in.
```
installList:
        - https://repo1.maven.org/maven2/org/opensearch/plugin/opensearch-anomaly-detection/2.2.0.0/opensearch-anomaly-detection-2.2.0.0.zip
```
Specify a plug-in using Maven coordinates:

Provide the Maven coordinates for the available artifacts and versions hosted on Maven Central.
```
installList:
        - org.opensearch.plugin:opensearch-anomaly-detection:2.2.0.0
```
NOTE
- Adding a new plug-in to the plugins.installList or removing a plug-in from the plugins.installList will result in restarting the OpenSearch related pods.
- To verify that a plug-in has installed successfully, make sure that no pod is in the CrashLoopBackOff state and the plug-in functionality is working fine.
- If there is any error during plug-in installation, then one of the OS master pods will go into the CrashLoopBackOff state, while other pods will still be in the Running state, and the OpenSearch cluster will be healthy and functional. Check the logs for the exact reason of the failure.
- Your environment must be able to connect to the Internet to access the provided plug-in URL or Maven Central to install the plug-in. In the case of an Internet issue, you might see SocketException or UnknownHostException exceptions in the logs. To resolve this issue, make sure that the pods are connected to the Internet.
- To be compatible, major, minor, and patch plug-in versions must match the OpenSearch major, minor, and patch versions. For example, plug-ins versions 2.3.0.x are compatible only with OpenSearch version 2.3.0.

For OpenSearch Dashboard, you can provide the plug-ins by defining the field spec.components.opensearch-dashboards.plugins in the Verrazzano custom resource.

Pre-built plug-ins for OpenSearch Dashboards

Here is a pre-built plug-in that is bundled with the OpenSearch Dashboard image: indexManagementDashboards

Here is a Verrazzano custom resource example to install plug-ins for the OpenSearch Dashboards:

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: custom-opensearch-example
spec:
  profile: dev
  components:
    opensearchDashboards:
      plugins:
        enabled: true
        installList:
          - <URL to OpenSearch Dashboard plugin ZIP file>

Docs: Persistent Storage

Mon, 01 Jan 0001 00:00:00 +0000

The following components can use persistent storage:

OpenSearch
OpenSearch Dashboards
Prometheus
Grafana
Keycloak/MySQL

By default, each Verrazzano installation profile has different storage characteristics. The dev profile uses ephemeral storage only, but in all other profiles, each of the listed components use persistent storage. For more information, see Profile Configurations.

NOTE

Ephemeral storage is not recommended for use in production; Kubernetes pods can be restarted at any time, leading to a loss of data and system instability if non-persistent storage is used. Persistent storage is recommended for all use cases beyond evaluation or development.

While each profile has its own default persistence settings, in each case you have the option to override the profile defaults to customize your persistence settings.

You can customize the persistence settings for these components through the VerrazzanoSpec, as follows:

Overriding the persistence settings for all components (Keycloak, Grafana, Prometheus, OpenSearch, and OpenSearch Dashboards) by using the defaultVolumeSource field.
Overriding the persistence settings for Keycloak by using the volumeSource field on that component’s configuration.

You can set the global defaultVolumeSource and component-level volumeSource fields to one of the following values:

Value	Storage
`emptyDir`	Ephemeral storage; should not be used for production scenarios.
`persistentVolumeClaim`	A `PersistentVolumeClaimVolumeSource` where the `claimSource` field references a named `volumeClaimSpecTemplate`.

When you want to use a persistentVolumeClaim to override the storage settings for components, you must do the following:

Create a volumeClaimSpecTemplate which identifies the desired persistence settings.
Configure a persistentVolumeClaim for the component where the claimName field references the template you created previously.

This lets you create named persistence settings that can be shared across multiple components within a Verrazzano configuration. Note that the existence of a persistence template in the volumeClaimSpecTemplates list does not directly result in the creation of a persistent volume, or affect any component storage settings until it is referenced by either defaultVolumeSource or a specific component’s volumeSource.

Examples

Review the following customizing persistent storage examples:

Customize persistence globally using defaultVolumeSource

NOTE

Beginning in version 1.5, persistent storage is always required for MySQL. If defaultVolumeSource is configured to use emptyDir, then MySQL will ignore that configuration setting.

If defaultVolumeSource is configured, then that setting will be used for all components that require storage.

For example, the following Verrazzano configuration uses the prod profile, but disables persistent storage for all components.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: no-storage-prod
spec:
  profile: prod
  defaultVolumeSource:
      emptyDir: {}

The following example uses persistentVolumeClaim to override persistence settings globally for a prod profile, to use 100Gi volumes for all components, instead of the default of 50Gi.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: prod-global-override
spec:
  profile: prod
  defaultVolumeSource:
    persistentVolumeClaim:
      claimName: globalOverride
  volumeClaimSpecTemplates:
    - metadata:
        name: globalOverride
      spec:
        resources:
          requests:
            storage: 100Gi

The following example uses a managed-cluster profile but overrides the persistence settings to use ephemeral storage.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: mgdcluster-empty-storage-example
spec:
  profile: managed-cluster
  defaultVolumeSource:
    emptyDir: {}  # Use emphemeral storage for all Components unless overridden

Customize PersistentVolumeClaim settings for Keycloak using volumeSource

The following example Verrazzano configuration enables a 100Gi PersistentVolumeClaim for the MySQL component in Keycloak in a dev profile configuration. This overrides the default of ephemeral storage for Keycloak in that profile, while retaining the default storage settings for other components.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: dev-mysql-storage-example
spec:
  profile: dev
  components:
    keycloak:
      mysql:
        volumeSource:
          persistentVolumeClaim:
            claimName: mysql  # Use the "mysql" PVC template for the MySQL volume configuration
  volumeClaimSpecTemplates:
  - metadata:
      name: mysql      
    spec:
      resources:
        requests:
          storage: 100Gi

Use global and local persistence settings together

The following example uses a dev installation profile, but overrides the profile persistence settings to:

Use 200Gi volumes for all components by default.
Use a 100Gi volume for the MySQL instance associated with Keycloak.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: dev-storage-example
spec:
  profile: dev
  defaultVolumeSource:
    persistentVolumeClaim:
      claimName: vmi     # Set storage globally for the metrics stack
  components:
    keycloak:
      mysql:
        volumeSource:
          persistentVolumeClaim:
            claimName: mysql  # Set storage separately for keycloak's MySql instance
  volumeClaimSpecTemplates:
    - metadata:
        name: mysql
      spec:
        resources:
          requests:
            storage: 100Gi
    - metadata:
        name: vmi
      spec:
        resources:
          requests:
            storage: 200Gi

Input	Description	Value
\(s\)	Stored data size in GiB (log size per day * days to retain).	User defined
\(sr\)	Shard replica count per index.	User defined
\(o\)	Overall overhead, which is a constant.	1.45

Input	Description	Value
\(io\)	Indexing overhead: Extra space used other than the actual data, which is generally 10% ( 0.1 ) of the index size.	1 + 0.1 = 1.1
\(lrs\)	Linux reserved space: Linux reserves 5% of the file system for the root user for some OS operations.	1- 0.05 = .95
\(oo\)	OpenSearch overhead: OpenSearch keeps a maximum 20% of the instance for segment merges, logs, and other internal operations.	1- 0.2 = 0.8

Input	Description	Value
\(ts\)	Total storage in GiB.	User defined
\(mem\)	Memory per data node in GiB.	User defined
\(md\)	Memory:data ratio (1:30 ratio means that you have 30 times more storage on the node than you have RAM; the value used would be 30).	User defined
\(fc\)	One data node for failover capacity, which is a constant.	1