Verrazzano Enterprise Container Platform – Networking

Docs: Network Security

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano manages and secures network traffic between Verrazzano system components and deployed applications. Verrazzano does not manage or secure traffic for the Kubernetes cluster itself, or for non-Verrazzano services or applications running in the cluster. Traffic is secured at two levels in the network stack:

ISO Layer 3/4: Using NetworkPolicies to control IP access to Pods.
ISO Layer 6: Using TLS and mutual TLS authentication (mTLS) to provide authentication, confidentiality, and integrity for connections within the cluster and for external connections.

NetworkPolicies

By default, all Pods in a Kubernetes cluster have network access to all other Pods in the cluster. Kubernetes has a NetworkPolicy resource that provides network level 3 and 4 security for Pods, restricting both ingress and egress IP traffic for a set of Pods in a namespace. Verrazzano configures all system components with NetworkPolicies to control ingress. Egress is not restricted.

NOTE: A NetworkPolicy resource needs a NetworkPolicy controller to implement the policy, otherwise the policy has no effect. You must install a Kubernetes Container Network Interface (CNI) plug-in that provides a NetworkPolicy controller, such as Calico, before installing Verrazzano, or else the policies are ignored.

NetworkPolicies for system components

Verrazzano installs a set of NetworkPolicies for system components to control ingress into the Pods. A policy is scoped to a namespace and uses selectors to specify the Pods that the policy applies to, along with the ingress and egress rules. For example, the following policy applies to the Verrazzano API Pod in the verrazzano-system namespace. This policy allows network traffic from NGINX Ingress Controller on port 8775 and from Prometheus on port 15090. No other Pods can reach those ports or any other ports of the Verrazzano API Pod. Notice that namespace selectors need to be used; the NetworkPolicy resource does not support specifying the namespace name.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
...
spec:
  PodSelector:
    matchLabels:
      app: verrazzano-api
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          verrazzano.io/namespace: ingress-nginx
      PodSelector:
        matchLabels:
          app.kubernetes.io/instance: ingress-controller
    ports:
    - port: 8775
      protocol: TCP
  - from:
    - namespaceSelector:
        matchLabels:
          verrazzano.io/namespace: verrazzano-system
      PodSelector:
        matchLabels:
          app: system-prometheus
    ports:
    - port: 15090
      protocol: TCP

The following table shows all of the ingresses that allow network traffic into system components. The ports shown are Pod ports, which is what NetworkPolicies require.

Component	Pod Port	From	Description
Argo CD	443	NGINX Ingress	Access from external client
Argo CD	8080	Argo CD Server, Internal	Argo CD Server data port
cert-manager	9402	Prometheus	Prometheus scraping
Coherence Operator	9443	Prometheus	Webhook entrypoint
Istio control plane	15012	Envoy	Envoy access to `istiod`
Istio control plane	15014	Prometheus	Prometheus scraping.
Istio control plane	15017	Kubernetes API Server	Webhook entrypoint
Istio egress gateway	8443	Mesh services	Application egress
Istio egress gateway	15090	Prometheus	Prometheus scraping
Istio ingress gateway	8443	External	Application ingress
Istio ingress gateway	15090	Prometheus	Prometheus scraping
Keycloak	8080	NGINX Ingress	Access from external client
Keycloak	15090	Prometheus	Prometheus scraping
MySql	15090	Prometheus	Prometheus scraping
MySql	3306	Keycloak	Keycloak datastore
Node exporter	9100	Prometheus	Prometheus scraping
OpenSearch	8775	Fluentd	Access from Fluentd
OpenSearch	9200	OpenSearch Dashboards, Internal	OpenSearch data port
OpenSearch	9300	Internal	OpenSearch cluster port
OpenSearch	15090	Prometheus	Envoy metrics scraping
OpenSearch	8775	NGINX Ingress	Access from external client
Prometheus	8775	NGINX Ingress	Access from external client
Prometheus	9090	Grafana	Access for Grafana console
Rancher	80	NGINX Ingress	Access from external client
Rancher	9443	Kubernetes API Server	Webhook entrypoint
Verrazzano Application Operator	9443	Kubernetes API Server	Webhook entrypoint
Verrazzano Authentication Proxy	8775	NGINX Ingress	Access from external client
Verrazzano Authentication Proxy	15090	Prometheus	Prometheus scraping
Verrazzano Console	8000	NGINX Ingress	Access from external client
Verrazzano Console	15090	Prometheus	Prometheus scraping
Verrazzano Platform Operator	9443	Kubernetes API Server	Webhook entrypoint

NetworkPolicies for applications

By default, applications do not have NetworkPolicies that restrict ingress into the application or egress from it. You can configure them for the application namespaces using the NetworkPolicy section of a Verrazzano project.

NOTE

Verrazzano requires specific ingress to and egress from application pods. If you add a NetworkPolicy for your application namespace or pods, you must add an additional policy to ensure that Verrazzano still has the required access it needs. The ingress policy is needed only if you restrict ingress. Likewise, the egress policy is needed only if you restrict egress. The following are the ingress and egress NetworkPolicies:

Ingress NetworkPolicies

  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          verrazzano.io/namespace: istio-system
      podSelector:
        matchLabels:
          app: istiod
  - from:
    - namespaceSelector:
        matchLabels:
          verrazzano.io/namespace: istio-system
      podSelector:
        matchLabels:
          app: istio-ingressgateway
  - from:
    - namespaceSelector:
        matchLabels:
          verrazzano.io/namespace: verrazzano-system
      podSelector:
        matchLabels:
          app: system-prometheus
  - from:
    - namespaceSelector:
        matchLabels:
          verrazzano.io/namespace: verrazzano-system
      podSelector:
        matchLabels:
          app: coherence-operator
  - from:
    - namespaceSelector:
        matchLabels:
          verrazzano.io/namespace: verrazzano-system
      podSelector:
        matchLabels:
          app: weblogic-operator

Egress NetworkPolicies

  egress:
  - ports:
    - port: 15012
      protocol: TCP
    to:
    - namespaceSelector:
        matchLabels:
          verrazzano.io/namespace: istio-system
      podSelector:
        matchLabels:
          app: istiod
  - to:
    - namespaceSelector:
        matchLabels:
          verrazzano.io/namespace: istio-system
      podSelector:
        matchLabels:
          app: istio-egressgateway
  - ports:
    - port: 53
      protocol: TCP
    - port: 53
      protocol: UDP
    to:
    - namespaceSelector:
        matchLabels:
          verrazzano.io/namespace: kube-system
  - ports:
    - port: 8000
      protocol: TCP
    to:
    - namespaceSelector:
        matchLabels:
          verrazzano.io/namespace: verrazzano-system
      podSelector:
        matchLabels:
          app: coherence-operator

NetworkPolicies for Envoy sidecar proxies

As mentioned, Envoy sidecar proxies run in both system component pods and application pods. Each proxy sends requests to the Istio control plane pod, istiod, for a variety of reasons. During installation, Verrazzano creates a NetworkPolicy named istiod-access in the istio-system namespace to give ingress to system component and application sidecar proxies.

Mutual TLS authentication (mTLS)

Istio can be enabled to use mTLS between services in the mesh, and also between the Istio gateways and Envoy sidecar proxies. There are various options to customize mTLS usage, for example it can be disabled on a per-port level. The Istio control plane, Istiod, is a CA and provides key and certificate rotation for the Envoy proxies, both gateways and sidecars.

Verrazzano configures Istio to have strict mTLS for the mesh. All components and applications put into the mesh will use mTLS, with the exception of Coherence clusters, which are not in the mesh. Also, all traffic between the Istio ingress gateway and mesh sidecars use mTLS, and the same is true between the proxy sidecars and the egress gateway.

Verrazzano sets up mTLS during installation with the PeerAuthentication resource as follows:

apiVersion: v1
items:
- apiVersion: security.istio.io/v1beta1
  kind: PeerAuthentication
  ...
  spec:
    mtls:
      mode: STRICT

TLS

TLS is used by external clients to access the cluster, both through the NGINX Ingress Controller and the Istio ingress gateway. The certificate used by these TLS connections vary; see Verrazzano security for details. All TLS connections are terminated at the ingress proxy. Traffic between the two proxies and the internal cluster Pods always uses mTLS, because those Pods are all in the Istio mesh.

Istio mesh

Istio provides extensive security protection for both authentication and authorization, as described in Istio Security. Access control and mTLS are two security features that Verrazzano configures. These security features are available in the context of a service mesh.

A service mesh is an infrastructure layer that provides certain capabilities like security, observability, load balancing, and such, for services. Istio defines a service mesh here. In the context of Istio on Kubernetes, a service in the mesh is a Kubernetes Service. Consider the Bob’s Books example application, which has several OAM Components defined. At runtime, there is a Kubernetes Service for each component, and each Service is in the mesh, with one or more Pods associated with the service. All services in the mesh have an Envoy proxy in front of their Pods, intercepting network traffic to and from the Pod. In Kubernetes, that proxy happens to be a sidecar running in each Pod.

There are various ways to put a service in the mesh. Verrazzano uses the namespace label, istio-injection: enabled, to designate that all Pods in a given namespace are in the mesh. When a Pod is created in that namespace, the Istio control plane mutating webhook, changes the Pod spec to add the Envoy proxy sidecar container, causing the Pod to be in the mesh.

Disabling sidecar injection

In certain cases, Verrazzano needs to disable sidecar injection for specific Pods in a namespace. This is done in two ways: first, during installation, Verrazzano modifies the istio-sidecar-injector ConfigMap using a Helm override file for the Istio chart. This excludes several components from the mesh, such as the Verrazzano application operator. Second, certain Pods, such as Coherence Pods, are labeled at runtime with sidecar.istio.io/inject="false" to exclude them from the mesh.

Components in the mesh

The following Verrazzano components are in the mesh and use mTLS for all service to service communication.

Argo CD
Fluentd
Grafana
Kiali
Keycloak
MySQL
NGINX Ingress Controller
OpenSearch
OpenSearch Dashboards
Prometheus
Verrazzano Authentication Proxy
Verrazzano Console
WebLogic Kubernetes Operator

Some of these components, have mesh-related details that are worth noting, as described in the following sections.

NGINX

The NGINX Ingress Controller listens for HTTPS traffic, and provides ingress into the cluster. NGINX is configured to do TLS termination of client connections. All traffic from NGINX to the mesh services use mTLS, which means that traffic is fully encrypted from the client to the target back-end services.

Keycloak and MySQL

Keycloak and MySQL are also in the mesh and use mTLS for network traffic. Because all of the components that use Keycloak are in the mesh, there is end to end mTLS security for all identity management handled by Keycloak. The following components access Keycloak:

Verrazzano Authentication Proxy
Verrazzano Console
OpenSearch
Prometheus
Grafana
Kiali
OpenSearch Dashboards

Prometheus

Although Prometheus is in the mesh, it is configured to use the Envoy sidecar and mTLS only when communicating with Keycloak. All the traffic related to scraping metrics, bypasses the sidecar proxy, doesn’t use the service IP address, but rather connects to the scrape target using the Pod IP address. If the scrape target is in the mesh, then HTTPS is used; otherwise, HTTP is used. For Verrazzano multicluster, Prometheus also connects from the admin cluster to the Prometheus server in the managed cluster by using the managed cluster NGINX Ingress, using HTTPS. Prometheus is in the managed cluster and never establishes connections to targets outside the cluster.

Because Prometheus is in the mesh, additional configuration is done to allow the Envoy sidecar to be bypassed when scraping Pods. This is done with the Prometheus Pod annotation traffic.sidecar.istio.io/includeOutboundIPRanges: <keycloak-service-ip>. This causes traffic bound for Keycloak to go through the Envoy sidecar, and all other traffic to bypass the sidecar.

WebLogic Kubernetes Operator

Applications in the mesh

Before you create a Verrazzano application, you should decide if it should be in the mesh. You control sidecar injection, for example, mesh inclusion, by labeling the application namespace with istio-injection=enabled or istio-injection=disabled. By default, applications will not be put in the mesh if that label is missing. If your application uses a Verrazzano project, then Verrazzano will label the namespaces in the project to enable injection. If the application is in the mesh, then mTLS will be used. You can change the PeerAuthentication mTLS mode as desired if you don’t want strict mTLS. Also, if you need to add mTLS port exceptions, you can do this with DestinationRules or by creating another PeerAuthentication resource in the application namespace. Consult the Istio documentation for more information.

WebLogic

When the WebLogic Kubernetes Operator creates a domain, it needs to communicate with the Pods in the domain. Verrazzano puts the operator in the mesh so that it can communicate with the domain Pods using mTLS. Because of that, the WebLogic domain must be created in the mesh. Also, because mTLS is used, do not configure WebLogic to use TLS. If you want to use a custom certificate for your application, you can specify that in the ApplicationConfiguration, but that TLS connection will be terminated at the Istio ingress gateway, which you configure using a Verrazzano IngressTrait.

Coherence

Coherence clusters are represented by the Coherence resource, and are not in the mesh. When Verrazzano creates a Coherence cluster in a namespace that is annotated to do sidecar injection, it disables injection of the Coherence resource using the sidecar.istio.io/inject="false" label shown previously. Furthermore, Verrazzano will create a DestinationRule in the application namespace to disable mTLS for the Coherence extend port 9000. This allows a service in the mesh to call the Coherence extend proxy. For an example, see Bobs Books.

Here is an example of a DestinationRule created for the Bob’s Books application which includes a Coherence cluster.

API Version:  networking.istio.io/v1beta1
Kind:         DestinationRule
...
Spec:
  Host:  *.bobs-books.svc.cluster.local
  Traffic Policy:
    Port Level Settings:
      Port:
        Number:  9000
      Tls:
    Tls:
      Mode:  ISTIO_MUTUAL

Istio access control

Istio lets you control access to your workload in the mesh using the AuthorizationPolicy resource. This lets you control which services or Pods can access your workloads. Some of these options require mTLS; for more information, see Authorization Policy.

Verrazzano always creates AuthorizationPolicies for applications but never for system components. During application deployment, Verrazzano creates the policy in the application namespace and configures it to allow access from the following:

Other Pods in the application
Istio ingress gateway
Prometheus scraper

This prevents other Pods in the cluster from gaining network access to the application Pods. Istio uses a service identity to determine the identity of the request’s origin; for Kubernetes this identity is a service account. Verrazzano creates a per-application AuthorizationPolicy as follows:

AuthorizationPolicy
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
...
spec:
  rules:
    - from:
    - source:
  principals:
    - cluster.local/ns/sales/sa/greeter
    - cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account
    - cluster.local/ns/verrazzano-system/sa/verrazzano-monitoring-operator

WebLogic domain access

For WebLogic applications, the WebLogic Kubernetes Operator must have access to the domain Pods for two reasons. First, it must access the domain servers to get health status; second, it must inject configuration into the Monitoring Exporter sidecar running in the domain server Pods. When a WebLogic domain is created, Verrazzano adds an additional source, cluster.local/ns/verrazzano-system/sa/weblogic-operator-sa to the principals section to permit that access.

Docs: Network Traffic

Mon, 01 Jan 0001 00:00:00 +0000

Network traffic refers to the data flowing across the network. In the context of this document, it is useful to think of network traffic from two perspectives: traffic based on direction and traffic related to component types, system, or applications. Traffic direction is either north-south traffic, which enters and leaves the cluster, or east-west traffic, which stays within the cluster.

First is a description of getting traffic into the cluster, then how traffic flows after it is in the cluster.

Ingress

Ingress is an overloaded term, so it needs to be understood in context. Sometimes the term means external access into the cluster, as in “ingress to the cluster.” The term also refers to the Kubernetes Ingress resource. In addition, it might be used to mean network ingress to a container in a Pod. Here, it’s used to refer to both general ingress into the cluster and the Kubernetes Ingress resource.

During installation, Verrazzano creates the necessary network resources to access both system components and applications. The following ingress and load balancers descriptions are in the context of a Verrazzano installation.

LoadBalancer Services

To reach Pods from outside a cluster, an external IP address must be exposed using a LoadBalancer or NodePort service. Verrazzano creates two LoadBalancer services, one for system component traffic and another for application traffic. The specifics of how the service gets traffic into the cluster depends on the underlying Kubernetes platform. With Oracle Cloud Infrastructure Container Engine for Kubernetes (OKE), creating a LoadBalancer type service will result in an Oracle Cloud Infrastructure load balancer being created and configured to load balance to a set of Pods.

Ingress for system components

To provide ingress to system components, Verrazzano installs a NGINX Ingress Controller, which includes a NGINX load balancer. Verrazzano also creates Kubernetes Ingress resources to configure ingress for each system component that requires ingress. An Ingress resource is used is to specify HTTP/HTTPS routes to Kubernetes services, along with an endpoint host name and a TLS certificate. An Ingress by itself doesn’t do anything; it is just a resource. An ingress controller is needed to watch Ingress resources and reconcile them, configuring the underlying Kubernetes load balancer to handle the service routing. The NGINX Ingress Controller processes Ingress resources and configures NGINX with the ingress route information, and such.

The NGINX Ingress Controller is a LoadBalancer service, as seen here:

$ kubectl get service -n ingress-nginx

# Sample output
ingress-controller-ingress-nginx-controller           LoadBalancer

Using the OKE example, traffic entering the Oracle Cloud Infrastructure load balancer is routed to the NGINX load balancer, then routed from there to the Pods belonging to the services described in the Ingress.

Ingress for applications

Verrazzano also provides ingress into applications, but uses an Istio ingress gateway, which is an Envoy proxy, instead of NGINX. Istio has a Gateway resource that provides load balancer information, such as hosts, ports, and certificates for traffic coming into the mesh. For more information, see Istio Gateway. Just as an Ingress needs a corresponding Ingress controller, the same is true for the Gateway resource, where there is a corresponding Istio ingress gateway controller. However, unlike the Ingress, the Gateway resource doesn’t have service routing information. That is handled by the Istio VirtualService resource. The combination of Gateway and VirtualService is basically a superset of Ingress, because the combination provides more features than Ingress. In summary, the Istio ingress gateway provides ingress to the cluster using information from both the Gateway and VirtualService resources.

Because Verrazzano doesn’t create any applications during installations, there is no need to create a Gateway and VirtualService at that time. However, during installation, Verrazzano does create the Istio ingress gateway, which is a LoadBalancer service, along with the Istio egress gateway, which is a ClusterIP service.

$ kubectl get service -n istio-system

# Sample output
istio-ingressgateway   LoadBalancer

Again, referring to the OKE use case, this means that there will another Oracle Cloud Infrastructure load balancer created, routing traffic to the Istio ingress gateway Pod, for example, the Envoy proxy.

External DNS

When you install Verrazzano, you can optionally specify an external DNS for your domain. If you do that, Verrazzano will not only create the DNS records, using ExternalDNS, but also it will configure your host name in the Ingress resources. You can then use that host name to access the system components through the NGINX Ingress Controller.

System traffic

System traffic includes all traffic that enters and leaves system Pods.

North-south system traffic

North-south traffic includes all system traffic that enters or leaves a Kubernetes cluster.

Ingress

The following lists the Verrazzano system components which are accessed through the NGINX Ingress Controller from a client external to the cluster:

argoCD
OpenSearch
Keycloak
OpenSearch Dashboards
Grafana
Prometheus
Rancher
Verrazzano Console
Verrazzano API

Egress

The following table shows Verrazzano system components that initiate requests to a destination outside the cluster.

Component	Destination	Description
argoCD	Git webhooks (GitHub, GitLab, Bitbucket)	Argo CD connection to Git webhooks for connecting to Git repositories.
cert-manager	Let’s Encrypt	Gets signed certificate.
ExternalDNS	External DNS	Creates and deletes DNS entries in an external DNS.
Fluentd	OpenSearch	Fluentd on the managed cluster calls OpenSearch on the admin cluster.
Prometheus	Prometheus	Prometheus on the admin cluster scrapes metrics from Prometheus on the managed cluster.
Rancher Agent	Rancher	Rancher agent on the managed cluster sends requests to Rancher on the admin cluster.
Verrazzano Authentication Proxy	Keycloak	Calls Keycloak for authentication, which includes redirects.
Verrazzano Platform Operator	Kubernetes API server	Multicluster agent on the managed cluster calls API server on the admin cluster.

East-west system traffic

The following tables show Verrazzano system components that send traffic to a destination inside the cluster, with the following exceptions:

Usage of CoreDNS: It can be assumed that any Pod in the cluster can access CoreDNS for name resolution.
Envoy to Istiod: The Envoy proxies all make requests to the Istio control plane to get dynamic configuration, and such. This includes both the gateways and the mesh sidecar proxies. That traffic is not shown.
Traffic within a component is not shown, for example, traffic between OpenSearch Pods.
Prometheus scraping traffic is shown in the second table.

Component	Destination	Description
argoCD	Kubernetes API server	Performs CRUD operations on Kubernetes resources.
cert-manager	Kubernetes API server	Performs CRUD operations on Kubernetes resources.
Fluentd	OpenSearch	Fluentd sends data to OpenSearch.
Grafana	Prometheus	Console for Prometheus data.
OpenSearch Dashboards	OpenSearch	Console for OpenSearch.
NGINX Ingress Controller	Kubernetes API server	Performs CRUD operations on Kubernetes resources.
Istio	Kubernetes API server	Performs CRUD operations on Kubernetes resources.
Rancher	Kubernetes API server	Performs CRUD operations on Kubernetes resources.
Verrazzano Authentication Proxy	Keycloak	Calls Keycloak for token authentication.
Verrazzano Authentication Proxy	VMI components	Access consoles for OpenSearch Dashboards, Grafana, and such.
Verrazzano Authentication Proxy	Kubernetes API server	Performs CRUD operations on Kubernetes resources.
Verrazzano Application Operator	Kubernetes API server	Performs CRUD operations on Kubernetes resources.
Verrazzano Monitoring Operator	Kubernetes API server	Performs CRUD operations on Kubernetes resources.
Verrazzano Operator	Kubernetes API server	Performs CRUD operations on Kubernetes resources.
Verrazzano Platform Operator	Kubernetes API server	Performs CRUD operations on Kubernetes resources.
Verrazzano Platform Operator	Rancher	Registers the managed cluster with Rancher.

Prometheus scraping traffic

This table shows Prometheus traffic for each system component scrape target.

Target	Description
argoCD	Envoy metrics
cadvisor	Kubernetes metrics
Grafana	Envoy metrics
Istiod	Istio control plane metrics
Istiod	Envoy metrics
Istio egress gateway	Envoy metrics
Istio ingress gateway	Envoy metrics
Keycloak	Envoy metrics
MySQL	Envoy metrics
NGINX Ingress Controller	Envoy metrics
NGINX Ingress Controller	NGINX metrics
NGINX default back end	Envoy metrics
Node exporter	Node metrics
OpenSearch	Envoy metrics
OpenSearch Dashboards	Envoy metrics
Prometheus	Envoy metrics
Prometheus	Prometheus metrics
Verrazzano Console	Envoy metrics
Verrazzano API	Envoy metrics
WebLogic operator	Envoy metrics

Webhooks

Several of the system components are controllers and some of those have webhooks. Webhooks are called by the Kubernetes API server on a component HTTPS port to validate or mutate API payloads before they reach the API server.

The following components use webhooks:

cert-manager
Coherence Operator
Istio
Rancher
Verrazzano Application Operator
Verrazzano Platform Operator

Application traffic

Application traffic includes all traffic to and from Verrazzano applications.

North-south application traffic

After Verrazzano is installed, you can deploy applications into the Istio mesh. When doing so, you will likely need ingress into the application. As previously mentioned, this can be done with Istio using the Gateway and VirtualService resources. Verrazzano will create those resources for you when you use an IngressTrait in your ApplicationConfiguration. The Istio ingress gateway created during installation will be shared by all applications in the mesh, and the Gateway resource is bound to the Istio ingress gateway that was created during installation. This is done by the selector field in the Gateway.

   selector:
     istio: ingressgateway

Verrazzano creates a Gateway/VirtualService pair for each IngressTrait. Following is an example of those two resources created by Verrazzano.

Here is the Gateway; in this case both the host name and certificate were generated by Verrazzano.

apiVersion: v1
items:
- apiVersion: networking.istio.io/v1beta1
  kind: Gateway
  metadata:
   ...
    name: hello-helidon-hello-helidon-gw
    namespace: hello-helidon
  ...
  spec:
    selector:
      istio: ingressgateway
    servers:
    - hosts:
      - hello-helidon-appconf.hello-helidon.1.2.3.4.nip.io
      port:
        name: HTTPS
        number: 443
        protocol: HTTPS
      tls:
        credentialName: hello-helidon-hello-helidon-appconf-cert-secret
        mode: SIMPLE

Here is the VirtualService; notice that it refers back to the Gateway and that it contains the service routing information.

apiVersion: v1
items:
- apiVersion: networking.istio.io/v1beta1
  kind: VirtualService
  metadata:
  ...
    name: hello-helidon-ingress-rule-0-vs
    namespace: hello-helidon
  spec:
    gateways:
    - hello-helidon-hello-helidon-gw
    hosts:
    - hello-helidon-appconf.hello-helidon.1.2.3.4.nip.io
    HTTP:
    - match:
      - uri:
          prefix: /greet
      route:
      - destination:
          host: hello-helidon
          port:
            number: 8080

East-west application traffic

To manage east-west traffic, each service in the mesh should be routed using a VirtualService and an optional DestinationRule. You can still send east-west traffic without either of these resources, but you won’t get any custom routing or load balancing. Verrazzano doesn’t configure east-west traffic. Consider bobbys-front-end in the Bob’s Books example at bobs-books-comp.yaml. When deploying Bob’s Books, a VirtualService is created for bobbys-front-end, because of the IngressTrait, but there are no VirtualServices for the other services in the application. When bobbys-front-end sends requests to bobbys-helidon-stock-application, this east-west traffic still goes to bobbys-helidon-stock-application through the Envoy sidecar proxies in the source and destination Pods, but there is no VirtualService representing bobbys-helidon-stock-application where you could specify a canary deployment or custom load balancing. This is something you could configure manually, but it is not configured by Verrazzano.

Proxies

Verrazzano uses network proxies in multiple places. The two proxy products are Envoy and NGINX. The following table shows which proxies are used and in which Pod they run.

Usage	Proxy	Pod	Namespace	Description
System ingress	NGINX	`ingress-controller-ingress-nginx-controller-*`	`ingress-nginx`	Provides external access to Verrazzano system components.
Verrazzano authentication proxy	NGINX	`verrazzano-authproxy-*`	`verrazzano-system`	Verrazzano authentication proxy server for Kubernetes API and Single Sign-On (SSO).
Application ingress	Envoy	`istio-ingressgateway-*`	`istio-system`	Provides external access to Verrazzano applications.
Application egress	Envoy	`istio-egressgateway-*`	`istio-system`	Provides control of application egress traffic.
Istio mesh sidecar	Envoy	`ingress-controller-ingress-nginx-controller-*`	`ingress-nginx`	NGINX Ingress Controller in the Istio mesh.
Istio mesh sidecar	Envoy	`ingress-controller-ingress-nginx-defaultbackend-*`	`ingress-nginx`	NGINX default backend in the Istio mesh.
Istio mesh sidecar	Envoy	`fluentd-*`	`verrazzano-system`	Fluentd in the Istio mesh.
Istio mesh sidecar	Envoy	`keycloak-*`	`keycloak`	Keycloak in the Istio mesh.
Istio mesh sidecar	Envoy	`mysql-*`	`keycloak`	MySQL used by Keycloak in the Istio mesh.
Istio mesh sidecar	Envoy	`verrazzano-api-*`	`verrazzano-system`	Verrazzano API in the Istio mesh.
Istio mesh sidecar	Envoy	`verrazzano-console-*`	`verrazzano-system`	Verrazzano Console in the Istio mesh.
Istio mesh sidecar	Envoy	`vmi-system-es-master-*`	`verrazzano-system`	OpenSearch in the Istio mesh.
Istio mesh sidecar	Envoy	`vmi-system-es-data-*`	`verrazzano-system`	OpenSearch in the Istio mesh.
Istio mesh sidecar	Envoy	`vmi-system-es-ingest-*`	`verrazzano-system`	OpenSearch in the Istio mesh.
Istio mesh sidecar	Envoy	`vmi-system-kibana-*`	`verrazzano-system`	OpenSearch Dashboards in the Istio mesh.
Istio mesh sidecar	Envoy	`vmi-system-grafana-*`	`verrazzano-system`	Grafana in the Istio mesh.
Istio mesh sidecar	Envoy	`weblogic-operator-*`	`verrazzano-system`	WebLogic Kubernetes Operator in the Istio mesh.
Istio mesh sidecar	Envoy	`prometheus-prometheus-operator-kube-p-prometheus-*`	`verrazzano-monitoring`	Prometheus in the Istio mesh.

Multicluster

Some Verrazzano components send traffic between Kubernetes clusters. Those components are the Verrazzano agent, Verrazzano authentication proxy, and Prometheus.

Multicluster egress

The following table shows Verrazzano system components that initiate requests between the admin and managed clusters. All of these requests go through the NGINX Ingress Controller on the respective destination cluster.

Traffic on port 443 needs to be allowed in both directions, from managed clusters to the admin cluster, and from the admin cluster to managed clusters. Additionally, if Rancher is not enabled on the admin cluster, then managed clusters will also need access to the admin cluster’s Kubernetes API server port (typically, this is port 6443).

Source Cluster	Source Component	Destination Cluster	Destination Component	Description
Admin	Prometheus	Managed	Prometheus	Scrapes metrics on managed clusters.
Admin	argoCD	Managed	Rancher Proxy	Argo CD connects to the Rancher proxy for creating resources required for the Argo CD managed cluster registration.
Admin	Verrazzano Console	Managed	Verrazzano Authentication Proxy	Admin cluster proxy sends Kubernetes API requests to managed cluster proxy.
Admin	Verrazzano Cluster Operator	Managed	Rancher Proxy	Admin cluster sends registration updates to managed cluster, and retrieves managed cluster CA certificate.
Managed	Fluentd	Admin	OpenSearch	Fluentd sends logs to OpenSearch.
Managed	Rancher Agent	Admin	Rancher	Rancher Agent sends requests Rancher.
Managed	Verrazzano Authentication Proxy	Admin	Keycloak	Proxy sends requests to Keycloak.
Managed	Verrazzano Agent	Admin	Rancher Proxy or Kubernetes API server	Managed cluster agent, in the application operator, sends requests to the Rancher proxy if Rancher is enabled, or to the admin cluster Kubernetes API server.

Verrazzano agent

In the multicluster topology, the Verrazzano platform operator has an agent thread running on the managed cluster that sends requests to the Kubernetes API server on the admin cluster. The URL for the admin cluster Kubernetes API server is registered on the managed cluster by the user.

Verrazzano authentication proxy

In a multicluster topology, the Verrazzano authentication proxy runs on both the admin and managed clusters. On the admin cluster, the authentication proxy connects to in-cluster Keycloak, using the Keycloak Service. On the managed cluster, the authentication proxy connects to Keycloak on the admin cluster through the NGINX Ingress Controller running on the admin cluster.

For Single Sign-On (SSO), the authentication proxy also needs to send requests to Keycloak, either in-cluster or through the cluster ingress. When a request comes into the authentication proxy without an authentication header, the proxy sends a request to Keycloak through the NGINX Ingress Controller, so the request exits the cluster. Otherwise, if the authentication proxy is on the admin cluster, then the request is sent directly to Keycloak within the cluster. If the authentication proxy is on the managed cluster, then it must send requests to Keycloak on the admin cluster.

Prometheus

A single Prometheus service in the cluster, scrapes metrics from Pods in system components and applications. It also scrapes Pods in the Istio mesh using HTTPS, and outside the mesh using HTTP. In the multicluster case, Prometheus on the admin cluster, scrapes metrics from Prometheus on the managed cluster, through the NGINX Ingress Controller on the managed cluster.

Docs: Customize Istio

Mon, 01 Jan 0001 00:00:00 +0000

You can customize the Verrazzano Istio component using settings in the Verrazzano custom resource.

The following table describes the fields in the Verrazzano custom resource pertaining to the Istio component.

Path to Field	Description
`spec.components.istio.egress.kubernetes.replicas`	The number of pods to replicate. The default is `2` for the `prod` profile and `1` for all other profiles.
`spec.components.istio.egress.kubernetes.affinity`	The pod affinity definition expressed as a standard Kubernetes affinity definition. The default configuration spreads the Istio gateway pods across the available nodes. spec: components: istio: egress: kubernetes: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchExpressions: - key: app operator: In values: - istio-egressgateway topologyKey: kubernetes.io/hostname
`spec.components.istio.ingress.kubernetes.replicas`	The number of pods to replicate. The default is `2` for the `prod` profile and `1` for all other profiles.
`spec.components.istio.ingress.kubernetes.affinity`	The pod affinity definition expressed as a standard Kubernetes affinity definition. The default configuration spreads the Istio gateway pods across the available nodes. spec: components: istio: ingress: kubernetes: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchExpressions: - key: app operator: In values: - istio-ingressgateway topologyKey: kubernetes.io/hostname

The following example customizes a Verrazzano prod profile as follows:

Increases the replicas count to 3 for istio-ingressgateway and istio-egressgateway
Changes the podAffinity configuration to use requiredDuringSchedulingIgnoredDuringExecution for istio-ingressgateway and istio-egressgateway
```
apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: prod
  components:
    istio:
      overrides:
      - values:
          apiVersion: install.istio.io/v1alpha1
          kind: IstioOperator
          spec:
            components:
              egressGateways:
                - enabled: true
                  k8s:
                    affinity:
                      podAntiAffinity:
                        requiredDuringSchedulingIgnoredDuringExecution:
                          - podAffinityTerm:
                              labelSelector:
                                matchExpressions:
                                  - key: app
                                    operator: In
                                    values:
                                      - istio-egressgateway
                              topologyKey: kubernetes.io/hostname
                            weight: 100
                    replicaCount: 3
                  name: istio-egressgateway
              ingressGateways:
                - enabled: true
                  k8s:
                    affinity:
                      podAntiAffinity:
                        requiredDuringSchedulingIgnoredDuringExecution:
                          - podAffinityTerm:
                              labelSelector:
                                matchExpressions:
                                  - key: app
                                    operator: In
                                    values:
                                      - istio-ingressgateway
                              topologyKey: kubernetes.io/hostname
                            weight: 100
                    replicaCount: 3
                    service:
                      type: LoadBalancer
                  name: istio-ingressgateway
```