When using the failover load balancing strategy, investigate and validate how resolution will be handled effectively when clusters are configured for mutual failover

[donovanmuller]

Related to this note on #46:

The existence of multiple "secondary" failover clusters should also be considered. For example, if there were 3 clusters (X, Y and Z) in the scenario 2 above, could the Ingress node IPs for both clusters (X and Z) be resolved and if so, how (in terms of "load balancing") would the Ingress node IPs across both those secondary/failover clusters be resolved? Would they use the default round robin strategy, if any strategy at all?

we need to validate that the IPs resolved from the secondary/failover clusters are handled correctlty and consistently in line with a load balancing strategy of their own.

[donovanmuller]

The following scenario should be catered for:

We have two applications, A and B deployed on two clusters, X and Y.

• Application A calls application B on cluster X (within the same cluster) through it's GLB host
• Same for applications A and B on cluster Y
• The Gslb resources for application B (applied on both clusters) both have their own cluster as primary. I.e. the primary cluster for application B on cluster X is X and the failover is Y and vice versa. For example:

# Applied on cluster X
apiVersion: ohmyglb.absa.oss/v1beta1
kind: Gslb
metadata:
  name: application-b-gslb
spec:
  ingress:
    rules:
      - host: b.cloud.example.com
        http:
          paths:
          - backend:
              serviceName: application-b
              servicePort: http
            path: /
  strategy:
    type: failover
    primaryGeoTag: x
---
# Applied on cluster Y
apiVersion: ohmyglb.absa.oss/v1beta1
kind: Gslb
metadata:
  name: application-b-gslb
spec:
  ingress:
    rules:
      - host: b.cloud.example.com
        http:
          paths:
          - backend:
              serviceName: application-b
              servicePort: http
            path: /
  strategy:
    type: failover
    primaryGeoTag: y

If application B becomes unhealthy on cluster X then application B on cluster Y should have it's node IPs resolved. So that application A on cluster X calls application B on the failover cluster, cluster Y.

[ytsarev]

To document the findings:

• From DNS response perspective ohmyglb behaves exactly as described
• Associated terratest TestOhmyglbSplitFailoverExample test proves it in automated way
• At the same time this 'pinning' is edge case scenario because in case of healthy workload on cluster X ( default situation) X ohmyglb will return its own records and Y ohmyglb will return its own records too.
• It will result in the rare situation when 2 ohmylgb clusters that usually working in sync and pair are returning different A records set
• Associated Zone Delegation includes both ohmyglb clusters in its configuration and it is desired design
• Having above in mind - assuming app A access app B through fqdn there will be no guarantee which of the clusters it will hit
• To enable this Geo 'pinning' we will have to point app pods to @ohmyglb-coredns cluster service effectively using ohmylgb coredns as a main DNS for resolution.
• The above can be achieved by changing forward plugin conf of main kubernetes coredns config to forward to ohmyglb one.
• In the meantime forward of ohmyglb coredns can be adjusted to recursively query upstream dns servers for external fqdns