Geoip does not forward requests to external k8gb instance

[ErikLundJensen]

I have create a setup with geoip in 2 Kubernetes clusters (site01 and site02). A lookup of "hello-world.poc.foo.bar" in each cluster works fine, however, if I scale hello-world replicaset to 0 in site02 then the DNS requests are not forwarded to site01.

Configuration in site02 (similar setup for site01):

  k8gb:
    dnsZoneNegTTL: 10
    clusterGeoTag: "site02"
    extGslbClustersGeoTags: "site01"
    log:
      level: debug
    dnsZone: poc.foo.bar
    edgeDNSZone: foo.bar
    edgeDNSServers:
     - "172.1.2.3"

  coredns:
    serviceType: LoadBalancer
    service:
      loadBalancerIP: "172.2.4.6"
    extraVolumes:
    - name: geo-data
      configMap:
        name: geo-data
    extraVolumeMounts:
    - name: geo-data 
      mountPath: /geoip.mmdb

DNS request towards site02:

dig -p 53 @172.2.4.6 hello-world.poc.foo.bar 

Below is logs from the k8gb controller. In the log I have replaced the edgeDNSServer IP and the DNS name to foo.bar.

From the log the nameserver is gslb-ns-site01-poc.foo.bar, however, we don't have a DNS name for the edgeDNSServer, just the given edgeDNSServer IP.

2022-11-17T15:42:42Z DBG github.com/k8gb-io/k8gb/controllers/gslb_controller_reconciliation.go:93 > Resolved strategy gslb=hello-world-gslb-east strategy={"dnsTtlSeconds":5,"splitBrainThresholdSeconds":300,"type":"geoip"}
2022-11-17T15:42:42Z INF github.com/k8gb-io/k8gb/controllers/providers/assistant/gslb.go:305 > Adding external Gslb targets from cluster cluster=gslb-ns-site01-poc.foo.bar
2022-11-17T15:42:42Z INF github.com/k8gb-io/k8gb/controllers/providers/assistant/gslb.go:314 > Resolved glue A record for NS edgeDNSServers=[{"Host":"172.1.2.3","Port":53}] glueARecord=null nameserver=gslb-ns-site01-poc.foo.bar
2022-11-17T15:42:42Z WRN github.com/k8gb-io/k8gb/controllers/providers/assistant/gslb.go:294 > can't resolve FQDN using nameservers error="exchange error: all dns servers were tried and none of them were able to resolve, err: dial udp: lookup gslb-ns-site01-poc.foo.bar on 169.254.25.10:53: no such host" fqdn=localtargets-hello-world.poc.foo.bar. nameservers=[{"Host":"gslb-ns-site01-poc.foo.bar","Port":53}]
2022-11-17T15:42:42Z INF github.com/k8gb-io/k8gb/controllers/dnsupdate.go:111 > No external targets have been found for host host=hello-world.poc.foo.bar
2022-11-17T15:42:42Z INF github.com/k8gb-io/k8gb/controllers/dnsupdate.go:118 > Final target list gslb=hello-world-gslb-east targets=[]

K8gb version: "v0.10.0"

[ErikLundJensen]

Adding hostAliases to the operator pod did solve the issue, however, the documentation nor the helm chart contains any information on this requirement.
For example:

  hostAliases:
    - hostnames:
        - gslb-ns-site01-poc.foo.bar
      ip: 172.1.2.3

Implementing this in the helm chart would require some work if more than one edgeDNSServer is specified.
If only one edgeDNSServer and one dnsZone are specified it could look like:

      hostAliases:
      - ip: {{ index .Values.k8gb.edgeDNSServers 0 | quote }}
        hostnames:
        - {{ printf "gslb-ns-%s-%s" .Values.k8gb.extGslbClustersGeoTags .Values.k8gb.dnsZone }}

However, the dnsZone is a comma separated list and there is not gurantee that the order of those zones will match the entries in the edgeDNSServers array.

Do we have other options for solving the lookup of the remote k8gb nameserver?

[ErikLundJensen]

Could it be that I'm missing installation of the modified external-dns according to https://github.com/k8gb-io/coredns-crd-plugin ?

[ytsarev]

Hi @ErikLundJensen , thanks a lot for trying out k8gb and welcome to the project :)

gslb-ns-* NS records and A glue records are supposed to be automatically created by external-dns integration. So they should be generally resolvable.

What type of edgeDNS do you use (route53/ns1/infoblox/smth else) ?

[ErikLundJensen]

I don't have an edge DNS that is updateable through external DNS in this setup. I have therefore pointed the edgeDNS to the k8gb coredns in the opposite cluster. That is why I don't get the gslb-ns-* records.