Missing endpoints in `localtargets.*` A records

[ytsarev]

Steps to reproduce

• Deploy two cross-communicating ohmyglb setup locally
  $ make deploy-full-local-setup
• Check generated localtargets.* dnsendpoint conf

$ kubectl -n test-gslb get dnsendpoints test-gslb -o yaml
...
spec:
  endpoints:
  - dnsName: localtargets.app3.cloud.example.com
    recordTTL: 30
    recordType: A
    targets:
    - 172.17.0.2
    - 172.17.0.3
    - 172.17.0.4
...

• Check if coredns returns matching A records

dig +short @localhost localtargets.app3.cloud.example.com
172.17.0.2
172.17.0.4
172.17.0.3

This is expected result. After some time localtargets.* can 'lose' one of the records in the following way:

• localtargets.* dnsendpoint conf is always consistent

$ kubectl -n test-gslb get dnsendpoints test-gslb -o yaml
...
spec:
  endpoints:
  - dnsName: localtargets.app3.cloud.example.com
    recordTTL: 30
    recordType: A
    targets:
    - 172.17.0.2
    - 172.17.0.3
    - 172.17.0.4
...

• Meanwhile actual DNS response might lose one of the A records

dig +short @localhost localtargets.app3.cloud.example.com
172.17.0.2
172.17.0.4

Issue is not really deterministic in its behaviour . Meanwhile we faced it several times over multiple deployments
In case of 2 cluster setup only single cluster is affected effectively making exposed through coredns only 5 out of 6 k8s worker.

DNSEndpoint CR generation looks always correct so the problem is somewhere in etcd coredns backend area.

make debug-test-etcd can help in debugging this issue runtime.

[ytsarev]

Example of degraded localtargets from within etcd:

/ # etcdctl get '' --from-key
/skydns/com/example/cloud/failover/16fccd7b
{"host":"172.17.0.5","ttl":30,"targetstrip":1}
/skydns/com/example/cloud/failover/24f6061f
{"host":"172.17.0.3","ttl":30,"targetstrip":1}
/skydns/com/example/cloud/failover/6c871674
{"host":"172.17.0.5","ttl":30,"targetstrip":1}
/skydns/com/example/cloud/failover/localtargets/13ee1210
{"host":"172.17.0.2","ttl":30,"targetstrip":1}
/skydns/com/example/cloud/failover/localtargets/2485c7ba
{"host":"172.17.0.5","ttl":30,"targetstrip":1}
/skydns/com/example/cloud/failover/localtargets/315f5958
{"host":"172.17.0.5","ttl":30,"targetstrip":1}

Notice the unexpected duplication of 172.17.0.5
At the same time related DNSEndpoint is fully ok:

k -n test-gslb get dnsendpoints -o yaml
....
    - dnsName: failover.cloud.example.com
      recordTTL: 30
      recordType: A
      targets:
      - 172.17.0.2
      - 172.17.0.3
      - 172.17.0.5

So the problem seems to be fully isolated to upstream components of external-dns+etcd ... coredns reads whatever it has in etcd backend so it is behaving correctly.
Digging into crd_source -> external-dns -> etcd propagation code ...

[ytsarev]

Made $ etcdctl del '' --from-key effectively wiping out local etcd database

Then it was gslb reconcile

External-dns logs during the even :

time="2020-03-19T21:32:52Z" level=info msg="Add/set key /skydns/com/example/cloud/failover/4088e0a8 to Host=172.17.0.2, Text=, TTL=30"
time="2020-03-19T21:32:52Z" level=info msg="Add/set key /skydns/com/example/cloud/failover/461535c9 to Host=172.17.0.3, Text=, TTL=30"
time="2020-03-19T21:32:52Z" level=info msg="Add/set key /skydns/com/example/cloud/failover/68ec81be to Host=172.17.0.5, Text=, TTL=30"
time="2020-03-19T21:32:52Z" level=info msg="Add/set key /skydns/com/example/cloud/failover/localtargets/57c491cc to Host=172.17.0.2, Text=, TTL=30"
time="2020-03-19T21:32:52Z" level=info msg="Add/set key /skydns/com/example/cloud/failover/localtargets/5afed140 to Host=172.17.0.3, Text=, TTL=30"
time="2020-03-19T21:32:52Z" level=info msg="Add/set key /skydns/com/example/cloud/failover/localtargets/6a0a107c to Host=172.17.0.5, Text=, TTL=30"
time="2020-03-19T21:33:52Z" level=info msg="Add/set key /skydns/com/example/cloud/failover/68ec81be to Host=172.17.0.2, Text=, TTL=30"
time="2020-03-19T21:33:52Z" level=info msg="Add/set key /skydns/com/example/cloud/failover/68ec81be to Host=172.17.0.3, Text=, TTL=30"
time="2020-03-19T21:33:52Z" level=info msg="Add/set key /skydns/com/example/cloud/failover/68ec81be to Host=172.17.0.5, Text=, TTL=30"
time="2020-03-19T21:33:52Z" level=info msg="Add/set key /skydns/com/example/cloud/failover/localtargets/6a0a107c to Host=172.17.0.2, Text=, TTL=30"
time="2020-03-19T21:33:52Z" level=info msg="Add/set key /skydns/com/example/cloud/failover/localtargets/6a0a107c to Host=172.17.0.3, Text=, TTL=30"
time="2020-03-19T21:33:52Z" level=info msg="Add/set key /skydns/com/example/cloud/failover/localtargets/6a0a107c to Host=172.17.0.5, Text=, TTL=30"

So the first chunk of the log looks totally correct - each entry at different path and different hash at the end. Consequently we observe weird output of different values ending up at the same hash.
Surprisingly after the wipe out and reconcile it did fix itself:

# etcdctl get '' --from-key
/skydns/com/example/cloud/failover/4088e0a8
{"host":"172.17.0.2","ttl":30,"targetstrip":1}
/skydns/com/example/cloud/failover/461535c9
{"host":"172.17.0.3","ttl":30,"targetstrip":1}
/skydns/com/example/cloud/failover/68ec81be
{"host":"172.17.0.5","ttl":30,"targetstrip":1}
/skydns/com/example/cloud/failover/localtargets/57c491cc
{"host":"172.17.0.2","ttl":30,"targetstrip":1}
/skydns/com/example/cloud/failover/localtargets/5afed140
{"host":"172.17.0.3","ttl":30,"targetstrip":1}
/skydns/com/example/cloud/failover/localtargets/6a0a107c
{"host":"172.17.0.5","ttl":30,"targetstrip":1}

No dups.
Something really weird happens here.