Missing targets (tested on local playground)

[kuritka]

make upgrade-candidate

make supertest I=100

using such framework in Makefile:

# make supertest I=100
MAKEIN =make -C .
I ?=2
supertest:
	for run in {1..$(I)}; do $(MAKEIN) testrr; sleep 3; done

testrr:
	@echo "[ep1,ep2]"
	@$(MAKEIN) ep1 ep2
	sleep 1
	@echo "[ep2,ep1]"
	@$(MAKEIN) ep2 ep1
	@sleep 1

ep1:
	@echo "EP1 targets"
	@kubectl get dnsendpoint test-gslb -oyaml  -n test-gslb --context=k3d-test-gslb1 -o jsonpath={.spec.endpoints[1].targets}
	@echo "\n"

ep2:
	@echo "EP2 targets"
	@kubectl get dnsendpoint test-gslb -oyaml  -n test-gslb --context=k3d-test-gslb1 -o jsonpath={.spec.endpoints[1].targets}
	@echo "\n"

apps:
	kubectl get pods -n test-gslb --context=k3d-test-gslb1
	kubectl get pods -n test-gslb --context=k3d-test-gslb1

The thing is that after some time of running, targets start to disappear and reappear (I expect that besides initialization, endpoints should contain all targets stably). I have tried on several commits (latest, e1af8, 793c486). The bug has been there for afew months at least.

Please investigate what is going on and fix the code if necessary.

tested on colima.

Details

EP2 targets


sleep 1
[ep2,ep1]
EP2 targets


EP1 targets


[ep1,ep2]
EP1 targets


EP2 targets


sleep 1
[ep2,ep1]
EP2 targets


EP1 targets


...

sleep 1
[ep2,ep1]
EP2 targets
[172.19.0.3 172.19.0.4]

EP1 targets
[172.19.0.3 172.19.0.4]

[ep1,ep2]
EP1 targets
[172.19.0.3 172.19.0.4 172.19.0.5 172.19.0.6]

EP2 targets
[172.19.0.3 172.19.0.4 172.19.0.5 172.19.0.6]

[kuritka]

ok, it's a podinfo reset. Their irregular restart will cause the targets of one of the regions to disappear after a while, and it takes a while for the targets to reappear in the endpoint.

NAME                                READY   STATUS      RESTARTS         
busybox                             0/1     Completed   0               
frontend-podinfo-7579d54897-nw7sx   1/1     Running     22 (3m14s ago) 

* it takes a while for the values to be written to the monitored/watched locations, so everything is delayed compared to the restart of the podinfo