Test implemented LB strategies for more than 2 cluster scenario

[ytsarev]

Most of the current use cases, automated tests and functional test scenarios were performed in context of 2 clusters only as it is most common and easier to test setup.

Meanwhile we need to be ready and test for >2 scenario cluster scenario.

This series of testing can reveal potential edge cases in current LB strategies implementation and can lead to LB logic extension and handling of >2 number of clusters

[jkremser]

For round robin, it's quite straightforward, but how this should actually work for the fail-over strategy? I think this actually can't be done with what we currently have. We allow to mark only 1 cluster w/ the primaryGeoTag, if this one goes down what's next? Should the rest of the clusters (in case there is more than one) overtake the load and behave in round robin fashion or should one of them overtake the role of the leader and become a new primaryGeoTag-marked cluster? If the latter is the expected behavior, we would need something like this:

...
  strategy:
    type: failover
    ordering: "eu,us" # probably all participating clusters can be mentioned here and the left most is the desired primary one (not good for adding a new cluster scenario)
...

or more backward compatible way would be something like:

...
  strategy:
    type: failover
    primaryGeoTag: "eu"
    fallback/backup: "us,cz"
...

[ytsarev]

@jkremser yes, failover is not ready for >2 clusters, so we need to design that stuff.

I like the ordering idea, maybe composed like.

  strategy:
    type: failover
    primaryGeoTag: "eu"
    failoverOrder:
       - us
       - za
       - cz

failoverOrder can be optional for cluster number < 3

[jkremser]

That would work and the array is definitely better than comma-separated strings. It's kind of redundant this way, but at the same time backward compatible and easier for N < 3.

I have the tests almost ready for the round-robin scenario, for the failover it would depend on the ^ implementation, so I'll start easy w/ the PR for round robin case :D

[somaritane]

#815