Multicloud cluster w/ cilium - setting node-ip and node-external-ip to node's public IP

[oasisvali]

I have created a multi-cloud cluster with k3s. The nodes are all on different private networks and cannot reach each other via their private IPs.

I was able to setup a working multicloud cluster using the flannel wireguard backend (https://docs.k3s.io/installation/network-options#distributed-hybrid-or-multicloud-cluster) which instructs flannel to use the nodes public IP.

However, due to certain networking requirements, we need to use cilium cni instead. To get the same multi cloud cluster working with cilium, I had to set both node-ip and node-external-ip to the nodes public IP for all nodes. This was the only way to ensure all nodes can reach each other as the private IPs are on different subnets.

I have encryption enabled with wireguard in the cilium install and have confirmed all pod-to-pod communication is happening via the encrypted wireguard interface.

Is this a secure setup or is setting both node-ip and node-external-ip to the nodes public IP considered bad practice? My understanding is that since all control plane communication is using https and all inter-pod communication is using wireguard, the cluster should be secure, but I'd like to ask the community's opinion.

[ammirator-administrator]

@oasisvali
Hey did you figured out in other places maybe if that is a good idea to use node-ip and external ip as the public ip
I'm trying to do the same thing and the moment and I'm interested if it worked for you?
I see none responded here, maybe you found some info anywhere

[oasisvali]

Hi, it's been a while so I don't remember exactly... but while I was able to get it working by setting the external IP and node IP to the same public IP, I eventually decided to not move forward with this approach as it seemed inherently insecure and an anti-pattern.

Having the nodes reach perform fkuster-internal communication over the public internet (even though this communication was all encrypted), did not fill me with confidence.

I ended up with creating a vpn with the nodes using tailscale. This gave them all an internal IP that I could then use as node IP and worked well. Hope this helps.

[babanfaraj]

hi @oasisvali , we are in a similar situation and have tried everything to get cilium working the same way you have. I do not like setting the node ip to be the same as the public ip as that does not even fully solve our problem with wireguard. (Wireguard then rejects all packets since packet source ip is somehow still the private ip)

I ended up with creating a vpn with the nodes using tailscale. This gave them all an internal IP that I could then use as node IP and worked well. Hope this helps.

In regards to your comment with using tailscale, are you saying tailscale gave each node an internal ip, connected all nodes together somehow, then installed cilium? I have been looking through their docs and I really do not see much regarding this. Mind shedding some more light on your process with tailscale + cilium?

[babanfaraj]

Did you utilize a tailscale subnet router? Also, how was the ordering of your installation? Tailscale in a pending state installed then cilium?