Can't route between internal subnets

[scm7mae]

Hi there,

I'm new to k3s and I'm sure this is not the correct place to be posting for help, but I've been stuck for days on this point. I cannot seem to route to specific subnets in my installation. It seems to be a routing issue, as if I do a trace route from inside a container, the subnet I am trying to route to goes to the default gateway of my home network rather than routing on the internal subnets defined by k3s. I'm using metallb swell, not sure if that's an issue or not. Can someone help, or point me in the correct direction of the appropriate forum to get some help please?

Thanks in advance,

Mark

[brandond]

I cannot seem to route to specific subnets in my installation.

Which subnets specifically are you unable to reach? What CIDRs are you using for your home network, and which subnets are you using for cluster and service CIDRs? What problem are you attempting to troubleshoot by running traceroute commands?

Note that much of the networking in Kubernetes is done with iptables forwarding traffic to specific IPs and ports; you're not necessarily going to be able to traceroute to everything.

[scm7mae]

Thanks for your response @brandond. I have a three node raspberry pi4 cluster, 1 master 2 worker nodes.

My home network range is limited by my ISP, but it is the range 192.168.0.x. The default gateway for this network is 192.168.0.1. I have allocated the master 192.168.0.201, whilst the workers are 192.168.0.202, and 192.168.0.203.

I am also using metallb so that I can allocate external IPs to loadBalancers, and this range is currently 192.168.0.220-230.

The issue I seem to have is routing between internal subnets. CNI0 interfaces have been allocated 10.42.0.1 for the master, and for the workers 10.42.1.1 and 10.42.4.1.

I have installed some applications within the default namespace, and I've also installed Grafana, prometheus within a monitoring namespace. Today I installed Loki into another namespace, and when trying to configure Grafana to point towards the service for Loki, it cannot see the specific subnet. This address is 10.43.54.17. and I've exec'd into a pod that is working on 10.42.4.162, with pre-installed trace route and other tools, and tried to see what would happen if I tried to access the subnet, specifically 10.43.54.17.

The result of the trace route is in my opinion a routing issue that I'm unsure on how to fix. Result is below, but I can see it routing out to the default gateway of my home network and not forwarding or routing packets to the correct destination. I've done a similar exercise with a number of IP addresses within other subnets, and get the same result.

bash-4.4# traceroute 10.43.54.17
traceroute to 10.43.54.17 (10.43.54.17), 30 hops max, 38 byte packets
1 10.42.4.1 (10.42.4.1) 0.028 ms 0.023 ms 0.014 ms
2 192.168.0.1 (192.168.0.1) 1.851 ms 1.597 ms 1.730 ms
3

I've looked at iptables, and cannot find any reference to some of these subnets, so I'm at a loss. What have I done wrong?

[scm7mae]

Thanks for your response @brandond. I have a three node raspberry pi4 cluster, 1 master 2 worker nodes.

My home network range is limited by my ISP, but it is the range 192.168.0.x. The default gateway for this network is 192.168.0.1. I have allocated the master 192.168.0.201, whilst the workers are 192.168.0.202, and 192.168.0.203.

I am also using metallb so that I can allocate external IPs to loadBalancers, and this range is currently 192.168.0.220-230.

The issue I seem to have is routing between internal subnets. CNI0 interfaces have been allocated 10.42.0.1 for the master, and for the workers 10.42.1.1 and 10.42.4.1.

I have installed some applications within the default namespace, and I've also installed Grafana, prometheus within a monitoring namespace. Today I installed Loki into another namespace, and when trying to configure Grafana to point towards the service for Loki, it cannot see the specific subnet. This address is 10.43.54.17. and I've exec'd into a pod that is working on 10.42.4.162, with pre-installed trace route and other tools, and tried to see what would happen if I tried to access the subnet, specifically 10.43.54.17.

The result of the trace route is in my opinion a routing issue that I'm unsure on how to fix. Result is below, but I can see it routing out to the default gateway of my home network and not forwarding or routing packets to the correct destination. I've done a similar exercise with a number of IP addresses within other subnets, and get the same result.

bash-4.4# traceroute 10.43.54.17
traceroute to 10.43.54.17 (10.43.54.17), 30 hops max, 38 byte packets
1 10.42.4.1 (10.42.4.1) 0.028 ms 0.023 ms 0.014 ms
2 192.168.0.1 (192.168.0.1) 1.851 ms 1.597 ms 1.730 ms
3

I've looked at iptables, and cannot find any reference to some of these subnets, so I'm at a loss. What have I done wrong?

[scm7mae]

Hi @brandond, sorry to hassle you but please can you help?

[megamxl]

I have also been stuck on the same problem for days now.

the install argument

curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 664

I suspect something with the kube-proxy is not working since the kube proxy should intercept requests going to the --service-cidr. As far as I understood.

To gather debug info I opened the busy box and queried the DNS server in the cluster

#nslookup nats.openfaas.svc.cluster.local
-------------------------------------------------
Server:         10.43.0.10
Address:        10.43.0.10:53


Name:   nats.openfaas.svc.cluster.local
Address: 10.43.226.18

When I try to traceroute to that IP

# traceroute 10.43.226.18
traceroute to 10.43.226.18 (10.43.226.18), 30 hops max, 46 byte packets
 1  10.42.0.1 (10.42.0.1)  0.014 ms  0.013 ms  0.009 ms
 2  192.168.178.1 (192.168.178.1)  0.442 ms  0.371 ms  0.297 ms

it gets forwarded to the My gateway and then to my ISP.

When I do get svc -A I get my nats server.

MANAMESPACE     NAME               TYPE           CLUSTER-IP      EXTERNAL-IP       PORT(S)                      AGE
openfaas                 nats                  ClusterIP     10.42.0.9          <none>               4222/TCP                   9m46s

traceroute 10.42.0.9 (10.42.0.9), 30 hops max, 46 byte packets
 1  10-42-0-9.nats.openfaas.svc.cluster.local (10.42.0.9)  0.016 ms  0.010 ms  0.010 ms

@brandond maybe you can help with this

I tried to debug this further and tried to assign the service cider to my eth0 ,cni0, lo, flannel.1 , interface but this also didn't fix it.

# ip route show

default via 192.168.178.1 dev eth0 proto dhcp src 192.168.178.115 metric 100
8.8.8.8 via 192.168.178.1 dev eth0 proto dhcp src 192.168.178.115 metric 100
10.42.0.0/24 dev cni0 proto kernel scope link src 10.42.0.1
192.168.178.0/24 dev eth0 proto kernel scope link src 192.168.178.115 metric 100
192.168.178.1 dev eth0 proto dhcp scope link src 192.168.178.115 metric 100

# ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 3e:9d:5f:5d:ae:10 brd ff:ff:ff:ff:ff:ff
    inet 192.168.178.115/24 metric 100 brd 192.168.178.255 scope global dynamic eth0
       valid_lft 856769sec preferred_lft 856769sec
    inet6 fd00::3c9d:5fff:fe5d:ae10/64 scope global dynamic mngtmpaddr noprefixroute
       valid_lft 6825sec preferred_lft 3225sec
    inet6 fe80::3c9d:5fff:fe5d:ae10/64 scope link
       valid_lft forever preferred_lft forever
95: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
    link/ether 96:72:f8:5e:6c:d6 brd ff:ff:ff:ff:ff:ff
    inet 10.42.0.0/32 scope global flannel.1
       valid_lft forever preferred_lft forever
    inet6 fe80::9472:f8ff:fe5e:6cd6/64 scope link
       valid_lft forever preferred_lft forever
96: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether d2:61:39:c4:2f:40 brd ff:ff:ff:ff:ff:ff
    inet 10.42.0.1/24 brd 10.42.0.255 scope global cni0
       valid_lft forever preferred_lft forever
    inet6 fe80::d061:39ff:fec4:2f40/64 scope link
       valid_lft forever preferred_lft forever
97: veth58dca6fb@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
    link/ether c2:a9:f7:b7:0d:71 brd ff:ff:ff:ff:ff:ff link-netns cni-ec183c7d-7fb3-a9b2-4813-18944b613090
    inet6 fe80::c0a9:f7ff:feb7:d71/64 scope link
       valid_lft forever preferred_lft forever
98: veth8cf9bf71@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
    link/ether 92:6b:87:5c:da:f2 brd ff:ff:ff:ff:ff:ff link-netns cni-e4236b53-12d1-6848-c21e-d7bf6132fdd9
    inet6 fe80::906b:87ff:fe5c:daf2/64 scope link
       valid_lft forever preferred_lft forever
99: veth2144c498@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
    link/ether 32:56:33:3d:79:97 brd ff:ff:ff:ff:ff:ff link-netns cni-91889a39-d0d0-36e4-7dc4-7f0f3ffc28b2
    inet6 fe80::3056:33ff:fe3d:7997/64 scope link
       valid_lft forever preferred_lft forever
102: vethd7340007@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
    link/ether 32:3f:9b:f9:1f:37 brd ff:ff:ff:ff:ff:ff link-netns cni-261d998a-4cee-9e0b-bcc3-59e959ab7446
    inet6 fe80::303f:9bff:fef9:1f37/64 scope link
       valid_lft forever preferred_lft forever
103: vetha102677f@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
    link/ether ee:05:15:cf:d4:07 brd ff:ff:ff:ff:ff:ff link-netns cni-6ee497a0-137c-ce42-93fd-5de2fb5136e8
    inet6 fe80::ec05:15ff:fecf:d407/64 scope link
       valid_lft forever preferred_lft forever

In my ip-tables I can find only this line containing 43

sudo iptables -S | grep 43
-A KUBE-ROUTER-INPUT -d 10.43.0.0/16 -m comment --comment "allow traffic to primary/secondary cluster IP range - EKROEGTNIJ3AP3LC" -j RETURN

[DamianoSamperi]

I have the same problem, did you solve it?

[scm7mae]

Yes. I started from scratch again. I wiped all the nodes and reinstalled and that fixed the issues

[DamianoSamperi]

thanks for the reply, unfortunately even reinstalling everything from scratch gives me the same problem

[megamxl]

For me the fix was to put a Software defined Router between my nodes and my Home Network. Something in the DNS Config of it or somewhere else in the Routing of my Home Router did not work with k3s.