NDB lost sync in apply() trying to set GRE tunnel configs

[carloslockward]

I found this issue while trying to change the configuration of an existing GRE(or any other kind) tunnel using pyroute2 version 0.6.5

Exception

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/snap/iotr-cr-sysmgr/x2/lib/python3.8/site-packages/pr2modules/ndb/auth_manager.py", line 69, in guard
    return f(obj, *argv, **kwarg)
  File "/snap/iotr-cr-sysmgr/x2/lib/python3.8/site-packages/pr2modules/ndb/objects/__init__.py", line 676, in commit
    self.apply()
  File "/snap/iotr-cr-sysmgr/x2/lib/python3.8/site-packages/pr2modules/ndb/auth_manager.py", line 69, in guard
    return f(obj, *argv, **kwarg)
  File "/snap/iotr-cr-sysmgr/x2/lib/python3.8/site-packages/pr2modules/ndb/objects/interface.py", line 702, in apply
    super(Interface, self).apply(rollback, req_filter)
  File "/snap/iotr-cr-sysmgr/x2/lib/python3.8/site-packages/pr2modules/ndb/auth_manager.py", line 69, in guard
    return f(obj, *argv, **kwarg)
  File "/snap/iotr-cr-sysmgr/x2/lib/python3.8/site-packages/pr2modules/ndb/objects/__init__.py", line 892, in apply
    raise Exception('lost sync in apply()')
Exception: lost sync in apply()

Steps to reproduce

1. Create a GRE tunnel ip.link("add", ifname="testtun", kind="gre", gre_local="10.10.10.2", gre_remote="10.10.100.3", gre_ttl=254)
2. Try to change it with NDB ndb.interfaces["testtun"].set("gre_local", "10.10.10.3").commit()

Notes

I also tried to change the tunnel using these commands but they didn't work(didn't throw an exception but value was not changed):

• ip.link("update", index=31, IFLA_GRE_LOCAL="10.10.10.3")
• ip.link("set", index=31, IFLA_GRE_LOCAL="10.10.10.3")

[svinota]

Got it, thanks! Fixing

[svinota]

The root cause has been found. Preparing the fix.

[svinota]

The issue is fixed.

For NDB it's as simple as

ndb
.interfaces["testtun"]
.set("gre_local", "10.10.10.3")
.commit()

For IPRoute it's a bit more complicated.

The GRE tunnels while setting an endpoint require both endpoints to be specified as well as some other GRE attributes. Thus if you want only to change IFLA_GRE_LOCAL, you need anyways to specify all the attributes:

ip.link('set',
        index=31,
        kind='gre',                # important! required to properly populate IFLA_LINKINFO
        gre_local='10.10.10.3',    # setup the new local ip
        gre_remote='10.10.100.3',  # the remote ip must be specified as well
        gre_ttl=254)               # preserve TTL

[carloslockward]

I commented a suggestion on your fix! I don't really know the full architecture of pyroute2 so I could be wrong but by looking at your commit I could infer that other tunnels kinds might need the same fix. Give it a look and let me know!

[svinota]

Yep, still testing other tunnel types.

[svinota]

Added some more supported types.

[carloslockward]

Seems like the problem is still not fixed on 0.6.6

Creating a GRE tunnel as before i get this tunnel.

{
    "flags": 144,
    "state": "down",
    "ifname": "TEST-GRE",
    "target": "localhost",
    "tflags": 0,
    "index": 8,
    "family": 0,
    "ifi_type": 778,
    "change": 4294967295,
    "address": "0a:0a:01:64:08:00",
    "broadcast": "0a:c8:0a:01:c4:00",
    "mtu": 1476,
    "link": 0,
    "qdisc": "noop",
    "master": None,
    "txqlen": 1000,
    "linkmode": 0,
    "net_ns_pid": None,
    "ifalias": None,
    "num_vf": None,
    "group": 0,
    "ext_mask": None,
    "promiscuity": 0,
    "num_tx_queues": 1,
    "num_rx_queues": 1,
    "carrier": 1,
    "carrier_changes": 0,
    "link_netnsid": None,
    "phys_port_name": None,
    "proto_down": 0,
    "gso_max_segs": 65535,
    "gso_max_size": 65536,
    "if_netnsid": None,
    "carrier_up_count": 0,
    "carrier_down_count": 0,
    "new_ifindex": None,
    "min_mtu": 0,
    "max_mtu": 0,
    "alt_ifname": None,
    "kind": "gre",
    "slave_kind": None,
    "xdp_attached": None,
    "xdp_flags": None,
    "xdp_prog_id": None,
    "xdp_drv_prog_id": None,
    "xdp_skb_prog_id": None,
    "xdp_hw_prog_id": None,
    "gre_link": 0,
    "gre_iflags": 0,
    "gre_oflags": 0,
    "gre_ikey": 0,
    "gre_okey": 0,
    "gre_local": "10.10.1.100",
    "gre_remote": "10.200.10.1",
    "gre_ttl": 0,
    "gre_tos": 0,
    "gre_pmtudisc": 1,
    "gre_encap_limit": None,
    "gre_flowinfo": None,
    "gre_flags": None,
    "gre_encap_type": 0,
    "gre_encap_flags": 0,
    "gre_encap_sport": 0,
    "gre_encap_dport": 0,
    "gre_ignore_df": 0,
    "gre_fwmark": 0,
}

I tried to modify it using NDB by doing:

ndb.interfaces["TEST-GRE"].set("gre_local", "10.1.1.201").commit()

Which didn't fail(progress!), however the tunnel value was not updated. As seen bellow.

>>> ndb.interfaces["TEST-GRE"]["gre_local"]
'10.10.1.100'

And ip.link("set") still didn't work either Even when specifying all the values as you suggested.

>>> ip.link("set", index=8, kind="gre", gre_local="10.10.200.1", gre_remote="10.200.10.1", gre_ttl=0)
>>> ndb.interfaces["TEST-GRE"]["gre_local"]
'10.10.1.100'

No exception thrown but the values are not updated.

Thanks for the support and let me know if you need any additional info.

[svinota]

Yeah, it's another issue. The down tunnels don't send updates, I ignore that in the code — but forgot to force attributes reading.

The interface attributes are changed in the system, but it's NDB that doesn't get updated. Will be fixed within an hour.

[svinota]

Done.