NDB cant add default route 'could not apply the changes' v0.6.12

[carloslockward]

NDB fails to add a default route with priority 0 with error: 'could not apply the changes'.

Exception

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/snap/iotr-cr-sysmgr/x125/lib/python3.8/site-packages/pr2modules/ndb/auth_manager.py", line 67, in guard
    return f(obj, *argv, **kwarg)
  File "/snap/iotr-cr-sysmgr/x125/lib/python3.8/site-packages/pr2modules/ndb/objects/__init__.py", line 694, in commit
    raise e_i
  File "/snap/iotr-cr-sysmgr/x125/lib/python3.8/site-packages/pr2modules/ndb/objects/__init__.py", line 683, in commit
    return self.apply(mode='commit')
  File "/snap/iotr-cr-sysmgr/x125/lib/python3.8/site-packages/pr2modules/ndb/auth_manager.py", line 67, in guard
    return f(obj, *argv, **kwarg)
  File "/snap/iotr-cr-sysmgr/x125/lib/python3.8/site-packages/pr2modules/ndb/objects/route.py", line 691, in apply
    return super(Route, self).apply(rollback, req_filter, mode)
  File "/snap/iotr-cr-sysmgr/x125/lib/python3.8/site-packages/pr2modules/ndb/auth_manager.py", line 67, in guard
    return f(obj, *argv, **kwarg)
  File "/snap/iotr-cr-sysmgr/x125/lib/python3.8/site-packages/pr2modules/ndb/objects/__init__.py", line 935, in apply
    raise Exception('could not apply the changes')
Exception: could not apply the changes

Relevant info.

I don't have concrete steps to reproduce this issue as I failed to reproduced myself on another machine. But here is the relevant info I have about the issue.

• We add the route like ndb.routes.create(**{"dst": "default", "gateway": "10.10.2.1", "oif": 2, "priority": 0, "table": 254}).commit(), this produces the error. However, if we try creating the same route but with priority 1 it works perfectly.
• Not including the priority has the same issue ndb.routes.create(**{"dst": "default", "gateway": "10.10.2.1", "oif": 2, "table": 254}).commit().
• I tested this on 2 machines, and 1 failed and 1 worked. The one that failed is running Ubuntu Core with an ARM processor, the one that worked is running regular Ubuntu on an Intel processor.
• Version 0.6.9 does not have the issue.

If you need any more info let me know.

Thanks

[svinota]

Sounds strange enough, thanks for reporting.

Would be nice to see the debug logs — if possible.

But meanwhile I'm trying to reproduce it on ARM + Ubuntu, I've got some RPi for that purpose.

[carloslockward]

I just tried the orignal route that had the problem but it worked. now it happened with another route. Here are the logs:

>>> parameters = {"dst": "beef:feed:fade::/112", "oif": 3, "priority": 0, "table": 254}
>>> ndb.routes.create(**parameters).commit()
2022-06-17 21:25:57,481    DEBUG pyroute2.ndb.281473114185888.rtnl_object: init
2022-06-17 21:25:57,483    DEBUG pyroute2.ndb.281473114185888.view.routes: check if the key {'dst': 'beef:feed:fade::', 'dst_len': 112, 'family': <AddressFamily.AF_INET6: 10>, 'oif': 3, 'priority': 0, 'table': 254, 'target': 'localhost'} exists in table routes
2022-06-17 21:25:57,485    DEBUG pyroute2.ndb.281473114185888.view.routes: not exists
2022-06-17 21:25:57,486    DEBUG pyroute2.ndb.281473114185888.rtnl_object: complete key {'dst': 'beef:feed:fade::', 'dst_len': 112, 'family': <AddressFamily.AF_INET6: 10>, 'oif': 3, 'priority': 0, 'table': 254, 'target': 'localhost'} from table routes
2022-06-17 21:25:57,489    DEBUG pyroute2.ndb.281473114185888.rtnl_object: got none
2022-06-17 21:25:57,491    DEBUG pyroute2.ndb.281473114185888.view.routes: check if the key {'dst': 'beef:feed:fade::', 'dst_len': 112, 'family': <AddressFamily.AF_INET6: 10>, 'oif': 3, 'priority': 0, 'table': 254, 'target': 'localhost', 'create': True} exists in table routes
2022-06-17 21:25:57,494    DEBUG pyroute2.ndb.281473114185888.view.routes: not exists
2022-06-17 21:25:57,494    DEBUG pyroute2.ndb.281473114185888.rtnl_object: commit: [(1655501157.4822505, 'invalid')]
2022-06-17 21:25:57,495    DEBUG pyroute2.ndb.281473114185888.rtnl_object: events log: [(1655501157.4822505, 'invalid')]
2022-06-17 21:25:57,495    DEBUG pyroute2.ndb.281473114185888.rtnl_object: run apply
2022-06-17 21:25:57,498    DEBUG pyroute2.ndb.281473114185888.rtnl_object: load_sql load: None
2022-06-17 21:25:57,499    DEBUG pyroute2.ndb.281473114185888.rtnl_object: load_sql names: ['target', 'tflags', 'family', 'dst_len', 'src_len', 'tos', 'table', 'proto', 'scope', 'type', 'flags', 'dst', 'src', 'iif', 'oif', 'gateway', 'priority', 'prefsrc', 'protoinfo', 'flow', 'table', 'mark', 'via', 'newdst', 'pref', 'encap_type', 'route_id', 'gc_mark', 'deps']
2022-06-17 21:25:57,499    DEBUG pyroute2.ndb.281473114185888.rtnl_object: apply req: {'target': 'localhost', 'family': <AddressFamily.AF_INET6: 10>, 'dst_len': 112, 'dst': 'beef:feed:fade::', 'oif': 3, 'priority': 0, 'table': 254}
2022-06-17 21:25:57,500    DEBUG pyroute2.ndb.281473114185888.rtnl_object: apply idx_req: {'target': 'localhost', 'family': <AddressFamily.AF_INET6: 10>, 'dst_len': 112, 'dst': 'beef:feed:fade::', 'oif': 3, 'priority': 0, 'table': 254}
2022-06-17 21:25:57,500    DEBUG pyroute2.ndb.281473114185888.rtnl_object: apply transition from: invalid
2022-06-17 21:25:57,500    DEBUG pyroute2.ndb.281473114185888.rtnl_object: apply method: add
2022-06-17 21:25:57,501    DEBUG pyroute2.ndb.281473114185888.rtnl_object: API call add ({'target': 'localhost', 'family': <AddressFamily.AF_INET6: 10>, 'dst_len': 112, 'dst': 'beef:feed:fade::', 'oif': 3, 'priority': 0, 'table': 254, 'multipath': [], 'metrics': {}, 'deps': 0})
2022-06-17 21:25:57,501    DEBUG pyroute2.ndb.281473114185888.sources.localhost: source api run route ('add',) {'target': 'localhost', 'family': <AddressFamily.AF_INET6: 10>, 'dst_len': 112, 'dst': 'beef:feed:fade::', 'oif': 3, 'priority': 0, 'table': 254, 'multipath': [], 'metrics': {}, 'deps': 0}
2022-06-17 21:25:57,508    DEBUG pyroute2.ndb.281473114185888.rtnl_object: stats: apply add {objid 281473063064096, wtime 0.0, mqsize 0, nqsize 0}
2022-06-17 21:25:57,524    DEBUG pyroute2.ndb.281473114185888.rtnl_object: load_sql load: None
2022-06-17 21:25:57,525    DEBUG pyroute2.ndb.281473114185888.rtnl_object: load_sql names: ['target', 'tflags', 'family', 'dst_len', 'src_len', 'tos', 'table', 'proto', 'scope', 'type', 'flags', 'dst', 'src', 'iif', 'oif', 'gateway', 'priority', 'prefsrc', 'protoinfo', 'flow', 'table', 'mark', 'via', 'newdst', 'pref', 'encap_type', 'route_id', 'gc_mark', 'deps']
2022-06-17 21:25:57,525    DEBUG pyroute2.ndb.281473114185888.rtnl_object: check: [(1655501157.4822505, 'invalid')]
2022-06-17 21:25:57,526    DEBUG pyroute2.ndb.281473114185888.rtnl_object: check state: False
2022-06-17 21:25:57,526    DEBUG pyroute2.ndb.281473114185888.rtnl_object: check failed
2022-06-17 21:25:57,526    DEBUG pyroute2.ndb.281473114185888.rtnl_object: API call add ({'target': 'localhost', 'family': <AddressFamily.AF_INET6: 10>, 'dst_len': 112, 'dst': 'beef:feed:fade::', 'oif': 3, 'priority': 0, 'table': 254, 'multipath': [], 'metrics': {}, 'deps': 0})
2022-06-17 21:25:57,527    DEBUG pyroute2.ndb.281473114185888.sources.localhost: source api run route ('add',) {'target': 'localhost', 'family': <AddressFamily.AF_INET6: 10>, 'dst_len': 112, 'dst': 'beef:feed:fade::', 'oif': 3, 'priority': 0, 'table': 254, 'multipath': [], 'metrics': {}, 'deps': 0}
2022-06-17 21:25:57,531    DEBUG pyroute2.ndb.281473114185888.rtnl_object: error: (17, 'File exists')
2022-06-17 21:25:57,531    DEBUG pyroute2.ndb.281473114185888.rtnl_object: ignore error 17 for {'multipath': [], 'metrics': {}, 'deps': 0, 'dst': 'beef:feed:fade::', 'family': <AddressFamily.AF_INET6: 10>, 'dst_len': 112, 'oif': 3, 'priority': 0, 'table': 254, 'target': 'localhost'}
2022-06-17 21:25:57,532    DEBUG pyroute2.ndb.281473114185888.rtnl_object: run fallback <function fallback_add at 0xffff906bcee0> ({'target': 'localhost', 'family': <AddressFamily.AF_INET6: 10>, 'dst_len': 112, 'dst': 'beef:feed:fade::', 'oif': 3, 'priority': 0, 'table': 254, 'multipath': [], 'metrics': {}, 'deps': 0})
2022-06-17 21:25:57,532    DEBUG pyroute2.ndb.281473114185888.rtnl_object: ignore incomplete idx_req in the fallback
2022-06-17 21:25:57,532    DEBUG pyroute2.ndb.281473114185888.rtnl_object: stats: apply add {objid 281473063064096, wtime 0.1, mqsize 0, nqsize 0}
2022-06-17 21:25:57,534    DEBUG pyroute2.ndb.281473114185888.rtnl_object: load_sql load: None
2022-06-17 21:25:57,535    DEBUG pyroute2.ndb.281473114185888.rtnl_object: load_sql names: ['target', 'tflags', 'family', 'dst_len', 'src_len', 'tos', 'table', 'proto', 'scope', 'type', 'flags', 'dst', 'src', 'iif', 'oif', 'gateway', 'priority', 'prefsrc', 'protoinfo', 'flow', 'table', 'mark', 'via', 'newdst', 'pref', 'encap_type', 'route_id', 'gc_mark', 'deps']
2022-06-17 21:25:57,535    DEBUG pyroute2.ndb.281473114185888.rtnl_object: check: [(1655501157.4822505, 'invalid')]
2022-06-17 21:25:57,535    DEBUG pyroute2.ndb.281473114185888.rtnl_object: check state: False
2022-06-17 21:25:57,536    DEBUG pyroute2.ndb.281473114185888.rtnl_object: check failed
2022-06-17 21:25:57,637    DEBUG pyroute2.ndb.281473114185888.rtnl_object: API call add ({'target': 'localhost', 'family': <AddressFamily.AF_INET6: 10>, 'dst_len': 112, 'dst': 'beef:feed:fade::', 'oif': 3, 'priority': 0, 'table': 254, 'multipath': [], 'metrics': {}, 'deps': 0})
2022-06-17 21:25:57,637    DEBUG pyroute2.ndb.281473114185888.sources.localhost: source api run route ('add',) {'target': 'localhost', 'family': <AddressFamily.AF_INET6: 10>, 'dst_len': 112, 'dst': 'beef:feed:fade::', 'oif': 3, 'priority': 0, 'table': 254, 'multipath': [], 'metrics': {}, 'deps': 0}
2022-06-17 21:25:57,641    DEBUG pyroute2.ndb.281473114185888.rtnl_object: error: (17, 'File exists')
2022-06-17 21:25:57,642    DEBUG pyroute2.ndb.281473114185888.rtnl_object: ignore error 17 for {'multipath': [], 'metrics': {}, 'deps': 0, 'dst': 'beef:feed:fade::', 'family': <AddressFamily.AF_INET6: 10>, 'dst_len': 112, 'oif': 3, 'priority': 0, 'table': 254, 'target': 'localhost'}
2022-06-17 21:25:57,642    DEBUG pyroute2.ndb.281473114185888.rtnl_object: run fallback <function fallback_add at 0xffff906bcee0> ({'target': 'localhost', 'family': <AddressFamily.AF_INET6: 10>, 'dst_len': 112, 'dst': 'beef:feed:fade::', 'oif': 3, 'priority': 0, 'table': 254, 'multipath': [], 'metrics': {}, 'deps': 0})
2022-06-17 21:25:57,643    DEBUG pyroute2.ndb.281473114185888.rtnl_object: ignore incomplete idx_req in the fallback
2022-06-17 21:25:57,643    DEBUG pyroute2.ndb.281473114185888.rtnl_object: stats: apply add {objid 281473063064096, wtime 0.2, mqsize 0, nqsize 0}
2022-06-17 21:25:57,645    DEBUG pyroute2.ndb.281473114185888.rtnl_object: load_sql load: None
2022-06-17 21:25:57,645    DEBUG pyroute2.ndb.281473114185888.rtnl_object: load_sql names: ['target', 'tflags', 'family', 'dst_len', 'src_len', 'tos', 'table', 'proto', 'scope', 'type', 'flags', 'dst', 'src', 'iif', 'oif', 'gateway', 'priority', 'prefsrc', 'protoinfo', 'flow', 'table', 'mark', 'via', 'newdst', 'pref', 'encap_type', 'route_id', 'gc_mark', 'deps']
2022-06-17 21:25:57,646    DEBUG pyroute2.ndb.281473114185888.rtnl_object: check: [(1655501157.4822505, 'invalid')]
2022-06-17 21:25:57,646    DEBUG pyroute2.ndb.281473114185888.rtnl_object: check state: False
2022-06-17 21:25:57,646    DEBUG pyroute2.ndb.281473114185888.rtnl_object: check failed
2022-06-17 21:25:57,847    DEBUG pyroute2.ndb.281473114185888.rtnl_object: API call add ({'target': 'localhost', 'family': <AddressFamily.AF_INET6: 10>, 'dst_len': 112, 'dst': 'beef:feed:fade::', 'oif': 3, 'priority': 0, 'table': 254, 'multipath': [], 'metrics': {}, 'deps': 0})
2022-06-17 21:25:57,848    DEBUG pyroute2.ndb.281473114185888.sources.localhost: source api run route ('add',) {'target': 'localhost', 'family': <AddressFamily.AF_INET6: 10>, 'dst_len': 112, 'dst': 'beef:feed:fade::', 'oif': 3, 'priority': 0, 'table': 254, 'multipath': [], 'metrics': {}, 'deps': 0}
2022-06-17 21:25:57,852    DEBUG pyroute2.ndb.281473114185888.rtnl_object: error: (17, 'File exists')
2022-06-17 21:25:57,852    DEBUG pyroute2.ndb.281473114185888.rtnl_object: ignore error 17 for {'multipath': [], 'metrics': {}, 'deps': 0, 'dst': 'beef:feed:fade::', 'family': <AddressFamily.AF_INET6: 10>, 'dst_len': 112, 'oif': 3, 'priority': 0, 'table': 254, 'target': 'localhost'}
2022-06-17 21:25:57,852    DEBUG pyroute2.ndb.281473114185888.rtnl_object: run fallback <function fallback_add at 0xffff906bcee0> ({'target': 'localhost', 'family': <AddressFamily.AF_INET6: 10>, 'dst_len': 112, 'dst': 'beef:feed:fade::', 'oif': 3, 'priority': 0, 'table': 254, 'multipath': [], 'metrics': {}, 'deps': 0})
2022-06-17 21:25:57,853    DEBUG pyroute2.ndb.281473114185888.rtnl_object: ignore incomplete idx_req in the fallback
2022-06-17 21:25:57,853    DEBUG pyroute2.ndb.281473114185888.rtnl_object: stats: apply add {objid 281473063064096, wtime 0.30000000000000004, mqsize 0, nqsize 0}
2022-06-17 21:25:57,855    DEBUG pyroute2.ndb.281473114185888.rtnl_object: load_sql load: None
2022-06-17 21:25:57,856    DEBUG pyroute2.ndb.281473114185888.rtnl_object: load_sql names: ['target', 'tflags', 'family', 'dst_len', 'src_len', 'tos', 'table', 'proto', 'scope', 'type', 'flags', 'dst', 'src', 'iif', 'oif', 'gateway', 'priority', 'prefsrc', 'protoinfo', 'flow', 'table', 'mark', 'via', 'newdst', 'pref', 'encap_type', 'route_id', 'gc_mark', 'deps']
2022-06-17 21:25:57,856    DEBUG pyroute2.ndb.281473114185888.rtnl_object: check: [(1655501157.4822505, 'invalid')]
2022-06-17 21:25:57,857    DEBUG pyroute2.ndb.281473114185888.rtnl_object: check state: False
2022-06-17 21:25:57,857    DEBUG pyroute2.ndb.281473114185888.rtnl_object: check failed
2022-06-17 21:25:58,158    DEBUG pyroute2.ndb.281473114185888.rtnl_object: API call add ({'target': 'localhost', 'family': <AddressFamily.AF_INET6: 10>, 'dst_len': 112, 'dst': 'beef:feed:fade::', 'oif': 3, 'priority': 0, 'table': 254, 'multipath': [], 'metrics': {}, 'deps': 0})
2022-06-17 21:25:58,158    DEBUG pyroute2.ndb.281473114185888.sources.localhost: source api run route ('add',) {'target': 'localhost', 'family': <AddressFamily.AF_INET6: 10>, 'dst_len': 112, 'dst': 'beef:feed:fade::', 'oif': 3, 'priority': 0, 'table': 254, 'multipath': [], 'metrics': {}, 'deps': 0}
2022-06-17 21:25:58,162    DEBUG pyroute2.ndb.281473114185888.rtnl_object: error: (17, 'File exists')
2022-06-17 21:25:58,163    DEBUG pyroute2.ndb.281473114185888.rtnl_object: ignore error 17 for {'multipath': [], 'metrics': {}, 'deps': 0, 'dst': 'beef:feed:fade::', 'family': <AddressFamily.AF_INET6: 10>, 'dst_len': 112, 'oif': 3, 'priority': 0, 'table': 254, 'target': 'localhost'}
2022-06-17 21:25:58,163    DEBUG pyroute2.ndb.281473114185888.rtnl_object: run fallback <function fallback_add at 0xffff906bcee0> ({'target': 'localhost', 'family': <AddressFamily.AF_INET6: 10>, 'dst_len': 112, 'dst': 'beef:feed:fade::', 'oif': 3, 'priority': 0, 'table': 254, 'multipath': [], 'metrics': {}, 'deps': 0})
2022-06-17 21:25:58,164    DEBUG pyroute2.ndb.281473114185888.rtnl_object: ignore incomplete idx_req in the fallback
2022-06-17 21:25:58,164    DEBUG pyroute2.ndb.281473114185888.rtnl_object: stats: apply add {objid 281473063064096, wtime 0.4, mqsize 0, nqsize 0}
2022-06-17 21:25:58,166    DEBUG pyroute2.ndb.281473114185888.rtnl_object: load_sql load: None
2022-06-17 21:25:58,167    DEBUG pyroute2.ndb.281473114185888.rtnl_object: load_sql names: ['target', 'tflags', 'family', 'dst_len', 'src_len', 'tos', 'table', 'proto', 'scope', 'type', 'flags', 'dst', 'src', 'iif', 'oif', 'gateway', 'priority', 'prefsrc', 'protoinfo', 'flow', 'table', 'mark', 'via', 'newdst', 'pref', 'encap_type', 'route_id', 'gc_mark', 'deps']
2022-06-17 21:25:58,167    DEBUG pyroute2.ndb.281473114185888.rtnl_object: check: [(1655501157.4822505, 'invalid')]
2022-06-17 21:25:58,167    DEBUG pyroute2.ndb.281473114185888.rtnl_object: check state: False
2022-06-17 21:25:58,168    DEBUG pyroute2.ndb.281473114185888.rtnl_object: check failed
2022-06-17 21:25:58,569    DEBUG pyroute2.ndb.281473114185888.rtnl_object: API call add ({'target': 'localhost', 'family': <AddressFamily.AF_INET6: 10>, 'dst_len': 112, 'dst': 'beef:feed:fade::', 'oif': 3, 'priority': 0, 'table': 254, 'multipath': [], 'metrics': {}, 'deps': 0})
2022-06-17 21:25:58,569    DEBUG pyroute2.ndb.281473114185888.sources.localhost: source api run route ('add',) {'target': 'localhost', 'family': <AddressFamily.AF_INET6: 10>, 'dst_len': 112, 'dst': 'beef:feed:fade::', 'oif': 3, 'priority': 0, 'table': 254, 'multipath': [], 'metrics': {}, 'deps': 0}
2022-06-17 21:25:58,573    DEBUG pyroute2.ndb.281473114185888.rtnl_object: error: (17, 'File exists')
2022-06-17 21:25:58,574    DEBUG pyroute2.ndb.281473114185888.rtnl_object: ignore error 17 for {'multipath': [], 'metrics': {}, 'deps': 0, 'dst': 'beef:feed:fade::', 'family': <AddressFamily.AF_INET6: 10>, 'dst_len': 112, 'oif': 3, 'priority': 0, 'table': 254, 'target': 'localhost'}
2022-06-17 21:25:58,574    DEBUG pyroute2.ndb.281473114185888.rtnl_object: run fallback <function fallback_add at 0xffff906bcee0> ({'target': 'localhost', 'family': <AddressFamily.AF_INET6: 10>, 'dst_len': 112, 'dst': 'beef:feed:fade::', 'oif': 3, 'priority': 0, 'table': 254, 'multipath': [], 'metrics': {}, 'deps': 0})
2022-06-17 21:25:58,575    DEBUG pyroute2.ndb.281473114185888.rtnl_object: ignore incomplete idx_req in the fallback
2022-06-17 21:25:58,575    DEBUG pyroute2.ndb.281473114185888.rtnl_object: stats: apply add {objid 281473063064096, wtime 0.5, mqsize 0, nqsize 0}
2022-06-17 21:25:58,577    DEBUG pyroute2.ndb.281473114185888.rtnl_object: load_sql load: None
2022-06-17 21:25:58,577    DEBUG pyroute2.ndb.281473114185888.rtnl_object: load_sql names: ['target', 'tflags', 'family', 'dst_len', 'src_len', 'tos', 'table', 'proto', 'scope', 'type', 'flags', 'dst', 'src', 'iif', 'oif', 'gateway', 'priority', 'prefsrc', 'protoinfo', 'flow', 'table', 'mark', 'via', 'newdst', 'pref', 'encap_type', 'route_id', 'gc_mark', 'deps']
2022-06-17 21:25:58,578    DEBUG pyroute2.ndb.281473114185888.rtnl_object: check: [(1655501157.4822505, 'invalid')]
2022-06-17 21:25:58,578    DEBUG pyroute2.ndb.281473114185888.rtnl_object: check state: False
2022-06-17 21:25:58,578    DEBUG pyroute2.ndb.281473114185888.rtnl_object: check failed
2022-06-17 21:25:59,079    DEBUG pyroute2.ndb.281473114185888.rtnl_object: API call add ({'target': 'localhost', 'family': <AddressFamily.AF_INET6: 10>, 'dst_len': 112, 'dst': 'beef:feed:fade::', 'oif': 3, 'priority': 0, 'table': 254, 'multipath': [], 'metrics': {}, 'deps': 0})
2022-06-17 21:25:59,080    DEBUG pyroute2.ndb.281473114185888.sources.localhost: source api run route ('add',) {'target': 'localhost', 'family': <AddressFamily.AF_INET6: 10>, 'dst_len': 112, 'dst': 'beef:feed:fade::', 'oif': 3, 'priority': 0, 'table': 254, 'multipath': [], 'metrics': {}, 'deps': 0}
2022-06-17 21:25:59,084    DEBUG pyroute2.ndb.281473114185888.rtnl_object: error: (17, 'File exists')
2022-06-17 21:25:59,085    DEBUG pyroute2.ndb.281473114185888.rtnl_object: ignore error 17 for {'multipath': [], 'metrics': {}, 'deps': 0, 'dst': 'beef:feed:fade::', 'family': <AddressFamily.AF_INET6: 10>, 'dst_len': 112, 'oif': 3, 'priority': 0, 'table': 254, 'target': 'localhost'}
2022-06-17 21:25:59,085    DEBUG pyroute2.ndb.281473114185888.rtnl_object: run fallback <function fallback_add at 0xffff906bcee0> ({'target': 'localhost', 'family': <AddressFamily.AF_INET6: 10>, 'dst_len': 112, 'dst': 'beef:feed:fade::', 'oif': 3, 'priority': 0, 'table': 254, 'multipath': [], 'metrics': {}, 'deps': 0})
2022-06-17 21:25:59,085    DEBUG pyroute2.ndb.281473114185888.rtnl_object: ignore incomplete idx_req in the fallback
2022-06-17 21:25:59,086    DEBUG pyroute2.ndb.281473114185888.rtnl_object: stats: apply add {objid 281473063064096, wtime 0.6000000000000001, mqsize 0, nqsize 0}
2022-06-17 21:25:59,088    DEBUG pyroute2.ndb.281473114185888.rtnl_object: load_sql load: None
2022-06-17 21:25:59,088    DEBUG pyroute2.ndb.281473114185888.rtnl_object: load_sql names: ['target', 'tflags', 'family', 'dst_len', 'src_len', 'tos', 'table', 'proto', 'scope', 'type', 'flags', 'dst', 'src', 'iif', 'oif', 'gateway', 'priority', 'prefsrc', 'protoinfo', 'flow', 'table', 'mark', 'via', 'newdst', 'pref', 'encap_type', 'route_id', 'gc_mark', 'deps']
2022-06-17 21:25:59,089    DEBUG pyroute2.ndb.281473114185888.rtnl_object: check: [(1655501157.4822505, 'invalid')]
2022-06-17 21:25:59,089    DEBUG pyroute2.ndb.281473114185888.rtnl_object: check state: False
2022-06-17 21:25:59,089    DEBUG pyroute2.ndb.281473114185888.rtnl_object: check failed
2022-06-17 21:25:59,690    DEBUG pyroute2.ndb.281473114185888.rtnl_object: API call add ({'target': 'localhost', 'family': <AddressFamily.AF_INET6: 10>, 'dst_len': 112, 'dst': 'beef:feed:fade::', 'oif': 3, 'priority': 0, 'table': 254, 'multipath': [], 'metrics': {}, 'deps': 0})
2022-06-17 21:25:59,691    DEBUG pyroute2.ndb.281473114185888.sources.localhost: source api run route ('add',) {'target': 'localhost', 'family': <AddressFamily.AF_INET6: 10>, 'dst_len': 112, 'dst': 'beef:feed:fade::', 'oif': 3, 'priority': 0, 'table': 254, 'multipath': [], 'metrics': {}, 'deps': 0}
2022-06-17 21:25:59,695    DEBUG pyroute2.ndb.281473114185888.rtnl_object: error: (17, 'File exists')
2022-06-17 21:25:59,695    DEBUG pyroute2.ndb.281473114185888.rtnl_object: ignore error 17 for {'multipath': [], 'metrics': {}, 'deps': 0, 'dst': 'beef:feed:fade::', 'family': <AddressFamily.AF_INET6: 10>, 'dst_len': 112, 'oif': 3, 'priority': 0, 'table': 254, 'target': 'localhost'}
2022-06-17 21:25:59,696    DEBUG pyroute2.ndb.281473114185888.rtnl_object: run fallback <function fallback_add at 0xffff906bcee0> ({'target': 'localhost', 'family': <AddressFamily.AF_INET6: 10>, 'dst_len': 112, 'dst': 'beef:feed:fade::', 'oif': 3, 'priority': 0, 'table': 254, 'multipath': [], 'metrics': {}, 'deps': 0})
2022-06-17 21:25:59,696    DEBUG pyroute2.ndb.281473114185888.rtnl_object: ignore incomplete idx_req in the fallback
2022-06-17 21:25:59,696    DEBUG pyroute2.ndb.281473114185888.rtnl_object: stats: apply add {objid 281473063064096, wtime 0.7000000000000001, mqsize 0, nqsize 0}
2022-06-17 21:25:59,698    DEBUG pyroute2.ndb.281473114185888.rtnl_object: load_sql load: None
2022-06-17 21:25:59,699    DEBUG pyroute2.ndb.281473114185888.rtnl_object: load_sql names: ['target', 'tflags', 'family', 'dst_len', 'src_len', 'tos', 'table', 'proto', 'scope', 'type', 'flags', 'dst', 'src', 'iif', 'oif', 'gateway', 'priority', 'prefsrc', 'protoinfo', 'flow', 'table', 'mark', 'via', 'newdst', 'pref', 'encap_type', 'route_id', 'gc_mark', 'deps']
2022-06-17 21:25:59,699    DEBUG pyroute2.ndb.281473114185888.rtnl_object: check: [(1655501157.4822505, 'invalid')]
2022-06-17 21:25:59,700    DEBUG pyroute2.ndb.281473114185888.rtnl_object: check state: False
2022-06-17 21:25:59,700    DEBUG pyroute2.ndb.281473114185888.rtnl_object: check failed
2022-06-17 21:26:00,401    DEBUG pyroute2.ndb.281473114185888.rtnl_object: API call add ({'target': 'localhost', 'family': <AddressFamily.AF_INET6: 10>, 'dst_len': 112, 'dst': 'beef:feed:fade::', 'oif': 3, 'priority': 0, 'table': 254, 'multipath': [], 'metrics': {}, 'deps': 0})
2022-06-17 21:26:00,401    DEBUG pyroute2.ndb.281473114185888.sources.localhost: source api run route ('add',) {'target': 'localhost', 'family': <AddressFamily.AF_INET6: 10>, 'dst_len': 112, 'dst': 'beef:feed:fade::', 'oif': 3, 'priority': 0, 'table': 254, 'multipath': [], 'metrics': {}, 'deps': 0}
2022-06-17 21:26:00,405    DEBUG pyroute2.ndb.281473114185888.rtnl_object: error: (17, 'File exists')
2022-06-17 21:26:00,406    DEBUG pyroute2.ndb.281473114185888.rtnl_object: ignore error 17 for {'multipath': [], 'metrics': {}, 'deps': 0, 'dst': 'beef:feed:fade::', 'family': <AddressFamily.AF_INET6: 10>, 'dst_len': 112, 'oif': 3, 'priority': 0, 'table': 254, 'target': 'localhost'}
2022-06-17 21:26:00,406    DEBUG pyroute2.ndb.281473114185888.rtnl_object: run fallback <function fallback_add at 0xffff906bcee0> ({'target': 'localhost', 'family': <AddressFamily.AF_INET6: 10>, 'dst_len': 112, 'dst': 'beef:feed:fade::', 'oif': 3, 'priority': 0, 'table': 254, 'multipath': [], 'metrics': {}, 'deps': 0})
2022-06-17 21:26:00,407    DEBUG pyroute2.ndb.281473114185888.rtnl_object: ignore incomplete idx_req in the fallback
2022-06-17 21:26:00,407    DEBUG pyroute2.ndb.281473114185888.rtnl_object: stats: apply add {objid 281473063064096, wtime 0.8, mqsize 0, nqsize 0}
2022-06-17 21:26:00,409    DEBUG pyroute2.ndb.281473114185888.rtnl_object: load_sql load: None
2022-06-17 21:26:00,410    DEBUG pyroute2.ndb.281473114185888.rtnl_object: load_sql names: ['target', 'tflags', 'family', 'dst_len', 'src_len', 'tos', 'table', 'proto', 'scope', 'type', 'flags', 'dst', 'src', 'iif', 'oif', 'gateway', 'priority', 'prefsrc', 'protoinfo', 'flow', 'table', 'mark', 'via', 'newdst', 'pref', 'encap_type', 'route_id', 'gc_mark', 'deps']
2022-06-17 21:26:00,410    DEBUG pyroute2.ndb.281473114185888.rtnl_object: check: [(1655501157.4822505, 'invalid')]
2022-06-17 21:26:00,410    DEBUG pyroute2.ndb.281473114185888.rtnl_object: check state: False
2022-06-17 21:26:00,411    DEBUG pyroute2.ndb.281473114185888.rtnl_object: check failed
2022-06-17 21:26:01,212    DEBUG pyroute2.ndb.281473114185888.rtnl_object: API call add ({'target': 'localhost', 'family': <AddressFamily.AF_INET6: 10>, 'dst_len': 112, 'dst': 'beef:feed:fade::', 'oif': 3, 'priority': 0, 'table': 254, 'multipath': [], 'metrics': {}, 'deps': 0})
2022-06-17 21:26:01,212    DEBUG pyroute2.ndb.281473114185888.sources.localhost: source api run route ('add',) {'target': 'localhost', 'family': <AddressFamily.AF_INET6: 10>, 'dst_len': 112, 'dst': 'beef:feed:fade::', 'oif': 3, 'priority': 0, 'table': 254, 'multipath': [], 'metrics': {}, 'deps': 0}
2022-06-17 21:26:01,216    DEBUG pyroute2.ndb.281473114185888.rtnl_object: error: (17, 'File exists')
2022-06-17 21:26:01,217    DEBUG pyroute2.ndb.281473114185888.rtnl_object: ignore error 17 for {'multipath': [], 'metrics': {}, 'deps': 0, 'dst': 'beef:feed:fade::', 'family': <AddressFamily.AF_INET6: 10>, 'dst_len': 112, 'oif': 3, 'priority': 0, 'table': 254, 'target': 'localhost'}
2022-06-17 21:26:01,217    DEBUG pyroute2.ndb.281473114185888.rtnl_object: run fallback <function fallback_add at 0xffff906bcee0> ({'target': 'localhost', 'family': <AddressFamily.AF_INET6: 10>, 'dst_len': 112, 'dst': 'beef:feed:fade::', 'oif': 3, 'priority': 0, 'table': 254, 'multipath': [], 'metrics': {}, 'deps': 0})
2022-06-17 21:26:01,217    DEBUG pyroute2.ndb.281473114185888.rtnl_object: ignore incomplete idx_req in the fallback
2022-06-17 21:26:01,218    DEBUG pyroute2.ndb.281473114185888.rtnl_object: stats: apply add {objid 281473063064096, wtime 0.9, mqsize 0, nqsize 0}
2022-06-17 21:26:01,220    DEBUG pyroute2.ndb.281473114185888.rtnl_object: load_sql load: None
2022-06-17 21:26:01,220    DEBUG pyroute2.ndb.281473114185888.rtnl_object: load_sql names: ['target', 'tflags', 'family', 'dst_len', 'src_len', 'tos', 'table', 'proto', 'scope', 'type', 'flags', 'dst', 'src', 'iif', 'oif', 'gateway', 'priority', 'prefsrc', 'protoinfo', 'flow', 'table', 'mark', 'via', 'newdst', 'pref', 'encap_type', 'route_id', 'gc_mark', 'deps']
2022-06-17 21:26:01,221    DEBUG pyroute2.ndb.281473114185888.rtnl_object: check: [(1655501157.4822505, 'invalid')]
2022-06-17 21:26:01,221    DEBUG pyroute2.ndb.281473114185888.rtnl_object: check state: False
2022-06-17 21:26:01,221    DEBUG pyroute2.ndb.281473114185888.rtnl_object: check failed
2022-06-17 21:26:02,122    DEBUG pyroute2.ndb.281473114185888.rtnl_object: stats: 281473063064096 apply add fail
2022-06-17 21:26:02,123    DEBUG pyroute2.ndb.281473114185888.rtnl_object: criteria <function RTNL_Object.apply.<locals>.<lambda> at 0xffff8dee0af0>
2022-06-17 21:26:02,123    DEBUG pyroute2.ndb.281473114185888.rtnl_object: method <bound method RTNL_Object.check of {'multipath': [], 'metrics': {}, 'deps': 0, 'dst': 'beef:feed:fade::', 'family': <AddressFamily.AF_INET6: 10>, 'dst_len': 112, 'oif': 3, 'priority': 0, 'table': 254, 'target': 'localhost'}>, [], {}
2022-06-17 21:26:02,125    DEBUG pyroute2.ndb.281473114185888.rtnl_object: load_sql load: None
2022-06-17 21:26:02,126    DEBUG pyroute2.ndb.281473114185888.rtnl_object: load_sql names: ['target', 'tflags', 'family', 'dst_len', 'src_len', 'tos', 'table', 'proto', 'scope', 'type', 'flags', 'dst', 'src', 'iif', 'oif', 'gateway', 'priority', 'prefsrc', 'protoinfo', 'flow', 'table', 'mark', 'via', 'newdst', 'pref', 'encap_type', 'route_id', 'gc_mark', 'deps']
2022-06-17 21:26:02,126    DEBUG pyroute2.ndb.281473114185888.rtnl_object: check: [(1655501157.4822505, 'invalid')]
2022-06-17 21:26:02,127    DEBUG pyroute2.ndb.281473114185888.rtnl_object: check state: False
2022-06-17 21:26:02,127    DEBUG pyroute2.ndb.281473114185888.rtnl_object: ret False
2022-06-17 21:26:02,127    DEBUG pyroute2.ndb.281473114185888.rtnl_object: resync the DB attempt 0
2022-06-17 21:26:02,140    DEBUG pyroute2.ndb.281473114185888.sources.localhost: source api run dump () {}
2022-06-17 21:26:02,295    DEBUG pyroute2.ndb.281473114185888.rtnl_object: load_rtnl: {'length': 52, 'type': 24, 'flags': 2, 'sequence_number': 279, 'pid': 3780768812, 'error': None, 'target': 'localhost', 'stats': Stats(qsize=0, delta=0, delay=0)}
2022-06-17 21:26:02,298    DEBUG pyroute2.ndb.281473114185888.rtnl_object: load_sql load: ('localhost', 0, 2, 24, 0, 0, 254, 4, 0, 1, 0, '1.2.3.0', None, None, 3, None, 0, None, None, None, 254, None, '', '', None, None, None, None, 0)
2022-06-17 21:26:02,298    DEBUG pyroute2.ndb.281473114185888.rtnl_object: load_sql names: ['target', 'tflags', 'family', 'dst_len', 'src_len', 'tos', 'table', 'proto', 'scope', 'type', 'flags', 'dst', 'src', 'iif', 'oif', 'gateway', 'priority', 'prefsrc', 'protoinfo', 'flow', 'table', 'mark', 'via', 'newdst', 'pref', 'encap_type', 'route_id', 'gc_mark', 'deps']
2022-06-17 21:26:02,458    DEBUG pyroute2.ndb.281473114185888.rtnl_object: load_sql load: None
2022-06-17 21:26:02,458    DEBUG pyroute2.ndb.281473114185888.rtnl_object: load_sql names: ['target', 'tflags', 'family', 'dst_len', 'src_len', 'tos', 'table', 'proto', 'scope', 'type', 'flags', 'dst', 'src', 'iif', 'oif', 'gateway', 'priority', 'prefsrc', 'protoinfo', 'flow', 'table', 'mark', 'via', 'newdst', 'pref', 'encap_type', 'route_id', 'gc_mark', 'deps']
2022-06-17 21:26:02,459    DEBUG pyroute2.ndb.281473114185888.rtnl_object: check: [(1655501157.4822505, 'invalid')]
2022-06-17 21:26:02,459    DEBUG pyroute2.ndb.281473114185888.rtnl_object: check state: False
2022-06-17 21:26:02,459    DEBUG pyroute2.ndb.281473114185888.rtnl_object: ret False
2022-06-17 21:26:02,459    DEBUG pyroute2.ndb.281473114185888.rtnl_object: resync the DB attempt 1
2022-06-17 21:26:02,468    DEBUG pyroute2.ndb.281473114185888.sources.localhost: source api run dump () {}
2022-06-17 21:26:02,625    DEBUG pyroute2.ndb.281473114185888.rtnl_object: load_rtnl: {'length': 52, 'type': 24, 'flags': 2, 'sequence_number': 287, 'pid': 3780768812, 'error': None, 'target': 'localhost', 'stats': Stats(qsize=0, delta=0, delay=0)}
2022-06-17 21:26:02,626    DEBUG pyroute2.ndb.281473114185888.rtnl_object: load_sql load: ('localhost', 0, 2, 24, 0, 0, 254, 4, 0, 1, 0, '1.2.3.0', None, None, 3, None, 0, None, None, None, 254, None, '', '', None, None, None, None, 0)
2022-06-17 21:26:02,627    DEBUG pyroute2.ndb.281473114185888.rtnl_object: load_sql names: ['target', 'tflags', 'family', 'dst_len', 'src_len', 'tos', 'table', 'proto', 'scope', 'type', 'flags', 'dst', 'src', 'iif', 'oif', 'gateway', 'priority', 'prefsrc', 'protoinfo', 'flow', 'table', 'mark', 'via', 'newdst', 'pref', 'encap_type', 'route_id', 'gc_mark', 'deps']
2022-06-17 21:26:02,797    DEBUG pyroute2.ndb.281473114185888.rtnl_object: load_sql load: None
2022-06-17 21:26:02,797    DEBUG pyroute2.ndb.281473114185888.rtnl_object: load_sql names: ['target', 'tflags', 'family', 'dst_len', 'src_len', 'tos', 'table', 'proto', 'scope', 'type', 'flags', 'dst', 'src', 'iif', 'oif', 'gateway', 'priority', 'prefsrc', 'protoinfo', 'flow', 'table', 'mark', 'via', 'newdst', 'pref', 'encap_type', 'route_id', 'gc_mark', 'deps']
2022-06-17 21:26:02,798    DEBUG pyroute2.ndb.281473114185888.rtnl_object: check: [(1655501157.4822505, 'invalid')]
2022-06-17 21:26:02,798    DEBUG pyroute2.ndb.281473114185888.rtnl_object: check state: False
2022-06-17 21:26:02,798    DEBUG pyroute2.ndb.281473114185888.rtnl_object: ret False
2022-06-17 21:26:02,798    DEBUG pyroute2.ndb.281473114185888.rtnl_object: resync the DB attempt 2
2022-06-17 21:26:02,807    DEBUG pyroute2.ndb.281473114185888.sources.localhost: source api run dump () {}
2022-06-17 21:26:02,964    DEBUG pyroute2.ndb.281473114185888.rtnl_object: load_rtnl: {'length': 52, 'type': 24, 'flags': 2, 'sequence_number': 295, 'pid': 3780768812, 'error': None, 'target': 'localhost', 'stats': Stats(qsize=0, delta=0, delay=0)}
2022-06-17 21:26:02,966    DEBUG pyroute2.ndb.281473114185888.rtnl_object: load_sql load: ('localhost', 0, 2, 24, 0, 0, 254, 4, 0, 1, 0, '1.2.3.0', None, None, 3, None, 0, None, None, None, 254, None, '', '', None, None, None, None, 0)
2022-06-17 21:26:02,966    DEBUG pyroute2.ndb.281473114185888.rtnl_object: load_sql names: ['target', 'tflags', 'family', 'dst_len', 'src_len', 'tos', 'table', 'proto', 'scope', 'type', 'flags', 'dst', 'src', 'iif', 'oif', 'gateway', 'priority', 'prefsrc', 'protoinfo', 'flow', 'table', 'mark', 'via', 'newdst', 'pref', 'encap_type', 'route_id', 'gc_mark', 'deps']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/snap/iotr-cr-sysmgr/x130/lib/python3.8/site-packages/pr2modules/ndb/auth_manager.py", line 67, in guard
    return f(obj, *argv, **kwarg)
  File "/snap/iotr-cr-sysmgr/x130/lib/python3.8/site-packages/pr2modules/ndb/objects/__init__.py", line 694, in commit
    raise e_i
  File "/snap/iotr-cr-sysmgr/x130/lib/python3.8/site-packages/pr2modules/ndb/objects/__init__.py", line 683, in commit
    return self.apply(mode='commit')
  File "/snap/iotr-cr-sysmgr/x130/lib/python3.8/site-packages/pr2modules/ndb/auth_manager.py", line 67, in guard
    return f(obj, *argv, **kwarg)
  File "/snap/iotr-cr-sysmgr/x130/lib/python3.8/site-packages/pr2modules/ndb/objects/route.py", line 691, in apply
    return super(Route, self).apply(rollback, req_filter, mode)
  File "/snap/iotr-cr-sysmgr/x130/lib/python3.8/site-packages/pr2modules/ndb/auth_manager.py", line 67, in guard
    return f(obj, *argv, **kwarg)
  File "/snap/iotr-cr-sysmgr/x130/lib/python3.8/site-packages/pr2modules/ndb/objects/__init__.py", line 935, in apply
    raise Exception('could not apply the changes')
Exception: could not apply the changes

This time the route was added but with priority 1024 instead of 0.

[svinota]

Thanks a lot, that helps!

[carloslockward]

Also, when I try to delete that route that was just added i get this:

>>> ndb.routes[{"oif": 3, "dst": "beef:feed:fade::/112"}]
2022-06-17 21:56:00,303    DEBUG pyroute2.ndb.281473114185888.rtnl_object: init
2022-06-17 21:56:00,305    DEBUG pyroute2.ndb.281473114185888.view.routes: check if the key {'oif': 3, 'dst': 'beef:feed:fade::', 'dst_len': 112, 'family': <AddressFamily.AF_INET6: 10>, 'target': 'localhost'} exists in table routes
2022-06-17 21:56:00,308    DEBUG pyroute2.ndb.281473114185888.view.routes: exists
2022-06-17 21:56:00,308    DEBUG pyroute2.ndb.281473114185888.rtnl_object: complete key {'oif': 3, 'dst': 'beef:feed:fade::', 'dst_len': 112, 'family': <AddressFamily.AF_INET6: 10>, 'target': 'localhost', 'table': 254} from table routes
2022-06-17 21:56:00,311    DEBUG pyroute2.ndb.281473114185888.rtnl_object: got {'oif': 3, 'dst': 'beef:feed:fade::', 'dst_len': 112, 'family': <AddressFamily.AF_INET6: 10>, 'target': 'localhost', 'table': 254, 'tflags': 0, 'tos': 0, 'scope': 0, 'RTA_DST': 'beef:feed:fade::', 'RTA_OIF': 3, 'RTA_PRIORITY': 1, 'RTA_TABLE': 254, 'RTA_VIA': '', 'RTA_NEWDST': ''}
2022-06-17 21:56:00,312    DEBUG pyroute2.ndb.281473114185888.view.routes: check if the key {'oif': 3, 'dst': 'beef:feed:fade::', 'dst_len': 112, 'family': <AddressFamily.AF_INET6: 10>, 'target': 'localhost'} exists in table routes
2022-06-17 21:56:00,315    DEBUG pyroute2.ndb.281473114185888.view.routes: exists
2022-06-17 21:56:00,318    DEBUG pyroute2.ndb.281473114185888.rtnl_object: load_sql load: ('localhost', 0, 10, 112, 0, 0, 254, 4, 0, 1, 0, 'beef:feed:fade::', None, None, 3, None, 1, None, None, None, 254, None, '', '', 0, None, None, None, 0)
2022-06-17 21:56:00,318    DEBUG pyroute2.ndb.281473114185888.rtnl_object: load_sql names: ['target', 'tflags', 'family', 'dst_len', 'src_len', 'tos', 'table', 'proto', 'scope', 'type', 'flags', 'dst', 'src', 'iif', 'oif', 'gateway', 'priority', 'prefsrc', 'protoinfo', 'flow', 'table', 'mark', 'via', 'newdst', 'pref', 'encap_type', 'route_id', 'gc_mark', 'deps']
2022-06-17 21:56:00,319    DEBUG pyroute2.ndb.281473114185888.view.routes: cache add (('target', 'localhost'), ('tflags', 0), ('family', <AddressFamily.AF_INET6: 10>), ('dst_len', 112), ('tos', 0), ('scope', 0), ('RTA_DST', 'beef:feed:fade::'), ('RTA_OIF', 3), ('RTA_PRIORITY', 1), ('RTA_TABLE', 254), ('RTA_VIA', ''), ('RTA_NEWDST', ''))
{'multipath': [], 'metrics': {}, 'deps': 0, 'oif': 3, 'dst': 'beef:feed:fade::', 'dst_len': 112, 'family': 10, 'target': 'localhost', 'table': 254, 'tflags': 0, 'tos': 0, 'scope': 0, 'priority': 1, 'via': '', 'newdst': '', 'src_len': 0, 'proto': 4, 'type': 1, 'flags': 0, 'src': None, 'iif': None, 'gateway': None, 'prefsrc': None, 'protoinfo': None, 'flow': None, 'mark': None, 'pref': 0, 'encap_type': None, 'route_id': None, 'gc_mark': None}
>>> ndb.routes[{"oif": 3, "dst": "beef:feed:fade::/112"}].remove().commit()
2022-06-17 21:56:12,248    DEBUG pyroute2.ndb.281473114185888.rtnl_object: init
2022-06-17 21:56:12,250    DEBUG pyroute2.ndb.281473114185888.view.routes: check if the key {'oif': 3, 'dst': 'beef:feed:fade::', 'dst_len': 112, 'family': <AddressFamily.AF_INET6: 10>, 'target': 'localhost'} exists in table routes
2022-06-17 21:56:12,252    DEBUG pyroute2.ndb.281473114185888.view.routes: exists
2022-06-17 21:56:12,252    DEBUG pyroute2.ndb.281473114185888.rtnl_object: complete key {'oif': 3, 'dst': 'beef:feed:fade::', 'dst_len': 112, 'family': <AddressFamily.AF_INET6: 10>, 'target': 'localhost', 'table': 254} from table routes
2022-06-17 21:56:12,255    DEBUG pyroute2.ndb.281473114185888.rtnl_object: got {'oif': 3, 'dst': 'beef:feed:fade::', 'dst_len': 112, 'family': <AddressFamily.AF_INET6: 10>, 'target': 'localhost', 'table': 254, 'tflags': 0, 'tos': 0, 'scope': 0, 'RTA_DST': 'beef:feed:fade::', 'RTA_OIF': 3, 'RTA_PRIORITY': 1, 'RTA_TABLE': 254, 'RTA_VIA': '', 'RTA_NEWDST': ''}
2022-06-17 21:56:12,256    DEBUG pyroute2.ndb.281473114185888.view.routes: cache hit (('target', 'localhost'), ('tflags', 0), ('family', <AddressFamily.AF_INET6: 10>), ('dst_len', 112), ('tos', 0), ('scope', 0), ('RTA_DST', 'beef:feed:fade::'), ('RTA_OIF', 3), ('RTA_PRIORITY', 1), ('RTA_TABLE', 254), ('RTA_VIA', ''), ('RTA_NEWDST', ''))
2022-06-17 21:56:12,256    DEBUG pyroute2.ndb.281473114185888.rtnl_object: commit: [(1655502960.3044271, 'invalid'), (1655502960.319205, 'system'), (1655502972.2566624, 'remove')]
2022-06-17 21:56:12,257    DEBUG pyroute2.ndb.281473114185888.rtnl_object: init
2022-06-17 21:56:12,259    DEBUG pyroute2.ndb.281473114185888.view.routes: check if the key {'target': 'localhost', 'tflags': 0, 'family': <AddressFamily.AF_INET: 2>, 'dst_len': 112, 'tos': 0, 'scope': 0, 'dst': 'beef:feed:fade::', 'oif': 3, 'priority': 1, 'table': 254, 'via': '', 'newdst': ''} exists in table routes
2022-06-17 21:56:12,262    DEBUG pyroute2.ndb.281473114185888.view.routes: not exists
2022-06-17 21:56:12,262    DEBUG pyroute2.ndb.281473114185888.rtnl_object: complete key {'target': 'localhost', 'tflags': 0, 'family': 10, 'dst_len': 112, 'tos': 0, 'scope': 0, 'RTA_DST': 'beef:feed:fade::', 'RTA_OIF': 3, 'RTA_PRIORITY': 1, 'RTA_TABLE': 254, 'RTA_VIA': '', 'RTA_NEWDST': '', 'table': 254} from table routes_281473061720128
2022-06-17 21:56:12,263    DEBUG pyroute2.ndb.281473114185888.rtnl_object: got {'target': 'localhost', 'tflags': 0, 'family': 10, 'dst_len': 112, 'tos': 0, 'scope': 0, 'RTA_DST': 'beef:feed:fade::', 'RTA_OIF': 3, 'RTA_PRIORITY': 1, 'RTA_TABLE': 254, 'RTA_VIA': '', 'RTA_NEWDST': '', 'table': 254}
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/snap/iotr-cr-sysmgr/x130/lib/python3.8/site-packages/pr2modules/ndb/auth_manager.py", line 67, in guard
    return f(obj, *argv, **kwarg)
  File "/snap/iotr-cr-sysmgr/x130/lib/python3.8/site-packages/pr2modules/ndb/objects/__init__.py", line 706, in commit
    self.last_save = self.snapshot()
  File "/snap/iotr-cr-sysmgr/x130/lib/python3.8/site-packages/pr2modules/ndb/auth_manager.py", line 67, in guard
    return f(obj, *argv, **kwarg)
  File "/snap/iotr-cr-sysmgr/x130/lib/python3.8/site-packages/pr2modules/ndb/objects/__init__.py", line 557, in snapshot
    snp = type(self)(
  File "/snap/iotr-cr-sysmgr/x130/lib/python3.8/site-packages/pr2modules/ndb/objects/route.py", line 570, in __init__
    super(Route, self).__init__(*argv, **kwarg)
  File "/snap/iotr-cr-sysmgr/x130/lib/python3.8/site-packages/pr2modules/ndb/objects/__init__.py", line 323, in __init__
    raise KeyError('object does not exists')
KeyError: 'object does not exists'

Even though that route exists.

I also noticed there are multiple route with different priorities on that same interface, maybe that has something to do with the issue.

ubuntu@IOTR1201211029111139:~$ ip -6 route
::1 dev lo proto kernel metric 256 pref medium
beef:feed:fade::ffee dev enp1s0 proto kernel metric 256 pref medium
beef:feed:fade::/112 dev enp1s0 proto static metric 1 pref medium
beef:feed:fade::/112 dev enp1s0 proto static metric 1024 pref medium
...

[svinota]

The issue has been found, the fix will be ready tonight. Thanks for the help!

[svinota]

Here's the reason for IPv6:

net/ipv6/route.c :

static int inet6_rtm_newroute(struct sk_buff *skb, struct nlmsghdr *nlh,
			      struct netlink_ext_ack *extack)
{
	struct fib6_config cfg;
	int err;

	err = rtm_to_fib6_config(skb, nlh, &cfg, extack);
	if (err < 0)
		return err;

	if (cfg.fc_metric == 0)                                    //  <- that's the reason
		cfg.fc_metric = IP6_RT_PRIO_USER;                  //

	if (cfg.fc_mp)
		return ip6_route_multipath_add(&cfg, extack);
	else
		return ip6_route_add(&cfg, GFP_KERNEL, extack);
}

IPv4 doesn't have such defaults

another possible issues source: ra_defrtr_metric

[svinota]

@carloslockward pls check if it's fixed for you now. The next release to be tagged tomorrow then.

[carloslockward]

I managed to get the logs for the ipv4 issue!

Here they are:

>>> parameters = {"dst": "default", "gateway": "10.10.2.1", "oif": 2, "table": 254}
>>> ndb.routes.create(**parameters).commit()
2022-06-18 20:32:37,867    DEBUG pyroute2.ndb.281473158053360.rtnl_object: init
2022-06-18 20:32:37,869    DEBUG pyroute2.ndb.281473158053360.view.routes: check if the key {'dst': '', 'gateway': '10.10.2.1', 'family': <AddressFamily.AF_INET: 2>, 'oif': 2, 'table': 254, 'target': 'localhost'} exists in table routes
2022-06-18 20:32:37,872    DEBUG pyroute2.ndb.281473158053360.view.routes: not exists
2022-06-18 20:32:37,872    DEBUG pyroute2.ndb.281473158053360.rtnl_object: complete key {'dst': '', 'gateway': '10.10.2.1', 'family': <AddressFamily.AF_INET: 2>, 'oif': 2, 'table': 254, 'target': 'localhost'} from table routes
2022-06-18 20:32:37,875    DEBUG pyroute2.ndb.281473158053360.rtnl_object: got none
2022-06-18 20:32:37,876    DEBUG pyroute2.ndb.281473158053360.view.routes: check if the key {'dst': '', 'gateway': '10.10.2.1', 'family': <AddressFamily.AF_INET: 2>, 'oif': 2, 'table': 254, 'target': 'localhost', 'create': True} exists in table routes
2022-06-18 20:32:37,879    DEBUG pyroute2.ndb.281473158053360.view.routes: not exists
2022-06-18 20:32:37,880    DEBUG pyroute2.ndb.281473158053360.rtnl_object: commit: [(1655584357.8683937, 'invalid')]
2022-06-18 20:32:37,880    DEBUG pyroute2.ndb.281473158053360.rtnl_object: events log: [(1655584357.8683937, 'invalid')]
2022-06-18 20:32:37,881    DEBUG pyroute2.ndb.281473158053360.rtnl_object: run apply
2022-06-18 20:32:37,884    DEBUG pyroute2.ndb.281473158053360.rtnl_object: load_sql load: None
2022-06-18 20:32:37,885    DEBUG pyroute2.ndb.281473158053360.rtnl_object: load_sql names: ['target', 'tflags', 'family', 'dst_len', 'src_len', 'tos', 'table', 'proto', 'scope', 'type', 'flags', 'dst', 'src', 'iif', 'oif', 'gateway', 'priority', 'prefsrc', 'protoinfo', 'flow', 'table', 'mark', 'via', 'newdst', 'pref', 'encap_type', 'route_id', 'gc_mark', 'deps']
2022-06-18 20:32:37,885    DEBUG pyroute2.ndb.281473158053360.rtnl_object: apply req: {'target': 'localhost', 'family': <AddressFamily.AF_INET: 2>, 'dst': '', 'oif': 2, 'table': 254, 'gateway': '10.10.2.1'}
2022-06-18 20:32:37,886    DEBUG pyroute2.ndb.281473158053360.rtnl_object: apply idx_req: {'target': 'localhost', 'family': <AddressFamily.AF_INET: 2>, 'dst': '', 'oif': 2, 'table': 254}
2022-06-18 20:32:37,886    DEBUG pyroute2.ndb.281473158053360.rtnl_object: apply transition from: invalid
2022-06-18 20:32:37,886    DEBUG pyroute2.ndb.281473158053360.rtnl_object: apply method: add
2022-06-18 20:32:37,887    DEBUG pyroute2.ndb.281473158053360.rtnl_object: API call add ({'target': 'localhost', 'family': <AddressFamily.AF_INET: 2>, 'dst': '', 'oif': 2, 'table': 254, 'gateway': '10.10.2.1', 'multipath': [], 'metrics': {}, 'deps': 0})
2022-06-18 20:32:37,887    DEBUG pyroute2.ndb.281473158053360.sources.localhost: source api run route ('add',) {'target': 'localhost', 'family': <AddressFamily.AF_INET: 2>, 'dst': '', 'oif': 2, 'table': 254, 'gateway': '10.10.2.1', 'multipath': [], 'metrics': {}, 'deps': 0}
2022-06-18 20:32:37,891    DEBUG pyroute2.ndb.281473158053360.rtnl_object: error: (17, 'File exists')
2022-06-18 20:32:37,892    DEBUG pyroute2.ndb.281473158053360.rtnl_object: ignore error 17 for {'multipath': [], 'metrics': {}, 'deps': 0, 'dst': '', 'gateway': '10.10.2.1', 'family': <AddressFamily.AF_INET: 2>, 'oif': 2, 'table': 254, 'target': 'localhost'}
2022-06-18 20:32:37,892    DEBUG pyroute2.ndb.281473158053360.rtnl_object: run fallback <function fallback_add at 0xffff930ddee0> ({'target': 'localhost', 'family': <AddressFamily.AF_INET: 2>, 'dst': '', 'oif': 2, 'table': 254, 'gateway': '10.10.2.1', 'multipath': [], 'metrics': {}, 'deps': 0})
2022-06-18 20:32:37,892    DEBUG pyroute2.ndb.281473158053360.rtnl_object: ignore incomplete idx_req in the fallback
2022-06-18 20:32:37,893    DEBUG pyroute2.ndb.281473158053360.rtnl_object: stats: apply add {objid 281473098346800, wtime 0.0, mqsize 0, nqsize 0}
2022-06-18 20:32:37,895    DEBUG pyroute2.ndb.281473158053360.rtnl_object: load_sql load: None
2022-06-18 20:32:37,895    DEBUG pyroute2.ndb.281473158053360.rtnl_object: load_sql names: ['target', 'tflags', 'family', 'dst_len', 'src_len', 'tos', 'table', 'proto', 'scope', 'type', 'flags', 'dst', 'src', 'iif', 'oif', 'gateway', 'priority', 'prefsrc', 'protoinfo', 'flow', 'table', 'mark', 'via', 'newdst', 'pref', 'encap_type', 'route_id', 'gc_mark', 'deps']
2022-06-18 20:32:37,896    DEBUG pyroute2.ndb.281473158053360.rtnl_object: check: [(1655584357.8683937, 'invalid')]
2022-06-18 20:32:37,896    DEBUG pyroute2.ndb.281473158053360.rtnl_object: check state: False
2022-06-18 20:32:37,896    DEBUG pyroute2.ndb.281473158053360.rtnl_object: check failed
2022-06-18 20:32:37,896    DEBUG pyroute2.ndb.281473158053360.rtnl_object: API call add ({'target': 'localhost', 'family': <AddressFamily.AF_INET: 2>, 'dst': '', 'oif': 2, 'table': 254, 'gateway': '10.10.2.1', 'multipath': [], 'metrics': {}, 'deps': 0})
2022-06-18 20:32:37,897    DEBUG pyroute2.ndb.281473158053360.sources.localhost: source api run route ('add',) {'target': 'localhost', 'family': <AddressFamily.AF_INET: 2>, 'dst': '', 'oif': 2, 'table': 254, 'gateway': '10.10.2.1', 'multipath': [], 'metrics': {}, 'deps': 0}
2022-06-18 20:32:37,901    DEBUG pyroute2.ndb.281473158053360.rtnl_object: error: (17, 'File exists')
2022-06-18 20:32:37,901    DEBUG pyroute2.ndb.281473158053360.rtnl_object: ignore error 17 for {'multipath': [], 'metrics': {}, 'deps': 0, 'dst': '', 'gateway': '10.10.2.1', 'family': <AddressFamily.AF_INET: 2>, 'oif': 2, 'table': 254, 'target': 'localhost'}
2022-06-18 20:32:37,901    DEBUG pyroute2.ndb.281473158053360.rtnl_object: run fallback <function fallback_add at 0xffff930ddee0> ({'target': 'localhost', 'family': <AddressFamily.AF_INET: 2>, 'dst': '', 'oif': 2, 'table': 254, 'gateway': '10.10.2.1', 'multipath': [], 'metrics': {}, 'deps': 0})
2022-06-18 20:32:37,902    DEBUG pyroute2.ndb.281473158053360.rtnl_object: ignore incomplete idx_req in the fallback
2022-06-18 20:32:37,902    DEBUG pyroute2.ndb.281473158053360.rtnl_object: stats: apply add {objid 281473098346800, wtime 0.1, mqsize 0, nqsize 0}
2022-06-18 20:32:37,904    DEBUG pyroute2.ndb.281473158053360.rtnl_object: load_sql load: None
2022-06-18 20:32:37,905    DEBUG pyroute2.ndb.281473158053360.rtnl_object: load_sql names: ['target', 'tflags', 'family', 'dst_len', 'src_len', 'tos', 'table', 'proto', 'scope', 'type', 'flags', 'dst', 'src', 'iif', 'oif', 'gateway', 'priority', 'prefsrc', 'protoinfo', 'flow', 'table', 'mark', 'via', 'newdst', 'pref', 'encap_type', 'route_id', 'gc_mark', 'deps']
2022-06-18 20:32:37,905    DEBUG pyroute2.ndb.281473158053360.rtnl_object: check: [(1655584357.8683937, 'invalid')]
2022-06-18 20:32:37,905    DEBUG pyroute2.ndb.281473158053360.rtnl_object: check state: False
2022-06-18 20:32:37,906    DEBUG pyroute2.ndb.281473158053360.rtnl_object: check failed
2022-06-18 20:32:38,007    DEBUG pyroute2.ndb.281473158053360.rtnl_object: API call add ({'target': 'localhost', 'family': <AddressFamily.AF_INET: 2>, 'dst': '', 'oif': 2, 'table': 254, 'gateway': '10.10.2.1', 'multipath': [], 'metrics': {}, 'deps': 0})
2022-06-18 20:32:38,007    DEBUG pyroute2.ndb.281473158053360.sources.localhost: source api run route ('add',) {'target': 'localhost', 'family': <AddressFamily.AF_INET: 2>, 'dst': '', 'oif': 2, 'table': 254, 'gateway': '10.10.2.1', 'multipath': [], 'metrics': {}, 'deps': 0}
2022-06-18 20:32:38,011    DEBUG pyroute2.ndb.281473158053360.rtnl_object: error: (17, 'File exists')
2022-06-18 20:32:38,011    DEBUG pyroute2.ndb.281473158053360.rtnl_object: ignore error 17 for {'multipath': [], 'metrics': {}, 'deps': 0, 'dst': '', 'gateway': '10.10.2.1', 'family': <AddressFamily.AF_INET: 2>, 'oif': 2, 'table': 254, 'target': 'localhost'}
2022-06-18 20:32:38,012    DEBUG pyroute2.ndb.281473158053360.rtnl_object: run fallback <function fallback_add at 0xffff930ddee0> ({'target': 'localhost', 'family': <AddressFamily.AF_INET: 2>, 'dst': '', 'oif': 2, 'table': 254, 'gateway': '10.10.2.1', 'multipath': [], 'metrics': {}, 'deps': 0})
2022-06-18 20:32:38,012    DEBUG pyroute2.ndb.281473158053360.rtnl_object: ignore incomplete idx_req in the fallback
2022-06-18 20:32:38,012    DEBUG pyroute2.ndb.281473158053360.rtnl_object: stats: apply add {objid 281473098346800, wtime 0.2, mqsize 0, nqsize 0}
2022-06-18 20:32:38,016    DEBUG pyroute2.ndb.281473158053360.rtnl_object: load_sql load: None
2022-06-18 20:32:38,017    DEBUG pyroute2.ndb.281473158053360.rtnl_object: load_sql names: ['target', 'tflags', 'family', 'dst_len', 'src_len', 'tos', 'table', 'proto', 'scope', 'type', 'flags', 'dst', 'src', 'iif', 'oif', 'gateway', 'priority', 'prefsrc', 'protoinfo', 'flow', 'table', 'mark', 'via', 'newdst', 'pref', 'encap_type', 'route_id', 'gc_mark', 'deps']
2022-06-18 20:32:38,017    DEBUG pyroute2.ndb.281473158053360.rtnl_object: check: [(1655584357.8683937, 'invalid')]
2022-06-18 20:32:38,017    DEBUG pyroute2.ndb.281473158053360.rtnl_object: check state: False
2022-06-18 20:32:38,017    DEBUG pyroute2.ndb.281473158053360.rtnl_object: check failed
2022-06-18 20:32:38,218    DEBUG pyroute2.ndb.281473158053360.rtnl_object: API call add ({'target': 'localhost', 'family': <AddressFamily.AF_INET: 2>, 'dst': '', 'oif': 2, 'table': 254, 'gateway': '10.10.2.1', 'multipath': [], 'metrics': {}, 'deps': 0})
2022-06-18 20:32:38,219    DEBUG pyroute2.ndb.281473158053360.sources.localhost: source api run route ('add',) {'target': 'localhost', 'family': <AddressFamily.AF_INET: 2>, 'dst': '', 'oif': 2, 'table': 254, 'gateway': '10.10.2.1', 'multipath': [], 'metrics': {}, 'deps': 0}
2022-06-18 20:32:38,222    DEBUG pyroute2.ndb.281473158053360.rtnl_object: error: (17, 'File exists')
2022-06-18 20:32:38,223    DEBUG pyroute2.ndb.281473158053360.rtnl_object: ignore error 17 for {'multipath': [], 'metrics': {}, 'deps': 0, 'dst': '', 'gateway': '10.10.2.1', 'family': <AddressFamily.AF_INET: 2>, 'oif': 2, 'table': 254, 'target': 'localhost'}
2022-06-18 20:32:38,223    DEBUG pyroute2.ndb.281473158053360.rtnl_object: run fallback <function fallback_add at 0xffff930ddee0> ({'target': 'localhost', 'family': <AddressFamily.AF_INET: 2>, 'dst': '', 'oif': 2, 'table': 254, 'gateway': '10.10.2.1', 'multipath': [], 'metrics': {}, 'deps': 0})
2022-06-18 20:32:38,224    DEBUG pyroute2.ndb.281473158053360.rtnl_object: ignore incomplete idx_req in the fallback
2022-06-18 20:32:38,224    DEBUG pyroute2.ndb.281473158053360.rtnl_object: stats: apply add {objid 281473098346800, wtime 0.30000000000000004, mqsize 0, nqsize 0}
2022-06-18 20:32:38,226    DEBUG pyroute2.ndb.281473158053360.rtnl_object: load_sql load: None
2022-06-18 20:32:38,227    DEBUG pyroute2.ndb.281473158053360.rtnl_object: load_sql names: ['target', 'tflags', 'family', 'dst_len', 'src_len', 'tos', 'table', 'proto', 'scope', 'type', 'flags', 'dst', 'src', 'iif', 'oif', 'gateway', 'priority', 'prefsrc', 'protoinfo', 'flow', 'table', 'mark', 'via', 'newdst', 'pref', 'encap_type', 'route_id', 'gc_mark', 'deps']
2022-06-18 20:32:38,227    DEBUG pyroute2.ndb.281473158053360.rtnl_object: check: [(1655584357.8683937, 'invalid')]
2022-06-18 20:32:38,228    DEBUG pyroute2.ndb.281473158053360.rtnl_object: check state: False
2022-06-18 20:32:38,228    DEBUG pyroute2.ndb.281473158053360.rtnl_object: check failed
2022-06-18 20:32:38,529    DEBUG pyroute2.ndb.281473158053360.rtnl_object: API call add ({'target': 'localhost', 'family': <AddressFamily.AF_INET: 2>, 'dst': '', 'oif': 2, 'table': 254, 'gateway': '10.10.2.1', 'multipath': [], 'metrics': {}, 'deps': 0})
2022-06-18 20:32:38,529    DEBUG pyroute2.ndb.281473158053360.sources.localhost: source api run route ('add',) {'target': 'localhost', 'family': <AddressFamily.AF_INET: 2>, 'dst': '', 'oif': 2, 'table': 254, 'gateway': '10.10.2.1', 'multipath': [], 'metrics': {}, 'deps': 0}
2022-06-18 20:32:38,533    DEBUG pyroute2.ndb.281473158053360.rtnl_object: error: (17, 'File exists')
2022-06-18 20:32:38,533    DEBUG pyroute2.ndb.281473158053360.rtnl_object: ignore error 17 for {'multipath': [], 'metrics': {}, 'deps': 0, 'dst': '', 'gateway': '10.10.2.1', 'family': <AddressFamily.AF_INET: 2>, 'oif': 2, 'table': 254, 'target': 'localhost'}
2022-06-18 20:32:38,534    DEBUG pyroute2.ndb.281473158053360.rtnl_object: run fallback <function fallback_add at 0xffff930ddee0> ({'target': 'localhost', 'family': <AddressFamily.AF_INET: 2>, 'dst': '', 'oif': 2, 'table': 254, 'gateway': '10.10.2.1', 'multipath': [], 'metrics': {}, 'deps': 0})
2022-06-18 20:32:38,534    DEBUG pyroute2.ndb.281473158053360.rtnl_object: ignore incomplete idx_req in the fallback
2022-06-18 20:32:38,534    DEBUG pyroute2.ndb.281473158053360.rtnl_object: stats: apply add {objid 281473098346800, wtime 0.4, mqsize 0, nqsize 0}
2022-06-18 20:32:38,537    DEBUG pyroute2.ndb.281473158053360.rtnl_object: load_sql load: None
2022-06-18 20:32:38,537    DEBUG pyroute2.ndb.281473158053360.rtnl_object: load_sql names: ['target', 'tflags', 'family', 'dst_len', 'src_len', 'tos', 'table', 'proto', 'scope', 'type', 'flags', 'dst', 'src', 'iif', 'oif', 'gateway', 'priority', 'prefsrc', 'protoinfo', 'flow', 'table', 'mark', 'via', 'newdst', 'pref', 'encap_type', 'route_id', 'gc_mark', 'deps']
2022-06-18 20:32:38,537    DEBUG pyroute2.ndb.281473158053360.rtnl_object: check: [(1655584357.8683937, 'invalid')]
2022-06-18 20:32:38,538    DEBUG pyroute2.ndb.281473158053360.rtnl_object: check state: False
2022-06-18 20:32:38,538    DEBUG pyroute2.ndb.281473158053360.rtnl_object: check failed
2022-06-18 20:32:38,939    DEBUG pyroute2.ndb.281473158053360.rtnl_object: API call add ({'target': 'localhost', 'family': <AddressFamily.AF_INET: 2>, 'dst': '', 'oif': 2, 'table': 254, 'gateway': '10.10.2.1', 'multipath': [], 'metrics': {}, 'deps': 0})
2022-06-18 20:32:38,940    DEBUG pyroute2.ndb.281473158053360.sources.localhost: source api run route ('add',) {'target': 'localhost', 'family': <AddressFamily.AF_INET: 2>, 'dst': '', 'oif': 2, 'table': 254, 'gateway': '10.10.2.1', 'multipath': [], 'metrics': {}, 'deps': 0}
2022-06-18 20:32:38,943    DEBUG pyroute2.ndb.281473158053360.rtnl_object: error: (17, 'File exists')
2022-06-18 20:32:38,944    DEBUG pyroute2.ndb.281473158053360.rtnl_object: ignore error 17 for {'multipath': [], 'metrics': {}, 'deps': 0, 'dst': '', 'gateway': '10.10.2.1', 'family': <AddressFamily.AF_INET: 2>, 'oif': 2, 'table': 254, 'target': 'localhost'}
2022-06-18 20:32:38,944    DEBUG pyroute2.ndb.281473158053360.rtnl_object: run fallback <function fallback_add at 0xffff930ddee0> ({'target': 'localhost', 'family': <AddressFamily.AF_INET: 2>, 'dst': '', 'oif': 2, 'table': 254, 'gateway': '10.10.2.1', 'multipath': [], 'metrics': {}, 'deps': 0})
2022-06-18 20:32:38,944    DEBUG pyroute2.ndb.281473158053360.rtnl_object: ignore incomplete idx_req in the fallback
2022-06-18 20:32:38,945    DEBUG pyroute2.ndb.281473158053360.rtnl_object: stats: apply add {objid 281473098346800, wtime 0.5, mqsize 0, nqsize 0}
2022-06-18 20:32:38,947    DEBUG pyroute2.ndb.281473158053360.rtnl_object: load_sql load: None
2022-06-18 20:32:38,948    DEBUG pyroute2.ndb.281473158053360.rtnl_object: load_sql names: ['target', 'tflags', 'family', 'dst_len', 'src_len', 'tos', 'table', 'proto', 'scope', 'type', 'flags', 'dst', 'src', 'iif', 'oif', 'gateway', 'priority', 'prefsrc', 'protoinfo', 'flow', 'table', 'mark', 'via', 'newdst', 'pref', 'encap_type', 'route_id', 'gc_mark', 'deps']
2022-06-18 20:32:38,948    DEBUG pyroute2.ndb.281473158053360.rtnl_object: check: [(1655584357.8683937, 'invalid')]
2022-06-18 20:32:38,948    DEBUG pyroute2.ndb.281473158053360.rtnl_object: check state: False
2022-06-18 20:32:38,948    DEBUG pyroute2.ndb.281473158053360.rtnl_object: check failed
2022-06-18 20:32:39,449    DEBUG pyroute2.ndb.281473158053360.rtnl_object: API call add ({'target': 'localhost', 'family': <AddressFamily.AF_INET: 2>, 'dst': '', 'oif': 2, 'table': 254, 'gateway': '10.10.2.1', 'multipath': [], 'metrics': {}, 'deps': 0})
2022-06-18 20:32:39,450    DEBUG pyroute2.ndb.281473158053360.sources.localhost: source api run route ('add',) {'target': 'localhost', 'family': <AddressFamily.AF_INET: 2>, 'dst': '', 'oif': 2, 'table': 254, 'gateway': '10.10.2.1', 'multipath': [], 'metrics': {}, 'deps': 0}
2022-06-18 20:32:39,454    DEBUG pyroute2.ndb.281473158053360.rtnl_object: error: (17, 'File exists')
2022-06-18 20:32:39,454    DEBUG pyroute2.ndb.281473158053360.rtnl_object: ignore error 17 for {'multipath': [], 'metrics': {}, 'deps': 0, 'dst': '', 'gateway': '10.10.2.1', 'family': <AddressFamily.AF_INET: 2>, 'oif': 2, 'table': 254, 'target': 'localhost'}
2022-06-18 20:32:39,455    DEBUG pyroute2.ndb.281473158053360.rtnl_object: run fallback <function fallback_add at 0xffff930ddee0> ({'target': 'localhost', 'family': <AddressFamily.AF_INET: 2>, 'dst': '', 'oif': 2, 'table': 254, 'gateway': '10.10.2.1', 'multipath': [], 'metrics': {}, 'deps': 0})
2022-06-18 20:32:39,455    DEBUG pyroute2.ndb.281473158053360.rtnl_object: ignore incomplete idx_req in the fallback
2022-06-18 20:32:39,455    DEBUG pyroute2.ndb.281473158053360.rtnl_object: stats: apply add {objid 281473098346800, wtime 0.6000000000000001, mqsize 0, nqsize 0}
2022-06-18 20:32:39,457    DEBUG pyroute2.ndb.281473158053360.rtnl_object: load_sql load: None
2022-06-18 20:32:39,458    DEBUG pyroute2.ndb.281473158053360.rtnl_object: load_sql names: ['target', 'tflags', 'family', 'dst_len', 'src_len', 'tos', 'table', 'proto', 'scope', 'type', 'flags', 'dst', 'src', 'iif', 'oif', 'gateway', 'priority', 'prefsrc', 'protoinfo', 'flow', 'table', 'mark', 'via', 'newdst', 'pref', 'encap_type', 'route_id', 'gc_mark', 'deps']
2022-06-18 20:32:39,458    DEBUG pyroute2.ndb.281473158053360.rtnl_object: check: [(1655584357.8683937, 'invalid')]
2022-06-18 20:32:39,459    DEBUG pyroute2.ndb.281473158053360.rtnl_object: check state: False
2022-06-18 20:32:39,459    DEBUG pyroute2.ndb.281473158053360.rtnl_object: check failed
2022-06-18 20:32:40,060    DEBUG pyroute2.ndb.281473158053360.rtnl_object: API call add ({'target': 'localhost', 'family': <AddressFamily.AF_INET: 2>, 'dst': '', 'oif': 2, 'table': 254, 'gateway': '10.10.2.1', 'multipath': [], 'metrics': {}, 'deps': 0})
2022-06-18 20:32:40,061    DEBUG pyroute2.ndb.281473158053360.sources.localhost: source api run route ('add',) {'target': 'localhost', 'family': <AddressFamily.AF_INET: 2>, 'dst': '', 'oif': 2, 'table': 254, 'gateway': '10.10.2.1', 'multipath': [], 'metrics': {}, 'deps': 0}
2022-06-18 20:32:40,064    DEBUG pyroute2.ndb.281473158053360.rtnl_object: error: (17, 'File exists')
2022-06-18 20:32:40,065    DEBUG pyroute2.ndb.281473158053360.rtnl_object: ignore error 17 for {'multipath': [], 'metrics': {}, 'deps': 0, 'dst': '', 'gateway': '10.10.2.1', 'family': <AddressFamily.AF_INET: 2>, 'oif': 2, 'table': 254, 'target': 'localhost'}
2022-06-18 20:32:40,065    DEBUG pyroute2.ndb.281473158053360.rtnl_object: run fallback <function fallback_add at 0xffff930ddee0> ({'target': 'localhost', 'family': <AddressFamily.AF_INET: 2>, 'dst': '', 'oif': 2, 'table': 254, 'gateway': '10.10.2.1', 'multipath': [], 'metrics': {}, 'deps': 0})
2022-06-18 20:32:40,065    DEBUG pyroute2.ndb.281473158053360.rtnl_object: ignore incomplete idx_req in the fallback
2022-06-18 20:32:40,066    DEBUG pyroute2.ndb.281473158053360.rtnl_object: stats: apply add {objid 281473098346800, wtime 0.7000000000000001, mqsize 0, nqsize 0}
2022-06-18 20:32:40,068    DEBUG pyroute2.ndb.281473158053360.rtnl_object: load_sql load: None
2022-06-18 20:32:40,068    DEBUG pyroute2.ndb.281473158053360.rtnl_object: load_sql names: ['target', 'tflags', 'family', 'dst_len', 'src_len', 'tos', 'table', 'proto', 'scope', 'type', 'flags', 'dst', 'src', 'iif', 'oif', 'gateway', 'priority', 'prefsrc', 'protoinfo', 'flow', 'table', 'mark', 'via', 'newdst', 'pref', 'encap_type', 'route_id', 'gc_mark', 'deps']
2022-06-18 20:32:40,069    DEBUG pyroute2.ndb.281473158053360.rtnl_object: check: [(1655584357.8683937, 'invalid')]
2022-06-18 20:32:40,069    DEBUG pyroute2.ndb.281473158053360.rtnl_object: check state: False
2022-06-18 20:32:40,069    DEBUG pyroute2.ndb.281473158053360.rtnl_object: check failed
2022-06-18 20:32:40,770    DEBUG pyroute2.ndb.281473158053360.rtnl_object: API call add ({'target': 'localhost', 'family': <AddressFamily.AF_INET: 2>, 'dst': '', 'oif': 2, 'table': 254, 'gateway': '10.10.2.1', 'multipath': [], 'metrics': {}, 'deps': 0})
2022-06-18 20:32:40,771    DEBUG pyroute2.ndb.281473158053360.sources.localhost: source api run route ('add',) {'target': 'localhost', 'family': <AddressFamily.AF_INET: 2>, 'dst': '', 'oif': 2, 'table': 254, 'gateway': '10.10.2.1', 'multipath': [], 'metrics': {}, 'deps': 0}
2022-06-18 20:32:40,775    DEBUG pyroute2.ndb.281473158053360.rtnl_object: error: (17, 'File exists')
2022-06-18 20:32:40,775    DEBUG pyroute2.ndb.281473158053360.rtnl_object: ignore error 17 for {'multipath': [], 'metrics': {}, 'deps': 0, 'dst': '', 'gateway': '10.10.2.1', 'family': <AddressFamily.AF_INET: 2>, 'oif': 2, 'table': 254, 'target': 'localhost'}
2022-06-18 20:32:40,776    DEBUG pyroute2.ndb.281473158053360.rtnl_object: run fallback <function fallback_add at 0xffff930ddee0> ({'target': 'localhost', 'family': <AddressFamily.AF_INET: 2>, 'dst': '', 'oif': 2, 'table': 254, 'gateway': '10.10.2.1', 'multipath': [], 'metrics': {}, 'deps': 0})
2022-06-18 20:32:40,776    DEBUG pyroute2.ndb.281473158053360.rtnl_object: ignore incomplete idx_req in the fallback
2022-06-18 20:32:40,776    DEBUG pyroute2.ndb.281473158053360.rtnl_object: stats: apply add {objid 281473098346800, wtime 0.8, mqsize 0, nqsize 0}
2022-06-18 20:32:40,778    DEBUG pyroute2.ndb.281473158053360.rtnl_object: load_sql load: None
2022-06-18 20:32:40,778    DEBUG pyroute2.ndb.281473158053360.rtnl_object: load_sql names: ['target', 'tflags', 'family', 'dst_len', 'src_len', 'tos', 'table', 'proto', 'scope', 'type', 'flags', 'dst', 'src', 'iif', 'oif', 'gateway', 'priority', 'prefsrc', 'protoinfo', 'flow', 'table', 'mark', 'via', 'newdst', 'pref', 'encap_type', 'route_id', 'gc_mark', 'deps']
2022-06-18 20:32:40,779    DEBUG pyroute2.ndb.281473158053360.rtnl_object: check: [(1655584357.8683937, 'invalid')]
2022-06-18 20:32:40,779    DEBUG pyroute2.ndb.281473158053360.rtnl_object: check state: False
2022-06-18 20:32:40,780    DEBUG pyroute2.ndb.281473158053360.rtnl_object: check failed
2022-06-18 20:32:41,580    DEBUG pyroute2.ndb.281473158053360.rtnl_object: API call add ({'target': 'localhost', 'family': <AddressFamily.AF_INET: 2>, 'dst': '', 'oif': 2, 'table': 254, 'gateway': '10.10.2.1', 'multipath': [], 'metrics': {}, 'deps': 0})
2022-06-18 20:32:41,581    DEBUG pyroute2.ndb.281473158053360.sources.localhost: source api run route ('add',) {'target': 'localhost', 'family': <AddressFamily.AF_INET: 2>, 'dst': '', 'oif': 2, 'table': 254, 'gateway': '10.10.2.1', 'multipath': [], 'metrics': {}, 'deps': 0}
2022-06-18 20:32:41,584    DEBUG pyroute2.ndb.281473158053360.rtnl_object: error: (17, 'File exists')
2022-06-18 20:32:41,585    DEBUG pyroute2.ndb.281473158053360.rtnl_object: ignore error 17 for {'multipath': [], 'metrics': {}, 'deps': 0, 'dst': '', 'gateway': '10.10.2.1', 'family': <AddressFamily.AF_INET: 2>, 'oif': 2, 'table': 254, 'target': 'localhost'}
2022-06-18 20:32:41,585    DEBUG pyroute2.ndb.281473158053360.rtnl_object: run fallback <function fallback_add at 0xffff930ddee0> ({'target': 'localhost', 'family': <AddressFamily.AF_INET: 2>, 'dst': '', 'oif': 2, 'table': 254, 'gateway': '10.10.2.1', 'multipath': [], 'metrics': {}, 'deps': 0})
2022-06-18 20:32:41,585    DEBUG pyroute2.ndb.281473158053360.rtnl_object: ignore incomplete idx_req in the fallback
2022-06-18 20:32:41,586    DEBUG pyroute2.ndb.281473158053360.rtnl_object: stats: apply add {objid 281473098346800, wtime 0.9, mqsize 0, nqsize 0}
2022-06-18 20:32:41,588    DEBUG pyroute2.ndb.281473158053360.rtnl_object: load_sql load: None
2022-06-18 20:32:41,589    DEBUG pyroute2.ndb.281473158053360.rtnl_object: load_sql names: ['target', 'tflags', 'family', 'dst_len', 'src_len', 'tos', 'table', 'proto', 'scope', 'type', 'flags', 'dst', 'src', 'iif', 'oif', 'gateway', 'priority', 'prefsrc', 'protoinfo', 'flow', 'table', 'mark', 'via', 'newdst', 'pref', 'encap_type', 'route_id', 'gc_mark', 'deps']
2022-06-18 20:32:41,589    DEBUG pyroute2.ndb.281473158053360.rtnl_object: check: [(1655584357.8683937, 'invalid')]
2022-06-18 20:32:41,589    DEBUG pyroute2.ndb.281473158053360.rtnl_object: check state: False
2022-06-18 20:32:41,589    DEBUG pyroute2.ndb.281473158053360.rtnl_object: check failed
2022-06-18 20:32:42,490    DEBUG pyroute2.ndb.281473158053360.rtnl_object: stats: 281473098346800 apply add fail
2022-06-18 20:32:42,491    DEBUG pyroute2.ndb.281473158053360.rtnl_object: criteria <function RTNL_Object.apply.<locals>.<lambda> at 0xffff908c5310>
2022-06-18 20:32:42,491    DEBUG pyroute2.ndb.281473158053360.rtnl_object: method <bound method RTNL_Object.check of {'multipath': [], 'metrics': {}, 'deps': 0, 'dst': '', 'gateway': '10.10.2.1', 'family': <AddressFamily.AF_INET: 2>, 'oif': 2, 'table': 254, 'target': 'localhost'}>, [], {}
2022-06-18 20:32:42,493    DEBUG pyroute2.ndb.281473158053360.rtnl_object: load_sql load: None
2022-06-18 20:32:42,494    DEBUG pyroute2.ndb.281473158053360.rtnl_object: load_sql names: ['target', 'tflags', 'family', 'dst_len', 'src_len', 'tos', 'table', 'proto', 'scope', 'type', 'flags', 'dst', 'src', 'iif', 'oif', 'gateway', 'priority', 'prefsrc', 'protoinfo', 'flow', 'table', 'mark', 'via', 'newdst', 'pref', 'encap_type', 'route_id', 'gc_mark', 'deps']
2022-06-18 20:32:42,494    DEBUG pyroute2.ndb.281473158053360.rtnl_object: check: [(1655584357.8683937, 'invalid')]
2022-06-18 20:32:42,494    DEBUG pyroute2.ndb.281473158053360.rtnl_object: check state: False
2022-06-18 20:32:42,495    DEBUG pyroute2.ndb.281473158053360.rtnl_object: ret False
2022-06-18 20:32:42,495    DEBUG pyroute2.ndb.281473158053360.rtnl_object: resync the DB attempt 0
2022-06-18 20:32:42,503    DEBUG pyroute2.ndb.281473158053360.sources.localhost: source api run dump () {}
2022-06-18 20:32:42,776    DEBUG pyroute2.ndb.281473158053360.rtnl_object: load_sql load: None
2022-06-18 20:32:42,776    DEBUG pyroute2.ndb.281473158053360.rtnl_object: load_sql names: ['target', 'tflags', 'family', 'dst_len', 'src_len', 'tos', 'table', 'proto', 'scope', 'type', 'flags', 'dst', 'src', 'iif', 'oif', 'gateway', 'priority', 'prefsrc', 'protoinfo', 'flow', 'table', 'mark', 'via', 'newdst', 'pref', 'encap_type', 'route_id', 'gc_mark', 'deps']
2022-06-18 20:32:42,777    DEBUG pyroute2.ndb.281473158053360.rtnl_object: check: [(1655584357.8683937, 'invalid')]
2022-06-18 20:32:42,778    DEBUG pyroute2.ndb.281473158053360.rtnl_object: check state: False
2022-06-18 20:32:42,779    DEBUG pyroute2.ndb.281473158053360.rtnl_object: ret False
2022-06-18 20:32:42,779    DEBUG pyroute2.ndb.281473158053360.rtnl_object: resync the DB attempt 1
2022-06-18 20:32:42,789    DEBUG pyroute2.ndb.281473158053360.sources.localhost: source api run dump () {}
2022-06-18 20:32:43,040    DEBUG pyroute2.ndb.281473158053360.rtnl_object: load_sql load: None
2022-06-18 20:32:43,040    DEBUG pyroute2.ndb.281473158053360.rtnl_object: load_sql names: ['target', 'tflags', 'family', 'dst_len', 'src_len', 'tos', 'table', 'proto', 'scope', 'type', 'flags', 'dst', 'src', 'iif', 'oif', 'gateway', 'priority', 'prefsrc', 'protoinfo', 'flow', 'table', 'mark', 'via', 'newdst', 'pref', 'encap_type', 'route_id', 'gc_mark', 'deps']
2022-06-18 20:32:43,041    DEBUG pyroute2.ndb.281473158053360.rtnl_object: check: [(1655584357.8683937, 'invalid')]
2022-06-18 20:32:43,041    DEBUG pyroute2.ndb.281473158053360.rtnl_object: check state: False
2022-06-18 20:32:43,041    DEBUG pyroute2.ndb.281473158053360.rtnl_object: ret False
2022-06-18 20:32:43,042    DEBUG pyroute2.ndb.281473158053360.rtnl_object: resync the DB attempt 2
2022-06-18 20:32:43,050    DEBUG pyroute2.ndb.281473158053360.sources.localhost: source api run dump () {}
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/snap/iotr-cr-sysmgr/x130/lib/python3.8/site-packages/pr2modules/ndb/auth_manager.py", line 67, in guard
    return f(obj, *argv, **kwarg)
  File "/snap/iotr-cr-sysmgr/x130/lib/python3.8/site-packages/pr2modules/ndb/objects/__init__.py", line 694, in commit
    raise e_i
  File "/snap/iotr-cr-sysmgr/x130/lib/python3.8/site-packages/pr2modules/ndb/objects/__init__.py", line 683, in commit
    return self.apply(mode='commit')
  File "/snap/iotr-cr-sysmgr/x130/lib/python3.8/site-packages/pr2modules/ndb/auth_manager.py", line 67, in guard
    return f(obj, *argv, **kwarg)
  File "/snap/iotr-cr-sysmgr/x130/lib/python3.8/site-packages/pr2modules/ndb/objects/route.py", line 691, in apply
    return super(Route, self).apply(rollback, req_filter, mode)
  File "/snap/iotr-cr-sysmgr/x130/lib/python3.8/site-packages/pr2modules/ndb/auth_manager.py", line 67, in guard
    return f(obj, *argv, **kwarg)
  File "/snap/iotr-cr-sysmgr/x130/lib/python3.8/site-packages/pr2modules/ndb/objects/__init__.py", line 935, in apply
    raise Exception('could not apply the changes')
Exception: could not apply the changes

Cant test IPv6 right now, but I'll do it monday.

[svinota]

Great! Thanks a lot

[svinota]

I managed to get the logs for the ipv4 issue!

What happened here: the first add returns error: (17, 'File exists'), and it can mean only that there is already a default route, but NDB doesn't match it with the spec {"dst": "default", "gateway": "10.10.2.1", "oif": 2, "table": 254}

There may be two reasons for that:

1. one of the unique index fields, not described in spec, differs from the default expected by NDB
2. or one of the unique index fields described in spec is decoded wrong by pyroute2 (an endianness issue?)

The index:

pyroute2/pyroute2.ndb/pr2modules/ndb/objects/route.py

Lines 282 to 293 in fefad92

	.unique_index(
	'family',
	'dst_len',
	'tos',
	'scope',
	'RTA_DST',
	'RTA_OIF',
	'RTA_PRIORITY',
	'RTA_TABLE',
	'RTA_VIA',
	'RTA_NEWDST',
	)

family, dst_len, dst, oif, table are defined in the request directly or implicitly (dst_len is 0), so the fields left are tos, scope, priority, via, newdst. The latter two are obviously not the case, so there are three possible fields to check.

Would be super nice to get also ndb.routes.dump().format('csv') from the system with IPv4 issue, but I'm trying to reproduce using the data we have so far.

[carloslockward]

That's weird because that default route doesn't seem to exist.

Here is the table (converted to markdown):

'target'	'tflags'	'family'	'dst_len'	'src_len'	'tos'	'table'	'proto'	'scope'	'type'	'flags'	'dst'	'src'	'iif'	'oif'	'gateway'	'priority'	'prefsrc'	'protoinfo'	'flow'	'table'	'mark'	'via'	'newdst'	'nh_flags'	'nh_hops'	'nh_oif'	'nh_dst'	'nh_src'	'nh_iif'	'nh_oif'	'nh_gateway'	'nh_priority'	'nh_prefsrc'	'nh_protoinfo'	'nh_flow'	'nh_table'	'nh_mark'	'nh_via'	'nh_newdst'	'metrics'	'encap'
'localhost'	0	2	0	0	0	254	3	0	1	0	''			3	'1.2.3.5'	0				254		''	''
'localhost'	0	2	24	0	0	254	2	253	1	0	'1.2.3.0'			3		0	'1.2.3.31'			254		''	''
'localhost'	0	2	24	0	0	254	2	253	1	0	'10.1.100.0'			4		0	'10.1.100.1'			254		''	''
'localhost'	0	2	24	0	0	254	2	253	1	0	'10.10.2.0'			2		0	'10.10.2.126'			254		''	''
'localhost'	0	2	32	0	0	255	2	253	3	0	'1.2.3.0'			3		0	'1.2.3.31'			255		''	''
'localhost'	0	2	32	0	0	255	2	254	2	0	'1.2.3.31'			3		0	'1.2.3.31'			255		''	''
'localhost'	0	2	32	0	0	255	2	253	3	0	'1.2.3.255'			3		0	'1.2.3.31'			255		''	''
'localhost'	0	2	32	0	0	255	2	253	3	0	'10.1.100.0'			4		0	'10.1.100.1'			255		''	''
'localhost'	0	2	32	0	0	255	2	254	2	0	'10.1.100.1'			4		0	'10.1.100.1'			255		''	''
'localhost'	0	2	32	0	0	255	2	253	3	0	'10.1.100.255'			4		0	'10.1.100.1'			255		''	''
'localhost'	0	2	32	0	0	255	2	253	3	0	'10.10.2.0'			2		0	'10.10.2.126'			255		''	''
'localhost'	0	2	32	0	0	255	2	254	2	0	'10.10.2.126'			2		0	'10.10.2.126'			255		''	''
'localhost'	0	2	32	0	0	255	2	253	3	0	'10.10.2.255'			2		0	'10.10.2.126'			255		''	''
'localhost'	0	2	32	0	0	255	2	253	3	0	'127.0.0.0'			1		0	'127.0.0.1'			255		''	''
'localhost'	0	2	8	0	0	255	2	254	2	0	'127.0.0.0'			1		0	'127.0.0.1'			255		''	''
'localhost'	0	2	32	0	0	255	2	254	2	0	'127.0.0.1'			1		0	'127.0.0.1'			255		''	''
'localhost'	0	2	32	0	0	255	2	253	3	0	'127.255.255.255'			1		0	'127.0.0.1'			255		''	''
'localhost'	0	10	128	0	0	254	2	0	1	0	'::1'			1		256				254		''	''
'localhost'	0	10	128	0	0	254	2	0	1	0	'beef:feed:fade::ffee'			3		256				254		''	''
'localhost'	0	10	112	0	0	254	4	0	1	0	'beef:feed:fade::'			3		1				254		''	''
'localhost'	0	10	64	0	0	254	2	0	1	0	'fd04:7c3e:be2f:202a::'			6		256				254		''	''
'localhost'	0	10	64	0	0	254	2	0	1	0	'fd04:7c3e:be2f:203d::'			6		256				254		''	''
'localhost'	0	10	64	0	0	254	2	0	1	0	'fe80::'			3		256				254		''	''
'localhost'	0	10	64	0	0	254	2	0	1	0	'fe80::'			6		256				254		''	''
'localhost'	0	10	64	0	0	254	2	0	1	0	'fe80::'			2		256				254		''	''
'localhost'	0	10	64	0	0	254	2	0	1	0	'fe80::'			4		256				254		''	''
'localhost'	0	10	0	0	0	254	3	0	1	0	''			6	'fd04:7c3e:be2f:202a:213:50ff:fe30:5b24'	1				254		''	''
'localhost'	0	10	128	0	0	255	2	0	2	0	'::1'			1		0				255		''	''
'localhost'	0	10	128	0	0	255	2	0	2	0	'beef:feed:fade::ffee'			3		0				255		''	''
'localhost'	0	10	128	0	0	255	2	0	4	0	'fd04:7c3e:be2f:202a::'			6		0				255		''	''
'localhost'	0	10	128	0	0	255	2	0	2	0	'fd04:7c3e:be2f:202a:213:50ff:fe30:ca28'			6		0				255		''	''
'localhost'	0	10	128	0	0	255	2	0	4	0	'fd04:7c3e:be2f:203d::'			6		0				255		''	''
'localhost'	0	10	128	0	0	255	2	0	2	0	'fd04:7c3e:be2f:203d:213:50ff:fe30:ca28'			6		0				255		''	''
'localhost'	0	10	128	0	0	255	2	0	4	0	'fe80::'			3		0				255		''	''
'localhost'	0	10	128	0	0	255	2	0	4	0	'fe80::'			6		0				255		''	''
'localhost'	0	10	128	0	0	255	2	0	4	0	'fe80::'			2		0				255		''	''
'localhost'	0	10	128	0	0	255	2	0	4	0	'fe80::'			4		0				255		''	''
'localhost'	0	10	128	0	0	255	2	0	2	0	'fe80::213:50ff:fe30:ca28'			6		0				255		''	''
'localhost'	0	10	128	0	0	255	2	0	2	0	'fe80::2e28:70c8:c4d8:f49c'			6		0				255		''	''
'localhost'	0	10	128	0	0	255	2	0	2	0	'fe80::7274:14ff:fe92:b7ac'			4		0				255		''	''
'localhost'	0	10	128	0	0	255	2	0	2	0	'fe80::72b3:d5ff:fe7c:9b73'			2		0				255		''	''
'localhost'	0	10	128	0	0	255	2	0	2	0	'fe80::72b3:d5ff:fe7c:9b74'			3		0				255		''	''
'localhost'	0	10	8	0	0	255	2	0	5	0	'ff00::'			3		256				255		''	''
'localhost'	0	10	8	0	0	255	2	0	5	0	'ff00::'			6		256				255		''	''
'localhost'	0	10	8	0	0	255	2	0	5	0	'ff00::'			2		256				255		''	''
'localhost'	0	10	8	0	0	255	2	0	5	0	'ff00::'			4		256				255		''	''

Or use this paste bin

Also here is the output of ip route for v4 and v6.

IPv4:

default via 1.2.3.5 dev enp1s0 
1.2.3.0/24 dev enp1s0 proto kernel scope link src 1.2.3.31 
10.1.100.0/24 dev wlan0 proto kernel scope link src 10.1.100.1 
10.10.2.0/24 dev eth0 proto kernel scope link src 10.10.2.126 

IPv6

::1 dev lo proto kernel metric 256 pref medium
beef:feed:fade::ffee dev enp1s0 proto kernel metric 256 pref medium
beef:feed:fade::/112 dev enp1s0 proto static metric 1 pref medium
fd04:7c3e:be2f:202a::/64 dev tun6 proto kernel metric 256 pref medium
fd04:7c3e:be2f:203d::/64 dev tun6 proto kernel metric 256 pref medium
fe80::/64 dev enp1s0 proto kernel metric 256 pref medium
fe80::/64 dev tun6 proto kernel metric 256 pref medium
fe80::/64 dev eth0 proto kernel metric 256 pref medium
fe80::/64 dev wlan0 proto kernel metric 256 pref medium
default via fd04:7c3e:be2f:202a:213:50ff:fe30:5b24 dev tun6 metric 1 pref medium

[svinota]

Still can not reproduce.

I'm preparing a workaround that should help, but the root cause is still unclear.

[carloslockward]

I'm having a hard time reproducing it myself... I'm going to have another round of testing to see if I can determine how to trigger the issue.