Unable to delete IPv6 interface routes using NDB

[internetionals]

I've encountered a bug while trying to delete direct to interface (route without a next hop) IPv6 routes.

Reproducer:

#!/usr/bin/env python3

from pyroute2 import NDB

ndb = NDB(log='on')

# This creates a route
ndb.routes.create(dst='fd12::3', dst_len=128, oif=ndb.interfaces['lo']['index']).commit()

# Which we can see here
route = ndb.routes['fd12::3/128']
print("Created route: {}".format(route))

# But then this fails
print("Trying to remove this route again")
route.remove().commit()

Output:

Created route: {'multipath': [], 'metrics': {}, 'deps': 0, 'target': 'localhost', 'dst': 'fd12::3', 'dst_len': 128, 'family': 10, 'table': 254, 'tflags': 0, 'tos': 0, 'scope': 0, 'oif': 1, 'priority': 1024, 'via': '', 'newdst': '', 'src_len': 0, 'proto': 4, 'type': 1, 'flags': 0, 'src': None, 'iif': None, 'gateway': None, 'prefsrc': None, 'protoinfo': None, 'flow': None, 'mark': None, 'pref': 0, 'encap_type': None, 'route_id': None, 'gc_mark': None}
Trying to remove this route again
Traceback (most recent call last):
  File "/co/min.py", line 16, in <module>
    route.remove().commit()
  File "/usr/local/lib/python3.10/site-packages/pr2modules/ndb/auth_manager.py", line 67, in guard
    return f(obj, *argv, **kwarg)
  File "/usr/local/lib/python3.10/site-packages/pr2modules/ndb/objects/__init__.py", line 706, in commit
    self.last_save = self.snapshot()
  File "/usr/local/lib/python3.10/site-packages/pr2modules/ndb/auth_manager.py", line 67, in guard
    return f(obj, *argv, **kwarg)
  File "/usr/local/lib/python3.10/site-packages/pr2modules/ndb/objects/__init__.py", line 557, in snapshot
    snp = type(self)(
  File "/usr/local/lib/python3.10/site-packages/pr2modules/ndb/objects/route.py", line 570, in __init__
    super(Route, self).__init__(*argv, **kwarg)
  File "/usr/local/lib/python3.10/site-packages/pr2modules/ndb/objects/__init__.py", line 323, in __init__
    raise KeyError('object does not exists')
KeyError: 'object does not exists'

Output of the same program with the logging set to 'debug'

2022-06-23 14:15:59,414    DEBUG pyroute2.ndb.140665294947040.sources.localhost: init
2022-06-23 14:15:59,414    DEBUG pyroute2.ndb.140665294947040.sources.localhost: starting the source
2022-06-23 14:15:59,415    DEBUG pyroute2.ndb.140665294947040.sources.localhost/nsmanager: init
2022-06-23 14:15:59,415    DEBUG pyroute2.ndb.140665294947040.sources.localhost: connecting
2022-06-23 14:15:59,416    DEBUG pyroute2.ndb.140665294947040.sources.localhost: loading
2022-06-23 14:15:59,417    DEBUG pyroute2.ndb.140665294947040.sources.localhost/nsmanager: starting the source
2022-06-23 14:15:59,419    DEBUG pyroute2.ndb.140665294947040.sources.localhost/nsmanager: connecting
2022-06-23 14:15:59,422    DEBUG pyroute2.ndb.140665294947040.sources.localhost/nsmanager: loading
2022-06-23 14:15:59,439    DEBUG pyroute2.ndb.140665294947040.sources.localhost/nsmanager: running
2022-06-23 14:15:59,439    DEBUG pyroute2.ndb.140665294947040.sources.localhost: running
2022-06-23 14:15:59,441    DEBUG pyroute2.ndb.140665294947040.rtnl_object: init
2022-06-23 14:15:59,441    DEBUG pyroute2.ndb.140665294947040.view.interfaces: check if the key {'target': 'localhost', 'ifname': 'lo'} exists in table interfaces
2022-06-23 14:15:59,442    DEBUG pyroute2.ndb.140665294947040.view.interfaces: exists
2022-06-23 14:15:59,442    DEBUG pyroute2.ndb.140665294947040.rtnl_object: complete key {'target': 'localhost', 'ifname': 'lo'} from table interfaces
2022-06-23 14:15:59,442    DEBUG pyroute2.ndb.140665294947040.rtnl_object: got {'target': 'localhost', 'ifname': 'lo', 'tflags': 0, 'index': 1}
2022-06-23 14:15:59,442    DEBUG pyroute2.ndb.140665294947040.view.interfaces: check if the key {'target': 'localhost', 'ifname': 'lo'} exists in table interfaces
2022-06-23 14:15:59,443    DEBUG pyroute2.ndb.140665294947040.view.interfaces: exists
2022-06-23 14:15:59,443    DEBUG pyroute2.ndb.140665294947040.rtnl_object: load_sql load: ('localhost', 0, 0, 772, 1, 65609, 0, '00:00:00:00:00:00', '00:00:00:00:00:00', 'lo', 65536, None, 'noqueue', None, 1000, 0, None, None, None, 0, None, 0, 1, 1, 1, 0, None, None, 0, 65535, 65536, None, None, None, 0, 0, None, 0, 0, None, None, None, None, None, None, 'up', None, None, None, None, None, None, None, None)
2022-06-23 14:15:59,444    DEBUG pyroute2.ndb.140665294947040.rtnl_object: load_sql names: ['target', 'tflags', 'family', 'ifi_type', 'index', 'flags', 'change', 'address', 'broadcast', 'ifname', 'mtu', 'link', 'qdisc', 'master', 'txqlen', 'linkmode', 'net_ns_pid', 'ifalias', 'num_vf', 'group', 'ext_mask', 'promiscuity', 'num_tx_queues', 'num_rx_queues', 'carrier', 'carrier_changes', 'link_netnsid', 'phys_port_name', 'proto_down', 'gso_max_segs', 'gso_max_size', 'event', 'new_netnsid', 'if_netnsid', 'carrier_up_count', 'carrier_down_count', 'new_ifindex', 'min_mtu', 'max_mtu', 'alt_ifname', 'parent_dev_name', 'parent_dev_bus_name', 'gro_max_size', 'tso_max_size', 'tso_max_segs', 'state', 'kind', 'slave_kind', 'xdp_attached', 'xdp_flags', 'xdp_prog_id', 'xdp_drv_prog_id', 'xdp_skb_prog_id', 'xdp_hw_prog_id']
2022-06-23 14:15:59,444    DEBUG pyroute2.ndb.140665294947040.view.interfaces: cache add (('target', 'localhost'), ('tflags', 0), ('index', 1))
2022-06-23 14:15:59,444    DEBUG pyroute2.ndb.140665294947040.rtnl_object: init
2022-06-23 14:15:59,445    DEBUG pyroute2.ndb.140665294947040.view.routes: check if the key {'dst': 'fd12::3', 'family': <AddressFamily.AF_INET6: 10>, 'dst_len': 128, 'oif': 1, 'target': 'localhost'} exists in table routes
2022-06-23 14:15:59,445    DEBUG pyroute2.ndb.140665294947040.view.routes: not exists
2022-06-23 14:15:59,445    DEBUG pyroute2.ndb.140665294947040.rtnl_object: complete key {'dst': 'fd12::3', 'family': <AddressFamily.AF_INET6: 10>, 'dst_len': 128, 'oif': 1, 'target': 'localhost', 'table': 254} from table routes
2022-06-23 14:15:59,446    DEBUG pyroute2.ndb.140665294947040.rtnl_object: got none
2022-06-23 14:15:59,446    DEBUG pyroute2.ndb.140665294947040.view.routes: check if the key {'dst': 'fd12::3', 'family': <AddressFamily.AF_INET6: 10>, 'dst_len': 128, 'oif': 1, 'target': 'localhost', 'create': True} exists in table routes
2022-06-23 14:15:59,447    DEBUG pyroute2.ndb.140665294947040.view.routes: not exists
2022-06-23 14:15:59,447    DEBUG pyroute2.ndb.140665294947040.rtnl_object: commit: [(1655993759.444943, 'invalid')]
2022-06-23 14:15:59,447    DEBUG pyroute2.ndb.140665294947040.rtnl_object: events log: [(1655993759.444943, 'invalid')]
2022-06-23 14:15:59,447    DEBUG pyroute2.ndb.140665294947040.rtnl_object: run apply
2022-06-23 14:15:59,448    DEBUG pyroute2.ndb.140665294947040.rtnl_object: load_sql load: None
2022-06-23 14:15:59,448    DEBUG pyroute2.ndb.140665294947040.rtnl_object: load_sql names: ['target', 'tflags', 'family', 'dst_len', 'src_len', 'tos', 'table', 'proto', 'scope', 'type', 'flags', 'dst', 'src', 'iif', 'oif', 'gateway', 'priority', 'prefsrc', 'protoinfo', 'flow', 'table', 'mark', 'via', 'newdst', 'pref', 'encap_type', 'route_id', 'gc_mark', 'deps']
2022-06-23 14:15:59,448    DEBUG pyroute2.ndb.140665294947040.rtnl_object: apply req: {'target': 'localhost', 'family': <AddressFamily.AF_INET6: 10>, 'dst_len': 128, 'dst': 'fd12::3', 'oif': 1}
2022-06-23 14:15:59,448    DEBUG pyroute2.ndb.140665294947040.rtnl_object: apply idx_req: {'target': 'localhost', 'family': <AddressFamily.AF_INET6: 10>, 'dst_len': 128, 'dst': 'fd12::3', 'oif': 1}
2022-06-23 14:15:59,449    DEBUG pyroute2.ndb.140665294947040.rtnl_object: apply transition from: invalid
2022-06-23 14:15:59,449    DEBUG pyroute2.ndb.140665294947040.rtnl_object: apply method: add
2022-06-23 14:15:59,449    DEBUG pyroute2.ndb.140665294947040.rtnl_object: API call add ({'target': 'localhost', 'family': <AddressFamily.AF_INET6: 10>, 'dst_len': 128, 'dst': 'fd12::3', 'oif': 1, 'multipath': [], 'metrics': {}, 'deps': 0})
2022-06-23 14:15:59,449    DEBUG pyroute2.ndb.140665294947040.sources.localhost: source api run route ('add',) {'target': 'localhost', 'family': <AddressFamily.AF_INET6: 10>, 'dst_len': 128, 'dst': 'fd12::3', 'oif': 1, 'multipath': [], 'metrics': {}, 'deps': 0}
2022-06-23 14:15:59,450    DEBUG pyroute2.ndb.140665294947040.rtnl_object: stats: apply add {objid 140665276551696, wtime 0.0, mqsize 0, nqsize 0}
2022-06-23 14:15:59,451    DEBUG pyroute2.ndb.140665294947040.rtnl_object: load_sql load: None
2022-06-23 14:15:59,451    DEBUG pyroute2.ndb.140665294947040.rtnl_object: load_sql names: ['target', 'tflags', 'family', 'dst_len', 'src_len', 'tos', 'table', 'proto', 'scope', 'type', 'flags', 'dst', 'src', 'iif', 'oif', 'gateway', 'priority', 'prefsrc', 'protoinfo', 'flow', 'table', 'mark', 'via', 'newdst', 'pref', 'encap_type', 'route_id', 'gc_mark', 'deps']
2022-06-23 14:15:59,452    DEBUG pyroute2.ndb.140665294947040.rtnl_object: check: [(1655993759.444943, 'invalid')]
2022-06-23 14:15:59,452    DEBUG pyroute2.ndb.140665294947040.rtnl_object: check state: False
2022-06-23 14:15:59,452    DEBUG pyroute2.ndb.140665294947040.rtnl_object: check failed
2022-06-23 14:15:59,452    DEBUG pyroute2.ndb.140665294947040.rtnl_object: API call add ({'target': 'localhost', 'family': <AddressFamily.AF_INET6: 10>, 'dst_len': 128, 'dst': 'fd12::3', 'oif': 1, 'multipath': [], 'metrics': {}, 'deps': 0})
2022-06-23 14:15:59,452    DEBUG pyroute2.ndb.140665294947040.sources.localhost: source api run route ('add',) {'target': 'localhost', 'family': <AddressFamily.AF_INET6: 10>, 'dst_len': 128, 'dst': 'fd12::3', 'oif': 1, 'multipath': [], 'metrics': {}, 'deps': 0}
2022-06-23 14:15:59,453    DEBUG pyroute2.ndb.140665294947040.rtnl_object: error: (17, 'File exists')
2022-06-23 14:15:59,454    DEBUG pyroute2.ndb.140665294947040.rtnl_object: ignore error 17 for {'multipath': [], 'metrics': {}, 'deps': 0, 'dst': 'fd12::3', 'family': <AddressFamily.AF_INET6: 10>, 'dst_len': 128, 'oif': 1, 'target': 'localhost'}
2022-06-23 14:15:59,454    DEBUG pyroute2.ndb.140665294947040.rtnl_object: run fallback <function fallback_add at 0x7fef30883010> ({'target': 'localhost', 'family': <AddressFamily.AF_INET6: 10>, 'dst_len': 128, 'dst': 'fd12::3', 'oif': 1, 'multipath': [], 'metrics': {}, 'deps': 0})
2022-06-23 14:15:59,454    DEBUG pyroute2.ndb.140665294947040.rtnl_object: ignore incomplete idx_req in the fallback
2022-06-23 14:15:59,454    DEBUG pyroute2.ndb.140665294947040.rtnl_object: stats: apply add {objid 140665276551696, wtime 0.1, mqsize 1, nqsize 0}
2022-06-23 14:15:59,455    DEBUG pyroute2.ndb.140665294947040.rtnl_object: load_sql load: ('localhost', 0, 10, 128, 0, 0, 254, 4, 0, 1, 0, 'fd12::3', None, None, 1, None, 1024, None, None, None, 254, None, '', '', 0, None, None, None, 0)
2022-06-23 14:15:59,455    DEBUG pyroute2.ndb.140665294947040.rtnl_object: load_sql names: ['target', 'tflags', 'family', 'dst_len', 'src_len', 'tos', 'table', 'proto', 'scope', 'type', 'flags', 'dst', 'src', 'iif', 'oif', 'gateway', 'priority', 'prefsrc', 'protoinfo', 'flow', 'table', 'mark', 'via', 'newdst', 'pref', 'encap_type', 'route_id', 'gc_mark', 'deps']
2022-06-23 14:15:59,455    DEBUG pyroute2.ndb.140665294947040.rtnl_object: check: [(1655993759.444943, 'invalid'), (1655993759.4556077, 'system')]
2022-06-23 14:15:59,455    DEBUG pyroute2.ndb.140665294947040.rtnl_object: check: True
2022-06-23 14:15:59,455    DEBUG pyroute2.ndb.140665294947040.rtnl_object: checked
2022-06-23 14:15:59,455    DEBUG pyroute2.ndb.140665294947040.rtnl_object: stats: 140665276551696 pass
2022-06-23 14:15:59,455    DEBUG pyroute2.ndb.140665294947040.rtnl_object: init
2022-06-23 14:15:59,456    DEBUG pyroute2.ndb.140665294947040.view.routes: check if the key {'target': 'localhost', 'dst': 'fd12::3', 'dst_len': 128, 'family': <AddressFamily.AF_INET6: 10>} exists in table routes
2022-06-23 14:15:59,457    DEBUG pyroute2.ndb.140665294947040.view.routes: exists
2022-06-23 14:15:59,457    DEBUG pyroute2.ndb.140665294947040.rtnl_object: complete key {'target': 'localhost', 'dst': 'fd12::3', 'dst_len': 128, 'family': <AddressFamily.AF_INET6: 10>, 'table': 254} from table routes
2022-06-23 14:15:59,457    DEBUG pyroute2.ndb.140665294947040.rtnl_object: got {'target': 'localhost', 'dst': 'fd12::3', 'dst_len': 128, 'family': <AddressFamily.AF_INET6: 10>, 'table': 254, 'tflags': 0, 'tos': 0, 'scope': 0, 'RTA_DST': 'fd12::3', 'RTA_OIF': 1, 'RTA_PRIORITY': 1024, 'RTA_TABLE': 254, 'RTA_VIA': '', 'RTA_NEWDST': ''}
2022-06-23 14:15:59,457    DEBUG pyroute2.ndb.140665294947040.view.routes: check if the key {'target': 'localhost', 'dst': 'fd12::3', 'dst_len': 128, 'family': <AddressFamily.AF_INET6: 10>} exists in table routes
2022-06-23 14:15:59,458    DEBUG pyroute2.ndb.140665294947040.view.routes: exists
2022-06-23 14:15:59,458    DEBUG pyroute2.ndb.140665294947040.rtnl_object: load_sql load: ('localhost', 0, 10, 128, 0, 0, 254, 4, 0, 1, 0, 'fd12::3', None, None, 1, None, 1024, None, None, None, 254, None, '', '', 0, None, None, None, 0)
2022-06-23 14:15:59,458    DEBUG pyroute2.ndb.140665294947040.rtnl_object: load_sql names: ['target', 'tflags', 'family', 'dst_len', 'src_len', 'tos', 'table', 'proto', 'scope', 'type', 'flags', 'dst', 'src', 'iif', 'oif', 'gateway', 'priority', 'prefsrc', 'protoinfo', 'flow', 'table', 'mark', 'via', 'newdst', 'pref', 'encap_type', 'route_id', 'gc_mark', 'deps']
2022-06-23 14:15:59,458    DEBUG pyroute2.ndb.140665294947040.view.routes: cache add (('target', 'localhost'), ('tflags', 0), ('family', <AddressFamily.AF_INET6: 10>), ('dst_len', 128), ('tos', 0), ('scope', 0), ('RTA_DST', 'fd12::3'), ('RTA_OIF', 1), ('RTA_PRIORITY', 1024), ('RTA_TABLE', 254), ('RTA_VIA', ''), ('RTA_NEWDST', ''))
2022-06-23 14:15:59,458    DEBUG pyroute2.ndb.140665294947040.rtnl_object: commit: [(1655993759.4560027, 'invalid'), (1655993759.4588256, 'system'), (1655993759.4589062, 'remove')]
2022-06-23 14:15:59,459    DEBUG pyroute2.ndb.140665294947040.rtnl_object: init
2022-06-23 14:15:59,459    DEBUG pyroute2.ndb.140665294947040.view.routes: check if the key {'target': 'localhost', 'tflags': 0, 'family': <AddressFamily.AF_INET: 2>, 'dst_len': 128, 'tos': 0, 'scope': 0, 'dst': 'fd12::3', 'oif': 1, 'priority': 1024, 'table': 254, 'via': '', 'newdst': ''} exists in table routes
Created route: {'multipath': [], 'metrics': {}, 'deps': 0, 'target': 'localhost', 'dst': 'fd12::3', 'dst_len': 128, 'family': 10, 'table': 254, 'tflags': 0, 'tos': 0, 'scope': 0, 'oif': 1, 'priority': 1024, 'via': '', 'newdst': '', 'src_len': 0, 'proto': 4, 'type': 1, 'flags': 0, 'src': None, 'iif': None, 'gateway': None, 'prefsrc': None, 'protoinfo': None, 'flow': None, 'mark': None, 'pref': 0, 'encap_type': None, 'route_id': None, 'gc_mark': None}
Trying to remove this route again
2022-06-23 14:15:59,459    DEBUG pyroute2.ndb.140665294947040.view.routes: not exists
2022-06-23 14:15:59,459    DEBUG pyroute2.ndb.140665294947040.rtnl_object: complete key {'target': 'localhost', 'tflags': 0, 'family': 10, 'dst_len': 128, 'tos': 0, 'scope': 0, 'RTA_DST': 'fd12::3', 'RTA_OIF': 1, 'RTA_PRIORITY': 1024, 'RTA_TABLE': 254, 'RTA_VIA': '', 'RTA_NEWDST': '', 'table': 254} from table routes_140665276245920
2022-06-23 14:15:59,459    DEBUG pyroute2.ndb.140665294947040.rtnl_object: got {'target': 'localhost', 'tflags': 0, 'family': 10, 'dst_len': 128, 'tos': 0, 'scope': 0, 'RTA_DST': 'fd12::3', 'RTA_OIF': 1, 'RTA_PRIORITY': 1024, 'RTA_TABLE': 254, 'RTA_VIA': '', 'RTA_NEWDST': '', 'table': 254}
Traceback (most recent call last):
  File "/co/min.py", line 16, in <module>
    route.remove().commit()
  File "/usr/local/lib/python3.10/site-packages/pr2modules/ndb/auth_manager.py", line 67, in guard
    return f(obj, *argv, **kwarg)
  File "/usr/local/lib/python3.10/site-packages/pr2modules/ndb/objects/__init__.py", line 706, in commit
    self.last_save = self.snapshot()
  File "/usr/local/lib/python3.10/site-packages/pr2modules/ndb/auth_manager.py", line 67, in guard
    return f(obj, *argv, **kwarg)
  File "/usr/local/lib/python3.10/site-packages/pr2modules/ndb/objects/__init__.py", line 557, in snapshot
    snp = type(self)(
  File "/usr/local/lib/python3.10/site-packages/pr2modules/ndb/objects/route.py", line 570, in __init__
    super(Route, self).__init__(*argv, **kwarg)
  File "/usr/local/lib/python3.10/site-packages/pr2modules/ndb/objects/__init__.py", line 323, in __init__
    raise KeyError('object does not exists')
KeyError: 'object does not exists'
2022-06-23 14:15:59,461    DEBUG pyroute2.ndb.140665294947040.sources.localhost: source shutdown
2022-06-23 14:15:59,462    DEBUG pyroute2.ndb.140665294947040.sources.localhost: stop
2022-06-23 14:15:59,462    DEBUG pyroute2.ndb.140665294947040.sources.localhost: sync
2022-06-23 14:15:59,462    DEBUG pyroute2.ndb.140665294947040.sources.localhost: shutdown handled by the main thread
2022-06-23 14:15:59,462    DEBUG pyroute2.ndb.140665294947040.sources.localhost: stopped
2022-06-23 14:15:59,462    DEBUG pyroute2.ndb.140665294947040.main: flush DB for the target localhost
2022-06-23 14:15:59,463    DEBUG pyroute2.ndb.140665294947040.sources.localhost/nsmanager: source shutdown
2022-06-23 14:15:59,463    DEBUG pyroute2.ndb.140665294947040.sources.localhost/nsmanager: stop
2022-06-23 14:15:59,463    DEBUG pyroute2.ndb.140665294947040.sources.localhost/nsmanager: sync
2022-06-23 14:15:59,463    DEBUG pyroute2.ndb.140665294947040.sources.localhost/nsmanager: shutdown handled by the main thread
2022-06-23 14:15:59,464    DEBUG pyroute2.ndb.140665294947040.sources.localhost/nsmanager: stopped
2022-06-23 14:15:59,464    DEBUG pyroute2.ndb.140665294947040.main: flush DB for the target localhost/nsmanager

The problem is that after the remove() call a check is made to see if the route exists and that check succeeds, but after the commit() the same check is performed and that one fails. The key difference is that the family for that second call is set tot AF_INET(2) instead of the expected AF_INET6(10) and thus it concludes that the object does not exist.

[internetionals]

This incorrect transformation happens in pyroute2.pr2modules.requests.RouteFieldFilter and specifically in pyroute2.pr2modules.requests.common.IPTargets (one of it's base classes).

The route in question has a via field with the value '', this causes the parse_target method of IPTargets to return {"family": AF_INET, "via": ""}, which in turn gets merged with the other fields and effectively overwrites the original family value.

A possible fix that I tried was adding a simple check for this corner case in that method that doesn't return family in the result if the original value is an empty string:

  def parse_target(self, key, context, value):
      # ...
      if value == '':
        return { key: '' }
      # ...

This works, but I don't know if this is a sensible fix for this problem or that I'm just papering over some consequences of a problem elsewhere.

[carloslockward]

I have observed this issue as well

[svinota]

Thanks for the report, the fix is coming tonight.

[svinota]

The fix is in the master, thanks @internetionals

Unit tests added

Final CI runs and tagging 0.6.13

[internetionals]

@svinota thanks for the quick action

[svinota]

During CI found another related issue, fixing as well.

[svinota]

Should be fixed now.

0.6.13 tagged

[svinota]

Pls let me know if I can close the ticket.

[internetionals]

Hi @svinota, I've confirmed that the fix works for me, you may close this ticket. Thanks again!

[svinota]

Cool, thanks.