BleakError: "service already present" with duplicate UUID when enumerating services

[kupiakos]

• bleak version: 0.5.1
• Python version: Python 3.7.3
• Operating System: Ubuntu 19.04
• BlueZ version (bluetoothctl -v) in case of Linux: 5.50

Description

First off, thanks for the library! It's been fun to tinker with so far. As a warning, I am new to Bluetooth generally.

I'm trying to connect to an esoteric BLE device, and so far Bleak has been throwing an exception during characteristics enumeration. It appears to be due to two characteristics having the same UUID. I saw #101, but this appears to be due to a device actually reporting a duplicate UUID rather than a missed initialization.

I can confirm this is the case with this device with gatttool. Notice how the UUID 00002a06-0000-1000-8000-00805f9b34fb is duplicated for handles 0x0024 and 0x0027:

$ gatttool -b f8:30:02:1e:9c:ba --primary
attr handle = 0x0001, end grp handle = 0x000b uuid: 00001800-0000-1000-8000-00805f9b34fb
attr handle = 0x000c, end grp handle = 0x000f uuid: 00001801-0000-1000-8000-00805f9b34fb
attr handle = 0x0010, end grp handle = 0x0022 uuid: 0000180a-0000-1000-8000-00805f9b34fb
attr handle = 0x0023, end grp handle = 0x0025 uuid: 00001803-0000-1000-8000-00805f9b34fb
attr handle = 0x0026, end grp handle = 0x0028 uuid: 00001802-0000-1000-8000-00805f9b34fb
attr handle = 0x0029, end grp handle = 0x002c uuid: 00001804-0000-1000-8000-00805f9b34fb
attr handle = 0x002d, end grp handle = 0x0032 uuid: 0000180f-0000-1000-8000-00805f9b34fb
attr handle = 0x0033, end grp handle = 0x003a uuid: 0000ccc0-0000-1000-8000-00805f9b34fb
attr handle = 0x003b, end grp handle = 0x0045 uuid: 0000aa40-0000-1000-8000-00805f9b34fb
attr handle = 0x0046, end grp handle = 0xffff uuid: f000ffc0-0451-4000-b000-000000000000
$ gatttool -b f8:30:02:1e:9c:ba --characteristics 
handle = 0x0002, char properties = 0x02, char value handle = 0x0003, uuid = 00002a00-0000-1000-8000-00805f9b34fb
handle = 0x0004, char properties = 0x02, char value handle = 0x0005, uuid = 00002a01-0000-1000-8000-00805f9b34fb
handle = 0x0006, char properties = 0x0a, char value handle = 0x0007, uuid = 00002a02-0000-1000-8000-00805f9b34fb
handle = 0x0008, char properties = 0x08, char value handle = 0x0009, uuid = 00002a03-0000-1000-8000-00805f9b34fb
handle = 0x000a, char properties = 0x02, char value handle = 0x000b, uuid = 00002a04-0000-1000-8000-00805f9b34fb
handle = 0x000d, char properties = 0x20, char value handle = 0x000e, uuid = 00002a05-0000-1000-8000-00805f9b34fb
handle = 0x0011, char properties = 0x02, char value handle = 0x0012, uuid = 00002a23-0000-1000-8000-00805f9b34fb
handle = 0x0013, char properties = 0x02, char value handle = 0x0014, uuid = 00002a24-0000-1000-8000-00805f9b34fb
handle = 0x0015, char properties = 0x02, char value handle = 0x0016, uuid = 00002a25-0000-1000-8000-00805f9b34fb
handle = 0x0017, char properties = 0x02, char value handle = 0x0018, uuid = 00002a26-0000-1000-8000-00805f9b34fb
handle = 0x0019, char properties = 0x02, char value handle = 0x001a, uuid = 00002a27-0000-1000-8000-00805f9b34fb
handle = 0x001b, char properties = 0x02, char value handle = 0x001c, uuid = 00002a28-0000-1000-8000-00805f9b34fb
handle = 0x001d, char properties = 0x02, char value handle = 0x001e, uuid = 00002a29-0000-1000-8000-00805f9b34fb
handle = 0x001f, char properties = 0x02, char value handle = 0x0020, uuid = 00002a2a-0000-1000-8000-00805f9b34fb
handle = 0x0021, char properties = 0x02, char value handle = 0x0022, uuid = 00002a50-0000-1000-8000-00805f9b34fb
handle = 0x0024, char properties = 0x0a, char value handle = 0x0025, uuid = 00002a06-0000-1000-8000-00805f9b34fb
handle = 0x0027, char properties = 0x04, char value handle = 0x0028, uuid = 00002a06-0000-1000-8000-00805f9b34fb
handle = 0x002a, char properties = 0x12, char value handle = 0x002b, uuid = 00002a07-0000-1000-8000-00805f9b34fb
handle = 0x002e, char properties = 0x12, char value handle = 0x002f, uuid = 00002a19-0000-1000-8000-00805f9b34fb
handle = 0x0034, char properties = 0x12, char value handle = 0x0035, uuid = 0000ccc1-0000-1000-8000-00805f9b34fb
handle = 0x0037, char properties = 0x08, char value handle = 0x0038, uuid = 0000ccc2-0000-1000-8000-00805f9b34fb
handle = 0x0039, char properties = 0x08, char value handle = 0x003a, uuid = 0000ccc3-0000-1000-8000-00805f9b34fb
handle = 0x003c, char properties = 0x12, char value handle = 0x003d, uuid = 0000aa41-0000-1000-8000-00805f9b34fb
handle = 0x003f, char properties = 0x0a, char value handle = 0x0040, uuid = 0000aa42-0000-1000-8000-00805f9b34fb
handle = 0x0041, char properties = 0x0a, char value handle = 0x0042, uuid = 0000aa44-0000-1000-8000-00805f9b34fb
handle = 0x0043, char properties = 0x02, char value handle = 0x0044, uuid = 0000aa43-0000-1000-8000-00805f9b34fb
handle = 0x0047, char properties = 0x1c, char value handle = 0x0048, uuid = f000ffc1-0451-4000-b000-000000000000
handle = 0x004b, char properties = 0x1c, char value handle = 0x004c, uuid = f000ffc2-0451-4000-b000-000000000000

I expect to still be able to send commands to the device, as gatttool has no problems doing so. However, Bleak can't handle it.

What I Did

Running a simple Bleak connection script with debug logging provides:

DEBUG:bleak.backends.bluezdbus.client:Connecting to BLE device @ f8:30:02:1e:9c:ba with hci0
DEBUG:bleak.backends.bluezdbus.client:Connection successful.
DEBUG:bleak.backends.bluezdbus.client:Get Services...
DEBUG:bleak.backends.bluezdbus.client:
Primary Service
	/org/bluez/hci0/dev_F8_30_02_1E_9C_BA/service0046
	f000ffc0-0451-4000-b000-000000000000
	Unknown
DEBUG:bleak.backends.bluezdbus.client:
Characteristic
	/org/bluez/hci0/dev_F8_30_02_1E_9C_BA/service0046/char004b
	f000ffc2-0451-4000-b000-000000000000
	Unknown
DEBUG:bleak.backends.bluezdbus.client:
Descriptor
	/org/bluez/hci0/dev_F8_30_02_1E_9C_BA/service0046/char004b/desc004e
	00002901-0000-1000-8000-00805f9b34fb
	Characteristic User Description
DEBUG:bleak.backends.bluezdbus.client:
Descriptor
	/org/bluez/hci0/dev_F8_30_02_1E_9C_BA/service0046/char004b/desc004d
	00002902-0000-1000-8000-00805f9b34fb
	Client Characteristic Configuration
DEBUG:bleak.backends.bluezdbus.client:
Characteristic
	/org/bluez/hci0/dev_F8_30_02_1E_9C_BA/service0046/char0047
	f000ffc1-0451-4000-b000-000000000000
	Unknown
DEBUG:bleak.backends.bluezdbus.client:
Descriptor
	/org/bluez/hci0/dev_F8_30_02_1E_9C_BA/service0046/char0047/desc004a
	00002901-0000-1000-8000-00805f9b34fb
	Characteristic User Description
DEBUG:bleak.backends.bluezdbus.client:
Descriptor
	/org/bluez/hci0/dev_F8_30_02_1E_9C_BA/service0046/char0047/desc0049
	00002902-0000-1000-8000-00805f9b34fb
	Client Characteristic Configuration
DEBUG:bleak.backends.bluezdbus.client:
Primary Service
	/org/bluez/hci0/dev_F8_30_02_1E_9C_BA/service003b
	0000aa40-0000-1000-8000-00805f9b34fb
	Vendor specific
DEBUG:bleak.backends.bluezdbus.client:
Characteristic
	/org/bluez/hci0/dev_F8_30_02_1E_9C_BA/service003b/char0043
	0000aa43-0000-1000-8000-00805f9b34fb
	Vendor specific
DEBUG:bleak.backends.bluezdbus.client:
Descriptor
	/org/bluez/hci0/dev_F8_30_02_1E_9C_BA/service003b/char0043/desc0045
	00002902-0000-1000-8000-00805f9b34fb
	Client Characteristic Configuration
DEBUG:bleak.backends.bluezdbus.client:
Characteristic
	/org/bluez/hci0/dev_F8_30_02_1E_9C_BA/service003b/char0041
	0000aa44-0000-1000-8000-00805f9b34fb
	Vendor specific
DEBUG:bleak.backends.bluezdbus.client:
Characteristic
	/org/bluez/hci0/dev_F8_30_02_1E_9C_BA/service003b/char003f
	0000aa42-0000-1000-8000-00805f9b34fb
	Vendor specific
DEBUG:bleak.backends.bluezdbus.client:
Characteristic
	/org/bluez/hci0/dev_F8_30_02_1E_9C_BA/service003b/char003c
	0000aa41-0000-1000-8000-00805f9b34fb
	Vendor specific
DEBUG:bleak.backends.bluezdbus.client:
Descriptor
	/org/bluez/hci0/dev_F8_30_02_1E_9C_BA/service003b/char003c/desc003e
	00002902-0000-1000-8000-00805f9b34fb
	Client Characteristic Configuration
DEBUG:bleak.backends.bluezdbus.client:
Primary Service
	/org/bluez/hci0/dev_F8_30_02_1E_9C_BA/service0033
	0000ccc0-0000-1000-8000-00805f9b34fb
	Vendor specific
DEBUG:bleak.backends.bluezdbus.client:
Characteristic
	/org/bluez/hci0/dev_F8_30_02_1E_9C_BA/service0033/char0039
	0000ccc3-0000-1000-8000-00805f9b34fb
	Vendor specific
DEBUG:bleak.backends.bluezdbus.client:
Characteristic
	/org/bluez/hci0/dev_F8_30_02_1E_9C_BA/service0033/char0037
	0000ccc2-0000-1000-8000-00805f9b34fb
	Vendor specific
DEBUG:bleak.backends.bluezdbus.client:
Characteristic
	/org/bluez/hci0/dev_F8_30_02_1E_9C_BA/service0033/char0034
	0000ccc1-0000-1000-8000-00805f9b34fb
	Vendor specific
DEBUG:bleak.backends.bluezdbus.client:
Descriptor
	/org/bluez/hci0/dev_F8_30_02_1E_9C_BA/service0033/char0034/desc0036
	00002902-0000-1000-8000-00805f9b34fb
	Client Characteristic Configuration
DEBUG:bleak.backends.bluezdbus.client:
Primary Service
	/org/bluez/hci0/dev_F8_30_02_1E_9C_BA/service0029
	00001804-0000-1000-8000-00805f9b34fb
	Tx Power
DEBUG:bleak.backends.bluezdbus.client:
Characteristic
	/org/bluez/hci0/dev_F8_30_02_1E_9C_BA/service0029/char002a
	00002a07-0000-1000-8000-00805f9b34fb
	Tx Power Level
DEBUG:bleak.backends.bluezdbus.client:
Descriptor
	/org/bluez/hci0/dev_F8_30_02_1E_9C_BA/service0029/char002a/desc002c
	00002902-0000-1000-8000-00805f9b34fb
	Client Characteristic Configuration
DEBUG:bleak.backends.bluezdbus.client:
Primary Service
	/org/bluez/hci0/dev_F8_30_02_1E_9C_BA/service0026
	00001802-0000-1000-8000-00805f9b34fb
	Immediate Alert
DEBUG:bleak.backends.bluezdbus.client:
Characteristic
	/org/bluez/hci0/dev_F8_30_02_1E_9C_BA/service0026/char0027
	00002a06-0000-1000-8000-00805f9b34fb
	Alert Level
DEBUG:bleak.backends.bluezdbus.client:
Primary Service
	/org/bluez/hci0/dev_F8_30_02_1E_9C_BA/service0023
	00001803-0000-1000-8000-00805f9b34fb
	Link Loss
DEBUG:bleak.backends.bluezdbus.client:
Characteristic
	/org/bluez/hci0/dev_F8_30_02_1E_9C_BA/service0023/char0024
	00002a06-0000-1000-8000-00805f9b34fb
	Alert Level
DEBUG:bleak.backends.bluezdbus.client:
Primary Service
	/org/bluez/hci0/dev_F8_30_02_1E_9C_BA/service0010
	0000180a-0000-1000-8000-00805f9b34fb
	Device Information
DEBUG:bleak.backends.bluezdbus.client:
Characteristic
	/org/bluez/hci0/dev_F8_30_02_1E_9C_BA/service0010/char0021
	00002a50-0000-1000-8000-00805f9b34fb
	PnP ID
DEBUG:bleak.backends.bluezdbus.client:
Characteristic
	/org/bluez/hci0/dev_F8_30_02_1E_9C_BA/service0010/char001f
	00002a2a-0000-1000-8000-00805f9b34fb
	IEEE 11073-20601 Regulatory Cert. Data List
DEBUG:bleak.backends.bluezdbus.client:
Characteristic
	/org/bluez/hci0/dev_F8_30_02_1E_9C_BA/service0010/char001d
	00002a29-0000-1000-8000-00805f9b34fb
	Manufacturer Name String
DEBUG:bleak.backends.bluezdbus.client:
Characteristic
	/org/bluez/hci0/dev_F8_30_02_1E_9C_BA/service0010/char001b
	00002a28-0000-1000-8000-00805f9b34fb
	Software Revision String
DEBUG:bleak.backends.bluezdbus.client:
Characteristic
	/org/bluez/hci0/dev_F8_30_02_1E_9C_BA/service0010/char0019
	00002a27-0000-1000-8000-00805f9b34fb
	Hardware Revision String
DEBUG:bleak.backends.bluezdbus.client:
Characteristic
	/org/bluez/hci0/dev_F8_30_02_1E_9C_BA/service0010/char0017
	00002a26-0000-1000-8000-00805f9b34fb
	Firmware Revision String
DEBUG:bleak.backends.bluezdbus.client:
Characteristic
	/org/bluez/hci0/dev_F8_30_02_1E_9C_BA/service0010/char0015
	00002a25-0000-1000-8000-00805f9b34fb
	Serial Number String
DEBUG:bleak.backends.bluezdbus.client:
Characteristic
	/org/bluez/hci0/dev_F8_30_02_1E_9C_BA/service0010/char0013
	00002a24-0000-1000-8000-00805f9b34fb
	Model Number String
DEBUG:bleak.backends.bluezdbus.client:
Characteristic
	/org/bluez/hci0/dev_F8_30_02_1E_9C_BA/service0010/char0011
	00002a23-0000-1000-8000-00805f9b34fb
	System ID
DEBUG:bleak.backends.bluezdbus.client:
Primary Service
	/org/bluez/hci0/dev_F8_30_02_1E_9C_BA/service000c
	00001801-0000-1000-8000-00805f9b34fb
	Generic Attribute Profile
DEBUG:bleak.backends.bluezdbus.client:
Characteristic
	/org/bluez/hci0/dev_F8_30_02_1E_9C_BA/service000c/char000d
	00002a05-0000-1000-8000-00805f9b34fb
	Service Changed
DEBUG:bleak.backends.bluezdbus.client:
Descriptor
	/org/bluez/hci0/dev_F8_30_02_1E_9C_BA/service000c/char000d/desc000f
	00002902-0000-1000-8000-00805f9b34fb
	Client Characteristic Configuration
Traceback (most recent call last):
  File "<workspace>/main.py", line 18, in <module>
    loop.run_until_complete(main(loop))
  File "/usr/lib/python3.7/asyncio/base_events.py", line 584, in run_until_complete
    return future.result()
  File "<workspace>/main.py", line 11, in main
    async with BleakClient(address=ADDRESS, loop=loop) as client:
  File "<workspace>/venv/lib/python3.7/site-packages/bleak/backends/client.py", line 43, in __aenter__
    await self.connect()
  File "<workspace>/venv/lib/python3.7/site-packages/bleak/backends/bluezdbus/client.py", line 136, in connect
    await self.get_services()
  File "<workspace>/venv/lib/python3.7/site-packages/bleak/backends/bluezdbus/client.py", line 250, in get_services
    BleakGATTCharacteristicBlueZDBus(char, object_path, _service[0].uuid)
  File "<workspace>/venv/lib/python3.7/site-packages/bleak/backends/service.py", line 120, in add_characteristic
    "This characteristic is already present in this BleakGATTServiceCollection!"
bleak.exc.BleakError: This characteristic is already present in this BleakGATTServiceCollection!

Process finished with exit code 1

[hbldh]

This is problematic. I can make the error go away by not throwing the exepction, but currently Bleak cannot handle two characteristics with the same uuid, since it looks for characteristics by the uuid value, not the handle value.

I have no solution for this right now. I will have to think about this for a while.

[pliniofpa]

@hbldh per Bluetooth BLUETOOTH CORE SPECIFICATION Version 5.2 | Vol 3, Part G:

So I believe bleak implementation should be changed to use handlers in the characteristics dictionary instead of UUIDs in order to comply with the Bluetooth specification.

[hbldh]

This is indeed a problematic situation; I finally see why all the libraries and backends have their own object representing a charactersitic, and not just the UUID... This was a fundamental difference in Bleak, that I would only use UUIDs and not anything else, to simplify things and only have to deal with str and uuid objects as char specifiers.

All bleak.exc.BleakError: This characteristic is already present in this BleakGATTServiceCollection! issues (#159, ...) are problems that exist due to this. Your PR is greatly apprecieated, and it solves the problem of addign multiples of the same char, but it still lack in handling them in write_gatt_char and read_gatt_char. To change those I have (I think so at least) to make breaking changes to the public API. I have to think about this for a while...

[bojanpotocnik]

I had the same problem, described in #56. Look under 3rd bullet point under What I Did. I have such cases almost all the time and I was also bothered that I always re-query characteristic instance via UUID instead of just saving the reference, so I modified BleakGATTService to have get_characteristic() method - in this case you can have multiple services (saved by references, not UUIDs) with the same characteristics because you query characteristic UUIDs per service, not client instance.

I will also take a look into this and try to contribute. I was working with BlueZ for the last half year (in C not Python, modifying Linux core BT drivers) but now I return back to this great project to use it for Python testing.

[geeksville]

Oh, yes, I'm also seeing this problem for my usecase as well. Is the current temporary/recommended solution for connecting to devices with this behavior to use @pliniofpa's PR?

[bsiever]

Just to add a little more to this, the UUID is a type and multiple instances of nearly everything can occur:

• Multiple instances of the same service on a device are allowed, like multiple heart rate monitors on the same device. (Although some services, like Device Information Service, include specifications that there will be only one instance on a device)
• Multiple instances of a characteristic type within a service
• Multiple instances of descriptors within a characteristic, like multiple Characteristic Presentation Formats.

Some possible fixes:

• UPDATE: I don't think this is a workable idea...hadn't really thought through it: When a UUID isn't unique it maps to a list of instances and when it is unique it maps to an instance, like now. This could mostly be accomplished through changes to base classes and backends without breaking the API....but a re-write to avoid direct access to characteristics may be best (i.e., require characteristics to be accessed via a service and descriptors via a characteristic rather than the direct access of a characteristic from a collection of services as it allows now)

  • For known devices it'd be pretty convenient. You'd know whether to expect a single instance, which is the norm, or multiple instances in a list.
  • For things like service discovery in a generic bluetooth browser it's a bit clunky because one has to check to see if a UUID maps to a list or a single item.

• Use the concept of a handle to access attribute data (like an int / index corresponding to the data item --- Plinio's fix)

• Use a more complex data structure that represents the entire device via a hierarchical proxy object.

• Maybe replace the service class with a hierarchy that internally uses BLE handles or the OS-specific equivalent, like objects. Services would contain characteristics that would contain descriptors. Provide new API methods to iterate through all of them, but the existing get_* would all search for the first matching UUID to maintain compatibility (until a major release when they can be dumped via new API).

[hbldh]

This issue needs a resolution. It will not be identical to @pliniofpa's solution, but somewhere similar.

It might lead to a backwards incompatible solution, but I aim to be as not-interruptive as possible and issue DeprecationWarnings if possible.

My goal is to have this ready to release by the end of June.