[BUG] Simplejpeg related ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject

[mgineer85]

Describe the bug

To Reproduce
Here is what I did:

• Install my photobooth app in venv using pipx:
• pipx install --system-site-packages photobooth-app --pip-args='--prefer-binary'
• Configure to use the picamera2 as backend

Unfortunately I cannot reproduce using example scripts because they do not live in the environment.

Expected behaviour
No error

Console Output, Screenshots

2025-02-17 21:29:56,900 [    INFO] service AquisitionService now EnumStatus.starting (base.py:63)
2025-02-17 21:29:57,431 [   ERROR] numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject (container.py:85)
Traceback (most recent call last):
  File "/home/pi/.local/pipx/venvs/photobooth-app/lib/python3.11/site-packages/photobooth/container.py", line 81, in start
    service.start()
  File "/home/pi/.local/pipx/venvs/photobooth-app/lib/python3.11/site-packages/photobooth/services/aquisition.py", line 43, in start
    backend: AbstractBackend = self._import_backend(backend_config.selected_device)(
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pi/.local/pipx/venvs/photobooth-app/lib/python3.11/site-packages/photobooth/services/aquisition.py", line 268, in _import_backend
    module = import_module(module_path, package=pkg)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1206, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/pi/.local/pipx/venvs/photobooth-app/lib/python3.11/site-packages/photobooth/services/backends/picamera2.py", line 14, in <module>
    from picamera2 import Picamera2
  File "/usr/lib/python3/dist-packages/picamera2/__init__.py", line 11, in <module>
    from .picamera2 import Picamera2, Preview
  File "/usr/lib/python3/dist-packages/picamera2/picamera2.py", line 27, in <module>
    from picamera2.encoders import Encoder, H264Encoder, MJPEGEncoder, Quality
  File "/usr/lib/python3/dist-packages/picamera2/encoders/__init__.py", line 6, in <module>
    from .jpeg_encoder import JpegEncoder
  File "/usr/lib/python3/dist-packages/picamera2/encoders/jpeg_encoder.py", line 3, in <module>
    import simplejpeg
  File "/usr/lib/python3/dist-packages/simplejpeg/__init__.py", line 1, in <module>
    from ._jpeg import encode_jpeg, encode_jpeg_yuv_planes
  File "simplejpeg/_jpeg.pyx", line 1, in init simplejpeg._jpeg
ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject
2025-02-17 21:29:57,436 [CRITICAL] could not start service (container.py:86)

Hardware :
Pi4 latest apt update and full-upgrade

Additional context
I think it is related to the recent changes in simplejpeg?

In my venv there are following dependencies:

dependencies = [
    "fastapi>=0.106.0",
    "opencv-python-headless>=4.10.0.84",
    "numpy>=2.0.0",
    "simplejpeg>=1.7.4", # might be installed globally already on Pi but in older version incompatible to numpy2
    "pillow>=10.0.0",
    "psutil>=5.9.0",
    "pydantic>=2.7.0",
    "pydantic-extra-types>=2.3.0",
    "pydantic-settings>=2.1.0",
    "requests>=2.31.0",
    "sse-starlette>=2.2.0",
    "uvicorn>=0.24",
    "python-dotenv>=1.0.0",
    "pyserial~=3.5",
    "jsonref>=1.1.0",
    "pilgram2>=2.0.6",
    "linuxpy>=0.10.0; platform_system == 'Linux'",
    "gpiozero~=2.0.0",
    "gphoto2~=2.5.0; platform_system == 'Linux' or platform_system == 'Darwin'",
    "python-statemachine>=2.3.3",
    "python-multipart>=0.0.6",
    "pyjwt[crypto]>=2.8.0",
    "wigglecam",
    "piexif>=1.1.3",
    "SQLAlchemy>=2.0.37",
    "pyturbojpeg>=1.7.0",
    "pluggy>=1.5.0",
]

numpy and simplejpeg are there which could cause the issue? On the system there is numpy 1.x installed from the normal repositories, maybe that conflicts?

To "fix" it, I tested to downgrade simplejpeg and it works:

pi@photobooth-chameleon:~/photobooth-data $ pipx inject photobooth-app simplejpeg==1.7.6
  injected package simplejpeg into venv photobooth-app
done!

[davidplowman]

Hi, thanks for the report. I'm not sure what to do about this, but will ask around here at Pi HQ. Let me explain what I think has happened.

I've upstreamed some new features to the simplejpeg library which gives us better encoding performance in various ways, and this was released in simplejpeg version 1.8.1. A Picamera2 update that uses these new features from simplejpeg will be released "soon", and when that happens, you will need the 1.8.1 simplejpeg.

So before the Picamera2 update, we need to get the new simplejpeg into an apt package (we can't "pip install" stuff in our OS distributions) which means building them against the standard versions in the Debian release. Debian Bookworm is back on numpy 1.24.2, so that's what our simplejpeg apt package picks up.

Unfortunately, it looks like you're using lots of more recent pip packages, so you've effectively had to move on to numpy 2.x, and as we know, numpy 1.x and 2.x are binary incompatible. I don't know how to distinguish numpy 1.x versions of simplejpeg from 2.x versions (or in even if that's possible).

The next Debian release, Trixie, still appears to be on numpy 1.26, so it's not like the problem will go away any time soon. Doing a pip install --force-reinstall simplejpeg will probably fix things for you and get the numpy 2.x version of simplejpeg 1.8.1, but I'll ask around here to see if anyone has any better ideas.