[FR] Support more image codecs in fiftyone.utils.image.read

[Laurent2916]

Proposal Summary

Support more codecs when reading images from fiftyone.utils.image.read

What areas of FiftyOne does this feature affect?

• App: FiftyOne application
• Core: Core fiftyone Python library
• Server: FiftyOne server

Details

read uses opencv, which doesn't support .avif images (well at least not from what I could see):

Example 1:

from fiftyone.utils.image import read

url = "https://aomediacodec.github.io/av1-avif/testFiles/Link-U/hato.profile0.8bpc.yuv420.no-cdef.avif"
img = read(url)

Traceback (most recent call last):
  File "/home/laurent/Documents/project/scratch.py", line 810, in <module>
    img = read(url)
          ^^^^^^^^^
  File "/home/laurent/Documents/project/.venv/lib/python3.12/site-packages/fiftyone/utils/image.py", line 37, in read
    return etai.read(path_or_url, include_alpha=include_alpha, flag=flag)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/laurent/Documents/project/.venv/lib/python3.12/site-packages/eta/core/image.py", line 476, in read
    return download(path_or_url, include_alpha=include_alpha, flag=flag)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/laurent/Documents/project/.venv/lib/python3.12/site-packages/eta/core/image.py", line 457, in download
    return decode(bytes, include_alpha=include_alpha, flag=flag)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/laurent/Documents/project/.venv/lib/python3.12/site-packages/eta/core/image.py", line 438, in decode
    return _exchange_rb(cv2.imdecode(vec, flag))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/laurent/Documents/project/.venv/lib/python3.12/site-packages/eta/core/image.py", line 1982, in _exchange_rb
    if is_gray(img):
       ^^^^^^^^^^^^
  File "/home/laurent/Documents/project/.venv/lib/python3.12/site-packages/eta/core/image.py", line 1297, in is_gray
    return img.ndim == 2 or (img.ndim == 3 and img.shape[2] == 1)
           ^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'ndim'

Example 2:

$ file test.avif
test.avif: ISO Media, AVIF Image

from fiftyone.utils.image import read

filepath = "test.avif"
img = read(filepath)

Traceback (most recent call last):
  File "/home/laurent/Documents/project/scratch.py", line 810, in <module>
    img = read(filepath)
          ^^^^^^^^^^^^^^
  File "/home/laurent/Documents/project/.venv/lib/python3.12/site-packages/fiftyone/utils/image.py", line 37, in read
    return etai.read(path_or_url, include_alpha=include_alpha, flag=flag)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/laurent/Documents/project/.venv/lib/python3.12/site-packages/eta/core/image.py", line 481, in read
    raise OSError("Image not found '%s'" % path_or_url)
OSError: Image not found 'test.avif'

The errors from the traceback aren't also very helpful.

Workaround

In my usecase, I encounter this issue because I use fiftyone.Dataset.apply_model (which calls internally fiftyone.utils.image.read).

Adding an option to the internal of fiftyone.Dataset.apply_model to forward only the sample to the model could work. Essentially adding a only_needs_sample argument or something here:

fiftyone/fiftyone/core/models.py

Lines 328 to 334 in f40ab0e

	img = foui.read(sample.filepath)
	if needs_samples:
	labels = model.predict(img, sample=sample)
	else:
	labels = model.predict(img)

Since I'm creating my own model, I can handle the loading of the image myself.

Willingness to contribute

The FiftyOne Community welcomes contributions! Would you or another member of your organization be willing to contribute an implementation of this feature?

• Yes. I can contribute this feature independently
• Yes. I would be willing to contribute this feature with guidance from the FiftyOne community
• No. I cannot contribute this feature at this time

[brimoor]

Hi @Laurent2916 👋

Looks like OpenCV recently added support for loading AVIF images via the libavif library in opencv/opencv#23596. See related feature request opencv/opencv#19271 for additional details.

Hopefully this will get you up and running with apply_model() on AVIF images without needing to implement a different Model mixin which isn't able to leverage data loaders to more efficiently load images for batch inference 😄