Move the decompression bomb detection in the TIFF reader into into
ImageInput::check_compression_ratio() so it can be used uniformly for
other input types as well.
Assisted-by: Claude Code / Claude Sonnet 5
Signed-off-by: Larry Gritz <lg@larrygritz.com>
A Softimage PIC file stores a chain of channel packets, each carrying
its own bit depth (8 or 16). The reader assumed one uniform per-channel
size for the whole image, so a file mixing depths (e.g. 16-bit R with
8-bit G) wrote past the end of the scanline buffer.
Fix: always expose one uniform pixel format, the widest depth among the
file's packets, and promote narrower channels to it on read via exact
bit replication (v*257 for 8->16), which preserves the normalized value
exactly. This keeps the pixel layout naturally aligned; representing
channels with their true differing formats instead would pack them at
non-uniform, sometimes misaligned byte offsets.
Assisted-by: Claude Code / claude-sonnet-5
Signed-off-by: Larry Gritz <lg@larrygritz.com>
read_layer_record computed a layer's width/height as `std::abs((int)right
- (int)left)`. The rectangle coordinates are signed 32-bit values (layers
can have negative offsets), but were stored in unsigned (uint32_t)
struct fields, and subtracting two ints near the extremes overflows --
undefined behavior that UBSan flagged on a fuzzed file (1615681375 -
-1605526113 cannot be represented in type 'int').
Retype Layer's and MaskData's top/left/bottom/right fields as int32_t,
matching what they actually hold, instead of reinterpreting bits at the
point of use. Widen to int64_t before subtracting so the computation
can't overflow; the absolute difference of two int32 values always fits
in uint32, and validate_resolution still rejects implausibly large
extents.
The mask data rect (used for layer masks) had the identical
std::abs((int)right - (int)left) overflow hazard in load_layer_channel
and gets the same fix.
Assisted-by: Claude Code / Sonnet 5
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Read10bitFilled() unpacks a scanline of 10-bit "filled" (Method A/B)
samples backwards, and for 1-channel images applies a workaround that
swaps the first and third datum of each group of 3 packed samples (so
column order comes out right). The swap didn't check that a full group
of 3 remained in bounds. Guard the swap so it only fires when a complete
group of 3 remains.
Add a regression test in testsuite/dpx.
Assisted-by: Claude Code / Sonnet 5
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Find OpenCV via its exported CMake config (PREFER_CONFIG) so OpenCV 5 is
located correctly.
Modify our imagebufalgo_opencv.h to work with OpenCV 5. But how to test
whether we did it correctly?
We never had an actual test for the OpenCV interop. So set one up.
Assisted-by: Claude Code / Sonnet 5
Signed-off-by: Larry Gritz <lg@larrygritz.com>
One small improvement in how decode_exif detects corruption, to ensure
that the initial header examination doesn't read past the array bounds
-- by starting off with a check that the exif blob's size is at least a
big as a TIFFHeader, otherwise parsing it is pointless.
Then the rest of the changes are finding places where readers call
decode_exif but ignore the results, effectively ignoring corrupt Exif
blobs. That is in fact the default intended behavior (if the rest of the
image file can be read properly), but we are supposed to be checking the
global "imageinput:strict" attribute and considering it a hard error
when in strict mode. There were several places where we didn't handle
that case, in the JPEG, PNG, WebP, and Heif readers.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Adds a hint "raw:bad_pixels" which passes a filename for bad pixel data
to libraw.
I haven't added any tests, can't think of a good way of testing this.
Open to suggestions.
This functionality has been tested in a client code.
Signed-off-by: Anton Dukhovnikov <antond@wetafx.co.nz>
read_native_tiles() on a tiled EXR whose dimensions aren't a multiple of
the tile size wrote past the caller's buffer when reading an edge tile
range: both readers used a row stride padded up to a whole number of
tiles (nxtiles*tile_width) instead of the caller's actual rectangle
width, overrunning the buffer whenever the last tile column is partial.
Fixed in both exrinput_c.cpp (Core reader) and exrinput.cpp (classic
reader) to use the requested rectangle width as the destination row
stride instead. exrinput.cpp additionally needs the pre-clamp width
specifically, since read_native_tile() forwards a tile-aligned xend past
the image edge for ordinary full-tile reads.
Adds testsuite/openexr-partialtile, a compiled regression test that
calls read_native_tiles() directly for both readers -- the only path
that exercises this code -- and checks pixels and buffer bounds.
Assisted-by: Claude Code / Claude Opus 4.8
Signed-off-by: Larry Gritz <lg@larrygritz.com>
(The same problem and set of fixes were applied to both ImageInput and
ImageOutput. For brevity, we will only discuss ImageInput below.)
ImageInput error messages are stored in a thread_local map keyed by a
unique per-instance id (m_id). The destructor's erase of that entry was
commented out (over caution about the static destruction order fiasco),
so every ImageInput that accumulated an error and was destroyed without
its error being cleared via geterror() leaked one map entry. The id is
monotonically increasing, so entries never get reused and the map grows
without bound.
How did we find this? A fuzzer opening hundreds of thousands of corrupt
files -- each of which errors, and each ImageInput::create() attempt
tries several readers -- grew the map to tens of MB and eventually
OOMed. The leak was traced to this issue.
So we re-enable the erase, guarded against the destruction order fiasco:
the thread_local map is wrapped in a struct whose destructor clears an
'alive' flag, and the ImageInput destructor only touches the map while
it is alive. Each ImageInput removes only its own entry (keyed by its
unique id); other inputs' errors are untouched.
Cross-thread cleanup is still a known limitation -- errors accumulated
in a different thread than the one that descroys the ImageInput are
still leaked, noted in a TODO. However, this should be exceptionally
rare, since not only do you need the two-thread situation, but the one
that made the call that produced an error needs to not check for errors
and return/clear the message. So only badly behaving user code that
fails to check and retrieve errors should have this minor leak. And we
are in the process of adding compile-time warnings for not capturing the
error status values from our APIs.
Assisted-by: Claude Code / Claude Opus 4.8
Signed-off-by: Larry Gritz <lg@larrygritz.com>
A fuzzed JPEG2000 caused a heap-buffer-overflow (caught by ASan) when
copy_scanline read past the end of openjpeg's decoded component buffer.
The file's image canvas was 32x32, but one of its three components had a
horizontal subsampling factor dx=249 (larger than the entire image).
OIIO derives the ImageSpec size from the union of each component's
x0+w*dx / y0+h*dy window, so that bogus component inflated the spec to
249x32. copy_scanline then iterated x over the inflated width while
indexing a component whose data array only held 1 column, walking off
the end. The per-sample guard was also broken: it compared the component
row index against reference-grid offsets (wrong units) and used `>`
instead of `>=` on the column.
Two fixes:
1. Detect the corruption at open(): the JPEG2000 canvas (x1-x0, y1-y0)
is the authoritative image size, and a component's subsampling factor
must be nonzero and no larger than the canvas. Reject files that violate
this instead of decoding a nonsensical, inflated image.
2. Harden copy_scanline by bounds-checking each access against the
actual component array geometry and skip the channel (zero-fill) when
comp.data is null or a subsampling factor is zero, which also closes a
latent divide-by-zero on comp.dx/comp.dy.
Assisted-by: Claude Code / Claude Opus 4.8
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Previously, we did not have a PtexInput::valid_file() method, and our
open() method just directly called PtexTexture::open(), which is really
bad with validity checks and can crash or OOM with corrupted files.
But Ptex has a fixed 64 byte header (cheap to read and inspect) that has
lots of fields that let us more accurately check for reasonable file
validity. Therefore, we both implement valid_file, and also have open()
do these checks (rejecting many forms of corruption) before turning the
job over to the Ptex library itself.
Assisted-by: Claude Code / opus-4.8
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Change several build-time options from `option()` to our `set_option()`,
which mostly has the effect of allowing them to be overridden by
environment variables of the same name, which is really handy for CI
variants.
In the process, realized that we had some real problems with handling of
EMBEDPLUGINS=0, it probably hasn't worked properly for years and went
unnoticed because the one CI job variant that was trying to test
EMBEDPLUGINS=0 actually *was not* because it was trying to do it by
setting the environment variable, unaware that it could not be used to
set the option until this patch. So there is a bit of cmake and linkage
fixing that needed to be done to get it squared away.
Assisted-by: Claude Code / opus-4.8
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The existing "limits:imagesize_MB" rejects files whose total
uncompressed size exceeds a cap, but it cannot catch an image that is
tiny in one dimension yet absurdly large in another (e.g. an image
claiming 32 x 50331680, which slips under the 32 GB default). Add a
companion global attribute, "limits:resolution", that caps the maximum
number of pixels along any single dimension.
Like imagesize_MB, this is enforced only in ImageInput::check_open(),
intended for validity checks when reading untrusted files.
Applications may set the attribute larger or smaller than this default,
depending on their need to deal with images larger than these limits,
and/or willingness to risk mistaking a corrupted or maliciously crafted
image for a legitimate one on this basis.
Now then, with this in place, we can clean up some of the values we used
in passing valid resolution ranges to check_open(). That was originally
meant to express the true limits of what was valid in the file formats,
but had in many cases been conflated with common sense "what value is so
big that it's probably bogus". Now we can disentangle these ideas -- the
value passed to check_open is the definitional limit of the file format
itself, and the "limits:resolution" is the overrideable global policy
for what we consider more likely to be malicious/corrupted than merely
legit big.
Assisted-by: Claude Code / Claude Opus 4.8
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Validate channel-row byte availability before interleaving scanlines in
PSD reads, and fail gracefully with an error instead of risking
out-of-bounds access on malformed files.
Assisted-by: GitHub Copilot / GPT-5.3-Codex
Signed-off-by: Larry Gritz <lg@larrygritz.com>
- Add recently published security advisories to SECURITY.md
- Revision of security policy and definitions of vulnerability.
- Also some minor forward-ports and adjustments of CHANGES.
The substantive change is explaining that our policy is to only consider
vulnerabilities if they (a) allow remote code execution, privilege
escalation, and other very severe problems (not crashes, OOM, or most
"DoS" scenaries), and (b) are exploitable in practice by untrusted
actors merely by presenting malicious routine input such as image files
(misuse of APIs, for example, is not a vulnerability).
Everything else should be reported through the usual Issue/PR mechanism,
and not use the security vulnerability reporting regime, which is
massively more overhead for the maintainers.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
PRs #5260 and #5254 interact: #5254 exposed the default type arg on the
module getattribute (`"type"_a = TypeUnknown`), so a one-arg call now
returns None instead of raising TypeError. #5260's test still expected
the old pybind behavior. Update the test and ref to the intended new
behavior.
Assisted-by: Claude Code / Claude Opus 4.8
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Exploring an idea Anton had about unifiying two python backends into
single source since they are so similar.
## Summary
This PR unifies the early nanobind migration with the existing pybind11
Python bindings by compiling **shared sources** from `src/python/` for
both backends, instead of maintaining parallel copies under
`src/python-nanobind/`.
Migrated modules so far: `ROI`, `TypeDesc`, `ImageSpec`, `ParamValue` /
`ParamValueList`, plus the core `OpenImageIO` module helpers in
`py_oiio.cpp`. Backend differences are isolated in `py_backend.h` (type
aliases, binding API macros like `.OIIO_PY_RW(...)`, and small
`oiio_py::` helpers). Shared Python↔C++ conversion logic lives in
`py_oiio.h` with `#if` only where the backends genuinely differ.
CMake is updated so that when `-DOIIO_PYTHON_BINDINGS_BACKEND=nanobind`
is selected, the module is built from the unified sources and installed
as the main `OpenImageIO` package in the expected location. With `both`,
pybind11 remains the primary module and nanobind builds as
`PyOpenImageIONanobind` for side-by-side testing.
Duplicate nanobind-only `.cpp` files are removed (~1.8k lines deleted
net). Testsuite coverage is extended with explicit roundtrip checks for
shared helper paths (attributes, buffers, tuples).
---
## Pros
- **Single source of truth** for migrated bindings — no more keeping two
copies of `py_roi.cpp`, `py_imagespec.cpp`, etc. in sync by hand.
- **Lower risk of backend drift** — a fix or API tweak in a migrated
module automatically applies to both pybind11 and nanobind.
- **Net code reduction** — removes a large duplicated tree under
`src/python-nanobind/` while centralizing shared logic in one place.
- **Clearer migration path** — `MIGRATION_STATUS.md` documents what is
fully migrated, partially migrated, and still pybind-only.
- **Better nanobind-only build layout** — on `main`, choosing nanobind
does not install into the same package layout as pybind11; this PR fixes
that for the migrated surface.
- **Side-by-side validation** — `OIIO_PYTHON_BINDINGS_BACKEND=both`
builds both modules so existing `testsuite/python-*` and `*.nanobind`
variants can be run against the same C++ sources.
- **Explicit parity tests** — new/extended tests cover shared conversion
paths (typed attributes, buffer→vector, tuple roundtrips) rather than
relying on import smoke tests alone.
- **Maintainable binding style** — method-name macros
(`.OIIO_PY_RW(...)`) keep clang-format happy without `clang-format off`
regions.
---
## Cons
- **Added abstraction in the pybind11 path** — even pybind-only builds
now go through `py_backend.h`, macros, and shared helpers; slightly more
indirection than plain `.def_readwrite(...)`.
- **Regression risk for pybind11** — adapting existing pybind code to
work for both backends could subtly change pybind11 behavior at the
edges (init, casting, error paths, ownership).
- **Incomplete migration** — most of the Python API (`ImageBuf`,
`ImageInput`/`ImageOutput`, `ImageBufAlgo`, texture system, etc.) is
still pybind-only; this PR is a foundation, not full nanobind coverage.
- **Known nanobind gaps remain** — module-level exports and helpers
listed in `MIGRATION_STATUS.md` are still missing or intentionally
different vs pybind11.
- **Higher review and test burden** — reviewers need to think about two
backends; CI/local testing should cover both pybind and `.nanobind`
variants for touched code.
- **Conditional compilation complexity** — `#if
defined(OIIO_PY_BACKEND_NANOBIND)` remains in a few places (e.g.
`ParamValue` constructors, buffer dispatch); easy to get wrong if
extended carelessly.
- **Slightly less obvious binding code** — `.OIIO_PY_RW("x", ...)` reads
less directly than `.def_readwrite("x", ...)` until you know the macro
map.
---------
Signed-off-by: Aleksandr Motsjonov <soswow@gmail.com>
Run check_open on JXL basic-info dimensions/channels before decode
output allocation to prevent oversized allocation paths on malformed
input.
Also update ref output for libjxl 0.12.0.
Assisted-by: GitHub Copilot / GPT-5.3-Codex
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Find corruptions by validating layer resolutions.
* Find corruptions when eof is hit in the middle of a string.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Truncating a corrupt uint32 length_tab entry into `int` could yield a
huge negative value, causing a runaway allocation and crash in
uncompress_rle_channel. Validate offset/length against the file size
instead.
Assisted-by: Claude Code / Sonnet 5
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This aligns FFmpeg/libswscale destination image buffers and validates
the buffer-size, frame-fill, and swscale return paths before decoded
pixels are consumed.
The change prevents narrow high-bit-depth VP9/GBR inputs from writing
past OIIO's destination buffer during swscale conversion, and adds a
regression case for the width-1 high-bit-depth FFmpeg path.
Signed-off-by: Dongju Lee <dongpago2@gmail.com>
This handles malformed GIF files whose logical canvas dimensions can
overflow legacy int-indexed canvas calculations.
The patch rejects oversized canvas dimensions before scanline/canvas
indexing and adds a regression test for the malformed oversized-canvas
case.
Signed-off-by: Dongju Lee <dongpago2@gmail.com>
This treats non-positive tiff:RowsPerStrip metadata as absent before
using it to align generic scanline read/write chunks. That avoids
feeding invalid chunking metadata into round_to_multiple while
preserving normal positive RowsPerStrip behavior.
The patch adds regression coverage for an OpenEXR file carrying
RowsPerStrip=0 through iinfo --hash, iinfo --stats, and an oiiotool
conversion path.
---------
Signed-off-by: Dongju Lee <dongpago2@gmail.com>
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Co-authored-by: Larry Gritz <lg@larrygritz.com>
GifSplitPalette() computed `numPixels * (splitElt - firstElt)` as a
32-bit int before dividing. For any image over roughly 16.9M pixels
(e.g. 4117x4117), that intermediate multiplication overflows INT_MAX and
is undefined behavior; in practice it produced a bogus negative/garbage
split count that could drive later indexing into `image` out of its
bounds.
Do the multiply in 64 bits and narrow back afterward -- the quotient is
always <= numPixels, so the narrowing itself is safe.
Assisted-by: Claude Code / Sonnet 5
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
TBHD width/height/channel fields were never cross-checked against how
much tile data the file actually contains, so a corrupt header could
claim an arbitrarily large image (e.g. 16k x 16k) backed by only a few
hundred bytes of real data. readimg() would then resize() a
multi-gigabyte buffer for a tiny malformed file, hanging (or very slowly
succeeding) instead of failing -- a resource-exhaustion DoS on untrusted
input, found via fuzzing.
Bound the claimed image size against how much tile data remains in the
file, generously allowing for RLE's worst-case 128x expansion ratio, and
fail fast if it's implausible.
Assisted-by: Claude Code / claude-sonnet-5
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
CineonInput::open() accepted any per-channel bit depth in [1,32], but
libcineon's ComponentDataSize()/ComponentByteCount() only recognize
{8,10,12,16,32,64} and assert(0) on anything else. A crafted file with
an unsupported bit depth (e.g. 26) passed the OIIO-side check and then
aborted deep inside the vendored library instead of failing cleanly.
Tighten the check to libcineon's actual whitelist so bogus bit depths
are rejected with a proper errorfmt() before ever reaching the vendored
code. Adds a regression fixture (broken_bitdepth2.cin) to the cineon
testsuite.
Assisted-by: Claude Code / Sonnet 5
Signed-off-by: Larry Gritz <lg@larrygritz.com>
open_raw set imgdata.rawparams.max_raw_memory_mb *after* calling
m_processor->unpack(), so LibRaw's allocation cap never protected the
unpack step -- which is exactly where LibRaw allocates the raw image
buffer. A corrupt raw file claiming huge dimensions could therefore
allocate unbounded and OOM (showing up on the fuzzer as a 0-length crash
artifact from an out-of-memory trip rather than a normal reproducer).
Move the max_raw_memory_mb assignment to immediately after open_file
succeeds, before unpack(), so the cap (default 2048 MB) actually bounds
the allocation. LibRaw then fails the file cleanly instead of OOMing.
Assisted-by: Claude Code / Claude Opus 4.8
Signed-off-by: Larry Gritz <lg@larrygritz.com>
While working on
https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/5254,
where we're introducing changes that could surface bugs in pybind11 code
that's in production today, I figured I'd try adding some tests.
This was a fairly loose-handed effort. To be upfront: most of it was
LLM-generated. I did some research into which modules were lacking
coverage the most, and those are what this PR targets. I haven't gone
through every line of the generated code.
I hate being asked to review something that was clearly vibe-coded with
no real effort put in, so I'm saying that here rather than pretending
otherwise. I can try to review it properly, but without deep codebase
knowledge it'll be slow and painful. Since these are just tests, my
thinking was: even if there are silly bits here and there, it shouldn't
hurt much, as long as things go green and we cover more logic and
branches. Worth offering as-is.
---------
Signed-off-by: Aleksandr Motsjonov <soswow@gmail.com>
We had missed a case where bit_convert needed outputs to go to the
scratch buffer: PHOTOMETRIC_SEPARATED, but not requesting raw color.
(Because the real number of channels differ from what we are presenting
to the caller.)
Assisted-by: Claude Code / Sonnet 5
Signed-off-by: Larry Gritz <lg@larrygritz.com>
10MB was conservatively chosen as big enough that anything larger is
definitely bogus for an ICC, Exif, or XMP block in a PSD file.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
read_native_scanlines declared task_set before the locals its queued
decompression lambdas capture (ok, compressed_scratch). On the
leftover-scanline early-return path those locals were destroyed before
~task_set() waited for the queued work, so workers could write through a
dangling ok and hand a freed compressed buffer to zlib.
Declare task_set after the captured locals so its destructor wait runs
while they are still alive, and propagate async decompression failures
via return ok instead of unconditionally returning true.
Assisted-by: Claude Code / opus-4.8
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
validate_header() let the oiio:RawColor/psd:RawData path return true
before the switch that rejects unknown m_header.color_mode values, so an
out-of-range or "hole" (undocumented) color mode reached setup(), which
unconditionally indexes the fixed 10-entry mode_channel_count /
mode_channel_names tables with the attacker-controlled value -- a global
out-of-bounds read (and, per external report, a path to unbounded/bogus
allocation) reachable via the public RawColor config attribute on a
crafted PSD.
Move the "is this a color mode we know about at all" check ahead of the
RawColor early return so it always runs, and keep the existing
Duotone/Lab-unsupported-without-raw check as a second pass after the
early return, preserving current behavior for every valid mode.
Assisted-by: Claude Code / claude-sonnet-5
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
An 18-byte TGA file consisting of just a header (no pixel data) can
claim an arbitrary width/height/bpp -- e.g. a 1472x12000 24bpp image --
and readimg() would allocate the full ~53MB buffer and only discover the
truncation once the first read hits EOF. Worse, the resulting
partially-formed ImageSpec could still propagate to a downstream writer
before the read failure was acted on.
Cross-check the header's width/height/bpp against how many bytes of
pixel data the file could plausibly contain before committing to the
allocation, mirroring the existing pattern in iff.imageio and
jpeg2000.imageio. The check compares against the *on-disk* pixel size
(palette/16-bit storage is smaller per pixel than the decoded, unpacked
ImageSpec) and allows up to 128x expansion for RLE-compressed images (a
1-byte run-length count can encode up to 128 output pixels).
Updated testsuite/targa/ref/out.txt: several existing corrupt-file
regression cases (crash1.tga, crash6.tga, crash3952.tga) are now caught
earlier with clearer messages, and two real oiio-images TGA files
(UCM8.TGA, UTC16.TGA) that use palette/16-bit storage smaller than their
decoded size continue to read correctly.
Assisted-by: Claude Code / Sonnet 5
Signed-off-by: Larry Gritz <lg@larrygritz.com>
PngReadCallback passed input->geterror(false).c_str() to
png_chunk_error() when a PNG read failed. png_chunk_error() does not
return -- it longjmp's back to libpng's setjmp point, which skips C++
destructors, so the temporary std::string returned by geterror() was
never freed. The fuzzer surfaced this as a small, intermittent leak (a
few KB across hundreds of thousands of runs).
Copy the message into a plain stack buffer (no destructor) before
calling png_chunk_error, so nothing heap-allocated is alive across the
longjmp.
Assisted-by: Claude Code / Claude Opus 4.8
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The mask-based uncompressed decode path copies m_Bpp bytes into a
uint32_t pixel variable. m_Bpp is validated up to 16 (for 128-bit DXGI
formats), but the legacy fmt.masks[] are uint32_t and can only address
32 bits. A corrupt file with m_Bpp > 4 caused a stack-buffer-overflow.
Clamp the copy to sizeof(pixel) = 4 bytes, which is all the masks can
reference anyway.
Assisted-by: Claude Code / claude-sonnet-4-6
Signed-off-by: Larry Gritz <lg@larrygritz.com>
A DDS file with a garbage pixel-format bpp field (e.g. fmt.bpp ~2.5e9)
yields a native bytes-per-pixel (m_Bpp) of hundreds of millions.
internal_readimg() then does `new uint8_t[width * m_Bpp]`, a multi-GB
allocation that OOMs. The core resolution/imagesize limits do not catch
this: they only see the small output ImageSpec, not the native pixel
size, and the existing fmt.bpp check is gated on RGB/LUM/YUV/ALPHA flags
that a corrupt header can simply omit.
Validate m_Bpp for uncompressed formats right after it is computed in
open(): the largest real uncompressed DDS/DXGI pixel is 128-bit (16
bytes), so anything outside [1,16] is a corrupt header and is rejected
before any allocation. The check is confined to uncompressed formats
because m_Bpp is neither set nor used for compressed (block) formats,
which size their data via GetStorageRequirements.
Assisted-by: Claude Code / Claude Opus 4.8
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Detect and reject corrupt chunk sizes, flags, and channel configurations
in the header.
Assisted-by: Claude Code / Claude Opus 4.8
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Somehow the check_open validation was only in exrinput.cpp (using the
old C++ AP), but not exrinput_c.cpp (using the C "Core API").
Also adjust the allowable z resolution to reflect that OpenEXR doesn't
support volumetric images. (This was obviously a cut and paste error
long ago.)
Signed-off-by: Larry Gritz <lg@larrygritz.com>
read_offset_tables() sized the RLE start/length tables from the raw
header (`ysize * zsize`), but read_native_scanline() indexes them by the
spec dimensions (`height * nchannels`, derived from the image's
dimension field). A corrupt header (e.g. dimension=1 with zsize=0) makes
`ysize*zsize` = 0 so the tables are empty, while indexing still uses
`height*nchannels` = 1 -- a null-pointer dereference and crash.
Size the tables by `m_spec.height * m_spec.nchannels`, the same quantity
used to index them, so the two can never desync. For well-formed files
this equals ysize*zsize, so behavior is unchanged; a corrupt file now
fails cleanly when the (too-short) table read hits EOF. Also guard
against non-positive table sizes, use vector::data() instead of `&v[0]`
(UB on empty), and fix an off-by-one in the scanline bounds check (y ==
height would index start_tab[-1]).
Add the fuzzer reproducer as testsuite/sgi/src/broken02.sgi.
Assisted-by: Claude Code / Claude Opus 4.8
Signed-off-by: Larry Gritz <lg@larrygritz.com>
`merge_deep_pixels()` calls `copy_deep_sample()`, which has an error
status return, so change `merge_deep_pixels()`'s return type from void
to bool.
For OIIO_NODISCARD_ERROR, all internal calls that previously ignored
return values are handled:
- OIIO_CONTRACT_ASSERT where the call can't fail in context, or the
caller is a void function or a constructor.
- `return false` where the caller already returns a bool error state.
- `(void)` cast for the deprecated internal overload.
Follow-up to #5252
Part of #4781
Signed-off-by: Luna Kim <177369799+luna-y-kim@users.noreply.github.com>
Upgrade the embedded/vendored pugixml to 1.16, also the default version
that we use for CI with build_pugixml.bash.
Raise PugiXML minimum to from 1.8 to 1.11 (late 2020, over 5 years ago).
Rely completely on pugixml exported cmake config, eliminate our
Findpugixml.cmake. This is possible because we raised our minimum
pugixml to 1.11, which is the point where its exported cmake configs
were reliable.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Add scripts for openjph auto-build when not found (and
OpenImageIO_BUILD_MISSING_DEPS specifies it, or "all"). Build openjph
0.30.1 when missing.
Since we can build it when missing, take away the various missing
openjph allowances from our CI; auto-build when needed for all variants.
More thorough exception catching for openjph in our jpeg2000input.cpp.
When I compile openjph with clang (including icx), I get some runtime
exceptions. Not when I build it with gcc! I don't have the time or
inclination to investigate why, so for our CI, rig it to build openjph
with gcc even when I'm building OIIO with clang/icx.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Circular IFD chain loop detection (identified by for CVE-2022-3970) was
fixed in libtiff by MR 386
(https://gitlab.com/libtiff/libtiff/-/merge_requests/386), released in
libtiff 4.5.0.
Raise the recommended minimum from 4.2 to 4.5 so users on older
installations get a build-time advisory. The hard minimum stays at 4.0
(for now) to preserve compatibility.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
IOMemReader::pread guarded its memcpy with `size + size_t(offset) >
m_buf.size()`. For a large or negative offset, that addition wraps
around size_t and can compare as in-bounds, skipping both the error path
and the size clamp, so the memcpy reads past (or before) the buffer.
Assisted-by: Claude Code / Claude Opus 4.8
Signed-off-by: Larry Gritz <lg@larrygritz.com>
When creating an ImageInput, if the initial guess (based on the file
extension) fails to open, we try all the others in turn.
This tries to make that a little cheaper by not doing a full open() but
instead first checking for valid_file, which in theory is a much less
expensive approximate check.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
read_native_scanline computed the scanline's file position as
m_bmp_header.offset + y * m_padded_scanline_size and seeked there
without checking it against the file size. A corrupt header (bogus
pixel-data offset or scanline size) can place that position far outside
the file.
Assisted-by: Claude Code / Claude Opus 4.8
Signed-off-by: Larry Gritz <lg@larrygritz.com>
- Detect corrupted userbuf size.
- Protection against integer multiply overflow in various places in the
DPX reading code.
- Also fix memory load alignment issue. Didn't seem to be a problem in
practice, but the sanitizers weren't happy on some platforms.
Assisted-by: Claude Code / sonnet-4.6
Signed-off-by: Larry Gritz <lg@larrygritz.com>
IBA::make_texture() only passes the full metadata to the first level of
a MIPmap, and that might include output behavior hints, too! Make this
hint "sticky" in the TIFF writer, so if any subimage's first level's
spec enables half output, all the MIP levels of that subimage will allow
it.
Fixes#5239
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Carelessly written hdr/rgbe files, or correctly written ones that are
checked out by git on windows if you're not careful, might use Windows
CRLF line endings. Be silently tolerant of that.
Assisted-by: Claude Code / opus-4.8
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We were allocating a temporary buffer big enough for the RGBA but
neglecting the z. Only for the 16 bit case.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Guard against corrupted images, which if they said they had more than
the 8 channels allowed by Cineon format, would overrun array bounds
while reading the file. And check for valid bit depths.
Also opportunistically fix a bug I spotted where a red channel was
incrementing the count of greyscale channels rather than the count of
red channels -- which must have been a copy and paste error from
literally 15 years ago. (But those counters are only used for channel
naming, and only if there is more than one greyscale or red channel in
the file. So I doubt anybody has ever hit this bug, or ever would have
in the future.)
Fix possible memory leak by having init() call `m_cin.SetInStream()`
with a null handle to clear out the internals.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
When copy-constructing an ImageCache-backed ImageBuf, the "cache-based
or deep" branch set `m_bufspan = {}`, a default-constructed image_span
whose strides are all zero. retile() addresses a pixel within a cache
tile as `offset = pixelindex * m_bufspan.xstride()`, so a zero xstride
made every pixel resolve to tile offset 0 -- the whole copied image read
back as a constant color (the value of pixel 0,0).
Fixes#5243
Added a regression test (test_copy_of_imagecache_backed) that copies a
cache-backed buffer with distinct per-pixel values and verifies the copy
does not read back a constant.
Assisted-by: Claude Code / Claude Opus 4.8
Signed-off-by: Larry Gritz <lg@larrygritz.com>
To avoid breaking ABI, the old int overload is kept, but its declaration
is hidden behind `#ifdef OIIO_INTERNAL`. Downstream code only sees the
int64_t version, but old binaries still work. The forwarding wrapper
handles it until the next first-digit version changes.
Also, the Python binding is adjusted using a wrapper to resolve the
overload ambiguity. While adding the wrapper, I noticed other bindings
still took `int pixel`, so widened them to `int64_t` for consistency.
Follow-up to #2363Fixes#5242
Signed-off-by: Luna Kim <177369799+luna-y-kim@users.noreply.github.com>
- Fix sort(), incorrectly using the front Z channel in a place where it should have instead used Zback channel.
- Widen pixel indices from int to int64_t (internal only, follow-up to #2363).
- Cleanup unreachable branches in merge_overlaps().
Modified the `DeepData::sort` test to include samples with a tied front
Z, so it verifies the Zback fallback.
Signed-off-by: Luna Kim <177369799+luna-y-kim@users.noreply.github.com>
FITS files are allowed to have multiple "header blocks".
Unfortunately, as originally written, our fits input used *recursion* to
read them. If there is a file with too many header blocks, it can crash
with a stack overflow.
Change to use a loop instead of recursion. And put a maximum iteration
limit of 10000 headers in one file before giving up.
(Note to reviewers: the diff looks bigger than it really is. By
surrounding the contents of read_fits_header with a `for` loop, it
reindented it, and that is the vast majority of the change. I wish
GitHub review diffs more clearly visualized which changed lines are only
whitespace changes.)
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Bump the "bleeding edge" test to C++26 (from 23), and the "latest
releases" Linux test to C++23 (from 20). There are still other variants
that test C++20 including anything based on the VFX2026 platform.
Just one tiny change needed to compile in C++26 mode, related to a
deprecated std type_traits template.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The InterpretingDeepPixels reference link expired. It moved to an
HTML page on the OpenEXR docs site, so update the comments to the
working URL.
Signed-off-by: Luna Kim <177369799+luna-y-kim@users.noreply.github.com>
This suppresses the warnings in the fmt library code which have started
appearing recently when building on a Mac.
I suspect the warnings were triggered by the clang toolchain being
updated to v21, but to be fair I haven't tried to downgrade to confirm
that.
No extra tests. The changes allow OIIO to build with "exit on warning"
on Mac with the latest build tools, whereas without the changes the
build fails.
Signed-off-by: Anton Dukhovnikov <antond@wetafx.co.nz>
python_tests was from 2010 and either was checked in inadvertently, or
has long, long been supplanted by our real testsuite feature.
The various TRAVIS_SKIP_LSAN file used to mark certain tests as
skippable when running the leak sanitizer... but we long ago removed the
code in runtest.py that recognized its presence and took action. So it
has served no purpose for quite some time.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
OpenImageIO uses the fmt library, but spdlog is also quite common and
uses the fmt library. This fix should allow to use precompiled fmt
library in both spdlog and oiio if they are used in same project.
Added `OIIO_USE_COMPILED_FMT` CMake option to allow linking against a
compiled fmt (fmt::fmt) instead of the header-only variant.
Update externalpackages.cmake to expose the option, select the
appropriate fmt target and include dir, and pass the value into
generated headers via oiioversion.h.
Adjust Config.cmake.in to avoid forcing find_dependency(fmt) when
header-only behavior is overridden. Modify detail/fmt.h to skip defining
`FMT_HEADER_ONLY `when compiled fmt is used, and update libutil
CMakeLists and ustring.cpp to link/use the compiled fmt target and use
fmt::runtime for dynamic format strings.
Assisted-by: Codex 5.5 ExtraHigh
---------
Signed-off-by: Vlad <shaamaan@gmail.com>
Also make our CONTRIBUTING guide link to, and be consistent with,
our SECURITY.md policy file.
Follow OpenEXR's lead by explaining briefly that we only consider
something to be a security issue if it can plausibly be triggered by
ordinary inputs coming from untrusted parties. For example, a
maliciously crafted input image may be a vulnerability, but incorrect
use of an API call is not, because only trusted parties should have
direct access to the APIs.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Some minor fixes and inconsistencies.
- Per the ASWF [repo best practices
guide](https://tac.aswf.io/project_best_practices_guides/repo.html), the
project's technical charter should NOT be redundantly stored in our
repo, but instead we should just link to the canonical copy that the
ASWF maintains.
- Fix incorrect relaive path to CONTRIBUTING.md.
- Fix leftover "interim" designations from TSC members.
- Remove Lewis Wakeland from the TSC roster (he retired some months
back, but somehow we neglected to remove him from the file)
- Add a link to the Google doc with meeting notes.
- Remove stray reference to a 2024 release.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
According to C++26, some newly required std container hardening for
std::span and std::string_view were not reflected in our own analogous
types. In particular, some more precondition checking was required for
span::subspan, first, last, and string_view::remove_prefix and
remove_suffix in order to match the expectations of true parity with
with the std versions.
This is important because some day, we will probably discard our own
string_view and span for the std ones (especially when we are using
compilers in which they are fully hardened).
I found a couple spots where we used remove_prefix and remove_suffix
that assumed the range would be clamped, as it is in our version, rather
than contractually require them to be valid ranges as in std. This would
eventually fail if replaced by std::string_view, so fixing now.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Bump the "bleeding edge" variant to gcc-16, and the "latest releases"
variant to gcc-15.
One minor fix needed to make gcc16 happy -- it is better than earlier
versions at discovering variables whose values are not used.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Mostly for the sake of bots (but also humans) reading plugin.h and
trying to figure out exactly what each function does, just plainly state
which Unix-like dlopen/dlsym/etc function it is the equivalent of.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
It was throwing things off to set CMAKE_CXX_COMPILER in this way,
wrecking the ability to just run as a utility from the command line
without it being set. It was not necessary, as far as I can tell.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Adds agent configuration supporting Claude Code, Cursor, GitHub
Copilot, OpenAI Codex, and Opencode. A single AGENTS.md and contents
of .agents subdirectory serve as the canonical source for
instructions, skills, and other things that can be used across
multiple tools.
A script can be invoked by users to do the local setup of links or
references to allow them to use a particular tool:
`.agents/setup-agent TOOL` Currently supported tools by this script
include Claude Code, Codex, Cursor, GitHub Copilot, and OpenCode.
This initial commit includes two skills that I wrote and have used for
the last couple monthly releases: release-notes-update and
prepare-patch-release. I assume that we will add more over time as we
discover tasks that we want to make standard skill automation for.
Assisted-by: Claude Code / claude-sonnet-4-6
The AGENTS.md, Claude skills, and general scheme are entirely from me.
I used Claude to help me establish the setup script for the other
tools I don't happen to use.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Fixes indexed-color PSD transparency handling when the transparent
palette index is `0`.
Before this change, the PSD reader used `m_transparency_index` both as the
palette index value and as the signal that a transparency-index resource was
present. That made index `0` ambiguous: it is a valid transparent palette index,
but it evaluated as false when deciding whether to add an alpha channel.
As a result, an indexed PSD with transparency index `0` could expose a
3-channel output spec, then later `indexed_to_rgb()` would treat
`m_transparency_index >= 0` as meaning transparency exists and write 4
bytes per pixel. That could write a 4-channel row into a caller buffer sized for
3 channels.
The fix separates those two pieces of state:
- `m_has_transparency_index` records whether the transparency resource exists.
- `m_transparency_index` stores only the numeric palette index.
- Index `0` now correctly adds an alpha channel.
- Transparency indexes outside the valid palette range `0..255` are rejected.
- `indexed_to_rgb()` now checks source and destination spans before writing.
### Tests
Added regression coverage to `testsuite/psd-colormodes`.
The test now generates indexed PSD variants from the existing
`pattern2-8-indexed.psd` fixture with transparency indexes:
- `0`: valid, should report/read as 4-channel PSD.
- `255`: valid, should report/read as 4-channel PSD.
- `256`: invalid, should be rejected.
Validated locally with:
```bash
python3 -m py_compile testsuite/psd-colormodes/run.py \
testsuite/psd-colormodes/src/make-indexed-transparency-psds.py
ninja -C build-psd oiiotool iconvert idiff
```
Also ran the generated PSD cases and a manual full psd-colormodes test;
final out.txt matched the updated reference
Assisted-by: Codex GPT5.5xHigh
---------
Signed-off-by: Vlad (Kuzmin) Erium <libalias@gmail.com>
Fixes#5207
Sometimes the `WxH+X+Y` and `+X+Y` notation (borrowed from X11!) can be
a little awkward. This patch extends the accepted syntax to allow commas
to separate the offset components, for example also accepting `WxY,X,Y`
and `X,Y`.
This affects the commands
--create
--crop
--cut
--fit
--fullsize
--origin
--originoffset
--paste
--pattern
--printstats
--resize
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
This patch optionally frees the contents of the ustring table as the
process is exiting. It's triggered one of two ways:
- `OIIO::attribute("ustring:cleanup", 1);`
- Environment variable `OIIO_USTRING_CLEANUP=1`
If neither is used, the teardown won't happen, thus avoiding any
pointless work when it should be exiting the process.
The purpose of this is that when using certain debugging tools, the
ustring table (which only grows and never frees) looks like a memory
leak, even though it's by design. Someimes, it's useful to make it not
appear ot leak, and you are happy for a large table to spend time
pointlessly freeing all the data.
Fixes#3913
Signed-off-by: Larry Gritz <lg@larrygritz.com>
In main (working toward the late-2026/2027 releases) these 2022
containers are both outside the promised 3 year support window, and also
are using CentOS 7-based containers that are essentially irrelevant now
that all the studios have moved to RHEL9 (and Rocky/Alma) compatible OS
distros. So there is really no need to continue CI testing on them for
branches aimed at the future.
Note that we still have Ubuntu-based "oldest gcc" and "oldest clang"
jobs, so dropping this does not actually impact our ability to test
against the oldest gcc, C++, and python that we claim to support.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
New release of libheif, which continues to live up to its habit of
changing pixel values ever so slightly with each release.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Background: this flag was originally added in commit c5494ac06
(2015-11-24) with the message "...now builds linux+gcc...". It was later
refactored in commit 62d7164d3 (2017-01-23). From investigation, it
seems to be preventative rather than reactive because it does not
actively suppress warnings, but `-Werror` is enabled under the condition
NOT MSVC + STOP_ON_WARNING (originally in CMakeLists.txt, now in
compiler.cmake:126).
Compile tests without `-Wno-unused-result` but with `-Wunused-result`
explicitly enabled were conducted on the following commits, and all
compiled successfully without unused result warnings. Also, tests
confirmed that the warning triggers if there is an unused return value.
- 312793af4 (2026-01-01)
- 682825f9d (2025-01-01)
- 48362b26c (2024-01-01)
- 3b8807df8 (2023-01-01)
- 62d7164d3 (2017-01-23): refactored
- c5494ac06 (2015-11-24): originally added
Decision: the flag can be safely removed. Removing it allows gcc to
catch unused return values, which is required for the `OIIO_NODISCARD`
and `OIIO_NODISCARD_ERROR` macros to work as intended.
Related: #5196
Assisted-by: Claude / Opus 4.7
Signed-off-by: Luna Kim <177369799+luna-y-kim@users.noreply.github.com>
This message is triggered by the check that the tag of an auto-build
dependency matches the SHA hash we use as a cross-check. This patch is
just amending the error message to reveal the way to override the check,
so people (like me) don't have to go hunting for it if they are
purposely trying to build from a nonstandard commit of a dependency.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
I recently discovered that OCIO's built-in configs' "file rules" that
come into play when using getColorSpaceFromFilepath actually do not
incorporate the search for color space name embedded in the filename
into the logic, and moreover, assume that files ending in ".exr" are
ACES2065-1, of all things. This seems an unfortunate choice because in
the real world, almost no exr files people will encounter actually use
the ap0 primaries.
This PR does the following to shore up this and related edge cases:
* For OCIO built-in configs, fully replace the existing file rules with
a blank slate that just uses the path parsing rule. Everything else is
categorized as "unknown". (Though many of our format readers already do
something better, setting the metadata if they actually know what the
file is.)
* Recognize "unknown" as equivalent to having an empty color space, an
error if you pass such a color space name to IBA color transform
functions.
* For all IBA color conversion, if the input space is not color managed,
be sure NOT TO mark the output color space as the original `to` request!
Input raw means output is also raw.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This is an emergency fix on top of #5201.
PR #5201 passed CI completely. Then I merged it, and now it fails. How
did that happen?
OOOOOOH. The imagespeed_test.cpp is not compiled at all for CI! Only for
local builds. So a build break in it will not be caught in CI. OK,
fixing that, too. It doesn't run in CI, it's not that kind of test, it's
a special benchmarking test meant to run locally and needs to be told a
specific image file to test with. But at least build it for CI so we
can't inadvertently break it again.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
- Annotate all ImageOutput write methods (write_image, write_scanline,
write_scanlines, write_tile, write_tiles, write_rectangle,
write_deep_scanlines, write_deep_tiles, write_deep_image), as well as
open, close, and copy_image, with OIIO_NODISCARD_ERROR.
- Also fix the handful of internal callsites (imagebuf.cpp,
maketexture.cpp, py_imagebuf.cpp, iconvert.cpp, imagebufalgo_test.cpp,
and imagespeed_test.cpp) that were previously ignoring the return value
of close(), either by propagating the error or by casting to void where
the error path is already being handled.
- Reference: #4781
### Tests
- tested against unit_imageinout, unit_imagebuf, unit_imagebufalgo
Assisted-by: Claude Sonnet 4.6
---------
Signed-off-by: rose413 <116857309+rose413@users.noreply.github.com>
##### PNM
The PNM reader did not provide an optimized `valid_file` implementation.
While `PNMInput::open` would perform basic validation of the PNM header,
it would only do this after having read the entire contents of the file
into memory. Besides the obvious risks/issues with having `open`
unconditionally read arbitrarily large files into memory, this also
meant that any call to `valid_file` would necessarily do the same (as
the default implementation delegates to `open` when ioproxy support is
present).
To fix these issues, the following changes were made:
* Provide an efficient implementation of `valid_file` for `PNMInput`
that only loads the header, and then validates magic number/dimensions.
* In `PNMInput::open`, first load only the header to memory, and
subsequently, only load the rest of the image data if the header is
parsed successfully, and the data contained within is valid.
* Added a rough limit to header read size of 1KiB.
* Added a rough limit to full file read size of 1GiB.
Note that, for now, if the file size exceeds the 1GiB limit, the read is
simply truncated to 1GiB, rather than failing altogether.
##### JPEGXL
The JXL reader already provided an efficient `valid_file` override,
implemented by validating the 128 byte JXL signature. The signature,
however, was not being validated in `JxlInput::open` before attempting
to decode the file.
The process of JXL decoding appears to read the entire contents of the
file to memory (at least for some non-JXL inputs). Taken in combination,
this means that when `JXLInput::open` was called on arbitrarily large
non-JXL inputs, the entire file was being read into memory before `open`
would fail.
As a simple fix for this, `JxlInput::open` now checks the result of
`valid_file` before attempting any decoding.
### Tests
I did not add a new testsuite case for this, but did test on my end that
for both PNM and JPEGXL, large invalid files are no longer being read
into memory, while valid images (of their respective types) are still
read/opened successfully.
---------
Signed-off-by: maxwelliverson <maxwelliverson@gmail.com>
Adds `OIIO_NODISCARD` and `OIIO_NODISCARD_ERROR` to `color.h`.
Deprecated functions are left unannotated.
Part of #4781
Assisted-by: Claude / Opus 4.7
---------
Signed-off-by: Luna Kim <177369799+luna-y-kim@users.noreply.github.com>
Our CI job that builds against VFX Platform 2023 via docker container
aswf/ci-osl:2023-clang15 broke after some dependency changes in a recent
aswf docker container update. It's not worth trying to "fix" an old
year, just lock down to the previous version of the container that is
known to work.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This PR adds a `--nchannels` flag to oiiotool, providing parity with the
existing flag in maketx.
### Problem:
When processing images from varied sources in automated pipelines, using
name-based channel selection (e.g., --ch R,G,B) is fragile. If an input
image uses non-standard names (e.g., X,Y,Z or 1,2,3), the command may
output black channels or issue warnings that are easily missed in large
logs.
### Solution:
While you can solve this by specifiying the indices of the channnels,
e.g --ch 0,1,2, the `--nchannels` flag offers a more intuitive and
robust "shortcut" to force an image to a specific channel count. maketx
has had one for years and I found that it works really well and often
wished oiiotool would have one too.
### Implementation:
The implementation acts as a wrapper around action_channels,
programmatically generating the index string. It uses ustring to intern
the generated string, ensuring memory safety even if the action
execution is deferred/postponed.
Assisted-by: Google Gemini (advice on ustring interning and
architectural structure).
### Tests
I added some tests to showcase what this is trying to solve
Signed-off-by: Danny Greenstein <danny.gr@gmail.com>
Background: OCIO 1.x configs were LUT based and so tended to have
different color space names for different data types/bit depths. We
capitalized on this to let `oiiotool --autocc` also select type/bits,
but this trick doesn't work so well for OCIO 2.x configs with procedural
(non-LUT) transformations that all look like "float". We don't want to
promote to float just because of this!
Solution: When using color space data type (nearly always float for new
configs and certainly the built-in configs) to infer output, DO NOT use
"float" as a real request. Treat it as not being a specific type, don't
let it override the intended output type for --autocc. (But a color
space still marked as a particular integer bit depth is still an
--autocc inferred request.)
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Add `ColorConfig::isData(name)` method to C++ and Python APIs.
* Fix `ColorConfig.isColorSpaceLinear()` that was missing from the
Python bindings. (Tests added as well.)
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
In PR #5164, we changed to not be quite as aggressive in treating OCIO
exceptions as real errors. All still good with that, but sometimes it is
helpful to know exactly what OCIO wants to tell us, for debugging.
In this PR, we allow our color debug mode (triggered when environment
variable OIIO_DEBUG_COLOR is set to nonzero) to print debugging messages
that reveal the exception messages. It's a debugging aid for developers,
no users will want this. In ordinary operations, we really do want to
just ignore those exceptions. But if something is misbehaving and we
don't know why, it may be helpful to know which of the exceptions is
being caught and what the message was.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We test on 3 Windows runner configurations currently -- Windows 2022
with VS2022, Windows 2025 with VS2022, and Windows 2025 with VS2026.
GitHub says the middle one will go away imminently (actually, it won't
"break", but it will redirect to Windows 2025 with VS2026, which we
already test). So take it out of our lineup, it's just waste.
The one thing the middle one did differently than the others is test
nanobind, so just move that to the VS2026 one.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
iconvert was the very first command line utility for OIIO, and its
syntax was just `iconvert [options] infile outfile`. YEARS later,
oiiotool came along, superceding iconvert with 100x the functionality,
and also differed by requiring `-o` to specify output file.
iconvert still exists for testing and examples -- iconvert uses just
plain old ImageInput + ImageOutput straightforwardly, whereas oiiotool
is a very complex app that's ImageBuf powered. It's often useful for
debugging to test the ImageInput/ImageOutput layer without any extra
complication, or to directly see how going through an ImageBuf might
differ.
The fact that oiiotool requires `-o` to specify an output file, whereas
iconvert considers that an error, is a minor annoyance when switching
back and forth while debugging, and it's really bugging me.
So here it is, a tiny patch to let iconvert silently accept -o, or not,
and do the obviously right thing. A little quality of life gift to
myself.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Preserve RGB of transparent color index pixels in GIF files.
When reading GIF files, pixels of transparent color index were skipped.
This left their RGB channels zeroed, discarding possible valid colormap
data. To preserve the RGB values, we change to always write the
colormap's RGB and set alpha conditionally.
According to [GIF spec](https://www.w3.org/Graphics/GIF/spec-gif89a.txt):
>viii) Transparency Index - The Transparency Index is such that when
encountered, the corresponding pixel of the display device is not
modified and processing goes on to the next pixel. The index is present
if and only if the Transparency Flag is set to 1.
So basically, it just defines a display compositing behavior but says
nothing about the color table entry for that index being invalid or
empty. In practice, encoders may still assign meaningful RGB values to
the transparent index, and those values are perfectly well-defined in
the colormap. For example, GIMP reads the valid RGB values of
transparent pixels (see attachment below).
### Tests
A new test image and test case is added:
https://github.com/AcademySoftwareFoundation/OpenImageIO-images/pull/11
<img width="847" height="695" alt="image"
src="https://github.com/user-attachments/assets/484c8b53-5a1b-4732-b95c-55a2f52021b5"
/>
Signed-off-by: Lumina Wang <lumina.wang@autodesk.com>
We used new/delete for some small allocations, but there was a possible
leak because early returns would skip the delete. But even when not
leaking, this was wasteful. Replace by a stack allocation using
OIIO_ALLOCA outside the loop -- both problems solved.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Fixes bounds checks for malformed JPEG APP1 Exif and APP2 ICC metadata
markers.
Before this change, APP1 markers were treated as Exif when they started
with
`"Exif"` and had at least 4 bytes. The reader then skipped 6 bytes for
the
`Exif\0\0` prefix, so short payloads such as `Exif` or `Exif\0` could
advance
past the saved marker buffer and underflow the remaining length passed
to the
Exif parser.
The APP2 ICC path also used `strcmp()` on raw marker bytes before
proving the
marker was long enough or NUL-terminated. Short APP2 markers could
therefore be
read past the saved marker buffer, and the code later accessed sequence
bytes
and subtracted `ICC_HEADER_SIZE` without first checking the marker
length.
This is reachable by normal OpenImageIO image-open paths, including
tools such
as `iinfo`, `iconvert`, and `oiiotool`, when opening a malformed JPEG.
The fix adds bounded marker-prefix helpers:
- APP1 Exif is accepted only when the full `Exif\0\0` prefix is present.
- APP2 ICC is accepted only when the marker is at least
`ICC_HEADER_SIZE` bytes
and matches `ICC_PROFILE\0` using bounded comparison.
### Tests
Added coverage to `testsuite/jpeg-corrupt` for short APP1/APP2 metadata
markers:
- APP1 Exif payload lengths 4 and 5
- APP2 ICC payload lengths 11, 12, and 13
Validated locally with:
`ctest --test-dir build-jpeg-only -R '^jpeg-corrupt$'
--output-on-failure`
Assisted-by: Codex GPT 5.5xHigh
Signed-off-by: Vlad (Kuzmin) Erium <libalias@gmail.com>
This just avoids needing to zero out the allocated buffer, since we know
we are allocating it merely to write into it. Uninitialized contents is
ok and faster.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
These are all items fixed and documented in the latest releases. I had
hoped that the CVE reference numbers would be ready before I pushed
those releases out. But let's just say that GitHub's "3 business days to
assign a CVE" claim is aspirational.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Fixes bounds handling for WebP EXIF metadata chunks.
Before this change, the WebP reader always skipped the first 6 bytes of
an EXIF chunk before calling `decode_exif()`:
```cpp
bytes + 6
size - 6
```
That assumed every WebP EXIF chunk was at least 6 bytes long and always
started with the optional Exif\0\0 prefix. A malformed WebP file with a shorter EXIF
chunk could therefore advance the pointer past the chunk payload and
underflow the remaining size passed to the EXIF parser.
This is reachable through normal OpenImageIO image-open paths, including
tools such as iinfo, iconvert, and oiiotool, when opening a malformed WebP
file.
The fix changes the WebP reader to treat the EXIF chunk as a bounded byte span.
Before calling decode_exif(), it now verifies that the payload contains enough
bytes for a TIFF header, either at the start of the chunk or after an optional
Exif\0\0 prefix. The WebP reader no longer performs an unconditional 6-byte
skip; the shared EXIF decoder already handles the optional prefix safely.
### Tests
Added WebP regression coverage for malformed short EXIF chunks.
The testsuite/webp test now generates WebP files with EXIF chunk payload
lengths:
- 0 bytes
- 4 bytes: Exif
- 5 bytes: Exif\0
- 6 bytes: Exif\0\0
- 13 bytes: Exif\0\0 plus a truncated TIFF header
Each generated file is opened through oiiotool to verify that malformed EXIF
metadata does not crash or read past the chunk buffer.
Validated locally with:
python3 -m py_compile testsuite/webp/src/make-short-exif-webp.py
git diff --check
cmake --build build-webp --target oiiotool -j 8
Also ran direct oiiotool smoke tests for all generated malformed WebP
inputs.
Assisted-by: Codex GPT5.5xHigh
---------
Signed-off-by: Vlad (Kuzmin) Erium <libalias@gmail.com>
* SWAPRGBABytes: convert to from raw pointers to span-based.
* A variety of overflow safety fixes, replace int (32 bit) with size_t
or int64_t and use safe_mult64.
* Use ImageInput::check_open() for dpx files to check for
reasonable/legal resolutions and channel counts
* Comment out more function declarations not used by OIIO.
Aside:
I should say, this code was originally imported/vendored from another
project by Patrick Palmer: https://github.com/PatrickPalmer/dpx
We imagined it might be an ongoing/improving work, so we made some very
minor changes here and there but endeavoured to make as little change as
possible -- even excluding it from our clang-format rules! -- so that we
could diff against the changing dpx project and pull in any changes or
even use it as an external dependency.
But as you can see if you go there, it hasn't had any modifications for
17 years! So we never needed the "feature" of minimizing divergence from
the original. And now I think with the rate of discovery and reporting
of vulnerabilities and bugs accelerating, the pressure is on to make
this code "safer," for example with these changes in this PR.
I think it's time to give up the pretense entirely and just allow
ourselves to fully absorb this code as our own, be unconcerned about
divergence. So after this PR is merged, I expect follow-ons to:
- Once and for all, fully remove the "dead code" parts that we commented
out because they aren't used in OIIO.
- Allow clang-format to process these files and bring them into
formatting unity with the rest of OIIO.
- Convert all the raw pointer use to spans
- Remove redundant code -- functions in the original dpx project that
were functionality equivalent to things already in OIIO -- where we kept
the originals in place for the sake of minimizing divergence. Let's just
use the OIIO ones we use everywhere else, in cases where they already
exist.
- Root out all remaining overflow and bounds issues, some of the new LLM
based tools are really good at finding those.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This will get us through the great Canonical attack of May 2026, where
it seems that `add-apt-repository ppa:strukturag` is still unreliable
even as we are now multiple days overdue to get our releases out.
We will revert this after everything is fixed. I don't like it silently
passing when we think it's testing heif files but it's really not, any
longer than we have to be in that state.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This is mostly to get it to pass when the GHA runners can't
add-apt-repository ppa:strukturag when it's glitchy down. It can fall
back to an older libheif 1.17 that's already on those runners, but needs
a matching reference output for that version. This is it.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Fix bugs when channel order isn't strictly R,G,B,A by using a
`m_channel_map`.
* Detect malformed file with too many channel packets.
* Gracefully handle duplicate channel bits across packets by not
thinking that there are duplicate channels.
* Guard against channel packets that contain no channels.
Assisted-by: Claude Code / Opus 4.6
(mostly for analysis/identification of the potential problems; touching
the code only for the out-of-order channels fix)
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Fix incorrect return check for fread in read_pixels_pure_run_length.
* Detect and report error for unsupported packet type.
* Fix possible infinite loop from malformed RLE packet with a zero
run-count.
* Check for valid y range in read_native_scanline.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Oh dear, somehow our heif reader never correctly overloaded the
current_subimage method, instead inheriting the one from the base class
that always returns 0. That's very bad for a file type that supports
multiple subimages.
Along the way, simplified how we deal with the fact that the "primary"
subimage doesn't need to be first in the file, but we always want to
present it as if it were first. There was a point where I thought that
was related to the bug so I was rewriting it; it turned out to not be
the problem, but I'd still like to keep the cleanup I had done.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
First PR on the way to a migration from pybind11 to nanobind.
New env flag introduced `OIIO_PYTHON_BINDINGS_BACKEND` = `pybind11` or
`nanobind` or `both`
> Select which Python binding backend(s) to configure. `both` keeps the
existing pybind11 module and also builds the experimental nanobind
module.
When it is nanobine one is build it is in
`PyOpenImageIONanobindExperimental` target.
```
cmake --fresh -S . -B build \
-DUSE_CCACHE=OFF \
-Dfmt_DIR=/opt/homebrew/lib/cmake/fmt \
-DOpenColorIO_DIR=/opt/homebrew/lib/cmake/OpenColorIO \
-DOIIO_INTERNALIZE_FMT=ON \
-DOIIO_PYTHON_BINDINGS_BACKEND=both
```
### Checklist:
<!-- Put an 'x' in the boxes as you complete the checklist items -->
- [x] **I have read the guidelines** on
[contributions](https://github.com/AcademySoftwareFoundation/OpenImageIO/blob/main/CONTRIBUTING.md)
and [code review
procedures](https://github.com/AcademySoftwareFoundation/OpenImageIO/blob/main/docs/dev/CodeReview.md).
- [x] **I have updated the documentation** if my PR adds features or
changes
behavior.
- [x] **I am sure that this PR's changes are tested somewhere in the
testsuite**.
- [x] **I have run and passed the testsuite in CI** *before* submitting
the
PR, by pushing the changes to my fork and seeing that the automated CI
passed there. (Exceptions: If most tests pass and you can't figure out
why
the remaining ones fail, it's ok to submit the PR and ask for help. Or
if
any failures seem entirely unrelated to your change; sometimes things
break
on the GitHub runners.)
- [x] **My code follows the prevailing code style of this project** and
I
fixed any problems reported by the clang-format CI test.
- [x] If I added or modified a public C++ API call, I have also amended
the
corresponding Python bindings. If altering ImageBufAlgo functions, I
also
exposed the new functionality as oiiotool options.
This code contribution was assisted by Cursor/Composer-2
---------
Signed-off-by: Aleksandr Motsjonov <soswow@gmail.com>
PR #5154 crossed streams with another that turned on more warnings for
unchecked errors. The check isn't necessary here, though.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
1. Harden against buffer overruns from malformed RLE blocks. The simple
fix is the `if (count > ...)` clauses on lines 503 & 510, but I also
took the opportunity to rewrite decode_rle_span using spans rather than
raw pointers.
Fixes#5152
2. Clean up seeking methodology to be more robust.
Fixes#5159
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Implements the FLIP algorithm from Andersson et al. 2020, rewritten with
OIIO idioms (no source code from the BSD-licensed reference FLIP.h is
directly incorporated). Included are C++ and Python API for
ImageBufAlgo::FLIP_diff(), and `oiiotool --flipdiff` command. The basic
operation is to compare two images and produce a per-pixel error map
that conveys perceptual difference to human observers.
I won't explain it all here; see the extensive comments in
imagebufalgo.h, imagebufalgo_flip.cpp, oiiotool.cpp, imagebufalgo.rst,
oiiotool.rst.
There are some important changes (especially for how we expose this via
oiiotool) versus the way the original NVIDIA reference implementation's
command line tool worked. Please read the extensive comments at the top
of imagebufalgo_flip.cpp for details.
This is a preliminary, experimental implementation. It's hidden behind
an `experimental` namespace (and the oiiotool command requires use of
the `--experimental` argument) to emphasize that it may change and is
not yet considered part of OIIO's public API, and thus is exempt from
our usual strict rules about breaking backward compatibility. Try it out
and give feedback, but do not rely on this yet!
Assisted-by: Claude Code / sonnet-4.6 + opus-4.6
I used Claude Code for the inital stab at transforming the NVIDIA
reference implementation into OIIO idiomatic equivalents. But to be
honest, that got me over the hump of the blank page, but I rewrote most
of it bit by bit as I continued to refine my design ideas for how it
should work and be exposed to users.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Move this from internal-only in imageio_pvt.h to be public in memory.h.
It's useful and we've used it in limited ways for a while without
needing any changes. It seems ready for prime-time (for example,
allowing it to be used in reader/writer plugins). None of the logic
changes, it's just moving from a private header to a public header and
out of the 'pvt' sub-namespace.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Try to bubble up errors, stop processing as soon as an error has
occurred, correct return code if a file could not be read properly.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
In PR #5114, just a few weeks ago, I was trying to catch every possible
OCIO exception. That was a good idea, but I was over-eager and treated
every OCIO exception as if something really bad happened. I should have
recognized that OCIO enjoys throwing exceptions for things where I would
have just used an empty return value to indicate a simple failure to do
what was asked.
So this patch is revising the approach as follows:
* Exceptions thrown because it couldn't read a config properly (it was
missing or broken), or couldn't produce a ColorProcessor when asked --
those are real problems that we should treat as serious errors.
* Exceptions thrown because it can't satisfy a simple request (like
"what is the name of the color space for this role" in a case where the
role you passed is not known by the config) -- we should catch those
exceptions, but it's not an "error" for OIIO, we just return an empty
string because there is no color space for that role.
Also beefed up some of the documentation of our ColorConfig class
methods to describe what happens with certain kinds of failures.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Fix setting and response of OIIO_CI -- it was less functional before
than I had thought because we didn't actually define the symbol for
compilation.
* More aggressive cost reduction when combining CI + Debug builds,
reduce resolution of some benchmarks, etc., in addition to reducing
iterations as we had done before.
* Mark the most expensive unit tests with a cost, so CTest knows to
start them first. (more explanation below)
* Add Testing/Temporary/CTestCostData.txt to the cache-save paths --
that preserves CTest's understanding of how much each test costs so it
can schedule it more intellegently for the next run.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Guard against corrupt XMP blocks with TIFF related names: if the XMP
item starts with "tiff:", just ignore it if it's not one of the few
TIFF-related XMP items we recognize of if it's marked redundant, and
don't *replace* a TIFF related item that already had a value in the
spec.
Fixes#5157
Signed-off-by: Larry Gritz <lg@larrygritz.com>
decode_xmp() only searched for `</rdf:Description>` end tags, so
self-closing `<rdf:Description ... />` elements (valid XML) were
silently skipped. Now detect both `/>` and `</rdf:Description>` endings,
plus simplify quite a bit generally, by rewriting decode_xmp to use
pugixml more thoroughly to do the whole parse job instead of trying to
hunt for tags ourselves.
Fixes#5021
Assisted-by: Claude Code / Opus 4.6
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Check that the palette isn't senselessly bigger than the bpp calls
for.
* Protection against integer overflow when addressing the palette.
* Fix misunderstanding of the role of the palette "start" index. Pixel
values are absolute, not relative to the start index. The bug would have
produced incorrect images (or incorrectly reported image corruptions) if
we ever came across an tga file with a non-zero palette start index.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This fixes a longstanding issue where saying
make test TEST="foo|bar"
would be malformed, instead of running tests that matched either foo or
bar (misinterpreting the `|` as a pipe). The problem was just not
quoting it properly.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
According to the TIFF spec, TIFF files with a missing rowsperstrip
should assume the whole image is one strip.
Somewhat related (found in debugging): add a DASSERT to help debug if
fmath.h's round_to_multiple() is given an invalid denominator.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This method is advertised as being able to convert any type of the
attribute into a string, but actually, it would only *search* for
attributes that were entered as strings, so there was no conversion
happening.
Beef up testing of paramlist and imagespec to cover the holes that let
this mistake happen.
Fixed a couple typos in imageio.h comments as well.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Rearrange the automatic setting of `OIIO_NODISCARD_ERROR_ENABLE` so that
for external (not OIIO itself) builds, we default to enabling for OIIO
>= 3.3, and disabling for older and current versions. But now, when
building OIIO itself, it is ENABLED regardless of version.
So this means that as we add OIIO_NODISCARD_ERROR annotations to
functions, we MUST fix any calls to those functions where we ourselves
don't check their return values, and can't accidentally add new breaking
code.
Also, had to fix a bunch of spots that were missed in the original PR,
places where we didn't patch up our sloppy uses of what was annotated in
that patch. I'm not sure I understand how these could have been missed
in that last PR, since that PR did fix other uses of the annotated
functions.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The security@openimageio.org is still fine, but we prefer that true
vulnerability reports come via the GitHub security advisory mechanism.
(That makes it easy for us to turn them into CVEs when needed, among
other administrative niceties.)
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Introducing this as a way to allow us to have experimental oiiotool
features or behaviors that are enabled only if this flag is also
present.
It's a way of allowing features to be turned on for feedback before
we're sure it's in a form that can be a permanent, supported feature,
thus allowing it to undergo further change that is exempt from our usual
rules about what may change in which versions. By forcing users to also
employ this flag to enable the trial behavior, there's no way that they
can inadvertently come to depend on a temporary behavior and be
surprised if it changes later.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
A few years ago, in PR #3301, we made a big effort to remove Imath types
as much as possible from the public APIs. Somehow, as part of that
refactor, we ended up with just the half-varieties of fmath.h's
convert_types() templates actually being implemented in fmath.cpp and
only declared (by default) in fmath.h. I'm honestly not sure why I did
that now, and the PR doesn't explain that particular point.
But that sets up a potential link incompatibility if OIIO used a
different release of Imath than the use is using when calling that
utility function.
So this PR makes it fully inline again, so the Imath version the
downstream package uses just doesn't need to match what was found/used
at the time that OIIO was built.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Change DASSERTs (which were correct, but did not enforce anything in
release builds) into real checks that will return error messages and
flags if the corruptions are found.
* Switch to using spans for extra checks if anything goes wrong.
* Use check_open.
* Check that various header items are valid enum values.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We didn't have a complete enough test for when it was safe to hwy on a
buffer. Encapsulate in new function HwySupports so that all the logic
lives in one place and doesn't have to be replicated everywhere. The
more complete test now checks for:
- contiguous scanlines
- right number of channels (only indirectly checked)
- 2D image, not 3D volume (not previously checked)
- pixel data window fully contains the ROI (not previously checked)
- correct data type (should always be true, but was not checked)
This eliminates some test failures, which were specifically cases
where the ROI being operated on did not overlap with the data area of
the ImageBufs, and the Highway related code did not handle those in
the way that the old ImageBuf::Iterator transparently handled running
off the edge and various wrap modes.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* imageio.h: Document 'enable_hwy' global option.
* imagebufalgo_hwy_pvt.h: Remove RoiNChannels -- it's the same as
roi.nchannels()
* Macros: we can simply test OIIO_USE_HWY, since it's always defined;
there is no need to also test defined(OIIO_USE_HWY).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
1. Some tests are repeated, run once with hwy off and once with hwy on.
The "on" versions append ".hwy" to the test name and set the
OPENIMAGEIO_ENABLE_HWY=1 environment variable to force it on while
running the test. The advantage to this approach is that the test itself
doesn't need to change at all, we just run it separately in each mode.
In this PR, I do this with oiiotool, oiiotool-xform, and
docs-examples-cpp.
For now, while hwy is not enabled by default, it's doing these extra
things to enable hwy. Eventually, if/when hwy is enabled by default,
we'll flip the sense and use the env variable to turn it off.
2. Some tests do certain operations both ways when hwy support was
enabled at build time. This is more convenient for certain unit test
executables, where perhaps we want to benchmark both ways and have those
show up right next to each other within the run.
In this PR, I do this with the imagebufalgo_test unit test.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Add a new ImageBuf method: `contiguous_scanline()`, which reveals if
each scanline is contiguous in the ImageBuf's internal memory (no
padding or unusual spacing between channels or pixels). It's analogous
to the existing `contiguous()`, but the latter also tests contiguity in
y and z.
Additional internal fixes:
* Always and only set m_bufspan via calls to `set_bufspan()`, so it's
consistent. In particular, it ensures that we correctly set the channel
size, which some call sites previously did not do correctly for untyped
"byte" buffers.
* Re-evaluate the contiguous flags whenever resetting m_bufspan.
* contiguous() calls need to call validate_pixels().
An IB from a file that hasn't been read yet into memory yet isn't
considered contiguous. But once it's read, it will be (if it's got the
strides right, etc.). So when somebody asks if it's contiguous, what
does that mean? They are asking because they are about to do something
that requires memory contiguity (meaning, it needs to be IN memory),
just like if they call localpixels(). So just like localpixels(), we
need to call validate_pixels() to assure that if the pixels are destined
for the local buffer, it's done now if not yet.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Recent PR #5081 added more detection of corruptions in (among other
places) the IPTC blocks in a JPEG file.
But we maybe got a little too aggressive, since there appear to be some
jpg files out there that have strange (maybe technically incorrect?)
IPTC blocks but are otherwise fine.
Anyway, I noticed that for ICC profiles in JPEG files, we use the OIIO
attribute "imageinput:strict" to decide whether a localized corruption
in the ICC profile ONLY causes the ICC profile to be skipped (reading
the pixels just fine), or a stricter mode where anything suspicious is
grounds to stop reading the file entirely and declare it corrupt.
So in this patch, I'm using the same control for the analogous purpose
when reading the IPTC blocks. By default, just skip the IPTC block if we
don't know how to decode it. But if in strict mode, make a full error
that will cause us to stop reading the file.
With this change, I needed to update the jpeg-corrupt test to force
strict node, and update some some ref outputs.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
## Summary
Adds `OIIO_NODISCARD` to `ImageInput::create()`, `open()`,
`read_image()`, and `read_scanlines()` methods. Callers ignoring return
values from these methods now get a compiler warning.
## Why this matters
As noted in #4781, it's easy to accidentally ignore the bool return from
`read_image()` or `open()`. These methods return false on failure, and
silently proceeding with bad data leads to hard-to-debug issues.
`create()` returns a unique_ptr that should never be discarded.
## Changes
- `src/include/OpenImageIO/imageio.h`: Added `OIIO_NODISCARD` to 14
method declarations in the `ImageInput` class, covering `create()` (2
overloads), `open()` (4 overloads), `read_image()` (4 overloads), and
`read_scanlines()` (4 overloads).
Scoped to `ImageInput` only per @lgritz's guidance to start with one
class and iterate.
## Testing
- Header compiles (syntax validated)
- No behavioral changes - annotation only
Partial fix for #4781
Assisted-by: Claude Code / Claude Opus 4.6 (1M context)
Signed-off-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Co-authored-by: Larry Gritz <lg@larrygritz.com>
* Have build_dependency_with_cmake print the error message if the git
repo can't be cloned.
* Fix mis-quoted env variable in status message.
* Improve grammar of status message.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
I think forcing fmt to disable exceptions is probably doing more harm
than good, probably does not work as expected for apps that use fmt
elsewhere (when we first started using fmt, it was not super popular, so
we weren't really concerned about how we might conflict with other
libraries in the same app), and disabling exceptions makes it behave
differently than std::format/print.
So let's stop doing this and see what happens! (In main.)
Also, declutter by removing the FMT_USE_GRISU definition, which was only
meaningful for very old fmt versions that are older than our minimum
supported version now.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
For the sake of Windows' need for wstring filenames, we do some
utf8<->utf16 conversion in strutil.cpp. That has always used the
std::codecvt set of functions, but they have been deprecated beginning
in C++17 and will be removed entirely in C++26, so their days are
numbered and we already have to suppress warnings to get the compiler to
shut up about it.
Assisted-by: Claude Code / Opus 4.6
Signed-off-by: Larry Gritz <lg@larrygritz.com>
It must have been among the homebrew packages in the base Mac
installation on GHA, until just a couple days ago?
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Fixes#5132.
Reading DDS volumetric files used to work, but was broken by PR #4669
last year, the one where we added all the span-oriented API calls.
At that time, I did not realize/remember that DDS files could be
volumetric, observed that all the (other) volumetric files only
supported tiled access, and so in the course of that API refactor, I
removed the z parameter from the new read_scanlines API.
I don't want to break ABI compatibility again for this, so the solution
I came up with in this PR is just to make DDS volume files report
themselves as "tiled", with each tile being (width x 1 x 1), i.e. what
used to be one scanline, and it all just works out.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Fix warnings for gcc15:
* Suppress warnings about deprecated utf conversion (already suppressed
for clang, needed to do so for gcc >= 15 as well).
* Rename some internal methods that gcc 15 was warning were shadowing
the names of virtual methods.
Also took the opportunity to update INSTALL.md to reflect the latest
dependency versions that we have tested against.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Optional SIMD optimizations for selected ImageBufAlgo operations using
the Google Highway library:
* add/sub
* mul/div
* mad
* resample
Adds CMake and build system support, new implementation helpers, and
developer documentation.
This is currently off by default, needing to be enabled at build time both by
setting CMake variable `OIIO_USE_HWY` to ON *and* having the Highway
library installed and able to be found (we do not yet auto-build it). And even
if enabled at build time, currently it is only enabled at runtime by setting the
environment variable `OPENIMAGEIO_ENABLE_HWY=1` *or* setting the
global attribute `OIIO::attribute("enable_hwy", 1)`. We hope that the defaults
might be changed to fully enabled by the time we release OIIO 3.2.
Assisted by: Opus4.5 and Codex GPT5.2
_Code mostly wrote using frontier Opus4.5 and Codex GPT5.2 High models
with a strict rules._
---------
Signed-off-by: Vlad (Kuzmin) Erium <libalias@gmail.com>
We have enough spurious failures that I don't want all of the variants
in a job matrix to fail just because one goes wrong.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This test did more than was really necessary. It resized a small image
(twice) to get big images, but the contents didn't matter, so just using
a gradient is good enough. Also, we can check results just by using the
hash of the image contents, don't need to do extra idiff calls.
It's barely noticeable on a regular release build, since the whole test
was only a couple seconds on my laptop and maybe 15 seconds on the GH
runners. But for the CI "sanitize" test, this cuts about 30 seconds from
this test! Every little bit helps, the sanitize tests are usually the
last to complete in a CI run.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This PR fixes an inconsistency in OIIO's XMP write path that prevents
some metadata fields from being emitted into XMP, even when they are
present in `ImageSpec::extra_attribs`.
**Problem**
`encode_xmp()` gathers attributes through `gather_xmp_attribs()`, which
currently does:
```cpp
const XMPtag* tag = xmp_tagmap_ref().find(p.name());
if (tag) {
if (!Strutil::iequals(p.name(), tag->oiioname))
continue;
...
}
```
At the same time, `xmp_tagmap_ref()` builds its lookup map using
`xmpname`, not `oiioname`.
This becomes self-contradictory for mappings where the XMP name and the
OIIO-facing attribute name differ.
A concrete example from `xmp.cpp` is:
```cpp
{ "xmp:Rating", "IPTC:Rating", TypeDesc::INT, 0 },
```
With the current implementation:
- If the spec contains `IPTC:Rating`, lookup fails because the map is
keyed by `xmpname`.
- If the spec contains `Xmp:Rating` / `xmp:Rating`, lookup succeeds, but
the attribute is then rejected because it does not match `oiioname`.
As a result, `encode_xmp()` does not emit the rating field, and format
writers that rely on `encode_xmp()` (for example JPEG and TIFF) may fail
to write an XMP packet for such metadata.
**Fix**
This patch makes the XMP tag lookup accept both spellings:
- the XMP name (`xmpname`)
- the OIIO metadata name (`oiioname`)
and allows `gather_xmp_attribs()` to accept either as a valid match.
This preserves existing behavior for fields where `xmpname == oiioname`,
while fixing aliased mappings such as rating.
**Why this is correct**
OIIO already models these fields as explicit mappings between an XMP
name and an OIIO attribute name. The write-side gather logic should
therefore be able to consume metadata stored under either spelling.
Without this, some entries in the XMP mapping table are not actually
encodable.
**Observed impact**
In a local reproduction using JPEG output:
- `IPTC:Rating` was present in `ImageSpec::extra_attribs`
- the JPEG writer called `encode_xmp()`
- no XMP APP1 segment was written to the final file
This was traced back to the lookup/match mismatch described above.
**Testing**
I recommend adding tests that cover both direct XMP naming and aliased
OIIO naming.
1. Unit test: `IPTC:Rating` is encoded as XMP rating
```cpp
ImageSpec spec(1, 1, 3, TypeDesc::UINT8);
spec.attribute("IPTC:Rating", 4);
std::string xmp = encode_xmp(spec, true);
CHECK(xmp.find("xmp:Rating=\"4\"") != std::string::npos);
```
2. Unit test: `xmp:Rating` is encoded as XMP rating
```cpp
ImageSpec spec(1, 1, 3, TypeDesc::UINT8);
spec.attribute("xmp:Rating", 4);
std::string xmp = encode_xmp(spec, true);
CHECK(xmp.find("xmp:Rating=\"4\"") != std::string::npos);
```
3. Optional integration test: JPEG writer emits an XMP APP1 segment when
rating is present
- Create a small image
- Add `IPTC:Rating` or `xmp:Rating`
- Write JPEG
- Inspect the output bytes for the standard XMP APP1 namespace header:
`http://ns.adobe.com/xap/1.0/\0`
This verifies the full `encode_xmp()` -> JPEG writer path, not just the
XML generator.
**Notes**
This patch is intentionally narrow:
- it does not change the XMP mapping table itself
- it only fixes how mapped attributes are discovered during XMP
generation
Signed-off-by: Qi-fly <72260719+Qi-fly@users.noreply.github.com>
Fixes#5124
`fancy_add_executable()` currently adds the Windows manifest resource
file for all
`WIN32` builds, but it also unconditionally passes `/MANIFEST:NO` to the
linker.
That switch is MSVC-specific, so MinGW ends up forwarding it to `ld` as
a path
and the executable link fails.
This change keeps embedding `src/windows/oiio_exe.rc` for all Windows
toolchains,
but only adds `/MANIFEST:NO` when the build is using MSVC.
That preserves the intended long-path manifest behavior from #5066 while
letting
MinGW link the Windows executables again.
Assisted-by: Codex / GPT-5
Tests:
- Ran a targeted CMake smoke test that configures a dummy executable
with
`WIN32=TRUE,MSVC=TRUE` and verified that `/MANIFEST:NO` is present in
`LINK_OPTIONS` and the `.rc` file is attached as a source.
- Ran the same smoke test with `WIN32=TRUE,MSVC=FALSE` and verified that
the
`.rc` file is still attached but `/MANIFEST:NO` is not added.
- I could not run an actual MinGW build locally in this environment
because a
MinGW toolchain is not installed here.
Signed-off-by: Asish Kumar <officialasishkumar@gmail.com>
fmtlib renamed its "master" branch to "main" yesterday. So now our
"bleeding edge" test that requests fmt "master" fails.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Some fuzzing stumbled across a very interesting edge case in the
rehashing logic of ustring internals:
If the initial hash location of a newly ustring-ized string is already
occupied, it rehashes. But we never considered what happens if it ends
up rehashing to the same slot! And in fact, that can happen if the true
hash is 0 (I'm not sure if it can/does for any other value). And we may
have thought that the only string that would ever hash to 0 is the empty
string (which is 0 hash by design, and doesn't need to rehash because
the table is seeded with that entry already in its correct slot).
So anyway, if there was some other string that happened to hash to 0, it
would get caught in an infinite loop of rehashing but never finding a
free slot, because it keeps rehashing to the very same occupied slot.
And would you believe that there IS such a string, and it is this one:
"\t\n\n\t\301\350`O\217c85?\202\200\251r|}\304\351\2517\337K'\217C\240"
(unprintable chars expressed as escaped octal values)
The crux of this patch is to notice when the old and new hash bit
pattern has not changed, and bump it by 7 (a prime number, so it should
not land on the same location again until every slot in the bin has been
exhausted).
Solution entirely from my own human brain, in case you're wondering.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
I was reminded by Eric Enderton that we should mention and cross
reference the recently-added AI policy docs in other places, since it
can easily be overlooked now, buried in the docs directory.
* Mention it in the pull request template's checklist.
* In the CONTRIBUTING guide.
* Briefly mention in the main README in the list of important
documentation for developers.
* A few very minor touch-ups and wordsmithing in other areas, no
substantive changes.
Also, responding to comments at the OSL TSC meeting this week, I added
a brief section explaining that we don't have any specific guidance for
how AI tools should or shouldn't be used for "artistically creative"
work, since this project repo doesn't really house such things. This
is mostly for the benefit of other projects that may use our policy
document as inspiration, to warn them that they may have issues that
need policy guidance but that we do not address.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
TL;DR:
* Bump ASWF containers to the latest versions for their years.
* Move the "oldest" and "hobbled" tests off the 2022 containers
entirely, do as close as possible to equivalent on Unbuntu.
Detailed explanation follows:
First of all, bump to the latest ASWF docker containers for each year.
We had been locking down, but hadn't bumped to the latest for a while.
(Exception: leave the sanitizer test where it is -- the newer containers
have additional checks that are catching UB that I am tracking down
separately.)
Also, we are trying to reduce the surface area of the CentOS 7-based
2022 aswf-docker containers, which are getting long in the tooth, cannot
upgrade to the latest "actions" (because of a tricky interplay between
runner configuration, node version the actions require, and glibc on the
old containers), and may have a limited time that they will continue
working.
In most film studios, CentOS is long gone or taking its last breaths, so
being able to support and test on 2022 era dependencies and OS distros
is of dwindling relevance. We do, however, want to keep testing versus
the oldest supported versions we can muster for many major dependencies.
So to this end, this PR also does the following:
* Move three tests from running in the 2022 containers to just running
on bare Ubutu and by hook or by crook using the right dependencies:
- the "oldest" versions we claim to support for as much of the toolchain
toolchain and dependencies as is easily achievable, built with gcc
- and the same, built with clang
- the "hobbled" test, which is similar to the "oldest" test but with as
many of the optional dependencies missing or disabled, to make sure we
really do build with only required dependencies (this also is the test
build for no SIMD use)
* Keep two last tests running on 2022 containers, these are only for the
purpose of testing "pure" CentOS VFX Platform 2022 compliance for just a
bit longer.
* A few minor related touch-ups necessary to force some older
dependencies on the bare Ubuntu runner.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Fixes#4942
IBA::fillholes_pushpull() produced incorrect results when the source
image had a non-zero data window origin (such as exr files with
overscan).
Two interrelated bugs:
1. The paste() call double-applied the data window offset. paste() maps
source pixel (0,0) to destination (xbegin,ybegin), but since the source
data starts at (x,y), passing (x,y) as the offset shifted every pixel by
an extra (x,y). Pixels in the negative-coordinate region landed outside
the destination buffer and were silently lost.
2. The top pyramid level preserved the original display/full window
while all smaller levels had full=data at (0,0). Since resize() uses the
full window for coordinate mapping, it only sampled the display window
portion of the top level, losing all overscan data.
Fix by shifting the top pyramid level to origin (0,0) with full=data,
making all levels coordinate-consistent. The initial paste translates
source pixels into the origin-based pyramid, and the final paste (which
was already using src origin offsets) correctly translates back.
Add test cases in testsuite/oiiotool for fillholes with an offset data
window, and an offset display window, to ensure that both cases work.
Also, for the existing fillholes related test, move the source image
from ref (never should have been there) to src.
Assisted-by: Claude Code / claude-opus-4-6
Signed-off-by: Larry Gritz <lg@larrygritz.com>
I've heard occasional reports of uncaught exceptions in print
statements. I haven't caught it in the act yet, but one set of reports
involved the following output:
OpenImageIO/detail/fmt/format-inl.h:40: assertion failed: string pointer
is null
So I'm proactively trying to find places that might be passing a null as
a string. I'm just guessing here, but these are plausible candidates.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
PR #5113 didn't fix all the problems after all. Revert it, because it's
not needed after the real solution.
The real solution is to mark tiff_datatype_to_typedesc() with
OIIO_NO_SANITIZE_UNDEFINED to have the sanitizer skip it. What's
happening is that corrupted files end up with a value in the data type
field that is not a valid value of the TIFFDataType enum, and ubsan is
flagging that. BUT... `tiff_datatype_to_typedesc()` itself is actually
safe in that circumstance, because it's got a 'default` clause that
handles the unknown enum values just fine.
In contrast, I've been running around in circles trying to find
something to do *within* that function to make it "safe" (by the
sanitizer's reckoning), and trying to check for valid values prior to
converting to a TIFFDataType is just too hard, partly because there are
places where that conversion happens unchecked inside libtiff (C
language, just does a cast), so it comes to us as a TIFFDataType already
with an invalid value. It's easier to just mark the whole function as
being ignored by ubsan, given that we can see by inspection that it has
totally deterministic and desired behavior for the illegal values.
Making this spurious ubsan error disappear allows us to upgrade the
container version we are using for the "sanitizer" CI test variant, so
we also push that all the way up to 2026.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Corrupted exif data could put a value in a "tiff data type" field that
is not one of the valid enum values. That's UB. Identified by running
the sanitizer with a newer clang than we usually do.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
When reading corrupt files, sometimes there are a lot of different
potential error messages reported by libtiff depending on the exact
version and platform, and which corruption it encounters or notices
first. So we tend to have to add new refs frequently as libtiff changes.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
https://opencolorio.readthedocs.io/en/latest/api/exceptions.html
"All functions in the Config class can potentially throw this
exception."
Oh dear, that is not something I previously fully understood the
implications of. This patch makes extra sure that there's try/catch
around every call to OCIO::Config methods. Some previously did, but some
that were less obvious that they might throw did not try/catch. No
matter how benign they look, I'm taking no chances now.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Fixes#5063
In that issue, it was reported that the printing of any unretrieved
error messages upon destruction of an ImageBuf could itself have an
exception (including if it, for some reason, could not correctly fwrite
the output). And if that is the case, it could terminate the app.
So for safety, this patch encloses this particular print statement in a
try/catch. And also found another in ErrorHolder in the internals of
Strutil, which is used for "global" errors issued to OIIO::errorfmt().
Assisted-by: Claude Code / Opus 4.6
I did the fix in ImageBuf, then Claude found the other spot in
ErrorHolder that also needed the fix.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Rather than use old actions across the board for a handful of job
variants that need old containers, use the obsolete actions only where
really needed. This involves duplicating the some steps within the
workflow, and choosing whether to use old or new based on job. We'll get
rid of the "old" versions when we retire the 2022 containers that
require them.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Summary of our core values:
- Human always in the loop and is the responsible party.
- You're still on the hook for fully understanding and standing behind what you submit.
- Interact with the project and community yourself, not by agent.
- Disclose what tools you used and how. At the very least, have an `Assisted-by TOOL / MODEL` line.
- Don't waste maintainer time with low quality PRs.
- Try to reserve the curated Dev Days and "Good First Issues" tags for human learning.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The jph_infile class itself did not change, it just moved up in the file
to be defined before it was used. It's the use of the persistent
unique_ptr that fixes the leak.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Validity-check resolution of RLA files with check_open. RLA file
headers contain int16_t values for left & right (and top/bottom) window
coordinate, leading to a maximum resolution of 2^16-1.
* Fix potential bug with sign extension in RLE decoding -- if a signed
char is -128, negating it can't make signed char 128 (no such thing), so
must widen the var to an int.
* Fix potential bug by detecting when the number of matte or auxiliary
bits is 0, but the number of matte or aux channels, respectively, is
not.
* Better bounds checking in decode_channel_group. We did the checks
before, but after some accesses that would have been out of bounds! Move
the checks earlier than all the accesses. It actually looks like was the
result of a cut and paste error long ago.
* More care in read_native_scanline for checking valid scanline numbers,
offset into m_sot, and check whether ioseek succeeded (i.e. whether the
offsets loaded from the file are within the range of the size of the
file).
Code and fixes all are from my own brain, but some of the analysis of
which spots have bounds issues were identified in part by conversation
with Claude Code Opus 4.6.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
If certain camera metadata was mangled and or otherwise wasn't one of
the known metadata names, it could run off the end of the list and
dereference a null pointer.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
A visual inspection shows that it works fine and the images are a visual
match. But some numeric changes in ffmpeg 8.1 make it need new hashes.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
1. Validity check the resolution/channels.
2. Also, in png_pvt.h's `read_info()`, found a spot where errors were
being sent to the global error reporter instead of to the PNG reader's
own error store. (It is relevant to this PR because that is called by
the ICO code for certain ico files that are really using PNG encoding.)
3. Handle scanline width that's not 8 pixel multiple without running
over buffer limits.
4. Guard against possible palette array overrun for 4bpp case. The 1-
and 8-bpp palette cases already had computed the palette index and
checked for being in range before accessing the array. But the 4-bpp
case did not!
Items 1 & 2 were entirely LG. Items 3 & 4 were identified and fixed with
the assistance of Claude Code / Opus 4.6 -- it was really helpful! I'm
honestly not sure if it saved me 5 hours or closer to 5 minutes, but
even if short, it would not have been at all fun to trace through by
hand to find a subtle case that we previously forgot to check.
Assisted-by: Claude Opus 4.6
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This had been biting our wheel building for a couple weeks before it hit
the regular CI jobs, I guess? And I didn't even know it or fully
understand the cause. But it's the same solution as the other fix from
yesterday: explicitly specify the path to the xcode version of clang, so
it doesn't accidentally pick up the homebrew copy of a different clang.
(Either is ok, but mixing and matching is no good.)
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Update action version pins to their latest releases, referenced by full
commit hash with version tag as a comment. Except for a couple, as noted
below.
| Action | Old | New |
|-------------------------------------|------------|------------|
| actions/checkout | v4.2.2 | v4.3.0 |
| actions/upload-artifact | v4.5.0 | v7.0.0 |
| actions/download-artifact | v4.1.8 | v8.0.0 |
| actions/setup-python | v5.2.0 | v6.2.0 |
| pypa/cibuildwheel | v3.2.1 | v3.4.0 |
| pypa/gh-action-pypi-publish | release/v1 | v1.13.0 |
| sonarsource/sonarqube-scan-action | v6.0.0 | v7.0.0 |
| ossf/scorecard-action | v2.3.1 | v2.4.3 |
| github/codeql-action/upload-sarif | v2.3.6 | v3.32.6 |
| sigstore/gh-action-sigstore-python | v3.0.0 | v3.2.0 |
| jmertic/slack-release-notifier | main | 0.1 |
Note that all are locked down to hashes now, including a couple which
hadn't been previously.
We wish we could upgrade actions/checkout all the way to the latest
versions. But in both cases, they require nodejs 24, which in turn needs
a glibc that's too new for our ASWF VFX Platform 2022 containers. So
we're going to have to keep them locked to old versions until we're
ready to retire the 2022 containers, which we hope is soon. The checkout
action is upgraded from 4.2.2 to 4.3.0 only, intead of the latest 6.0.2,
and the cache actions had to remain on 4.3.0 as it was before, instead
of upgrading to the latest, 5.0.3.
The initial gruntwork of tracking down the latest versions and upgrading
was done by Claude Code using the Sonnet 4.6 model using the following
prompt:
```
For all the workflows in @.github/workflows/ , check every action
version reference to see if it is using the latest release
version. Upgrade any that are not the latest. Be sure to refer to the
action by full hash, not version number (but put the corresponding
version number as a comment).
```
That upgraded everything possible without me needing to track down each
action myself, though it was only a partial solution: the newest actions
failed on the old containers (as noted above) and it took me a while to
sort out by hand which ones needed to be held back to run on the old
containers. So Claude helped with the first 80% and then I did the
remaining 80% of the work.
Assisted-By: Claude Sonnet 4.6
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Don't copy from memory using the stride, as this can get padded and read
past the end of the input pixel data. The 10 and 12 bit paths were
correct.
Test added, this only failed with address sanitizer.
Signed-off-by: Brecht Van Lommel <brecht@blender.org>
1. Validate channel count against color mode to prevent heap buffer
overflow.
2. Corrupted PSD files can declare a color mode (e.g. RGB, needing 3
channels) with transparency (needing +1) but report fewer channels
in the header. This caused an out-of-bounds read in
read_native_scanline when setup() built channel pointer arrays
using mode_channel_count, which exceeded the actual
m_image_data.channel_info size.
3. Additional buffer overflow protection, error checking and
propagation.
4. Avoid leak by properly freeing z_stream resources in decompress_zip.
Used Claude Code / Opus 4.6 to help narrow down and confirm the bugs 1
and 2.
Assisted-by: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Reading the TIFF 6.0 spec, on p. 23, it sure looks like palette images
can only be 4 or 8 bits per sample. But the libtiff set of test images
indeed has 2, 4, 8, and yes, 16 bps palette images among the examples.
So I think we should support up to 16, since we might encounter it in
the wild, but no higher.
We didn't previously check for this, though, and that could lead to
accepting invalid TIFF files, and then potentially have problems with
`(1 << m_bitspersample)` which appears several times. That of course is
safe if m_bitspersample can't be > 16.
Also check and proper error for a file that has a bitspersample that is
not one of the valid choices (1, 2, 4, 6, 8, 10, 12, 14, 16, 24, 32,
64), and proper errors for if the sampleformat was an unknown value or
was not a valid value for the number of bits per sample.
Note that this changes the reference output of some tests because the
new checks can alter the first error discovered in a corrupt file.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Also change the x and y counters to uint64_t to prevent any possible
overflow as it's computing offsets.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Be even more aggressive for the runs that are both debug mode (including
sanitizers) and a CI run. Speeds up the slowest GHA test runs. Should
not affect ordinary builds.
This shaves about 8 minutes off the long "sanitizer" run, which is often
the last job to finish in our CI workflows.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The files that match the pattern "src/cmake/build_<dependency>.cmake"
are all similar and each one is called by our build system if a
dependency is not found, to download and built it locally to be
incorporated into our build. One example is src/cmake/build_TIFF.cmake.
Within these files, the version to be retrieved is determined by a pair
of lines such as
```
set_cache (TIFF_BUILD_VERSION 4.7.1 "TIFF version for local builds")
set_cache (TIFF_GIT_TAG "v${TIFF_BUILD_VERSION}" "Git branch or tag")
```
The tag is passed to `build_dependency_with_cmake` with the `GIT_TAG`
argument. There is also an optional `GIT_COMMIT` argument, that we
usually do not use (previously, only needed if we wanted to build from a
commit that didn't have a version tag).
But it turns out that it's probably not safe to blindly retrieve a tag,
in case the dependency's repo has been hijacked and the tag reset to
point to a malicious commit.
So here is how to address this:
- `build_dependency_with_cmake` modifies its internal logic:
- If only the GIT_COMMIT is passed and GIT_TAG is blank, use the commit.
- If both GIT_COMMIT and GIT_TAG are passed, check out the tag but
verify that it is the commit specified, and if not, then consider it an
error and do not proceed with building the dependency.
- If only GIT_TAG is passed and not GIT_COMMIT, warn about the potential
safety problem.
- But as a special option for dangerous tasks, an override called
OpenImageIO_DEPENDENCY_BUILD_ALLOW_UNVERIFIED_TAGS allows an unverified
git tag to be checked out. This is how we make the "bleeding edge" test
still build against dependecy main/masters.
- For each of the `build_<dependency>.cmake` files, change it as
follows:
- Determine the commit hash of the specified version by checking the
repository.
- Add a line setting `<dependency>_GIT_COMMIT` to the corresponding
hash.
- Pass both arguments to build_dependency_with_cmake.
This patch was made with the help of Claude Code using Opus 4.6. I
started the dialog with a prompt that was substantially similar to the
description above, detailing my proposed plan.
I first suggested that if only the TAG is present, to allow it as long
as it turned out to be an immutable git release tag, but it pointed out
that (a) this requires a build-time dependency on the `gh` command line
util, (b) not all our dependencies are on GitHub (such as libtiff on
GitLab), so we modified the plan to warn in that case rather than try to
determine if it's immutable. The rest proceeded more or less as
expected, with a few rounds of back and forth and corrections.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Certain CMYK files (maybe only corrupt ones?) hit an edge case that
caused us to read the original 4 channels but into space for only the 3
channels we ultimately report back. Fix that logic.
Also noticed several spots where we didn't correctly report back error
messages for newer versions of libtiff. This was some kind of oversight
from long ago.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* `decide_iptc_iim()` (semi-public) : change from taking a pointer &
length to a string_view, and internal safety checks.
* `jpeg_decode_iptc()` (non-public): change from taking an unlengthed
pointer (!) to a string_view, and internal safety checks.
* JpgInput::open(): a number of extra checks for memory bounds validity.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* fix check_open used m_spec instead of parameter spec!
The function advertised that its doing checks on the parameter passed
to it, but -- oops -- it was really doing it on this->m_spec. They
were almost always (or always always?) the same, but still. Bug.
* Check zero-sized images separately from nonsensical negative sizes.
By being togther, there was some confusion about whether "noimage"
feature should negate the error (that feature should allow
zero-sized images, but not negative dimensions).
* Add new checks for the full_width/etc to also be sensible.
* Some other minor refactoring of check_open.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The docs for TexturesSystem::attribute() had a summary of ImageCache
options that were accepted and passed along. Over the years, a couple
had been added to IC without being reflected in the TS docs.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The m_legacy_texture_blur flag is set in TextureSystemImpl::Init, which
overrides the default initialisation depending on the OIIO version.
In other words, the texture blur fix was always enabled, regardless of
the OIIO version.
I also updated the help description for this flag in testtex.
This fixes minor shortcomings of #5071
---------
Signed-off-by: Pascal Lecocq <pascal.lecocq@gmail.com>
Encountered at work: when using certain libheif dependency builds that
can read, but not write, avif files, we needed an additional reference
output.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Background: oiiotool is pretty good at remembering the data type of the
first file it reads in a command, and ensuring that the output preserves
the same types, despite the fact that internally, it tends to keep
everything in float. (As a further reminder: -d can override the
heuristics that make this work.)
But there was an exception: when `-i` was used with the optinal modifier
`-i:ch=...` to select a subset of channels or reorder channels upon
read, the underlying call to IBA::channels would cause it to not have
that information in the "nativespec" at the right time that it was
collected.
This PR primarily ensures that the `-i:ch=...` preserves the information
about which channels had which data types, despite the channel shuffle.
Some other bits and pieces along the way:
* A new `TypeDesc::basetype_merge()` variety that take a
`cspan<TypeDesc>`.
* A new utility `TypeDesc::all_types_equal(cspan<TypeDesc>)`
* Cleanup in IBA::channels() to more correctly compute the common type
of the channel subset that you actually specify.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Adds an embedded manifest to OIIO's generated executables on Windows,
which enables "long path" awareness.
This is done via a "resource" file, with the manifest resource ID
specified. From what I've read, this is more broadly compatible with
e.g. non-MSVC build toolchains like MinGW.
The manifest file was created by building OIIO from `main`, extracting
the manifest from the resulting `oiiotool` (using the `mt` tool), and
manually adding in the XML snippet from
https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=registry#application-manifest-updates-to-declare-long-path-capability.
Fixes#5064
Added a new Windows-only test to validate that `oiiotool` can correctly
read long image paths.
The one executable that I have not tested is `iv`. The only reason I'm
more curious about it than others is that it is a GUI executable, so I'm
not sure if its default
---------
Signed-off-by: Nathan Rusch <nrusch@users.noreply.github.com>
It's not in any pybind11 tagged release yet, but in its master, a change
has been introduced that causes it to allow functions with float
paramters to automatically match int arguments. This is fatal for us, as
we have set up several cases of methods that overload on int vs float,
and with this change, the float one gets called when an int is passed.
Pybind11 docs say that the `.noconvert()` modifier on the argument will
do the trick, but I'm finding that it doesn't help for overloaded
function names.
The solution is to find these cases, change our declaration to a generic
py::object, and then sort out what type was passed with py::isinstance.
It seems to mainly affect `attribute()` style calls.
In the process, minor changes to the pybind11 auto-builder allow it to
take a git commit hash directly. Also, I took the opportunity to remove
a few reference outputs that are no longer needed because they were
python2-specific.
Meanwhile, just in case anybody had been paying attention, the problem
in pybind11 master that had caused us to temporarily stop CI testing
against master seems to have been resolved. So re-enable the bleeding
edge test to use pybind11 master (which will fail if not for this PR
here), and bump the "latest version" tests to the latest pybind11 3.0.2.
I diagnosed the problem myself, wrote the solution for one class and
verified that it solved the issues with that class. Then I used Claude
Code to make the analogous set of changes to the other classes that
needed them and I reviewed that it was correct.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Fixes#5069
This PR fixes the texture blur overstimate reported in #5069
We provide the following changes:
* Use the mathematically correct Pythagorean form to properly adjust the
ellispse footprint with st-blur parameters.
* Add a new legacy_texture_blur attribute in TextureSystem to opt into
the fix without breaking existing renders.
* Add a new --fix-texture-blur option in testtex to enable the fix.
* Add a new texture-blurfix test.
Note: This PR has been partially edited using the Claude coding
assistant.
---------
Signed-off-by: Pascal Lecocq <pascal.lecocq@gmail.com>
The comments that explain how to use the OIIO_MAKE_VERSION and related
macros actually were using formatting that would have been rejected by
our own clang-format settings. Change those examples so that they use
the correct formatting that woule be accepted by the project.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
I don't know why, maybe some slightly different version of libtiff
rolled out to the GHA runners, but despite no changes to our TIFF
handling in the recent past, just today we started intermittently
getting different output for this test on some of the Linux runners.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Clarify in the documentation that `openexr:dwaCompressionLevel` is a
read-only attribute and cannot be used to set the compression level on
write.
Not sure if this is an obvious change or not, but it wasn't immediately
clear if the attribute is meant to be writable.
Signed-off-by: Aamir Raza <aamir_raza@outlook.com>
Lately, some Mac Intel CI runs have taken forever or failed because of
timeout. It appears to be because some packages are maybe not cached
(bottled) for Intel anymore? It only happens sometimes, so maybe it
fails when the package version has just been bumped, but eventually
somebody published a pre-built bottle and then it's ok?
openvdb and its dependencies seem to be a frequent culprit. Let's just
skip them for this one soon-to-be-obosolete variant.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Most visible change: The TIFF reader and writer now correctly handle GPS
tags.
Fixes#5049
But along the way, and to get it completely right, a fairly extensive
refactoring of our TIFF tag handling was needed. The libtiff behavior
around different tags is extremely convoluted, and there were a bunch of
cases we either didn't handle correctly, or were dropping on the floor.
So there are also some other non-GPS tags that we missed all along but
now read properly.
Note that the support for separate IFDs for GPS tags is only supported
via the API in libtiff 4.2 and newer, so this is disabled when bulding
against older libtiff versions. (Our support of old libtiff versions
extends extraordinarily far back in time.)
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
`oiiotool --compression XXX` just used the argument as a literal, and
wasn't able to use expression substitution for the value. That made it
behave differently than the supposedly synonymous `--attrib compression
XXX` in the case that the argument was an expression rather than a
literal value.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
PR #5036 incidentally added a new TypeURational alias, but I neglected
to make the corresponding change on the Python bindings, to update some
of the docs, or to fully test it. This PR completes those items.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The docs described a syntax where you could use `-d` to change the data
type of all channels in one subimage of a multi-subimage image. But it
didn't work, as revealed in #5047.
Fixes#5047
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Problem identified by @irieger Ingmar Rieger, solves a problem that pops
up when Conan build provides the targets.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
I tested out the JPEG XL CICP support and noticed that color primaries
12 was not supported. This pull request is looking to extend P3 support
for color primaries 12.
Note: color primaries 11 uses the DCI white point and color primaries 12
uses the D65 white point.
The JxlPrimaries enum only covers P3 primaries as value 11 and not 12.
See,
https://github.com/libjxl/libjxl/blob/main/lib/include/jxl/color_encoding.h#L55-L75
Further code is therefore required to account for this on read and
write.
Tests for read and write of color primaries 11 and 12 were added.
Signed-off-by: Shane Smith <shane.smith@dreamworks.com>
I was seeing warnings with instantiation of the ispow2 function template
for unsigned type, where the `x >= 0` clause is meaningless. Use a
constexpr if to eliminate that pointless test for unsigned types.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The CI stub generation has been broken for a few days, failing CI every
time. The checked-in stub files seem fine. Just turn off this check
until we can figure out why it is broken.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The previous minimum, 7.0, dated from mid-2020.
We are raising now (in main / future 3.2 only) to 9.0, which dates from
mid-2022, so we're still supporting several versions and/or years back.
Because this changes minimum dependency versions, it will NOT be
backported to release branches (3.1 or earlier).
I had to remove the CI test variant for icc, because ancient icc can't
correctly build newer versions of fmt, it seems. There is a separate PR
to simply drop icc from our list of supported compilers.
If anybody wants to argue for pulling the minimum up even farther (say,
to fmt 10.0, released in 2023, so still supporting 3 years back), which
would simplify even more places, I would consider it.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Intel icc is deprecated and hasn't had a release for a few years. It's
holding us back, both by making us work around an ever growing number of
icc bugs and limitation that will never be fixed, as well as not
allowing us to upgrade minimum versions of certain dependencies, because
icc can't correctly compile newer versions (as an example, it cannot use
a 'fmt' library newer than the oldest we support, 7.0).
So it's time to thank icc for its service and put it on the ice floe for
the polar bears to eat. This is of course in main (future 3.2), and will
not be backported to release branches, since we never stop support of a
dependency or toolchain of existing releases. People requiring icc for
whatever reason may keep using OIIO 3.1 or older.
We will continue to support and test icx, the fully supported Intel
LLVM-based compiler.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* cmake utility build_dependency_with_cmake was unconditionally doing a
shallow clone and using `clone -b`, but that only works if it's got a
branch or tag name, not if it has a commit hash. So change the logic so
it does a shallow clone only if GIT_TAG is specified but GIT_COMMIT is
not.
* pybind11 self-builder is modified to allow a git commit override.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Fixes#5044
Oops, the logic was a little mixed up when there were exactly two
images. One reason that this was a special case is that conceptually,
there is just a stack, but the implementation is that there is a
separate variable for the top item, and then the actual stack is all the
other items.
Also add more thorough testing of TOP/BOTTOM, including what happens for
2, 1, and also 0 items on the image stack (errors in that last case).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This allows us to correctly read the ExifVersion and FlashPixVersion
metadata in an EXIF block of a TIFF file.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Implement support for reading and writing monochrome. Reading requires
libheif 1.17+ for heif_image_handle_get_preferred_decoding_colorspace.
Previously writing a single channel image would cause an exception due
to
wrong parameters, but close() would continue writing the image and
crash.
Destroy m_ctx on exception to prevent that for other potential errors.
Test added for monochrome read and write.
---------
Signed-off-by: Brecht Van Lommel <brecht@blender.org>
This is a PR proposing to keep the gamma precision.
In some cases, we need more precise gamma values, while the existing
rounding operation loses most of the precision. This change will
continue to use rounded values to calculate and store color space
information, but retain the original value in the "Gamma" parameter. In
addition, it can also tidy up existing code.
I've verified with png/exif.png & python-colorconfig tests. No
regression is introduced.
Signed-off-by: Lumina Wang <lumina.wang@autodesk.com>
This is a more comprehensive fix for issues discovered in PR #5035.
The original problem reported in Issue #5023 was a crash when writing
TIFF information that was supposed to be arrays of more than one
rational, it was reading past the end of a memory array.
#5035 is a minimal, immediate fix to address the crashes. But in the
process, I saw a number of ways in which we were dropping metadata on
the floor when the types didn't exactly match, but that we *could*
handle with automatic conversion.
The new cases that we handle with this PR are:
* Exif RESOLUTIONUNIT tag is a short, but by convention we store it by
the name as a string in OIIO metadata, so we need to convert back to a
code (we did so for the main TIFF metadata, but not for Exif in TIFF).
* Handle Exif "version" and "flashpixversion" metadata which have
unusual encoding in TIFF files (they are 4-character strings, but must
be stored in a TIFF tag of type BYTES, not as the usual type ASCII that
most strings use.
* Handle things that TIFF insists are ASCII but that come to us as
metadata that's strings. Easy -- our `ParamValue.get_string()`
automatically converts ther things like ints or floats into string
representation.
* Much more flexibility in automatically converting among the signed and
unsigned, 16 and 32 bit, integer types when the metadata in our
ImageSpec is integer but not the specific type of integer that TIFF/Exif
thinks it should be.
This doesn't appear to change the results of anything in our testsuite,
but it's possible that some non-TIFF-to-TIFF image conversions that
contain Exif data may now do certain type conversions properly instead
of just silently dropping the metadata that had non-matching (but
reasonably valid) types.
Additionally, to do this nicely, I ended up adding a new TypeURational
alias in typedesc.h (similar to TypeRational, but the case where both
numerator and denominator are unsigned ints).
And also fixed a random comment typo I noticed in tiffinput.cpp.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Bump the version of 'fmt' library that we download and build (if not
found) from 10.2 to 12.1.
Some other touch-ups in build_fmt.cmake.
Also, we have seen that recent fmt versions will fail to compile on MSVC
unless using the `/utf-8` compiler flag, so ensure that is used and also
passed on to other clients of libOpenImageIO_Util (which expose
templates using those headers).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Fixes#5023
This was crashing when writing TIFF information that was supposed to be
arrays of more than one rational, but in fact was provided as a single
value, it was reading past the end of a memory array.
I noticed that this whole region needs a cleanup, this is not the only
problem. But a full overhaul seems too risky to backport, so my strategy
is as follows:
* THIS fix first, which I will backport right away to 3.0 and 3.1.
* I will then submit a separate PR (already implemented and tested) that
is a much more complete fix and overhaul of this portion of the code
(and other places). That will get merged into main when approved.
* After the second PR is merged, I'll hold it in main for a while to
test its safety, and then decide if it seems ok to backport to 3.1 (but
definitely not 3.0).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Extra protections for corrupted BMP files that claim to be palette
images, but have a BPP that doesn't support palette images. Also an
extra guard around accessing the palette array if it is empty.
Add an extra test case for this kind of corruption.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Mac Intel is getting long in the tooth, and quite often the Homebrew
packages for Intel are found to be uncached and will try to build from
source. When it's OpenCV, that's disastrous for our CI build times, it
can get stalled for hours building all of OpenCV and its dependencies.
So disable it for that one build variant.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Review: we have long had two assertion macros: OIIO_ASSERT which aborts
upon failure in Debug builds and prints but continues in Release builds,
and OIIO_DASSERT which aborts in Debug builds and is completely inactive
for Relase builds.
Inspired by C++26 contracts, and increasingly available "hardening
modes" in major compilers (especially with the LLVM/clang project's
libc++), I'm introducing some new verification helpers.
New macro `OIIO_CONTRACT_ASSERT` more closely mimics C++26
contract_assert in many ways, and perhaps will simply wrap C++
contract_assert when C++26 is on our menu.
Important ways that OIIO_CONTRACT_ASSERT differs from OIIO_ASSERT and
OIIO_DASSERT:
* Keeping in line with C++ contracts, there are 4 possible responses to
a failed contract assertion: Ignore, Observe (print only), Enforce
(print and abort) and Quick-Enforce (just abort).
* Also define hardening levels: None, Fast, Extensive, and Debug,
mimicking the levels of libc++. The idea is that maybe there will be
some CONTRACT_ASSERT checks you only want to do for certain hardening
levels.
* By default, the contract failure response is Enforce, unless it's both
a release build and the hardening level is set to None (in which case
the response will be Ignore). But it's also overrideable optionally on a
per-translation-unit basis by setting OIIO_ASSERTION_RESPONSE_DEFAULT
before any OIIO headers are included (though obviously that only applies
to inline functions or templates, not to any already-compiled code in
the library).
* Macros for explicit hardening levels: OIIO_HARDENING_ASSERT_FAST(),
EXTENSIVE(), and DEBUG(), which call CONTRACT_ASSERT only when the
hardening level is what's required or stricter.
I also changed the bounds checking in operator[] of string_view, span,
and image_span to use the contract assertions. Note that this adds a
tiny bit of overhead, since the default is "enforce" for release builds
(previously, using OIIO_DASSERT, it did no checks for release builds).
But the benchmarks seem to idicate that the perf difference is barely
measurable.
I added some benchmarking that proves that the bounds check adds a
minute overhead to an element access for a trivial `span<float>`, maybe
even indescernable. Here are benchmarks comparing raw pointer access,
std::array access, span access with the new checks, span access
carefully bypassing the tests.
Linux workstation, gcc-11, on my work computer:
pointer operator[]: 647.8 ns (+/- 0.1ns)
std::array operator[]: 647.8 ns (+/- 0.1ns)
span operator[] : 657.6 ns (+/- 0.5ns)
span unsafe indexing: 648.2 ns (+/- 0.2ns)
span range : 648.1 ns (+/- 0.1ns)
These are the most stable tests I have, with the least trial-to-trial
variation, and show about a 1.5% speed hit on the bounds-checked span
access itself, which I think will be truly un-measurable in the context
of being interleaved with any other operations that you do with the data
you pull from the span.
Mac Intel, Apple Clang 17, on my (old) personal laptop: (much more
variable timing, probably from MacOS scheduler quirks)
pointer operator[]: 929.2 ns (+/- 6.7ns)
std::array operator[]: 913.1 ns (+/- 20.6ns)
span operator[] : 905.8 ns (+/- 13.3ns)
span unsafe indexing: 913.9 ns (+/- 16.6ns)
span range : 916.4 ns (+/- 20.3ns)
You can see that here there is no obvious penalty, in fact it appears a
little faster, but all within the timing uncertainty of the multiple
trials, so statistically it's hard to discern any penalty.
And a couple more for good measure from our CI, but note that because
these are uncontrolled machines somewhere on the GitHub cloud, the
timings might not be as reliable:
Windows, MSVS 2022:
pointer operator[]: 3716.3 ns (+/- 6.3ns)
std::array operator[]: 3715.5 ns (+/- 3.4ns)
span operator[] : 3715.6 ns (+/- 2.6ns)
span unsafe indexing: 3712.1 ns (+/- 0.7ns)
span range : 3714.2 ns (+/- 2.9ns)
Linux, gcc-14, C++20:
pointer operator[]: 1130.9 ns (+/- 0.2ns), 884.2 k/s
std::array operator[]: 1132.0 ns (+/- 0.4ns), 883.4 k/s
span operator[] : 1133.7 ns (+/- 0.4ns), 882.1 k/s
span unsafe indexing: 1134.2 ns (+/- 1.6ns), 881.7 k/s
span range : 1133.9 ns (+/- 0.7ns), 881.9 k/s
MacOS ARM:
pointer operator[]: 3456.6 ns (+/- 7.5ns)
std::array operator[]: 3466.8 ns (+/- 12.2ns)
span operator[] : 3610.9 ns (+/- 11.0ns)
span unsafe indexing: 3607.4 ns (+/- 4.9ns)
span range : 3612.4 ns (+/- 12.2ns)
Windows with MSVS and Linux with newer g++ don't appear to show any
penalty, and the bracketing of trial times indicates that maybe it's
consistent enough to be meaningful? I can't think of anything I'm doing
wrong here that would throw off the timing or disable the range checking
on these tests.
For MacOS ARM, the span looks like it has about a 4% penalty versus raw
pointers? But OTOH, span bounds-checked vs non-checked vs range-for are
all the same, so maybe the speed vs raw pointer is something else
entirely?
Also please note that a preferred way to avoid these extra bounds checks
entirely is to change an index-oriented loop like
span s;
for (size_t i = 0; i < s.size(); ++i)
foo(s[i]); // maybe bounds check on each iteration?
to a range based loop:
span s;
for (auto& v : s)
foo(v);
which should be inherently safe and require no in-loop checks at all.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Since the OpenImageIO 2.5 series, when calls to `check_open` were added,
any format that did not declare support for "tiles" would immediately
fail to open. But many of the formats which attempted to emulate tiles,
by buffering the contents and writing it all as scanlines at the end,
were not updated. All of the tile emulation code for these formats is
effectively dead-code and untested.
Remove the tile emulation code from these formats.
An example of what the failure currently looks like:
```python
>>> out = oiio.ImageOutput.create("test.png")
>>> spec = oiio.ImageSpec(64, 64, 3, 'uint8')
>>> spec.tile_width = 64
>>> out.open("test.png", spec)
False
>>> out.geterror()
'png does not support tiled images'
```
No tests were impacted.
Signed-off-by: Jesse Yurkovich <jesse.y@gmail.com>
Need to test some MSVS-specific macros to determine what architecture to
report.
And especially, if it doesn't know the processor architecture, it still
should be *appending* that to the platform, not replacing it! This
caused MSVS-compiled OIIO on Windows to report "unknown arch?"
Signed-off-by: Larry Gritz <lg@larrygritz.com>
I think it was basically harmless, since we do all the metadata name
comparisons using case-insensitive comparisons. But we use "Exif:" as
our prefix for Exif data throughout OIIO by convention, and there was
this tiny handful of places where we said "exif:".
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Even though we have CI testing on Mac with ARM CPU that were passing,
after getting a new laptop, I saw some test failures that were due to
just a few pixels on a few tests needing a higher comparision threshold.
Results are correct, just different due to the math. I guess this
machine (CPU? build flags? specific compiler or library versions?) is
ever so slightly different than the CI Macs, so I caught a few more
instances that needed to be adjusted.
I tried to increase the thresholds as little as possible to fix the
problem.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Fixes#4715
- Added save functionality
- Added flipping and rotation capabilities via metadata.
- Hotkeys: Ctrl-Shift-R and L to rotate the image left and right.
Note that the actual pixel data remains unchanged, so only image formats
that support the rotation metadata will reflect the changes.
I thought that it made sense for the flip and rotate to be under the
tool section, similar to where the macbook Preview app places it.
Keyboard shortcuts also follow the shortcuts that Preview uses.
Tests
Used a .jpeg photo to test:
- saving
- making changes with/without saving and re-opening the image
- flipping and rotation from different initial rotation/mirrored states
Otherwise there doesn't seem to be a test suite for iv?
Signed-off-by: valery <71804886+vangeliq@users.noreply.github.com>
There's something in pybind11 master at the moment that is crashing in
its destructors. It's been causing our "bleeding edge" test to fail for
over a week now. I'm tired of our test failing, so I'm locking down to
the last known working version. Will check back periodically and return
to testing against pybind11 master after they have fixed it.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Makes OpenColorIO 2.5.1 the default version when building OCIO locally.
This will cause the released python wheels to build against the
currently-latest version of OCIO.
Signed-off-by: Zach Lewis <zachcanbereached@gmail.com>
Like other file formats, the returned ImageSpec should indicate if the
image contains unassociated alpha.
This was an oversight in #4770.
Test added.
Signed-off-by: Brecht Van Lommel <brecht@blender.org>
Add IOProxy support similar to other file formats.
MyHeifWriter was renamed for consistency with other code.
All input and output now goes through the proxy, so this is covered by
existing tests.
Signed-off-by: Brecht Van Lommel <brecht@blender.org>
Utilities to ask the IB for the local pixels as an untyped span of
bytes. This is a "safe" alternative to `localpixels()`, which just
returned a single pointer, instead returning an image_span that
understands the sizes and strides of the buffer.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Nearly all IBA functions take an optional parameter controlling the
threading. But make_texture() did not, so there was no way to control
its thread usage (it would always use the default number of threads).
Without changing the call signature or ABI, this patch merely makes
make_texture honor any "maketx:threads" hint passed in the config
ImageSpec that it already takes to convey all sorts of controls over the
texture creation process. Then this value is passed to any IBA
functions, use of parallel_image, and anything else in the
implementation of make_texture that would end up using the thread pool.
Fixes#4254
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The WebP format is very limited in the maximum resolution it supports.
We need to change the values we allow to be much smaller.
Signed-off-by: Jesse Yurkovich <jesse.y@gmail.com>
Initializing the encoder to `heif_compression_HEVC` by default throws an
exception when libheif was built without HEVC support, which prevents it
from being used entirely even if there is AVIF support.
So delay that initialization until we are sure which encoding we want.
Not really any practical way to automatically test this.
Signed-off-by: Brecht Van Lommel <brecht@blender.org>
We previously merely *marked* tests as "broken" in this case, and
depended on ctest being launched with `-E broken` to ensure those were
skipped. Not everybody did. Let's just truly skip them.
Fixes#4979
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Needed to change the Libheif::Libheif target we set up ages ago to the
"heif" name that the package's exported config uses.
Also, need to give the warning about static libraries occur any time we
are using static libheif, even if we didn't set LIBSTATIC to try to
force all static libraries.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* ImageBuf internal buffer span lacked correct chansize. The internal
`m_bufspan` is an `image_span<byte>`, and as such, it needs to remember
the size of the original data type. Otherwise, there's a cascade of
potential errors when it thinks that the individual values are byte
sized.
* In both ImageInput and ImageOutput, several sanity checks of
image_span size versus expectations were incorrect. They were only
checking if the total byte sizes matched expectations, but they are
allowed to disagree when you consider type conversions (in which case,
it's the total number of values that need to match, not the total byte
sizes.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Yay, I love how these things just break with no notice.
Fixes CI that broke a couple days agi.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Needed for some systems after the changes of #4963.
Ever so slightly different LSB somewhere makes the hashes not match, but
it's a correct visual result.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Due to typos in the option name in compiler.cmake (HARDENING vs
OIIO_HARDENING, oops), I think we were never really setting the intended
compiler flags. Fix that all up, and repair the other problems that it
revealed -- some compiler version and option combinations weren't happy
with each other, etc.
One notable change is in encode_iptc_iim_one_tag: I think I made it
safer by taking an early out for 0-sized data, and also it needed a
warning suppression when certain gcc hardening levels were used.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The Intel MacOS 15 CI testing is getting dicier... lots of times,
Homebrew doesn't have cached versions of updated packages, so it tries
to build from source, which takes forever. The big culprit today is Qt.
So, basically, just on this one CI job variant, don't ask it to install
Qt. If it's there, it's there. If not, just skip it. It's tested plenty
in other variants.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* CI test vs the latest freetype 2.14.1
* Bump the version of freetype that we auto-build to the latest (from
2.13.2)
* Simplify BZip2 finding logic, switch to using targets
Signed-off-by: Larry Gritz <lg@larrygritz.com>
For IBA::resample() when bilinear interpolation is used, almost all of
the expense was due to its relying on ImageBuf::interppixel which is
simple but constructs a new ImageBuf::ConstIterator EVERY TIME, which is
very expensive.
Reimplement in a way that reuses a single iterator. This speeds up
IBA::resample by 20x or more typicaly.
Also refactor resample to pull the handling of deep images into a
separate helper function and out of the main inner loop. And add some
benchmarking.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Starting with 1.21, libheif seems to change behavior: When no CICP
metadata is present, libheif now returns 2,2,2 (all unspecified) on
read. OIIO convention, though, is to not set the attribute if valid CICP
data is not in the file.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Implement RLE compression support for the SGI output plugin. Reading RLE
encoded images was already supported, but writing was never done up
until this point.
The existing sgi test seems sufficient to catch issues and it covers
input/output of both 1 byte-per-pixel and 2 byte-per-pixel files.
The documentation for the image plugins are sometimes not very clear
about which attributes are relevant for input vs. output. There's
usually 3 sections: Attributes, Attributes for Input, and Attributes for
Output.
Before this PR, SGI mentioned the "compression" attribute in the
"general" Attributes section (rather than say just the Input section),
which caused a bit of grief as the only way to discover that RLE was not
implemented for Output was to glance at the file size of the resulting
file... I had assumed that compression was supported for output too but
discovered that it was not.
Now that this PR implements the attribute for output I've left the
documentation as-is in the "general" Attributes section since it applies
to both read/writing now. But I'm open for suggestions here.
Signed-off-by: Jesse Yurkovich <jesse.y@gmail.com>
Rearrangements in 3.1 dropped the list of recognized attributes from
the visible online docs and failed to document the span varieties. We
fix and also reword a lot of the descriptions for clarity and uniformity.
The previous organization was that there were several varieties of attribute(). In the header, the first one had the overall long explanation, including the list of all the recognized attributes. The other ones had short explanations of how they differed. In the docs, each one was referenced explicitly, pulling in its attendant bit of documentation.
What really happened is that in the header, I made the new span-based version the "flagship" one with the full explanation, but I neglected to reference it in the docs, so the long description disappeared.
I could have fixed by just adding refs to the new functions to the docs, as I originally meant to. But while I was there, I took the opportunity to surround the whole collection with a group marker, and then include the lot of them with a single reference to the group, rather than need to refer to each function variant individually. And while I was at it, I also reworded (and hopefully improved) some of those explanations.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The JPEG XL color encoding metadata only supports a subset of CICP. So
for
example `srgb_p3d65_display` and `pq_rec2020_display` are supported, but
`g26_xyzd65_display` is not.
Custom primaries, custom white point and arbitrary gamma could be used
to
support more, but I didn't implement that.
Tests for read and write were added.
Signed-off-by: Brecht Van Lommel <brecht@blender.org>
If the colorspace exists and has an interop ID in an OCIO 2.5 config,
use that.
Otherwise check if the colorspace is equivalent to a known color interop
ID.
Tests were added.
Signed-off-by: Brecht Van Lommel <brecht@blender.org>
Periodic revisiting of the phrasing in the hopes of improving PR and
Issue quality / actionability. Not that I think they are bad now, just
always looking to guide people to improvements.
Will it work? 🤷 Let's try and see if we think PRs improve. If not, we
can revise again later.
Also make the issue title prefixes less yelly and shorter (makes it
easier to see the actual title on phone or other limited width display
of messages).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Each file writer that is capable of writing arbitrary metadata is
supposed to tace care to suppress hints meant for other writers, or for
OIIO in general, not interpret them as literal metadata to write to the
file. PNG was not doing so, and ended up writing hints as literal
metadata. This brings it in line with our behavior when writing OpenEXR
and other formats.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The GaussianPyramid::value() method used by IBA::compare_Yee() was
incorrectly accessing channel 1 instead of channel 0. This fix corrects the
channel access to properly retrieve the luminance value from the
single-channel pyramid levels.
One oddity this led to is that `idiff -p` would appear to PASS for
some clearly different images if they had different data windows.
Update reference outputs to reflect the corrected perceptual diff results
(Max error changes from 1 to 10 for the test images).
Fixes#4948
Signed-off-by: pmady <pavan4devops@gmail.com>
The default Ubuntu images keep shifing around. Last week, some packages
that used to be installed disappeared. Need to adjust what we install in
gh-installdeps.bash, and slightly adjust the optional deps in ci.yml to
allow some things that seem to be only on some runners to be missing.
Also bump the clang test to clang18 on an ubuntu-24 runner (from clang
15 on ubuntu 22).
Gets us back to all-passing CI after a week of all sorts of failures in
tests that used to work.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
If we don't specify anything, it seems that Ptex will think it's using
C++98, which it is in fact not capable of doing.
Meanwhile, since OIIO's own minimum requirement is C++17, it certainly
meand that any compiler we're using can support 17 as a minimum when
building Ptex as well.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
There seems to be a couple seconds overhead for each "brew install".
Refactor install_homebrew_deps.bash to first assemble the list of all
packages to install, and then do a single `brew install` for the whole
list.
Also remove mention of packages that are pre-installed on the Mac GHA
runners or that we don't need.
In the process, by using the `OIIO_BREW_INSTALL_PACKAGES` variable and
only setting it if empty, we now have the ability to fully override the
list of packages that brew will install on a per-job basis.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
On Windows, `Filesystem::exists` forward the call to
`std::filesystem::exists`, and finally calls `_std_fs_get_stats` .
`_std_fs_get_stats` is slow if the argument is a netwok path.
Signed-off-by: Jackson Sun <jackson.sun@autodesk.com>
When reading image files with CICP metadata, automatically set the
corresponding "oiio:ColorSpace". When writing files that support CICP
and no other colorspace metadata can represent "oiio:ColorSpace",
automatically write CICP metadata.
Setting "oiio:ColorSpace" on read prefers scene referred over display
referred color spaces, changing existing behavior as little as possible.
The alternative would have been to interpret the presence of CICP
metadata as an indication that the image is likely display referred,
which might be reasonable too. Either way it's a guess.
There is no automatic mapping from `g22_rec709_display` to CICP
currently to keep behavior unchanged, as this is often not what you want
and further discussion is needed to decided on the right behavior.
Also add new ColorConfig `get_cicp` and `get_color_interop_id` API
functions to share logic between file formats.
## Tests
Tests were updated, the auto detected color space name appears in the
output.
Commands like these should also do the right thing automatically.
```
oiiotool in.exr --ociodisplay "Rec.2100-PQ - Display" "ACES 2.0 - HDR 1000 nits (P3 D65)" -o out.avif
oiiotool out.avif --autocc -o out.exr
```
---------
Signed-off-by: Brecht Van Lommel <brecht@blender.org>
Move write_complete_data() to close() for proper scanline order support
The WebP output plugin incorrectly assumed that scanlines would be
written in sequential order, triggering the final write when y == height
- 1. This violated the write_scanline API contract which allows
scanlines to be written in any order (the plugin even advertises
'random_access' support).
Changes:
- Remove premature write_complete_data() call from write_scanline()
- Move final encoding/writing to close() where it belongs
- Add m_image_complete flag to track write state
- Move buffer cleanup to after write_complete_data() in close()
Fixes the issue where writing scanlines out of order would result in
incomplete or corrupted WebP files.
Signed-off-by: pmady <pavan4devops@gmail.com>
- Mount the host's root under /host/root so it's visible in the
containers.
- Remove stuff we don't need to free lots of disk space.
- While we're at it, arrange for the commands (copied from OSL) that
allow the containers to see a CPU, when available.
This is freeing 20+ GB!
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
I think something changed with Homebrew on the runners, this worked
until last week.
Emergency fix, will merge immediately if CI finally passes.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The EXR reading code was correct. These attribute names are case
sensitive in OpenEXR.
Test output was updated.
Signed-off-by: Brecht Van Lommel <brecht@blender.org>
Was just wrong. Presumably broken recently.
In the process, switched internal cmyk->rgb conversion to be span based.
Fixes#4962
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We were supporting Exif 2.3, but in 2024, Exif 3.0 was published and it
adds some more tags for us to recognize.
The only part I am not sure about here is that the ExifVersion we output
is still marked as 2.3, not 3.0, because I'm worried that if we mark it
as 3.0, older software reading the images will reject it. Let's see what
happens. If downstream apps care, maybe we can bump to 3.0, or even bump
to 3.0 only if any of the 3.0 tags are used (that is more involved).
Fixes#4960
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Result of our annual license scan.
* Add license notices some places we had missed (mostly documentation).
* Double check some of the code files that are still marked as a mix of
Apache 2.0 + BSD, see if there are any remaining lines that are from
authors that hadn't relicensed their contributions -- there were a few
that had none of those old lines left, so they can be changed to
advertise that they are purely Apache 2.0.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
A few days ago, the icc and icx CI tests started failing, saying they
were out of disk space while installing dependencies (specifically, the
intel compilers themselves).
Now, the VFX Platform 2023 asf docker image I'm using hasn't changed,
and neither has the years-old Intel rpms with the compilers. So maybe
the runners themselves just changed again, in a way that adds some
pre-installs and leaving us not quite enough disk space for everything
else we need?
Asking for a few less things to get installed seems to free up just
enough space. And for the icx one, I also swtiched to the 2025
container, which let me use a slightly newer icx version.
Also: Don't try to install giflib needlessly, which speeds up the setup
time. And remove some dead code in my dependency setup script.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Recently, when building OIIO using `OpenImageIO_BUILD_LOCAL_DEPS=all` cmake option, I noticed a reference to `lerc` that fails during the linker stage, when compiling the simd tests:
```
[ 71%] Building CXX object src/libutil/CMakeFiles/simd_test.dir/simd_test.cpp.o
[ 71%] Linking CXX executable ../../bin/simd_test
/usr/bin/ld: ../../lib/libOpenImageIO.so.3.1.7: undefined reference to `lerc_decode'
/usr/bin/ld: ../../lib/libOpenImageIO.so.3.1.7: undefined reference to `lerc_encodeForVersion'
/usr/bin/ld: ../../lib/libOpenImageIO.so.3.1.7: undefined reference to `lerc_getBlobInfo'
collect2: error: ld returned 1 exit status
gmake[2]: *** [src/libutil/CMakeFiles/simd_test.dir/build.make💯 bin/simd_test] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:2158: src/libutil/CMakeFiles/simd_test.dir/all] Error 2
gmake: *** [Makefile:166: all] Error 2
```
`lerc` is an image compression library from ESRI: https://github.com/Esri/lerc
A deeper dive of this reference to lerc shows that it is referred in `libtiff`. `libtiff`, however, did not seem to correctly export this dependency to OIIO when being built locally.
If someone was building OIIO with a system-installed `libtiff`, this error is unlikely to happen.
By disabling `lerc` in `libTiff`, the local build scripts ( `OpenImageIO_BUILD_LOCAL_DEPS=all` ) runs without a problem.
Tested build OIIO with `OpenImageIO_BUILD_LOCAL_DEPS=all` with this patch and was successful in finishing the build.
In exr output, we were doing attribute name substitutions using
case-insensitive comparisions, whereas on the exr input side, we were
using case-sensitive comparisons. This was leading to some strange
behavior where certain attributes were not properly "round tripping"
when reading and writing exr files, particularly when it involved
attribute names that differed only in case from certain
reserved/standard names.
Also be a little more flexible in adjusting exr attributes that have the
wrong type, by being able to convert int or float to a value that's
required to be a string.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Two Ubuntu-based cases (debug build and static build) started failing
this past week, unrelated to us. I think the runners changed yet again,
it seems to be having particular trouble because those tests were trying
to use gcc-9. But there's no particular need for that anyway, it just
reflects how long ago we last modified those tests.
So this spurred a grab bag of CI fixes:
* Move the debug and static linkage tests off of raw ubuntu runners and
to using the ASWF docker VFX Platform 2025 containers. This makes them
more modern and also more stable.
* Turns out the "static" test hadn't been working properly lately -- it
was accidentally building dynamic libs, not static at all! Fix that, but
then it uncovered some problems that are out of scope to fix in this PR.
I will look into it separately, but in the mean time I have commented
out the static build test to suppress the failures.
* Bump a few of the "latest" dependencies to the most recent.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Almost all the other unit tests do this, just adding it to
image_span_test: automatic reduction in test load for CI and debug
renders by doing fewer time trials. This cuts down on runtime for CI,
but stil exercises the test code. Command line arguments allow you to
override, in cases where you need really good timing fidelity even for a
debug run.
Also conform simd_test to the usual standard `-iters` used elsewhere
instead of being the one place that uses `-iterations`.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
I think that in a recent checkin, we accidentally made some ref outputs
for this test be identical, that were supposed to be for when libheif is
built without avif support.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Ugh, used the wrong option name. Also, it's sufficient to just do it
with the env variable, don't need extra logic in ci-build.bash.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Closes#4791
This PR introduces the `oiio:ACESContainer` hint for OpenEXR outputs.
The hint can take one of the following values: `none` (default),
`strict`, or `relaxed`. If not `none`, the spec will be checked to see
if it is compliant with the ACES Container format defined in [ST
2065-4](https://pub.smpte.org/pub/st2065-4/st2065-4-2023.pdf). If it is,
chromaticities will be set to the ACES AP0 ones, and the
acesImageContainerFlag attribute will be set to 1. In `strict` mode, if
the spec is non-compliant, the output will throw an error and avoid
writing the image. While in `relaxed` mode, if the spec in
non-compliant, only a warning will be printed and the attributes
mentioned above will not be written to the spec.
I've added several tests under `openexr-suite`. These use oiiotool to
attempt writing several EXRs with the `oiio:ACESContainer` hint.
---------
Signed-off-by: glowies <self@oktaycomu.com>
Signed-off-by: Oktay Comu <self@oktaycomu.com>
## Description
This PR fixes and improves a handful of wheels-related items:
- Fix an issue where wheels produced with v3.1 releases were not getting
uploaded to pypi
- Update cibuildwheel to latest version (v3.2.1)
- Remove the manylinux2014 builds, replace with manylinux_2_28 builds
where applicable
- Add CPython 3.14 wheels
- Update macos-x86 runner to `macos-15-intel`
- Update Python typestubs
## Notes
- This PR ends support for pre-built wheels for glibc < 2.27 (CentOS-7
etc). However, the module can still be pip-installed locally, provided
devtoolset-7+ (e.g., gcc-9+) is available.
- Unless we arrive at a more elegant solution, we must remember to
change the conditional in the "upload_pypi" step for every major OIIO
release.
---------
Signed-off-by: Zach Lewis <zachcanbereached@gmail.com>
Due to apparent limitations or bugs in swscale, there is a problem
reading the 16 bit green channel that is not 4 byte aligned like red and
blue. Reading each channel into its own plane solves the issue.
This also addresses an existing comment about 32 bit float being read as
16 bit. This needed the same mechanism of reading separate planes, so
the same code is reused to implement that.
Resolves#4901
I added a 10 bit test file encoded with VP9 and Matroska, same as the
existing 8 bit test file to maximize compatibility. I verified this now
converts correctly to a HDR PNG file.
Signed-off-by: Brecht Van Lommel <brecht@blender.org>
A small portion of time -- maybe 10%, just enough for many CI workflow
runs to have a single job fail -- our attempt to `dnf install ffmpeg`
fails.
I thought maybe it was an intermittent, transient failure. But looping
to try multiple times with delays in between does not help. A job that
fails once either succeeds initially or fails over and over. I'm now
leaning toward the belief that rather than a transient failures, there
are just a subset of GHA runners that will always fail, and the
semi-random nature of our job failures is just random luck of the draw
in runner selection.
Anyway, so I'm not going to merge the try-in-a-loop approach. Instead,
this PR just makes ffmpeg an optional dependency, so the failures to get
the dependency won't fail the OIIO build and therefore the CI. It seems
like most of the time we really do get the dependency, but I'm out of
better ideas at the moment.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Support for retrieving and setting the idManifest for deep image files
with object ID channels. This is an OpenEXR feature where the image
header can contian metadata that maps all the object ID channel integer
codes to strings of the object names.
We report this in the ImageSpec as the metadata
"openexr:compressedIDManifest", whose type is an array of uint8 (byte)
values. It is encoded the same way it appears in the exr file itself:
the first 8 bytes are a little endian uint64_t giving the *uncompressed*
size of a serialized OpenEXR IDManifest object, and then starting at
byte 8, the zip-compressed seriailized IDManifest. Therefore, the
compressed block size is the size of the metadata blob, minus 8 bytes
for the length. It is up to the caller to use the OpenEXR APIs to turn
this compressed serialized IDManifest into a fully expanded IDManifest,
if that's what they want to do.
It works for output, too, so it should copy the manifests correctly
between files, even if the app doesn't know what to do with the binary
data it contains.
Also added support for deepImageState, which we previously had ignored.
We report this as the metadata "openexr:deepImageState", a string, which
if present has one of the values "messy", "sorted", "non_overlapping",
or "tidy".
Signed-off-by: Larry Gritz <lg@larrygritz.com>
IffInput::read_native_tile was simply incorrect for images with nonzero
data window origin. Fix!
Also switch the forumulation to use spancpy rather than memcpy, to be a
little more careful that we are't overwriting the presumed sizes of the
buffers.
And while I'm in there, I realized we can hold the mutex for less time.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The CMake (or env) variable OpenImageIO_BUILD_MISSING_DEPS enumerates
dependencies to try to locally auto-build if not found. If set to "all",
it will build them all.
This patch also allows you to set it to "required", which will try to
auto-build any missing dependences that are required, but not bother
with non-required missing packages.
It also works to use required and also have list items naming additional
optional dependencies, such as
-DOpenImageIO_BUILD_MISSING_DEPS="required;WebP"
build all required dependecies, and also WebP (which is optional), if
they are no found already installed.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Two places where the docs and header for ImageBuf::set_pixels did not
match.
In one case, the docs were right, the declaration was supposed to have a
parameter that was a `const image_span<T>&` but the `&` was left out.
Now, if this was an actual linkable function, we wouldn't be able to fix
it until a compatibility-breaking boundary. But because it's a template
and inline, I believe it is actually safe to fix it in situ without
breaking ABI compatibility (and it seems to pass our ABI check in CI).
In the second case, the header was correct but the docs used the wrong
declaration (leaving out a `const`) and made the docs fail to pull in
the right explanation.
Fixes#4923
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Fixes#4641
Has something to do with mixing Windows compiler versions, there's a
subtle ABI compatibility issue that this sidesteps.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Building the python wheels takes a long time! I really hate waiting for
that when testing PRs, there must be a way to speed it up.
* For the wheel workflow, add cache actions to save and restore the
CCACHE_DIR. Hey, it's no easy feat to figure out what the path should be
to that directory, especially on Linux where the wheels are built in a
container, so the paths inside the container (where the wheel is built
and the C++ compilation happens) don't match the paths outside the
container (where the cache restore and save actions execute).
* I had a heck of a time on Linux trying to get a pre-built ccache
installed and had to resort to writing a bash script to build ccache
itself from scratch, which is much more expensive than I'd like, but
we'll have to come back to fix that separately.
* Changed our "auto-build" utility build_dependency_with_cmake to print
the amount of time it takes to build each dependency.
* When auto-building, pass along CMAKE_CXX_COMPILER_LAUNCHER so that the
dependenencies also for sure use ccache.
* Use CMAKE_BUILD_PARALLEL_LEVEL on the wheel run to use all the cores
and compile in parallel. (We did that on the regular CI but I think not
for the wheel building.)
* Fixes to the logic in our compiler.cmake where it tries to use ccache
even if the magic CMAKE_CXX_COMPILER_LAUNCHER isn't set -- I have come
to believe we were doing it wrong before, it was having no effect, and
all along we only got ccache working on CI because we *also* set the env
variable.
* For CI, set CCACHE_COMPRESSION=1 to make the caches take less space
against the precious limit of how much cache we can use total on GHA.
So, the result of all this:
**Previous times (typical), and first wheel run for any git branch**
| platform | total | compile OIIO + deps |
| ----------- | ----- | ------------------- |
| Linux Intel | 10:35 | 500s |
| Linux ARM | 7:20 | 294s |
| Mac Intel | 20:18 | 1146s |
| Mac ARM | 7:19 | 388s |
| Windows | 14:00 | 759s |
**With ccache active, 2nd or more wheel run for a git branch**
| platform | total | compile OIIO + deps |
| ----------- | ----- | ------------------- |
| Linux Intel | 3:33 | 98s |
| Linux ARM | 3:34 | 83s |
| Mac Intel | 5:01 | 212s |
| Mac ARM | 2:30 | 95s |
| Windows | N/A | not using ccache |
The "compile OIIO + deps" column is the isolated time to build OIIO and
also any auto-building of dependencies from source that we do. But it
does not include any other setup, such as 40-60s of container setup on
Linux, 20-40s setting up Python on Mac, or -- ick! -- 45-60 seconds to
build cmake itself from scratch.
So this is considerably better, speeding up the full wheel workflow by
2-4x on all platforms but Windows. Remaining room for improvement
(possibly in subsequent PRs, hopefully not necessarily by me):
* Is there a way to use ccache on Windows? I'm not sure if there is,
when using MSVS.
* Find a way to install pre-built binaries on Linux rather than building
ccache from scratch, which would save almost a whole minute per job.
* Organize each platform to build all of its wheels (i.e. all of the
python versions we are building for on that platform) in a single job
rather than as completely independent jobs, allowing (a) the fixed
per-job overhead such as container initialization, and installing
python, and building of certain dependencies to happen ONCE per platform
instead of separately for each wheel, and (b) the magic of ccache to
speed up the builds *across* those wheels, since only a tiny amount of
OIIO source codes depend on python at all.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Fixes#4387
Added build_GIF.cmake module for GIF library dependency management.
Since GIFLIB does not provide a CMakeLists.txt, I created a custom one
that gets added to the cloned repository.
The flow is similar to other libraries with cmake support, so I added a
conditional statement in the build_dependency_with_cmake macro that, if
a cmakelists.txt path is provided, will add/replace existing
CMakeLists.txt files on the cloned repository.
Signed-off-by: Valery Angelique <71804886+vangeliq@users.noreply.github.com>
Version 3.1 removed this definition, thinking it was safe because we
didn't use it anywhere, didn't document it, and what it expands to
(`using namespace OIIO`) isn't any longer than the macro name, so is
pointless to use.
But downstream projects did use it. Oops.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
A test needed updating because the set of color spaces in the built-in
config are different than for 2.4.
Fixes#4915
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Raise the version we "autobuild" for some packages:
* OpenJPEG 2.5.2 -> 2.5.4
* pybind11 3.0.0 -> 3.0.1
* libtiff 4.6.0 -> 4.7.1
* webp 1.4.0 -> 1.6.0
* libjpeg-turbo 3.0.4 -> 3.1.2
And some minor debugging and other aids along the way:
* After a build, on linux, do a `ldd` to verify what libraries are
linked dynamically for the logs.
* Enhance build_dependency_with_cmake macro to allow optional commands:
GIT_COMMIT, GIT_SHALLOW, QUIET
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Two small fixes to compile on musl libc systems.
The first commit is about the use of `__WORDSIZE`, which is not a
standard define and is defined another header in musl libc. I decided to
fix this by following what was done in `libdnf` in [this
commit](https://github.com/rpm-software-management/libdnf/pull/1159/commits/82a07dbe6e53fccfe3a8f8e0d951618e0aa72999).
However, there are other ways, such as including `bits/reg.h` or
`bits/user.h`, which define `__WORDSIZE` in musl libc, or using
`__LP64__` to check for 64 bit instead (inspired by [Alpine Linux's
patch](https://gitlab.alpinelinux.org/alpine/aports/-/blob/0f71e9504b59049e0ec1abb486ba455ab2650c83/community/openimageio/0001-fix-compile-error.patch)):
```c
#if defined(__GNUC__) && (__WORDSIZE == 64 || (defined(__LP64__) && __LP64__)) && !(defined(__APPLE__) && defined(__MACH__)) || defined(__NetBSD__)
```
The second commit is about the use of the `feenableexcept` function.
This function is only defined by glibc. The original code assumed all
Linux systems use glibc (checking `__linux__` define), so I just changed
it to check for glibc directly (by checking the `__GLIBC__` define).
I have only tested it on my musl libc system. I think just compiling it
on different systems is enough of a test in this case...
---------
Signed-off-by: omcaif <103504667+omcaif@users.noreply.github.com>
Fixes#4799
When trimming an image that contains an empty subimage, the resulting
ROI unexpectedly included the top-left pixel even if zero in all
subimages.
To avoid this, empty ROIs are ignored when computing the overall
non-zero region.
If all subimages are empty, the first subimage ROI is used and doctored
to be 1 pixel.
Two tests have been added to the oiiotool testsuite:
- 1 to check the trimming of an empty image
- 1 to check the trimming of a non-empty image with an empty subimage
Signed-off-by: Carine Touraille <carine@raynault.com>
Closes https://github.com/AcademySoftwareFoundation/OpenImageIO/issues/4752
Add Python type hints into documentation, in places where example code
defines a function. That ended up being just
`src/doc/pythonbindings.rst`. I also updated `testsuite/runtest.py` and
the functions in `testsuite/docs-examples-python/src/docs-examples-*.py`.
While I was in `pythonbindings.rst`, I noticed a couple places with
`py::method` (note the double colon). These lines didn't render in the
docs, so I removed the extra colon to match other usage.
I locally built the docs to make sure html looks good. I ran the tests
with `ctest` and the tests ran, so I'm taking that as a sign that
`runtest.py` still works.
Signed-off-by: Connie Chang <connie.chang@dreamworks.com>
We recently had a problem where our use of a dependency was broken, and
it had gone unnoticed because none of our CI jobs actually had the
dependency present and tested it. This can easily happen because so many
of our dependencies are "optional" -- our build will just give a warning
and silently disable the functionality that would have been supported by
the missing dependency.
We have long had build-time options `OpenImageIO_REQUIRED_DEPS`, to
consider specific dependencies (or "all") required even if they would
ordinarily be optional, and `OpenImageIO_OPTIONAL_DEPS`, to make
exceptions. But we didn't use these in CI.
So this PR makes sets `OpenImageIO_REQUIRED_DEPS=all` to make all
dependencies ostensibly required, and then list all exceptions
explicitly. This should make it much more difficult in the future to
make a mistake where use of a dependency is completely untested in our
CI without our being aware of it. And it gives us a visible "hit list"
of untested or under-tested dependencies to slowly whittle down.
Some changes that came long for the ride:
* checked_find_package: explicitly disabled packages are treated as
optional.
* cuda_macros.cmake: Only look for CUDA on platforms that might
conceivably have it (i.e., don't even look on Mac).
* Separate linux-aswf from linux-ubuntu into separate job groups,
because that makes it easier to have shared commonalities of which
dependencies they test.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
On Windows, using a wildcard input path containing `\` back slashes
would lead to failures because it would be compared with file paths
containing `/` forward slashes. Use the existing
`Filesystem::generic_filepath` API (which calls `path::generic_string`)
to get a normalized path containing consistent directory separators.
An example of the existing failure:
```
.\oiiotool.exe -v t:\render\foo_#.png --resize 160x90 -o t:\smaller_#.png
oiiotool WARNING : No frame number or views matched the wildcards
oiiotool ERROR: read : File does not exist: "t:\render\foo_#.png"
Full command line was:
> oiiotool.exe -v t:\\render\\foo_#.png --resize 160x90 -o t:\\smaller_#.png
```
Workaround - without this PR you would have to use an input path
containing `/`:
```
.\oiiotool.exe -v t:/render/foo_#.png --resize 160x90 -o t:\smaller_#.png
Reading t:/render/foo_0001.png
Writing t:\smaller_0001.png
Reading t:/render/foo_0002.png
Writing t:\smaller_0002.png
```
Signed-off-by: Jesse Yurkovich <jesse.y@gmail.com>
These additional things need not trigger a CI test build of the online
documentation:
* Build scripts
* Markdown docs that aren't part of the documentation we're generating
(like the main README.md, INSTALL.md, etc.)
Doing so just makes CI take longer and wastes resources.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We recreated the SonarCloud project to clear the 404s. This commit fixes
the workflow condition by comparing to the string '1', so the Sonar scan
step runs and uploads results.
Updates the GHA workflow guard so the Sonar scan actually executes:
- Change `if: inputs.sonar == 1` → `if: inputs.sonar == '1'` because
`inputs.sonar` is a string (from `workflow_call` inputs).
No unit tests added (CI-only change). Validation: run the “Analysis”
workflow and confirm SonarCloud receives results.
Signed-off-by: Vanessa Valderrama <vvalderrama@linuxfoundation.org>
PR #4875, switched to using openjph's exported cmake config instead of
our own FindOpenJPH.cmake, and in the process also renamed OpenJPH ->
openjph to follow their convention.
But I botched it, still using the old OPENJPH_LIBRARIES (etc.)
variables instead of fully switching to the correct targets exported
from their config.
This PR minimally fixes that error, re-enabling use of openjph.
Fixes#4893
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The version of open that takes an array of subimage specs didn't use
them properly for finding the FramesPerSecond hint (or the ioproxy!).
Fixes#3716
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This adds thumbnail support to the raw input plugin.
Fixes#4107
This supports 3 of the 4 formats of thumbnails Libraw provides: BMP,
JPEG, JPEGXL(untested). H265 is not supported, I don't know of any
camera using it, and expect the code being significantly different from
the other formats, given that the H265 is a movie container.
One thing I don't like about this implementation - I understand there is
no way to query the number of thumbnails available in the file? We are
expected to iterate, until `get_thumbnail()` returns false? If
`get_thumbnail()` returns false not because there is no thumbnail for
the requested index, but any other reason, we are not going to try
reading past that index.
## Tests
I have tested manually with a bunch of raw images having either JPEG or
BMP thumbnails.
I have not tested the JPEGXL branch, as I'm not aware of any cameras
using that for thumbnails. The code is identical to the JPEG branch
though, so I'm not expecting any issues with that.
I couldn't see any unit tests around the thumbnail functionality at all.
Should I add some? I assume we can't use `oiiotool` for this?
---------
Signed-off-by: Anton Dukhovnikov <antond@wetafx.co.nz>
PR #4880 checked in some output reference that contained version
numbers, which passed in main but failed when backported to 3.1. This
repairs it and makes the output suitable for any version that contains
the patch.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Fix SonarCloud analysis failures by switching the C/C++ configuration
from the
deprecated `sonar.cfamily.build-wrapper-output` property to the current
`sonar.cfamily.compile-commands` property.
**Changes**
- Remove `sonar.cfamily.build-wrapper-output` from
`sonar-project.properties`.
- In `.github/workflows/build-steps.yml`, pass
`--define
sonar.cfamily.compile-commands="$BUILD_WRAPPER_OUT_DIR/compile_commands.json"`.
**Root cause**
The SonarCloud config still used the deprecated
`sonar.cfamily.build-wrapper-output`
property. The current CFamily analyzer expects a `compile_commands.json`
(produced by build-wrapper). Using the old property caused scanner
failures and
blocked analysis.
**Validation**
Regenerated `bw_output/compile_commands.json`, ran `sonar-scanner`, and
confirmed the scan completed and uploaded successfully.
**Tests**
CI-only change. Verified locally by running the scanner against a fresh
build-wrapper capture.
---------
Signed-off-by: Vanessa Valderrama <vvalderrama@linuxfoundation.org>
There are still a few historical authors who never signed on to the
relicensing of OpenImageIO from BSD-3-Clause to Apache-2.0 two years
ago, usually because we could find no current way to contact them.
The few (< 25) files that still had their extant code were therefore
marked as using both licenses. They they almost always are down to just
a few lines in the whole file that are still original to those authors
(and therefore BSD), so every once in a while, natural code churn and
rewrites of what they touched remove the very last of these lines from a
file, leaving that file entirely Apache-2.0. I noticed that this was the
case for these three files, and so removed the reference to BSD license,
which no longer applies.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Many codecs don't provide `bits_per_raw_sample`. The bit depth of the
luma channel is the closest equivalent to a single bit depth then, while
the chroma channels may be subsampled.
Tested on basically all the video files on my computer, they all give a
non-zero bit depth that looks correct now.
Signed-off-by: Brecht Van Lommel <brecht@blender.org>
* The "oldest" didn't really test against the oldest we claim to
support.
* The "latest versions" didn't really test against the latest versions.
* The "bleeding edge" didn't really test against libpng master.
* The auto-build was unnecessarily a few releases behind.
* Documentation touch-ups
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Read and write 10 and 12 bit images
* Read and write CICP
This makes it possible to write HDR images that are much smaller than
PNGs, in a file format that is supported in all modern web browsers.
By setting the matrix coefficients, the library will perform RGB to YUV
conversion on write. The YUV to RBG conversion was already happening on
read automatically.
A future improvement would be a better default choice of matrix
coefficients for writing. When no CICP is specified or when CICP is read
from PNG, then no conversion to YUV will be performed and compression
will not work as well. This is the same behavior as before. When there
is support for conversion between CICP and display interop ID, a good
default choice for matrix coefficients could be made as part of that.
Tests for CICP and 10 bit were added.
---------
Signed-off-by: Brecht Van Lommel <brecht@blender.org>
This patch implements the new scheme that we hope will preserve ABI
compatibility across minor (yearly) releases.
Brief summary:
* We will strive to have annual minor releases (e.g., 3.1.x -> 3.2.x)
be ABI/link backwards-compatible, as strongly as monthly patch
releases (3.1.5 -> 3.1.6) always have been. Every-several-years
major releases (3.x -> 4.x) are still a potential full compatibility
break.
* Consumers of OpenImageIO don't need to change their code, and can
just continue to refer to `OIIO::Foo` or `OIIO::Bar()` as always.
* Internally, all items in the public OIIO APIs will be in a versioned
inner namespace, and will continue to live in the versioned
namespace where they were first introduced, forever. (Well, until the
next first-digit-changing major release, at which point we will clean
up the old cruft and version everything up.) If anything needs to
change in an ABI-breaking way, it will be *duplicated* in the newer
namespaces, so the old version continues to be available in the old
namespace.
* The new header nsversions.h contains a long and detailed comment
explaining how all the new declarations work.
---
Detailed explanation:
"Major" or "first digit" releases, which only happen once every
several years for us, are full ABI+API breaks. Previously, "minor" or
"second digit" releases, which are annual, have fully incompatible
ABIs, with separate namespaces to enforce it. Here, we are striving to
allow our annual/minor releases to no longer break ABI, to make
upgrading
easier for downstream users.
The basics are that once a symbol is introduced, it will forever live
in the namespace of that release. Subsequent release years will have
their own namespaces, but will either alias or duplicate the original
symbols. This is enabled by our recent (in the lead-up to 3.1)
introduction of a split 2-part namespacing scheme. So 3.2 is the first
release that has the potential to perserve compatibility with 3.1 in
this way.
In this PR, I have done this for the whole codebase. It works! -- as
measured by being able to have our automated ABI checker show 100%
back compatibility with 3.1. So currently, even though this is the
nascent 3.2 tree, everything in the public APIs live in the 3.1 ABI
namespace. And so they will remain, except for changes that cannot be
done without breaking ABI back-compatibility, which will then end up
with multiple versions in different versioned namespaces.
Let's look this over and decide if we like it. It does involve our
developers to do some hoop jumping and take extra care as we develop,
to keep things associated with the right namespaces.
Users of OpenImageIO shouldn't need to change anything on their end,
nor to be aware of this at all.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Something must have changed with the containers, it was failing on the
build step, before ever getting to the actual sonar analysis.
Switch to the same container and OCIO version that we use for the
regular 2024 era container CI.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The actual CICP reading code is trivial.
Also recognize mkv (Matroska) extension for the test.
Add test using Matroska + VP9, to maximize the chance of an ffmpeg build
supporting them. They are royalty free, built into ffmpeg and relatively
old.
Signed-off-by: Brecht Van Lommel <brecht@blender.org>
Enable ICCProfile attribute support for WEBP for both input and output.
Like other formats this support is passive; pixel data will not be
altered by the mere presence of the ICCProfile attribute (i.e. it
functions like other informational metadata only).
The WebP API is a bit odd when it comes to writing the ICCProfile. We
must use the "mux" API set; the "advanced" API is apparently not
advanced enough to add the metadata to the file. This wouldn't be too
bad except that the mux API is at odds with how we write out the data.
Before: When it comes time to do the file write, we would pass our own
file writer into webp. The writes would go straight to disk. Now: If ICC
profile data is necessary to write, we let webp use its own writer to
write to memory. We then assemble the in-memory file and the ICC profile
data together and pass that entire package to our writer as a final
step.
When no ICC profile data is used, the old write behavior is kept.
Added `oiiotool` test variations which uses ICC profiles with WEBP.
---------
Signed-off-by: Jesse Yurkovich <jesse.y@gmail.com>
This PR addresses the primary concern of #4678 -- implement support for
reading and writing the `cICP` chunk for PNGs.
CICP (Coding Independent Code Points) is a means for using a tuple of
integers to communicate characteristics of a variety of, traditionally,
video encodings. The tuple is a series of four values that map to
integer enumerations (codified in [ITU-T
H.273](https://www.itu.int/rec/T-REC-H.273-202407-I/en)) representing a
certain ColourPrimaries, TransferCharacteristics, and
MatrixCoefficients, as well as a VideoFullRangeFlag.
In particular, CICP is used for describing HDR encodings to browsers and
image viewers capable of displaying the image as intended. For example,
if one were to write P3-D65 PQ-encoded RGB values to a PNG, it will only
look "correct" if there's an appropriate `cICP` chunk describing the
primaries (14) and transfer function (16); without such metadata,
PQ-encoded images will appear significantly darker and lower in
contrast.
Internally, CICP metadata is stored in a `int[4]` type `CICP` ImageSpec
attribute.
This PR adds the following:
- An oiiotool `--cicp` flag for setting, modifying, or removing CICP
metadata for the top image
- Methods for reading and writing PNG `cICP` chunk metadata <-->
ImageSpec `CICP`
- Tests
I've included tests. I've also embedded CICP metadata in the existing
16-bit test png in the testsuite.
But to see what this is all about with your own eyes, you can quickly
convert a scene-linear AP0-encoded EXR to a PQ-encoded P3D65 HDR PNG
with the following command:
`$ oiiotool -i input_linap0.exr --ociodisplay:from=ACES2065-1
"ST2084-P3-D65 - Display" "ACES 1.1 - HDR Video (1000 nits & P3 lim)"
--cicp "12,16,0,1" -o output_p3d65pq.png`
If you compare in a capable image viewer on a capable display the image
produced with the above command compared to another one that lacks CICP
metadata, you should see a pretty significant difference:
`$ oiiotool -i output_p3d65pq.png --cicp "" -o output_no_cicp.png`
("Preview" on a ~5 year old Macbook should suffice).
---------
Signed-off-by: Zach Lewis <zachcanbereached@gmail.com>
Freetype 2.14 was recently released and just became the default for
homebrew.
It seems that no changes are needed on our end and no output references
even need to change!
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Original changes and goal:
* Bump our "latest releases" CI tests to OpenEXR 3.4.0
* cleanup: dead guard removal for openexr -- our minimum OpenEXR release
supports float vectors, so we don't need any `#if` guards for it
anymore.
* Fix warnings related to mixing Core and C++ library constants for
H2J2K compression enums within a switch statement (only noticed on ARM
Linux with Clang 18).
But it seems that OpenEXR 3.4's auto-building of OpenJPH (to support
exr's new h2j2k compression methods) revealed that our CI wasn't
building and testing against OpenJPG as well as we thought. So:
* Clean up some openjph API use and type mismatches
* Suppress warnings on clang and recent gcc versions had with the
OpenJPH headers and our use of them.
* Fix our test scripts and reference output for the htj2k tests; I think
maybe they weren't running at all before, and nad some incorrect file
paths.
Finally,
* When no OpenEXR is found, update the auto-build version to 3.3.5. I
wish I could go all the way to 3.4.0, but the OpenEXR 3.4.0 currently
auto-builds OpenJPH in a way that (I think incorrectly) installs the
OpenJPH components in ways that can overwrite or break existing
installations.
* Remove FindOpenJPH.cmake, the nomenclature clashes with the lowercase
"openjph" used by the new OpenJPH exported cmake configs.
* But those are only generated for OpenJPH >= 0.21.2, so make that our
new minimum. (This only affects OIIO 3.1 and higher, so it's safe to do
this.)
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Say you need to take a multi-part exr file and split it into separate
files for each part. (Note: what OpenEXR calls "parts", OIIO calls
"subimages.") You could do this
oiiotool multipart.exr --sisplit -o:all=1 "part.%04d.exr"
But this would name the parts part.0001.exr, part.0002.exr, etc. Maybe
you would prefer to split them into files that contain the subimage name
rather than just the inscrutible index number (which may also not be
consistent across files)? You could try this dance to loop over the
subimages and write each one with a name extracted from its metadata:
oiiotool multipart.exr --for i {TOP.SUBIMAGES} --dup --subimage {i} -o
"subimage_{TOP.'oiio:subimagename'}.exr
That works, but that's a little awkward and hard to remember off the top
of your head. Perhaps instead we would prefer a simpler version:
oiiotool multipart.exr --sisplit -o:all=1
"part.{TOP.'oiio:subimagename'}.exr"
This PR makes that last command work. It actually only required a small
change, which is that the function that handles the `-o` command needs
to defer the evaluation of the expression subsitution in the filename to
occur separately for each subimage being output, rather than just once
for the whole `-o:all=1` (which would have baked the FIRST subimage name
into all the output file names).
Docs are updated to give examples of how to do this.
Also fixed a silly typo in the test script.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Try a build against ffmpeg 8 -- seems to work out of the box, so
document that we support that version in INSTALL.md.
* Augment FindFFmpeg.cmake to correctly identify 7.1 and 8.0 by the very
non-intuitive versioning scheme of ffmpeg's headers.
* Add a FFMPEG_EXTRA_LIBRARIES variable to FindFFmpeg.cmake to allow us
to easily inject extra library dependencies (coincidentally, I need this
at work to force it to use static libraries).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Enforce gcc, clang, and apple clang minimum versions.
* Raise advertised minimum clang version to 10.0. Error for clang < 5,
which we know is the true minimum because earlier than that won't
support C++17 which we require. But in truth, we haven't tested
versions older than 10.0 for a long time, so we should not be
implying we know older versions will work or that we will support
them.
* Simplify cases no longer relevant for older gcc and clang.
* Remove unneeded gcc guard in simd.h. I think it was important only
for gcc 5! But it was also preventing the slightly faster construct
from activating for clang. Seems to pass all our CI jobs with this
simplified.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This doesn't really change any of the color management algorithms per
se, or change how we use OCIO. That will come later.
But this tries as best as we can to shift to CIF recommended interop
names for color spaces.
* "srgb_rec709_scene" as the canonical name for what we used to refer to
as "sRGB".
* "lin_rec709_scene" as the canonical name for what we used to refer to
as "lin_rec709" and several other names.
* "lin_ap1_scene" as the canonical name for ACEScg.
* Stop setting "oiio:Gamma" metadata (though we still accept it for back
compatibility), instead using space names like "g22_rec709_scene" and so
on. We never came across files with truly arbitrary gamma.
If I've done it right, the old names will continue to be accepted as
aliases.
I recognize that changing the output (when printing metadata) to the
canonical names might be an inconvenience at first, but in the long run,
I think it's better to comply with the conventions that will be used for
USD, MaterialX, and OpenEXR.
I'm fine with the new wordy and pedantic names as what is stored
underneath as the "oiio:ColorSpace" alias or stored in an OpenEXR file
in the colorInteropID metadata. But I gotta say, from a UI/UX
perspective, I do not feel good about replacing "srgb" and "acescg" as
things that are easy to type and are known/understood by the vast
majority of OIIO users with a change to these wonky, error-prone,
hard-to-type names. So I'm very uncertain about changing oiiotool
documentation, say, away from the short and widely understood aliases.
Seeking feedback on what to do here!
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Co-authored-by: Doug Walker <doug.walker@autodesk.com>
* Test against alpha of openexr 3.4.
* Document that we build against several things we know work but hadn't
updated INSTALL.md in a while.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
And a bunch of other minor updates:
* Change from 'testing' to regular containers
* Add forgotten 'export'
* Update default self-build pybind11 to 3.0.0
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* CI build as C++23 for bleeding edge test
* Fixes to disambiguate OIIO::print from std::print. Apparently, at
least for gcc14, std::print ends up in the global namespace and really
messes with things.
* Change to the library that contains the implementation of C++23
stacktrace in gcc14 vs 12/13.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Remove the CI jobs that tested VFXPlatform 2021 era, which we only were
still doing for the sake of python 3.7.
Also stop building Python wheels for any python versions older than 3.9
(we never built 3.7 wheels, but we did build 3.8).
This is just in main, a.k.a. OIIO 3.1, to be released in fall 2025 and
expected to be the main release series people are using in 2026.
VFX Platform has been 3.9+ since 2022, that's more than the "3 years
back" rule dictates. And in the real world, it hasn't been "current"
since 2020.
Just to be extra safe, I'm documenting 3.9 as the minimum and removing
all 3.7 tests, but I'm not having the build affirmatively reject 3.7 for
now, just in case somebody needs it. It should continue to work if
someone tries it, with no guarantees or support. But probably by the
time we do a 3.1 release in September, we will make it actually reject
3.7.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
ABI changed slightly after #4827. (Allowed in main but requires an
update of the reference.)
Also, bump to using the 2025 container for the ABI check.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
PRs #4840 and #4841 crossed in flight with their tests passing
individually, but they needed one small adjustment to reference output
in order to to account for their interaction in main.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* span-ify ParamValue (including ctr and init)
* add span versions of attribute() and getattribute() to the classes
that have them: ParamValueList, ImageSpec, ImageCache, TextureSystem.
Much like the previous span/image_span additions we've made elsewhere,
now each of these calls has 3 flavors:
1. The previously existing -- but acknowledged "unsafe" -- version that
takes a TypeDesc and a raw pointer (which had better match the TypeDesc
in type and size, but who knows?)
2. A version that takes a TypeDesc and a `span<std::byte>` delineating
the raw memory to use, that must match the size implied by the type, and
is implemented underneath (for now) by calling the pointer version after
checking that the size is adequate.
3. A template version that takes a TypeDesc and a `span<T>` that must
match both in total size and appropriate type and asserts in debug
builds if they don't. It's implemented by calling the raw byte version
after the type check.
The third variety is the preferred one for apps to call, whenever
possible, but the others are still exposed for special cases.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Allow CMAKE_MSVC_RUNTIME_LIBRARY to be initialized from an environment
variable of the same name. (CMake doesn't do that on its own.)
* If set, propagate CMAKE_MSVC_RUNTIME_LIBRARY to any packages we
auto-build within our build system.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
New logic for reading OpenEXR files:
1. Recognize "acesImageContainerFlag" as always being lin_ap0_scene.
2. If the new colorInteropID is present in the metadata, that's the single source of truth about color space.
3. Otherwise, leave it blank, don't make any assumptions about color space.
It was a mistake for OIIO 3.0 to assume that completely unlabeled openexr
files are lin_rec709.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Inside ImageInput::create(), there's a section where the created
ImageInput tries to do a quick call to its valid_file() method to do an
inexpensive check that the file appears to be the right file format
corresponding to the ImageInput subclass. However, it always called the
variety of valid_file() that takes a string filename, ignoring the fact
that create() itself may have been passed an IOProxy, which should
always take precedence over trying to open a real file.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
When passing numpy arrays for pixel blocks (say, to
ImageOutput.write_image), the sorting out of what the numpy arrays meant
had an error for the 2D case. A typo (or maybe copy pasta) got the
stride wrong.
This took a long time to be discovered because if there is more than one
channel, it gets passed as a 3D numpy array (height, width, chans). So
only for 1-channel images would it stumble across the bad code if it
were passed as a 2D (height, width) numpy array with no dimension for
channels.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Submitting this from my work as an intern on the RenderMan development
team.
- Bug fix with prman tile size for bumpslopes textures.
- `--slopefilter`: maketx has used sobel, txmake uses centraldiff
because sobel sometimes has ringing artifacts
- `--bumpinverts`: invert slopes in the s dir, txmake default behavior
has this on
- `--bumpinvertt`: invert slopes in the t dir, txmake default behavior
has this on
- `--bumpscale`: scale the strength of the resulting map
- `--bumprange`: support normal map values on either [-1,1] or [0,1]
- `--keepaspect`: sets PixelAspectRatio metadata to match txmake `up`
resize behavior.
`txmake -bumprough 1 0 0 0 0 0`: `maketx --bumpslopes --bumpinverts
--bumpinvertt --slopefilter centraldiff`
`txmake -bumprough 2.5 0 0 1 0 0`: `maketx --bumpslopes --bumpscale 2.5
--bumpinvertt --slopefilter centraldiff`
---------
Signed-off-by: Scott Milner <smilner@pixar.com>
Recent OpenEXR internals changes make header reads via the C++ library
dramatically slower if a Imf::IStream subclass does not implement the
size() and stateless-read method.
See https://github.com/AcademySoftwareFoundation/openexr/issues/1984
On my laptop, when forcing OIIO to avoid the "core" library and use the
C++ API to OpenEXR, this speeds up the reading of small exr files by
about 6x.
These changes were introduced with OpenEXR 3.3 and we never changed our
side.
(Also opportunistically added a couple DASSERT's to the existing
functions.)
Signed-off-by: Larry Gritz <lg@larrygritz.com>
avcodec_close (AVCodecContext *avctx) removing in ffmpeg 7.2 and was
deprecated for couple years
Global avcodec_close (AVCodecContext *avctx)
Do not use this function. Use avcodec_free_context() to destroy a codec
context (either open or closed). Opening and closing a codec context
multiple times is not supported anymore – use multiple codec contexts
instead.
In ffmpeg master, it has already been removed. But still work in v7.1
We checked the ffmpeg changelog, and avcodec_free_context was added 7 years ago and should be present in ffmpeg 4.0, the oldest version we support.
Signed-off-by: Vlad (Kuzmin) Erium <libalias@gmail.com>
The main thing is no longer using a custom namespace prefix for
auto-built libpng. We're already building it statically by default and
hiding the symbols, so the custom prefix only serves to make occasional
build problems by having different symbols than typically found in the
library. We still keep it as an option that can be enabled.
Also, be a little more careful in clearing unneeded variable that may
have been set by an earier attempt to find the package but that failed
to turn up a version we were happy with.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Add missing C++ header file
On the F41 and F42 compilation using LLVM based Clang++ failed.
```
[ 23%] Building CXX object
src/libOpenImageIO/CMakeFiles/OpenImageIO.dir/__/jpeg2000.imageio/jpeg2000input.cpp.o
/usr/src/github.com/1div0/OpenImageIO/src/jpeg2000.imageio/jpeg2000input.cpp:462:9:
error: use of undeclared identifier 'va_start'
462 | va_start(args, fmt);
| ^
/usr/src/github.com/1div0/OpenImageIO/src/jpeg2000.imageio/jpeg2000input.cpp:464:9:
error: use of undeclared identifier 'va_end'
464 | va_end(args);
| ^
2 errors generated.
```
Signed-off-by: Peter Kovář <peter.kovar@reflexion.tv>
This changeset has bunch of improvements for pixel closeup window
feature:
- Min/max/avg values
- RGB channels are colored with respective color
- Pixel colors in preview window are showing actual colors (previously
there was semi-dimming overlay on top of them)
- Center pixel is now marked including corner cases when close to the
edge
- During follow cursor mode window switches the the left from the cursor
when too close to right edge, and at the top when too close to bottom
edge
- Number of previeable pixels is configurable
- Number of pixels in a preview used for averaging is configurable
-- When avg subset is smaller then main closeup window there is an
indicator showing are that being averaged
---------
Signed-off-by: Aleksandr Motsjonov <soswow@gmail.com>
Also found one stray test that was using a pybind11 version older than
we claim to use, so bump that up to the advertised minimum.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Switch to posix_spawnp for macOS background launch to enable PATH
lookup, this is a fix for:
https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/4640
Description:
Previously, the code used posix_spawn with argv[0] as the executable
name. This failed when argv[0] was not an absolute or relative path
(e.g., just "iv"), because posix_spawn does not perform PATH lookups.
Switched to posix_spawnp, which behaves like execvp and searches PATH
when argv[0] does not contain a '/' character. This allows the program
to re-spawn in the background using the original command name, without
requiring _NSGetExecutablePath() to resolve the full path.
Signed-off-by: Mikael Sundell <mikael.sundell@gmail.com>
Wow, my fingers are so used to typing "exr", and I don't use the OS that
would make me ever type "exe", so...
Signed-off-by: Larry Gritz <lg@larrygritz.com>
pybind11 3.0 changed the generated stubs, causing CI to fail, so lock
down to no later than the last version before it broke.
We think the pybind11 3.0 generated stubs are mostly better, so at some
point soon, when we have more time to look at it carefully, we will come back
to this topic and bump our wheel workflow to use pybind11 3.0 and update
the stubs accordintly.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
[As reported on Slack](https://academysoftwarefdn.slack.com/archives/C05782U3806/p1751642730171029),
between OIIO 2.5 and 3.0, we had a big slowdown when doing a
multithreaded `maketx --envlatl`, which was traced almost entirely to
the new mutex lock in ImageBuf::IteratorBase::init_ib(), which were all
happening inside the call stack of `resize_block_<float>()`.
The reason was that it was calling ImageBuf::interppixel_NDC for every
pixel, which in turn works by creating (internally) an
ImageBuf::ConstIterator to sample the 4 nearby pixels. But every one of
those ConstIterator constructors called init_ib, which briefly grabbed
the mutex for the IB, which in addition to being wasteful on its own,
caused the threads to block on each other.
The solution is straightforward: there is no need to construct a new
Iterator for every pixel. We can create the iterator once (a single call
to the init_ib for each thread region of the image) and then for each
pixel, just call its `rerange()` method to reset the set of pixels to
loop over for the samples needed for that pixel.
I also opportunistically eliminated a few redundant spec() calls in
various routines in imagebuf.cpp.
On my Mac laptop, when doing a maketx --latlong on a 16k image with 16
threads, it was previously taking 170.5s (including 117.8s for the
initial resize and 42.8s for the MIP computations). With this change,
the same operation takes 12.4s (including 3.7s for the initial resize
and 1.6s for he MIP computations). That's almost a 14x speedup. YMMV,
depending on platform, compiler, image size, and number of threads.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
It was implemented as int, documented as float. Change the docs to match
what was always indended (int), and also switch the implementation so
that either int or float will be transparently accepted.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Just my periodic updating of the CHANGES doc in main.
Will merge immediately, as it's administrative docs.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Fixes#4811
This patch makes oiiotool's initial cursory scan of the command line
notice a `--threads` and carfully honor it when parallelizing the frame
loop.
Explanatory background info:
The cited issue shows something subtle happening with oiiotool's
`--parallel-frames` argument. The symptom was that it was using too many
threads and running out of memory, and seemed impervious to the use of
`--threads` to properly control it.
The reason is that oiiotool first scans the command line args in a
cursory way looking for signs of a "sequence", like a `--frames` or any
filename containing frame wildcards (like `linear.%04d.exr`) and also
noting if there is a `--parallel-frames`, but not actually taking any
actions for any of the commands yet.
Having seen that there are frame wildcards, what it then does is execute
the whole command line, once for each frame number that matched the
pattern. And since it noted that a `--parallel-frames` argument was
present somewhere, it does it with a parallel for loop. But how many
threads does it use? Well, enough for all the cores, since in some sense
it has not yet noticed the `--threads` argument. Moving the `--threads`
to be placed prior to the `--parallel-frames` doesn't help, either.
The suggested workaround was to set the default number of threads with
the environment variable `OPENIMAGEIO_THREADS` prior to launching
oiiotool. But this patch is a more robust solution.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Seeing build failures when building against libheif 1.20:
```
/private/tmp/openimageio-20250701-18561-96sl0/OpenImageIO-3.0.8.0/src/heif.imageio/heifinput.cpp:390:37: error: no matching member function for call to 'get_plane'
390 | const uint8_t* hdata = m_himage.get_plane(heif_channel_interleaved,
| ~~~~~~~~~^~~~~~~~~
/opt/homebrew/include/libheif/heif_cxx.h:909:32: note: candidate function not viable: no known conversion from 'int *' to 'size_t *' (aka 'unsigned long *') for 2nd argument
909 | inline const uint8_t* Image::get_plane(enum heif_channel channel, size_t* out_stride) const noexcept
| ^ ~~~~~~~~~~~~~~~~~~
/opt/homebrew/include/libheif/heif_cxx.h:914:26: note: candidate function not viable: no known conversion from 'int *' to 'size_t *' (aka 'unsigned long *') for 2nd argument
914 | inline uint8_t* Image::get_plane(enum heif_channel channel, size_t* out_stride) noexcept
| ^ ~~~~~~~~~~~~~~~~~~
1 error generated.
```
relates to https://github.com/Homebrew/homebrew-core/pull/228740
Starting with 1.20, libheif changed the type of a stride parameter.
---------
Signed-off-by: Rui Chen <rui@chenrui.dev>
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Co-authored-by: Larry Gritz <lg@larrygritz.com>
We've already done ImageInput, ImageOutput, ImageCache, TextureSystem.
This applies the analogous set of changes to the public APIs of ImageBuf
-- adding span and image_span versions of all the API calls that
previously required raw pointers and strides. Many of the internals are
still using the old versions, but at least this takes care of the
interface.
Specifically, this does constructors, reset(), and get_pixels().
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Fixes#4584
A minor edit with large consequences.
Previously the setting would encode uncompressed files due to a bug,
causing the much faster encoding as stated in the old description.
In the upcoming v0.12 libjxl release, [my
PR](https://github.com/libjxl/libjxl/pull/4201) overhauls the setting,
to correctly scale decode speed at the cost of density.
---------
Signed-off-by: Jonathan Brown <jonathanbr30@gmail.com>
Some actual bumps in CI, others just bringing the INSTALL documentation
up to date with what we have been testing and know works.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The upload_pypi step only happened for tags, not for other events like
PRs. But it would TRY (and fail) if you pushed a v3.0.* tag to your own
fork. This patch makes it not even attempt it, except when the tag is
pushed to the official repo.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We have been having intermittent CI failures when the auto-build of
OpenColorIO triggers the auto-build of Pystring, and then sometimes
immediately seems to think the version it installed is not correct.
I'm jiggling some things here to see if I can get it to stop
occasionally failing. I've run the testsuite on this patch several times
without seeing any failures that in recent days had been happening
typically in 1-2 jobs per CI run. So maybe this fixes it? I'd be lying
if I claimed to have a coherent and full explanation for exactly why
these changes help.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Delete some reference test output for older toolchains and dependency
versions that seem no longer needed.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We had a couple documents that were temporary checklists and status
reports from back when the project was first coming into the ASWF,
reminding us of tasks that needed to be performed.
That was years ago! All done, and OIIO is even a fully
"adopted/graduated" project. Remove the files, which are misleadingly
out of date and no longer needed.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Among other simplifications, remove all vcpkg builds from windows CI,
don't need them. Just rely on our auto-build. Shortens the build time on
Windows!
Signed-off-by: Larry Gritz <lg@larrygritz.com>
An important clarification: ImageInput::seek_subimage() returns false
but does not set an error message if the only thing wrong is
requesting a non-existent subimage or miplevel, and similarly,
ImageInput::spec() and spec_dimensions() set an empty spec (easy to
tell by spec.format == TypeUnknown) in that case. All three of these
only set an error message if the reason it couldn't perform the task
because there was actually something wrong with the file or the
I/O. (Put another way, it's common to use these to simply probe which
subimages or mip levels exist, and returning false for "no it doesn't"
isn't really an error, it's expected behavior.)
OK, then.
We weren't always following these rules, sometimes setting
error messages when we shouldn't.
Other times (in different routines) when we didn't set error messages
but should have. And also, there are some downstream places such as
ImageBuf::init_spec() that assumed that seek_subimage / spec /
spec_dimensions set an error message (but they didn't, by design) and
should have raised their own errors when discovering that a requested
subimage or miplevel didn't exist.
Also, there were a couple spots in ImageBuf internals where we tried
to do the right thing, but instead of calling ImageBufImpl::error(),
we called errorfmt(), which didn't exist in ImageBufImpl, and so was
calling the global OIIO::errorfmt(), which therefore was setting the
error on the wrong object entirely. (Fix that by adding an errorfmt
method in ImageBufImpl so that mistake is not possible. Should have
been like that all along.)
All this leads to more consistent error reporting (and not reporting,
in the cases we weren't supposed to), especially for people calling
ImageBuf::read() and ImageBuf::init_spec().
Signed-off-by: Larry Gritz <lg@larrygritz.com>
I'm unsure how this could have worked correctly. There were three
different variables used before, now there is only one.
Signed-off-by: Jesse Yurkovich <jesse.y@gmail.com>
- The parameter 'pattern' now supports a special value 'auto', which is
also default.
- The parameter 'layout' now supports a special value 'auto', which is
also default.
- The parameter 'algorithm' now supports a special value 'auto', which
is also default.
- New parameter 'white_balance_mode.
- The raw input plugin now adds an attribute 'raw:WhiteBalance' in
addition to the libraw-specific 'raw:cam_mul', carrying the same values.
This changes the current behaviour, please read below.
Previously, the parameter 'pattern' defaulted to 'bayer'. The new default
value of 'auto' makes OIIO look for the input ImageBuf's attribute
'raw:FilterPattern', and deduct the pattern from its value. If the
attribute is missing, 'bayer' is assumed.
Previously, the parameter 'layout' defaulted to 'RGGB' for Bayer pattern,
or 'GRBGBR BGGRGG RGGBGG GBRGRB RGGBGG BGGRGG' for X-Trans. The new
default value of 'auto' makes OIIO look for the input ImageBuf's
attribute 'raw:FilterPattern', and deduct the layout from its value. If
the attribute is missing, the old default values are assumed.
Technically the 2 changes above the current behaviour may change, but
only in a very rare use case: when the user relies on the old default
values to decode the image, and there is metadata present in the
ImageBuf saying that the pattern and/or layout is different from the
default.
Previously, the parameter 'algorithm' defaulted to 'linear'. The new
default value 'auto' maps to 'MHC' for the Bayer pattern, and 'linear'
for the X-Trans.
The newly added parameter 'white_balance_mode' can have one of the
following three values:
- 'auto' (default) - the white-balancing weights are read from the input
ImageBuf's attribute 'raw:WhiteBalance', defaulting to (1, 1, 1, 1).
- 'manual' - the white-balancing weights are read from the parameter
'white_balance'.
- 'none' - same as 'manual' with the weights of (1, 1, 1, 1).
This changes the current behaviour where the white-balancing weights
from the parameter 'white_balance' get applied if present. Now, in
addition to providing them, you need to explicitly set
'white_balance_mode' to 'manual'.
I've updated the unit tests to cover the new 'auto' mode.
---------
Signed-off-by: Anton Dukhovnikov <antond@wetafx.co.nz>
ImageOutput::supports() recognized queries for "multiimage" and
"mipmap". But ImageInput::supports() only recognized "multiimage". Add
"mipmap" to make it easier for client software to know if the image file
format they're reading from allows the possibility of MIP-mapping.
As far as I can tell, only OpenEXR, TIFF, Ptex, and DDS do, among the
file types we currently support.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
On Windows ARM64, when building with clang-cl, LLVM can sometimes use it's own built-in intrin.h file, located in `<LLVM INSTALL PATH>\lib\clang\20\include`.
This file, as opposed to the built-in one that ships with the Windows SDK that MSVC uses, doesn't have an inclusion of arm_neon.h. This PR adds a small check to the simd.h file to include this header.
I didn't add an extra check to specifically only do this for clang, as it seemed a little messy - I can do this, but it's relatively harmless for MSVC given intrin.h includes it anyway for that compiler.
This change has already been in place for some time in Blender, this is just an upstreaming PR for that change: https://projects.blender.org/blender/blender/src/commit/ea604b6b80e49adb51d12df4e4edd5af1c51f446/build_files/build_environment/patches/oiio_windows_arm64.diff
Fix some `oiiotool --missingfile` behavioral corner cases:
When substituting, warn rather than making a true error. This didn't
seem like a big deal for a single command line, but when doing a loop
over frames, it would prevent correct behavior from pressing on to the
next iteration.
How big is the substitute image? It uses the rez and channels of the
first image successfully read (if available). But it when in a frame
loop, it resets this on every iteration, so any missing frame looks like
it doesn't have a good size guess -- but it does, as long as it's not
the first iteration of the loop. This involves safely gettingthe first
image dimensions from the *parent* (each iteration is a child Oiiotool
object).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
When doing oiiotool command line expression substitution, asking for
this metadata evaluates to the number of subimages in the referenced
image.
The purpose is that asking to retrieve, say, `{TOP.'oiio:subimages'}`
might look right, but generally it's only added to images (or format
readers) that are potentially multi-image. Whereas SUBIMAGES is not
actual image metadata, it's just an evaluation token that will always be
the number of subimages.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Improvements to the way our CI utilizes ccache to speed up build. Ported
from OSL:
- Split caching into separate restore/save, and rearrange the step order
so we save cache before tests run. This allows us to save the ccache
even for jobs with failed tests (where the all-in-one unsplit approach
would skip the cache save step for a failed job), or for jobs that we
purposely end early (maybe because we see failing tests), not only when
all tests finish and succeed. This makes subsequent builds, that we do
on the same branch while attempting to fix failures, go faster by using
the cache.
- Make subsequent pushes to the same PR or REF to cancel any previous
jobs, so if we push updates before the last CI run of the same branch
completes, it won't keep those going as useless zombie jobs.
- Change the cache key names we use to remove the github ref to
encourage cache reuse across different commits or branches (I think),
and to fix a couple errors where we had mis-named things and were
sharing caches between jobs when we shouldn't or vice versa.
- Be more careful about the location of the cache, making sure we were
for sure telling the GHA cache action to be saving/restoring the same
directory location that we told ccache to use as its cache location.
- Use CCACHE_COMPRESSION env var to be extra sure we're using compressed
caches to be sure we get the best bang for buck on the limited cache
storage we're allowed.
- Set CMAKE_CXX_COMPILER_LAUNCHER and CMAKE_C_COMPILER_LAUNCHER in the
shell for CI in the hopes of making more dependencies we build also use
ccache.
- Alternate launcher structure -- use LANG_COMPILER_LAUNCHER, not
RULE_LAUNCH.
A sampling of typical results:
test build deps build oiio
----------------------- ---------- ----------
VP 2024 (cold) N/A 6:22
VP 2024 (cached) N/A 0:55 (7x)
latest rels (cold) 8:29 5:30
latest rels (cached) 2:24 (3.5x) 0:45 (7.3x)
(The N/A is because that run gets all its dependencies from the
container and doesn't have to build any from scratch.)
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Just on the "bleeding edge" CI job, we were refraining from doing the
apt-get update. It was breaking something last year. Now NOT doing it is
breaking something in this CI test, just since this week (presumably
some default package on the GH runners changed, I haven't been able to
isolate any other reason). But bringing all the apt packages up to date
makes it go away.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Sorry for not doing it in a first PR. I thought to see if I can make
VSCode / Cursor to actually syntax highlight that raw string literal
content. And I found a way. Seems like if you change the delimiter using
language of content syntax name you make VSCode to syntax highlight it:
I don't think those clang exceptions are needed now.
Signed-off-by: Aleksandr Motsjonov <soswow@gmail.com>
This PR fixes and improves a tiny constellation of OCIO-related bits,
directly or indirectly pertaining to OCIO Viewing Rules and File Rules,
and the propagation of `oiio:ColorSpace`.
## Description
The focus of this PR:
- feat: Permit the `ociodisplay` ImageBufAlgo to dynamically select a
default View for the `fromspace` (or current `oiio:ColorSpace` value),
as directed by the OCIO config's Viewing Rules (if present)
- feat: Have the `ociodisplay` IBA appropriately set the destination
ImageSpec's color space.
As part of this effort, a handful of additional changes and improvements
are addressed:
- fix: Re-enable the "looks" override functionality, matching the
previous OCIO-1-style behavior.
- fix: Improve `ColorConfig::getDisplayViewColorSpaceName(...)`'s
handling of Shared Views
- feat: Overload `ColorConfig::getDefaultViewName(...)` with function
that takes an additional `inputColorSpace` argument
- change: Alter the `ociofiletransform` IBA's behavior pertaining to
setting the destination spec's color space: instead of storing the
filepath as the `oiio:ColorSpace` value, attempt to get the color space
from the filepath; else, leave the `oiio:ColorSpace` attribute alone.
This isn't 100% foolproof, obviously, but I'd prefer to reserve the
`oiio:ColorSpace` attribute for color space names, unless there's a
reason not to. In a future PR, we can additionally attempt to divine a
valid color space string from specific CLF and CTF metadata, if present.
- feat: Implement a `ColorConfig::filepathOnlyMatchesDefaultRule(...)`
function, which wraps the OCIO::Config function of the same name.
- change: Alter `ColorConfig::getColorSpaceFromFilepath(...)` to behave
more like the OCIO::Config function of the same name. In particular,
return the Config's default color space name (instead of failing over to
OCIO-1 string-parsing heuristics, which, in turn, fail over to an empty
string); and provide an overloaded variant for implementing the legacy
(i.e., current) behavior. Update all usages in the codebase (i.e., in
oiiotool, w.r.t. autocc) to use the overloaded variant, so there's no
apparent change in behavior to how autocc functions.
## Background
This PR was inspired by #4753, which adds to the Python ImageBuf
bindings a `_repr_png_` magic method which returns a PNG bytestring
(which Jupyter notebooks can display inline), similar in behavior to
other libraries, like Pillow. This is great for previewing gamma-encoded
output-referred buffer data, but less great for previewing "linear"
encodings. Ideally, the `_repr_png_` method would apply an `ociodisplay`
ImageBufAlgo just before writing the PNG; but there would need to be
some means of controlling for the buffer "image state" when determining
how / whether to apply an `ociodisplay` IBA. For example, it would be
appropriate to apply the default ColorConfig's "ACES 1.0 - SDR Video"
View to an image buffer that contains scene-referred data; but very
evidently inappropriate to apply that same View to an image buffer that
holds display-referred data, yielding a "double-LUTted" appearance.
Fortunately, OCIO-2 `Viewing Rules` provides a solution. Viewing Rules
allow OCIO Config authors to selectively filter the set of Views
available for a given Display, on a per-input-colorspace basis.
Ostensibly, applications use the OCIO API's DisplayViewHelper class for
building View menu UX widgets that dynamically update as a function of
both the selected Display and Input Color Space; but there are
implications for headless applications, as well -- crucially, the Config
author may order Active Views such that different Input Color Spaces
elicit different default Views.
## Tests
I've included a test OCIO config with some Viewing Rules and File Rules
in testsuite/python-colorconfig; I'm testing that the new and updated
python bindings behave as expected; and I'm testing that the
`ociodisplay` IBA is OCIO-Viewing-Rules-aware and is properly
propagating `oiio:ColorSpace` to the dst buffer's spec.
Notably:
```python
import os
import OpenImageIO as oiio
import numpy as np
os.environ["OCIO"] = "/path/to/testsuite/python-colorconfig/src/oiio_test_v0.9.2.ocio"
a = oiio.ImageBuf(np.array([[[0.1, 0.2, 0.3]]]))
# The first time the ociodisplay IBA is applied, the lack of an `oiio:ColorSpace`
# attribute will be interpreted as the OCIO config's default color space -- in this case,
# "lin_rec709", which corresponds to a scene-referred default view (ACES 1.0 - SDR Video).
b = oiio.ImageBufAlgo.ociodisplay(a, "", "")
# The second time around, the `oiio:ColorSpace` attribute is now "sRGB (~2.22) - Display",
# a display-referred color space, thanks to the previous operation; as such, the IBA uses
# a different colorimetric "pass-thru" View as its default ("Colorimetry"), instead of the
# ACES View. Upshot is, values remain unchanged this time.
assert b.spec()['oiio:ColorSpace'], 'sRGB (~2.22) - Display'
c = oiio.ImageBufAlgo.ociodisplay(b, "", "")
assert c.spec()['oiio:ColorSpace'], 'sRGB (~2.22) - Display'
# Subsequent applications will be noops as well.
d = oiio.ImageBufAlgo.ociodisplay(c, "", "")
assert np.allclose(b.get_pixels(), c.get_pixels())
assert np.allclose(b.get_pixels(), d.get_pixels())
assert not np.allclose(b.get_pixels(), a.get_pixels())
```
## TL;DR
This PR enables OCIO config authors to dynamically control `ociodisplay`
IBA behavior w.r.t. the `oiio:ColorSpace` attribute, such that
`ImageBufAlgo.ociodisplay(buf, "", "")` always "does the right thing",
regardless of whether `buf` holds scene-referred or display-referred
data.
---------
Signed-off-by: Zach Lewis <zachcanbereached@gmail.com>
This flavor of runner will be retired at the end of the month and is
already experiencing intentional brown-outs to warn us.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Oops, my bad. Looks like #4664 (merged about a week ago) and #4783
(merged today, but authored before #4664 was merged) interacted in a way
that produced a build break. This tiny fix is essential to unbreak it.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
I think this is all the span-izing that the public interface of
ImageCache and TextureSystem needs. And it doesn't break ABI
compatibility (unlike the ImageInput and ImageOutput changes recently)
because these aren't virtual methods.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Fixes#4714
This PR adds support for selecting a rectangular region within an image
to compute and display per-channel minimum, maximum, and average values.
The feature can be activated through the Tools menu or by using a
keyboard shortcut (Ctrl-A / Cmd-A) . Once active, the user can click and
drag with the left mouse button to define a rectangular area, which is
visually indicated by a blue outline. Upon releasing the mouse button,
the tool calculates statistics for all pixels within the selected area
and displays the results in the status bar.
---------
Signed-off-by: Danielle Imogu <dannieim689@gmail.com>
OIIO did already have a decompressor for HTJ2K via the OpenJpeg library,
but nothing for encoding. This adds the encoder using the OpenJPH
library, which will be needed for the HTJ2K encoding for OpenEXR too.
This is a merge with the OpenJpeg plugin, the reader will attempt to
use the OpenJPH library (if found) to read the input file, if it cannot
read it, it will fall back on OpenJpeg to read it. For writing with the
OpenJpeg library, if the compression flag is set to htj2k or the file
extension is j2c the OpenJPH library will be called to write out the
file.
If you want to do a lossy encode you can do it with:
```
oiiotool -i INPUTFILE --attrib jph:qstep 0.03 -o OUTPUTFILE.j2c
oiiotool -i INPUTFILE --compression htj2k --attrib jph:qstep 0.03 -o
OUTPUTFILE.j2k
```
Other options include:
```oiiotool -i INPUTFILE --compression htj2k --attrib jph:qstep 0.03
--attrib jph:num_decomps 3 --attrib jph:block_size 32,32 --attrib
jph:prog_order CPRL --attrib jph:precincts 128,128,256,256 -o
OUTPUTFILE.j2c
```
Added a htj2k test, it tests encoding a dox and a jpeg file, both
lossless and lossy.
---------
Signed-off-by: Sam.Richards@taurich.org <Sam.Richards@taurich.org>
As a reminder, the global OIIO attribute "missingcolor", or the
file-specific ImageInput open hint "oiio:missingcolor" is designed to
treat missing scanlines or tiles as not a true error, but instead just
pass over them and substitute the missingcolor value for those missing
areas on the file.
The nature of this PR is that while we had implemented this behavior
fully for tile-based exr files, we never did so for scanline-based
files. So here we do.
Now then, unfortunately, it does not work. I think it's a bug in the
OpenEXR library itself, but we're still trying to figure that out. But
nonetheless, I believe this change is correct.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We documented this function as checking the valid range of pixel
coordinates, but we CALLED it everywhere as if we were only checking the
resolution.
But we IMPLEMENTED as if we were checking the range, generating errors
for outputting exr files with negative pixel window origins. But why was
nobody seeing those errors when writing exr files with negative origins?
Because the openexr writer BOTCHED it by not noticing the return value
and considering it a failure.
Oh boy. All my fault.
So the fix:
* Change the comment documenting check_open() to match the way we had
been using it at the call sites.
* Change the implementation of check_open() to match this by no longer
checking whether all four corners were in the range -- only check if
each dimension's size is within the limit for that dimension.
* Fix the OpenEXR writer's failure to notice the check_open failure.
* Does this eliminate necessary checks? No! We already separately
checked that nonzero origins were only allowed for formats that support
it, and that negative origins were only allowed for formats that support
that. So the flawed check was redundant anyway. Ha!
* Add a test verifying that we can write exr files with negative
origins. (Technically, that would have passed before, but only because
we were ignoring the incorrect error.)
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
A utility to confirm that an image_span (possibly with non-contiguous
strides) entirely fits within another contiguous span.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
After adding the new `htj2k` compression codec to OpenEXR, here we
enable it in the OIIO stack so it will be recognized by the library and
binary tools, such as using `oiiotool` to directly change compression
type into htj2k.
---------
Signed-off-by: lji <lji@ilm.com>
Backend changes for issue
https://github.com/AcademySoftwareFoundation/OpenImageIO/issues/4436.
## Description
- add `ImageDims` struct to store only a subset of `ImageSpec` or use it
as a view on an existing ImageSpec.
- move `ImageDims` and `ImageSpec` ownership to `ImageCacheFile` and use
them as a pool
- add `ImageSpec* m_spec` in `SubimageInfo` to access the descriptors
that are common across all mip level
- add `ImageDims* m_dims` in `LevelInfo` to access the despcriptors that
differ per mip level
- add functions to access to mip level dimensions:
```c++
const Dimensions& SubimageInfo::leveldims(int miplevel) const;
```
- do the plumbing across all `libtexture` to use the new interface
## Tests
I performed a batch of testing to compare the memory usage and
performance of different strategies using 17 `ptex` files from the PTex
library samples (some from the Moana Island dataset, so they are quite
heavy). The baseline numbers are approximates but gives a good estimate
of the memory usage.
Baseline | Total (MB) | Specs (MB) (%) | # specs | Subimages (MB) (%)
-- | -- | -- | -- | --
OIIO 3.0 main | 1500 | 1200 (80%) | 4936364 | 300 (20%)
Method | Total (MB) | Specs (MB) (%) | # specs | Subimages (MB) (%)
-- | -- | -- | -- | --
No reuse | 787,2 | 454,3 (58%) | 4936381 | 332,5 (42%)
Reuse dist=1 | 442,2 | 109,3 (25%) | 1193485 | 332,5 (75%)
Reuse dist=10 | 368,5 | 35,6 (10%) | 380506 | 332,5 (90%)
Reuse dist=inf | 333,2 | 0,292 (<1%) | 3020 | 332,5 (>99%)
These numbers shows that the memory is now bounded by the `SubimageInfo`
+ `LevelInfo` structure sizes, and not from the descriptors `ImageSpec`
+ `Dimensions` with the right amount of reuse across subimages and mip
levels.
---------
Signed-off-by: Basile Fraboni <basile.fraboni@gmail.com>
Two PRs crossed paths in flight last week -- one adding the ARM tests,
the other doing something that changed the reference output of this test
slightly. Both passed on their own, but the combo left one platform's
output needing the update.
Also, edit nicknames of two tests, they were not unique.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
GHA offers free runners for Linux on ARM based machines. Let's throw
them into the testing mix!
A very small number of tests needed additional reference outputs saved,
due to a few LSB type errors in the math.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The INSTALL.md Windows building instructions were out of date in two
important ways: (1) they neglected to mention OpenColorIO as a (newly)
required dependency nor give instructions for how to build it; (2) it
doesn't mention the new BUILD_MISSING_DEPS capabilities.
So basically, this PR modifies our Windows build guidance to set
OpenImageIO_BUILD_MISSING_DEPS=all and rely on that without needing to
document any additional steps for dependencies.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Extended eraseattrib to be able to read regex patterns from a file,
one per line, with `--eraseattrib:fromfile=1 filename`.
Erases all metadata attribute fields that match.
Added a new test to `oiiotool-attribs` which has been verified
painstakenly to be working!
Also tried running the command directly from dist/bin and verified with
`oiiotool --info -v`.
More detail on the motivation:
It's common in a studio to have tools insert all sorts of metadata into files as they're rendered, etc. But a lot of that are things that SHOULD NOT leave the facility -- like security issues from internal pathnames or network info, or secrets like plot details revealed by sequence or character names. So generally you want to scrub the metadata from the final images before they are shipped to the client or to marketing or whatever.
oiiotool's --eraseattrib is handy for this, but it takes a single regex. But we were thinking that:
- You may have several patterns to fully describe the metadata you want to be sure not to ship. This could be an unwieldy regex or a cluttered series of multiple --eraseattrib commands.
- That list of patterns may change over time, and it would be error prone to have to change it in every script that calls oiiotool.
- That list of patterns may need to be different per show.
We thought a solution to facilitate this might be a version of --eraseattrib that takes a filename that contains a list of regexes, one per line of the text file. So that master list of things to scrub could be maintained in a central place and not need to update every oiiotool-calling script in order to update it. Furthermore, you could have the command say something like
$ oiiotool ready_to_ship.exr --eraseattrib:fromfile=1 /blah/$SHOW/nometadata.txt -o clean.exr
to make a single command that allows per-show customization.
---------
Signed-off-by: Lydia Zheng <lydiamzheng@gmail.com>
Fixes#4598.
It was noted in the issue that running `oiiotool` with output filename
inside a folder that doesn't exist will cause OIIO to consume time and
resources, but fail eventually due to the folder not existing.
Take for example the command:
```
oiiotool --create-dir -i input.png -o ./output_folder/output.png -o ./output_folder_2/output.png
```
During the output action, it creates the directories if needed
(`output_folder` and `output_folder_2`), and outputs the images into the
folder.
Added unit tests. Tested it locally, and it creates both the above
folders and outputs the image.
---------
Signed-off-by: DharshanV <dharshanv1999@gmail.com>
WEBP uses the `WebPConfig::factor` setting to also control how hard it
tries to compress when lossless compression is used. It also uses
another `WebPConfig::method` setting to further tune both lossless and
lossy schemes. [1]
Expose both of these settings so client applications using OIIO can
adjust these values as they see fit.
This PR makes 4 changes:
- Adds support for using a `lossless:factor` value for the `compression`
attribute. The 'factor' value controls the effort that WEBP uses during
the compression process.
- Adds support for using a `webp:method` int attribute. The value of
this attribute controls the `WebPConfig::method` setting. Note: this is
different than the compression settings but will also affect speed and
quality in various ways.
- Sets `WebPPicture::use_argb` to `true` when using lossless compression
(per the WEBP documentation[2])
- Changes default lossless "quality" from 100 to 70. This results in
much faster image output at a very minor increase in file size (e.g. a
2048x2048 test image goes from 1300ms/364kb to 800ms/371kb after this
change)
Before - just 2 basic forms were possible:
```
Client uses "compression=lossless"
- OIIO uses lossless=true, method=6, quality=100, use_argb=false
Client uses "compression=webp:90"
- OIIO uses lossless=false, method=6, quality=90, use_argb=false
```
After - several additional variations are possible:
```
Client uses "compression=lossless"
- OIIO uses lossless=true, method=6, quality=70, use_argb=true
Client uses "compression=lossless:100" and "webp:method=4"
- OIIO uses lossless=true, method=4, quality=100, use_argb=true
Client uses "compression=webp:90"
- OIIO uses lossless=false, method=6, quality=90, use_argb=false
Client uses "compression=webp:90" and "webp:method=4"
- OIIO uses lossless=false, method=4, quality=90, use_argb=false
```
[1]
https://developers.google.com/speed/webp/docs/api#advanced_encoding_api
[2] See notes around `use_argb` from the above link and see also the
internal Encode helper function which keeps lossless and use_argb in
sync:
https://github.com/webmproject/libwebp/blob/main/src/enc/picture_enc.c#L241
Signed-off-by: Jesse Yurkovich <jesse.y@gmail.com>
Only symptomatic on debug builds, but I was seeing exceptions from
pybind11 about bad GIL release. I traced it to the recent
implementations of demosaic. Apparently it's not safe to release the GIL
until AFTER the call to py_to_stdvector. Which makes sense when you
think about it -- we need the lock whenever we are interacting directly
with Python calls or data structures (such as what py_to_stdvector is
doing), but we release it when we're done with the python and only
calling OIIO internals (such as IBA::demosaic() itself).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
ImageInput methods that read scanlines, tiles, and image are given new
flavors that take image_span, much like the changes of PR #4727 did for
ImageOutput.
Generally, for each, there are three versions: (1) a low-level virtual
method base case that takes an `image_span<byte>` that describes the
underlying memory, and an explicit TypeDesc that specifies the data type
conversion (including allowing TypeUnknown to indicate keeping channels
in their native per-channel format from the file); (2) a templated
version taking an `image_span<T>` that infers the data type conversion
from `T` (must be a single type for all channels); (3) a templated
version taking a `span<T>` that further implies a contiguous buffer in
all dimensions.
For now, the default implementations of these new ImageInput methods are
just wrappers that call the old pointer-based ones. One by one, over
time, we can swap them, changing the format implementations to have a
full implementation of the new bounded versions, and make their raw
pointer versions call the wrappers. The raw pointer ones will be
understood to be "unsafe", merely assuming that the pointers always
refer to appropriately-sized memory areas. Meanwhile, the ones using
spans and image_spans will, due to assertions in their implementations,
make it easier to verify (at least in debug mode), that we never touch
memory outside these bounds.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Make it so that CI test cases that set the GHA variable "benchmark" to 1
will add a benchmarking step to the workflow that runs a new
ci-benchmark.bash script.
The script runs selected unit tests containing benchmarks (currently,
only image_span_test, but we can amend later as needed). Those
designated tests are run, and their output both echoed to the log for
that step and also put in build/benchmarks/TESTNAME and saved as a build
artifact for optional download.
Most test cases will not turn benchmarking on -- it probably will end up
adding a few minutes so do it very selectively (once per major platform
or compiler version is plenty).
I would have previously guessed that any attempts at benchmarking on GHA
runners was doomed, but in practice, I'm surprised to find that there's
almost as much run-to-run consistency as I find doing casual benchmarks
on my own machine. As such, I think this can be a handy way to do some
rough benchmarking using CI, to compare platforms or compilers, or
verify that changes we want to make don't introduce performance
regressions.
Caveats to remember in the future:
* Take it all with a big grain of salt, and watch the benchmark numbers
for the trial-to-trial range of times -- wide variation means that the
numbers probably can't be trusted.
* The GH runners themselves may change without warning, so beware
benchmark stability over time, or if they ever have pools of
heterogeneous machine generations/configurations.
* While my results indicate a decent amount of timing reliability for
purely computational tests, I assume that there will be enormous
run-to-run variation in anything involving I/O or networking. So this is
unlikely to be a fruitful way of testing for performance regressions in
image format I/O speed (but probably is useful for a variety of
in-memory operations).
* As we add more unit tests to what we benchmark in the future, keep an
eye how much time we're spending running these benchmarks. A few minutes
on a small subset of the test jobs is probably fine, but I wouldn't want
it to make the overall wait for a full CI run to become substantially
longer because of it.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Master version of the iv today when opening on mac sometimes opens
without a focus, which causes top menu bar not to be populated (just
`iv` on the left). Workaround is to switch back and forth once to and
from another app for menu bar too get populated.
It is unclear to me how reliably replicate the bug so that I have a
solid prove, that this fix works. But since building with this line it
never happened to me yet.
Maybe more knowledgeable in Qt can verify if it makes sense.
Signed-off-by: Aleksandr Motsjonov <soswow@gmail.com>
This was causing the Rust bindings to fail because the pkgconfig was
pointing to a path that didn't exist (the libdir was the full root
appended to itself).
Signed-off-by: Scott Wilson <scott@propersquid.com>
Similar to other formats like PNG, make WEBP respect the
`oiio:UnassociatedAlpha` attribute value in case client applications are
already handling the alpha association themselves.
---------
Signed-off-by: Jesse Yurkovich <jesse.y@gmail.com>
II and IO were documented as initializing their internal "how many
threads to use when reading/writing this particular file" to the global
"threads" OIIO attribute. But they did not, they always initialized to 0
("all the threads") even if the global attribute had been set
differently prior to creation of the reader or writer.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Our fmath_test (a.k.a. unit_fmath) test was failing with an exception
thrown by the fmt library because we used a width specifier when
formatting a TypeDesc.
It was really only an issue when using old fmt < 10.0, so I'm just
solving it trivially by ignoring that specifier for old fmt. It doesn't
seem necessary to make it work correctly with width specifiers for old
versions, when we don't in any way rely on it.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Fixes#4680
Adds a `_repr_png_` implementation to the Python bindings of ImageBuf.
This allows ImageBuf objects to be displayed in Jupyter
notebooks, as requested in #4680
Uses the IOVecOutput IOProxy to write the ImageBuf as a png into memory.
Then, reads and returns that as a python bytes object.
---------
Signed-off-by: glowies <comuokta@gmail.com>
Fixes#4718
See the demo of problems that were fixed and of fixes in action over
here: https://www.loom.com/share/3bdee20b32434e0bbba47d393928d4f7
There are fixed for three problems:
1) Window is not resizing when new image is loaded in (it still does
when image being loaded is the only one)
2) When you using scroll mouse wheel it is taking into account where the
cursor is (there is some corner case when it doesn't do it perfectly,
that I can't catch when it doesn't follow it exactly, but a bit of
panning fixes it)
~~3) Switching between images recenters (if image is smaller) or making
some other panning nessery to have image in the viewport. I believe it
is doing same stuff what happens when window is resized.~~ This has been
reverted.
---------
Signed-off-by: Aleksandr Motsjonov <soswow@gmail.com>
Freetype master (presumably a change that will be in the next release)
broke our CI this week by changing how the accents on certain unicode
characters are rasterized. So we needed a new reference image on our
end.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Apparently, the runners used to check things out on drive D:, but today
they are doing so on C:, and that changes some filenames in the
reference
output.
Also, fix missing nametag needed for Windows artifacts.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
I've found a couple issues with iv when launched with a directory name
as input, while iterating through the files in the directory:
1. Crash on .DS_Store file
2. The files with extension in capital letters are not recognised as
valid images, due to case-sensitive compare. Capitalised extensions are
common in raw camera images.
Signed-off-by: Anton Dukhovnikov <antond@wetafx.co.nz>
If the python stubs test fails (because the API changed in some way that needs
the stubs regenerated, but the submitter of the PR did not or could not,
the wheel CI will regenerate the stubs and include them in the failure artifact.
This provides a path for users who can't build the stubs locally.
Signed-off-by: Chad Dombrova <chadrik@gmail.com>
Add python stub files for code completion and static analysis.
Fixes
https://github.com/AcademySoftwareFoundation/OpenImageIO/issues/4682
No new test cases, since there is not any change to the behavior. I do
have an idea for a kind of test that might validate the stubs against
the python unit tests, but I don't know if it's worth it.
---------
Signed-off-by: Chad Dombrova <chadrik@gmail.com>
While I was investigating some SonarQube warnings about our simd
`shuffle<>` templates (which are a false positive and I have a separate
PR to simply silence it), I did get to thinking about the naming, and
want to make a couple changes:
* I decided that the 1-template-argument version of this function,
`shuffle<int i>(simd_type)` would actually be more clear and
self-documenting if renamed `broadcast_element` to emphasize that it is
taking just one simd lane/element and broadcasting it to all lanes. (The
multi-argument shuffle really is doing a true shuffle, giving an index
for each lane to make a permutation of swizzle, so I'm not renaming that
one.) To avoid breaking source compatibility, I am leaving the old name
as well as a synonym, but commenting it as deprecated and I will phase
out its use. It will disappear entirely from a future OIIO version
that's safe to break compatibility.
* For 16-wide simd, the 1-arg template we called shuffle was not doing
the same operation -- it was replicating one group of 4 elements instead
of a single element. We didn't use it anywhere, so I'm redefining it to
do the analogous thing as it does for 4-wide and 8-wide.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* ImageOutput methods that write scanlines, tiles, and images, which are
the main customization points for format output implementations, are
given additional methods that take `image_span` in place of raw pointers
and strides.
* Generally, for each, there is a templated version that takes
`image_span<T>` that communicates both the memory area and the data type
conversion requested, a "base case" that takes an explicit TypeDesc for
the conversion type and a `image_view<std::byte>` giving the raw memory
layout, and a `span<T>` convenience version for when there are
contiguous strides. Note that when reading mixed channel data types in
"native" mode (no type conversion, just leave the data in its original
types), you have to use the std::byte image_span version, since the idea
is not to do any format conversion, and there may not be a single type
involved.
* For now, the default implementations of these new ImageOutput methods
are just wrappers that call the old pointer-based ones. One by one, over
time, we can swap them, changing the format implementations to have a
full implementation of the new bounded versions, and make their raw
pointer versions call the wrappers. The raw pointer ones will be
understood to be "unsafe", merely assuming that the pointers always
refer to appropriately-sized memory areas. Meanwhile, the ones using
spans and image_spans will, due to assertions in their implementations,
make it easier to verify (at least in debug mode), that we never touch
memory outside these bounds.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Added support for reading and writing ZBUF chunks, also known as
Z-buffers or depth maps.
Removed noproxy versions, no longer needed.
---------
Signed-off-by: Mikael Sundell <mikael.sundell@gmail.com>
A constellation of closely related fixes and additions related to
oiiotool `-i` behavior:
* When using `-i:now=1` to force for immediate read rather than using an
ImageCache, it would "forget" any prior setting of `--native`. This is
fixed now to correctly combine desire for bypassing the cache with
hinting to maintain native data type.
* When restricting channel subset with `-i:ch=...`, use of the
`--native` flag was incorrect in some cases -- it would use the overall
best single data format of all channels, rather than restricting that to
the selected channel range. So, for example, if you said `-i:ch=R,G,B,A`
and those channels were all `half`, but channel 17 (not selected) was
`float`, you would get a conversion to float rather than keeping the
selected channels at their native half. The solution here is that the
type chosen for `--native` behavior should only consider the selected
channel subset, not all channels.
* Add a new `-i` modifier: `:native=1`, allowing you to select the
"native" behavior on a per-input-file basis (the existing `--native`
command sets the default).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
PR #4736 bumped the version of sphinx we use on readthedocs, and that
was incompatible with the old version of furo we specified.
I'm not sure how the problem was not caught by our "docs" workflow.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Certain version combinations of fmt and gcc were warning, and it all
boiled down to the knots we tied ourselves into to replace the function
fmt uses instead of throwing exceptions, when they are disabled. I can't
seem to find any harm in letting it do its default thing, so simplify
our side to silence the warnings.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
I discovered, by testing against the latest libultrahdr, that only
starting in 1.3 does it support the "64 bit 'half' RGBA" variety of UHDR
that is the flavor used by the testsuite example. But our "auto-build"
is pegged to 1.2, which means that all along, we were not decoding that
test file properly as UHDR and our test reference output was therefore
wrong. We also didn't detect libuhdr's version at all, let alone exclude
versions that were too old.
Remediations:
* Amend Findlibuhdr.cmake to detect the version from symbols in its
header files.
* Have our search for libuhdr enforce a minimum version of 1.3. (I feel
bad about "changing minimum dependencies" in a release branch, which we
usually disallow, but I think this time is an exception since we know
versions < 1.3 are broken.)
* Change the auto-build selected version to 1.4, the latest.
* Mention libuhdr in the INSTALL description of dependencies and
acknowledgement in the docs, where it had previously been omitted in
those places.
* Update the testsuite/jpeg-ultrahdr test reference output. (You can
tell it was wrong before because the "max" channel value was 1.0, and
now it's 4.6.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Seems to shave about 5s off of each and every CI run variant. Every
little bit helps.
Also update the versions of some actions we're using to the latest.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
I think what we were doing here was fine all along, but the idiom was
confusing to static analyzers who identified a danger that we were
memcpy'ing into a field that was potentially not big enough. A minor
restructuring of the code and a new assertion should verify that it's
safe and also make it clear to the static analyzer that we aren't
falling into the case it warned about.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Sonar was complaining that initialize_cuda() contained a loop that would
only iterate once. Upon looking, this was true, but mainly because we
just didn't adequately check errors. Fix up, which should both make the
code more robust and also silence the Sonar complaint.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This useless symbol was removed in PR #4724, but maybe downstream things
used it and will break simply because it was removed, even though it
never would have been nonzero. It should not have been completely
removed except at a compatibility-breaking version boundary.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* `image_span<T,Rank>` is a 2D-to-4D bounded span with byte-measured
strides (describing data layout with dimensions for channel, x, y, z)
for us to have future APIs deal with bounded+strided regions rather than
YOLOing raw pointers. Among other advantages, image_span will have
robust bounds checking in debug builds. It also simplifies interfaces to
pass a single image_span that encapsulates everything about an up-to-4D
data layout, rather than needing to pass a long list of individual
pointers, sizes, and strides.
* I've templated image_span on Rank, but default to Rank=4 and am using
that version only in practice. The templating by rank allows future
expansion to describe one scanline (rank 2) or 2D image plane (Rank 3)
if we want to do so for certain API calls, but I'm not sure yet whether
to bother with that, so for now, the ability to template by Rank is just
flexibility for the future.
* Add image_span based versions of copy_image(), contiguize(),
convert_pixel_values(), convert_image(), and parallel_convert_image()
utilities. I added tests and benchmarks to verify their correctness and
that they have similar performance to the pointer based versions.
* Deprecate image_view, which we did not ever use internally to OIIO or
in its APIs. But since we published the headers, there's a chance it's
used downstream and would be unwise to change its behavior or data
layout. The new image_span is what we will use moving forward so that we
are free to modify it and start with a fresh slate. Keeping the
definition/header for now, but marking it with a deprecation warning.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* We passed a bad flag to _mm512_cvtps_ph intrinsics because of what I
can only imagine was a bad cut-and-paste years ago. (Harmless because it
set bits never used, but a warning on Windows.)
* Remove clauses for AVX512-EL, I'm not sure why they were there,
because those instructions were only for "Knights Landing" which was
never made available to the general public and has been discontinued. It
must be that years ago, I got a little overzealous with the SIMD
intrinsics.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Somewhere along the road to 3.0, the build files changed in such a way
that OIIO_SITE was not correctly passed down to C++ symbols, and some
SPI-specific (sorry) color space handling was inadvertently disabled as
a result. This restores things to the way it was in 2.5, and should not
affect anybody else.
Also eliminate some old SPI-specific code that's no longer needed (and
clearly hasn't been needed for a long time, because we had been naming
the site "spi" but testing "SPI", and no problems were evident with
skipping that ancient code).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Use constexpr inside BaseTypeToC templates.
* Add BaseTypeToC_v and TypeDescToC_v convenience aliases, much like is
done in C++ standard type trait helpers.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Inspired by a link from EmilDohne in an unrelated PR, this introduces
`default_init_allocator`, an experimental custom allocator that default
initializes, rather than value-0-initializes, POD/non-class types.
This is useful for being able to make a std::vector with this allocator
(`default_init_vector`), thus having the property of not incurring a
large cost of needlessly initializing the elements when they are
allocated, as a default std::vector does.
I picked two spots sort of randomly, in which we previously avoided
using std::vector and instead did a unique_ptr / new[] dance to do
scoped heap allocation without paying the vector initialization tax. By
using default_init_vector, we avoid the extra cost while still using the
much more convenient vector type.
These two places are just to prove it works. There's no particular need
to proactively root out every use of the unique_ptr trick. But for
future development, we can use now use std::vector without the
construction cost when we allocate.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
INSTALL.md bolds *changed* minimums of dependencies since the last
release. Now that 3.0 is released and main is working toward 3.1, remove
those highlights from the last release so they won't get mixed up with
future changes for 3.1.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Raise auto-build version of Robinmap to 1.4.
* Add the magic CMAKE_POLICY_VERSION_MINIMUM to our auto-build of
Robinmap in case an older version is selected for auto-build. (Mimicking
the fix we did for other packages.)
* Add Robin-map to the INSTALL.md list of dependencies, somehow it had
been overlooked before.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The raw input plugin populates the attributes in the ImageSpec from the
libraw structures in the RawInput::open_raw() method. That is done
before the actual image data being decompressed on demand in
RawInput::read_native_scanline(). Libraw re-calculates some values after
the pixels get available, it is impossible to access those, as the
ImageBuf's spec does not get updated, and the updated ImageSpec is not
available from outside of RawInput when/after the pixels are being
processed.
An example: the initial values for the white balancing weights get
populated from the image file metadata, those get copied into the
ImageSpec. If the libraw was configured to white balance by averaging
the pixels of the whole image or a region, the white balancing weights
would get re-calculated by libraw after processing the pixels, but the
updated numbers are impossible to access from outside of the RawInput
plugin.
This change adds a hint "raw:ForceLoad", which, if set to 1, forces
libraw to decompress and process the image in RawInput::open_raw(), so
the image attributes returned from that method would contain the final
values.
I am not certain if this is the best way to achieve what I needed. I
have also looked into making the ImageBuf retain the ImageInput object
instead of spawning multiple transient objects, but that seemed too
complicated.
I have not added any unit tests, not sure how to approach that. I have
tested that the image does not get decompressed multiple times in a
bunch of scenarios.
Signed-off-by: Anton Dukhovnikov <antond@wetafx.co.nz>
This change embeds version information into our Windows binaries. This
is done by ways of a `version_win32.rc` resource file[1] populated at
CMake configure time with our major.minor.patch version information.
This not only allows at a glance to discover what version a particular
OIIO binary is (see screenshot), but it also impacts certain MSI
installer behaviors. If this is a MSI install which updates a prior
installation, and the shared libs/dlls don't have version information,
then the installer assumes the binaries are equal and just skips them.
This can be problematic for applications that package OIIO etc.
Old on left, New on right:
[1]
https://learn.microsoft.com/en-us/windows/win32/menurc/versioninfo-resource
---------
Signed-off-by: Jesse Yurkovich <jesse.y@gmail.com>
`--render_text:measure=1` causes the render_text command to set user
variables `TEXT_X`, `TEXT_Y`, `TEXT_WIDTH`, and `TEXT_HEIGHT` (as if set
by `--set`) to the origin offset and dimensions of the rendered text
size. These can then be accessed in subsequent expression evaluation.
`--render_text:render=0` causes the text to not actually be rendered
into the image. (The default is 1.) This is primarily of use in
conjunction with `measure=1` if you want to measure the size that text
would have taken, then use the variables for subsequent commands, but
not actually draw the text.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The SGI IRIS format is big-endian so incoming multi-byte fields need to
be swapped. This wasn't being done in the `valid_file` call so the magic
test would always fail on little-endian platforms.
Signed-off-by: Jesse Yurkovich <jesse.y@gmail.com>
From the PDF:
https://pub.smpte.org/doc/st268/20030505-pub/st0268-2003.pdf
3.31 X scanned size: Defines the horizontal size of the original scanned
optical image in millimeters. [42.1]
3.32 Y scanned size: Defines the vertical size of the original scanned
optical image in millimeters. [42.2]
----
This is a small PR that simply clarifies the units in the documentation.
Signed-off-by: Campbell Barton <ideasman42@gmail.com>
These values are read & written as integers
and don't work unless they are accessed using int accessors.
m_dib_header.hres = m_spec.get_int_attribute("XResolution");
Correct documentation.
BMP uses integer access:
- `./src/bmp.imageio/bmpinput.cpp:`, `m_spec.attribute("XResolution",
(int)m_dib_header.hres);`
- `./src/bmp.imageio/bmpoutput.cpp`, `m_dib_header.hres =
m_spec.get_int_attribute("XResolution");`
Signed-off-by: Campbell Barton <ideasman42@gmail.com>
Seems that for quite some time, there have been particular circumstances
in which the Python bindings have leaked ImageInput or ImageOutput
objects. That's bad enough on its own, but it also leaves file handles
open if code was depending on the fact that the ImageInput and
ImageOutput destructors close the files if they haven't already been
closed.
I think the offending part is that the `unique_ptr<ImageInput>` extracts
its raw pointer and gives it to Python, on the mistaken assumption that
it would be reference counted as a python object. But apparently, it's
not. Oops.
Leaving it as a unique_ptr turns out to be fine, as far as pybind11 is
concerned! And on top of that, if you give an uninitialized unique_ptr
to pybind11, it considers that a Python `None`, which means we can
further simplify some places where we explicitly had to return a
py::object and figure out if it should be None or not, but actually it's
all ok to just give the unique_ptr and pybind11 figures the whole thing
out for us.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We recently started getting some CI failures on the "bleeding edge" test
that uses the new gcc 14, seems that the newest cut of that compiler was
giving us new warnings where it suspected we were writing past the valid
bounds of some memory areas, but it was very hard to decipher the
circumstances and either find the bug or convince myself that is was a
false positive.
To help me debug it, I switched some of the internals of exif.cpp that
passed around raw pointers in very hard-to-validate ways to instead
use spans with explicit lengths, in the hopes that the debug-mode
bounds checking it provides would allow me to more easily track down the
problem.
Now it no longer gets compiler warnings.
Did I fix a very subtle bug in the process?
Does the reformulation with spans make it easier for the compiler to
verify that we aren't stomping on memory, and that suppresses the
warning?
Either way, it now passes all CI tests and I think the new code is safer
and aligned with our strategy of slowly changing raw unbounded pointers
to using spans in order to make these kinds of bugs harder to introduce.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The latest cmake in some package repos is 4.0. CMake 4.0 drops support
for compatibility with cmake < 3.5, and thus errors when building our
own dependencies that happen to have their own cmake scripts asking for
compatibility with something older than that.
Setting CMAKE_POLICY_VERSION_MINIMUM=3.5 environment and/or cmake
variable seems like a hack that could hide true incompatibilities, but
will get us past this for now.
Change our own minimum to the range of 3.18.2...4.0 to correctly reflect
the range of versions that we believe will work.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Add new ImageInput varieties of read_native_scanlines,
read_native_tiles, and a new read_native_voluemetric_tiles (all-channel
and channel-subset varieties of each) that take a `span<byte>`
describing the area to deposit the contiguous native-datatype values.
Moving forward, these will be the primary pixel-reading customization
points of the format readers. For now, these default to calling the
pointer-based varieties (so that old plugins keep working), but
eventually, the pointer-based varieties will be deprecated.
Notes:
* Omit the single-scanline and single-tile versions. Just keep the more
general multi-scanline, multi-tile versions, which of course people are
free to call with ranges that request just one scanline or tile.
* Omit the z coordintes for read_native_scanlines, since we know of no
volumetric format that allows reading of individual "scanlines".
* Omit the z coordinates from read_native_tiles to simplify the common
case of 2D images, but add a new read_native_volumetric_tiles() to
handle the volumetric case separately. This only needs to be supplied
for volume formats,
* The default implementations of the new methods is simply to call the
old, unsafe, raw pointer based versions. Thus, an ImageInput that still
supplies pointer based versions but not the span-based ones will not
break. Eventually, we will switch to the other way around, where all
ImageInput subclasses are expected to fully implement the span version,
and the pointer version will default to just be wrappers around the span
version for backward incompatibility. Way, way down the line, we'll
probably remove the pointer based varieties entirely.
Note that for this PR, we are only tinkering with the `read_native_`
functions that the ImageInput format readers supply. In a later PR,
we'll handle all the client-facing API calls to have span-based
varieties, as well as the analogous functions on the ImageOutput side.
In this PR, we do the overhaul on the JPEG and TIFF readers, to prove
that the whole scheme works. Over time, we will do the similar overhaul
of the other format readers.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Bump to the latest in several cases. Also make sure the INSTALL.md is in
line with what we support and/or test.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
When using `OpenImageIO_BUILD_LOCAL_DEPS` environment variable to build
all OIIO's dependency locally, OIIO's build script specifies a set of
default versions of the dependent libraries. In particular,
`ebiggers/libdeflate` is set to `1.20`.
This 1.20 version of libdeflate fails to build on platforms where AVX512
is not available and the make tool chain did not correctly tell the C
compiler about it.
```
-- Building package TIFF locally
-- Building package libdeflate locally
-- Building local libdeflate 1.20 from https://github.com/ebiggers/libdeflate
/usr/tmp/ccCelbo5.s: Assembler messages:
/usr/tmp/ccCelbo5.s:649: Error: unsupported instruction `vpdpbusd'
/usr/tmp/ccCelbo5.s:650: Error: unsupported instruction `vpdpbusd'
```
This issue is fixed in libdeflate Merge Request 389
https://github.com/ebiggers/libdeflate/pull/389, by explicitly calling
CMake facilities to detect whether AVX512 is present, and set compiler
preprocessor flags accordingly in the cmake file. This libdeflate MR is
after version 1.20.
To fix OIIO's build error when building this dependency locally, one can
either set `libdeflate_BUILD_VERSION` to include this patch, (for
example `export libdeflate_BUILD_VERSION=1.23`) or change the default
version in OIIO's local dependency build script to 1.23. This PR makes
the change in OIIO's local dependency build script.
Assembler build error resolved after using libdeflate v1.23 for local
TIFF build.
Signed-off-by: lji <lji@ilm.com>
The Intel Macs had this problem last week and #4668 fixed it. Now
suddenly the ARM Macs had it this week. Seems to be something changing
on the skbuild side.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
I still think that the "abi break" is a false positive. But I really
didn't need the new constructor that seemed to confuse the abi checker,
so just to be extra sure that there is no break that I simply
misunderstand, change to make abicheck happy.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
After merging #4672, we noticed some things that could be improved, so
this is a further revision (and partial retraction).
There was already an oiiotool flag, `--metamerge`, which in retrospect
make it seem unnecessarily confusing to add --mergemeta. So this PR
removes this extra new command, instead simply piggy-backing on the
existing --pastemeta by adding to it optional modifiers:
- `:merge=` controls the merge strategy, 0 (default) means discard the
existing metadata entirely and use only the pasted, matching the
original --pastemeta behavior; 1 means merge nondestructively (no
existing metadata will be altered or removed); 2 means destructive
merge, where pasted metadata will overwrite existing metadata having the
same name, but conflict-free existing metadata will remain.
- `:pattern=` regex selects only some metadata.
Additionally, we noticed an interesting corner case that surprised users
when testing the new metadata merging feature from the last PR: What
happens if the metadata source image has fewer subimages than the pixel
source images? The behavior generally for oiiotool is that when binary
operations (those taking two images as input) are on images with
differing numbers of subimages, the output gets the number of subimges
as the FIRST input (assuming -a instructed us to consider subimages at
all). But that's not what we want here! If the metadata source has one
subimage and the pixel source has many, clearly the intuitive result is
for that one metadata set to be pasted into all of the subimages from
the pixel-supplying image.
So we fix that so that for --pastemeta, it's the second input image (not
the first) that takes the lead in determining the number of subimages we
will process. This in turn took a bit of extra infrastructure fixing in
the OiiotoolOp class that controls this logic.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Clarify that IB::copy_metadata and oiiotool --pastemea fully replace all
existing metadata with that of the other IB.
Add a new IB::merge_metadata() and oiiotool --mergemeta that merges in
metadata from the other IB, preserving metadata in `this` that was not
in the other IB. Optionally, metadata that existed in both can be
overridden. Optionally, a subset of metadata to be merged can be
selected via a regex.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Give names to some CI steps that didn't have them -- makes it nicer to
read the logs and CI status dashboard.
* Make sure no job nametag is a substring of another, or it might find
the wrong cache, which is based on the job name.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The following command fails for images with mip map levels due to a bug:
```
oiiotool mipmap_images.*.exr --siappendall -o all_images.exr
```
If the option `-a` is not appended to the command, mip levels higher
than 0 should be ignored, instead it just segfault due to out of bound
access; I just fixed the it in the PR cf. [my comment
below](https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/4671#discussion_r1999705885).
Signed-off-by: Basile Fraboni <basile.fraboni@gmail.com>
The default value for m_bitspersample should be 1 when
TIFFTAG_BITSPERSAMPLE is not present in the directory.
Signed-off-by: vernalchen <shaochun.chen@autodesk.com>
Additions to `OIIO::span<>`:
* New flavors of `as_bytes()`, `as_writable_bytes()` that take a raw
pointer + length, turning them into `cspan<byte>` and `span<byte>`,
respectively.
* `span_cast<T>()`: "cast" a span of one type into a span of a different
type covering the same memory. Use with caution!
Safety improvements to existing functionality:
* front(), back(): Add a debug assertion to validate that there is at
least one element.
* first(), last(), subspan(): clamp offsets and counts so that the
result can't possibly return a span referencing memory that wasn't
inside the original bounds.
* The copy constructor that takes a non-matching span was much too
flexible. Now it uses enable_if to ensure that it can't be used to
change data types or lengths, it can only turn span of non-const into
span of const, and span of fixed extent into span of dynamic extent.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* The GitHub account has changed (the old one redirects, but still,
let's update to the current canonical name).
* Update the version we use in CI and for "autobuild" to the latest
tagged release instead of one that's years old.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This used to work until a few days ago. Then it stopped. I think that
GitHub changed the Mac runner configuration in some way that broke it,
now it's complaining about not being able to find ninja, so easily fixed
by instructing it to generate regular makefiles instead.
Along for the ride: If ONLY the wheel making workflow changes, there's
no need to rerun the regular CI, docs, or analysis workflows.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
From an array of known size, we can implicitly construct a span of
dynamic length, or of fixed length equal to the array length. Use
template enable_if trick to restrict to these cases.
Previously, the constructor would have allowed a fixed-length span to be
constructed from an array of differing length, which should obviously
have been disallowed.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Also detect patch level version. E.g. useful when checking for included
security fixes.
I ran `oiiotool --help` manually, and checked that the macro is
available already in 2.1.0.
Signed-off-by: Joachim Reichel <43646584+jreichel-nvidia@users.noreply.github.com>
There were some psd/psb files that were not opening with oiiotool, failing
on the signature.
Some keys were missing for the additional layer information, leading to
the pointer to misalign when reading the file, expecting the next 4
bytes rather than 8 bytes where needed for parsing.
No additional test suites added.
After deliberating, it is unclear if we should have a strict mode, or
keep parsing the rest of the file after a failed signature, given that
in this case the signature is not the part that's actually causing the
issue. Might be useful as a future feature, if there is an use case for
that?
---------
Signed-off-by: Lydia Zheng <lydiamzheng@gmail.com>
Signed-off-by: Lydia Zheng <36296057+lydia-zheng@users.noreply.github.com>
The current width/height limit enforced by the `check_open` API is too
low for PNG files. Each dimension can technically be up to `2^31 - 1`
but using that value would defeat the purpose of the check here. Use `1
<< 20` like what is used for TIFF and EXR instead.
The situation was triggered while trying to write an 85000x28400 px
image, which is smaller than 65536*65536 but the dimensions are checked
individually rather than as a total pixel count or similar.
Also looks like the PNG output was allowing too many channels so fix
that too.
Signed-off-by: Jesse Yurkovich <jesse.y@gmail.com>
Since C++17 defines std::size(C) (defaulting to returning C.size()), we
don't need to define our own now that C++17 is our minimum.
It seemed harmless to define it anyway for quite a while... but then a
few days ago, the GitHub Windows CI runners upgraded their version of
MSVS, and we started getting some very odd test failures that I was able
to trace to std::size somehow returning the wrong size in some cases,
and printf debugging revealed that the one we supplied wasn't even the
one being called.
I can't say I understand exactly what was going wrong. I was debugging
by repeatedly submitting small changes with debugging scaffolding to GH
to see how it works in a CI run, since I have no access to a local
Windows machine (at about 1/2 hour turnaround time for each iteration!).
Once I discovered that not supplying the redundant definition somehow
fixes the problem, I couldn't justify spending more of my weekend on
this to find out why. "Because MSVS" is a sufficient explanation for a
lot of weirdness in the computing world.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Old: Do a build with target 'clang-format', relying on our build system.
But that means that we need to do the cmake config step, and need a
bunch of dependencies set up so that the config doesn't fail.
New: Make a new run-clang-format.bash utility to issue the commands for
the right files. No "build" is requred, and the only dependencies we
need are the minimum it takes to run clang-format itself.
These changes reduce the approximate total elapsed time of a CI
clang-format job from nearly 7:00 to around 0:45.
Also, I noticed that we never had been clang-format testing the .cpp and
.h files in the testsuite itself. So enable that now, and commit the
small amount of reformatting that it identified.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
When test building the docs, lock down breathe and furo versions to
exactly what we use on RTD.
Also, to speed up the workflow: run on native Ubuntu rather than on an
ASWF container (eliminates needless container download), and make sure
that we don't waste time installing dependencies that we don't need.
These changes speed up the docs workflow from around 3:40 to around
2:00.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The main thing here is that it turns out that when trying to parse the
ICC profiles from Photoshop PSD/PSB files, we weren't storing the
attributes in the correct way. Everything else, we shoved in both the
"common" and "composite" imagespecs, but we left ICC data out of the
composite spec.
While I'm at it, I made two other ICC related changes:
* Among the several formats with ICC profiles, we were inconsistent
about whether we report errors for corrupt ICC profiles and treat the
whole image as broken, or just ignore the errors entirely. Fixed in
various places to make a consistent policy of always checking the
errors, and reporting/terminating if the "imageinput:strict" global
attribute is true, allowing it to just ignore the broken ICC record if
that attribute is false.
* Found a spot in JPEG handling of ICC where we were dangerously
trusting of the sizes of things. Replace a risky memcpy with a safer
spancpy that can't overrun the buffer bounds when copying.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Replaces the system() call with posix_spawn() on macOS to preserve
environment variables. This is essential for Debug builds using
DYLD_IMAGE_SUFFIX=_debug to prevent Qt from loading mixed
configurations.
## Tests
Run the iv application in background mode on macOS without the -F flag,
using a Debug configuration. Previously, the system() call did not
inherit the parent environment variables, causing a mix of Debug and
Release versions of Qt to be loaded.
---------
Signed-off-by: Mikael Sundell <mikael.sundell@gmail.com>
Existing versions could return the expected bytes for a scanline, tile,
or image for either the native per-channel types, or the single
composite channel type.
The new flavor lets you pass in any data type and it tells you how many
bytes you'd need for a scanline, tile, or image if all channels were the
passed type. (As a special case, the answer for TypeUnknown is the
"native" size, same as `thing_bytes(true)`).
This is handy for computing how big your buffer needs to be to hold a
type-converted scanline or tile.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Always use patch-specific SO versioning for scikit-build-core-based
builds, even when OpenImageIO_SUPPORTED_RELEASE=ON.
Fixes#4621
Ensures that Python-based builds only bundle a single copy each of
libOpenImageIO and libOpenImageIO_Util shared libraries.
## Tests
Previously, `$ uv sync -v --reinstall
-Ccmake.define.OpenImageIO_SUPPORTED_RELEASE=1` would bundle both
.so.3.0 and .so.3.0.3 copies of libraries when installing to a local
virtual environment.
With this fix, only .so.3.0.3 copies of libraries are installed.
Signed-off-by: Zach Lewis <zachcanbereached@gmail.com>
Also, make the wheel workflow work on pushes when the branch name
contains the substring "python" or "wheel", and when any of the cmake
build files in src/cmake (which control building of dependencies,
among other things), even if none of the python bindings themselves
are touched. This means that if you suspect your work might cause a
wheel to break even though you're not modifying the python bindings
directly, you can ensure that the wheel workflow will run on CI simply
by naming your branch to contain one of those words.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Included two additional metadata boolean checks for constant and black
frame images, so that handling for those image cases are facilitated.
IS_CONSTANT and IS_BLACK return 1 when true, and 0 when false.
Signed-off-by: Lydia Zheng <lydiamzheng@gmail.com>
There was a bug in span_from_buffer when strides were negative
(basically, we were subtracting, when adding was correct since those
strides were negative values).
Also added a negative stride test to imagebuf_test.cpp. Embarrassed that
we didn't already have this, since it would have caught the problem.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Fill in OpenEXR lineOrder attribute when parsing the header in
OpenEXRCoreInput::PartInfo::parse_header()
---------
Signed-off-by: vernalchen <shaochun.chen@autodesk.com>
When you use frame or view wildcards, for example:
oiiotool foo.%04d.tif -o bar.%04d.exr
we noticed that if NO files matched, no iteration would occur. You would
get this output:
```
$ oiiotool foo.%04d.tif -o bar.%04d.exr
oiiotool WARNING : oiiotool produced no output. Did you forget -o?
```
But maybe this is not helpful and it should be some kind of error to ask
for a match and not get one? Or at least give a more clear explanation
of what happened.
This PR detects the case where there is no match of the wildcard at all,
warns about that, and then tries interpreting the command line literally
(no wildcard expansion). So now the output is
```
$ oiiotool foo.%04d.tif -o bar.%04d.exr
oiiotool WARNING : No frame number or views matched the wildcards
oiiotool ERROR: read : File does not exist: "foo.%04d.tif"
Full command line was:
> oiiotool foo.%04d.tif -o bar.%04d.exr
```
Closes#4623
Signed-off-by: Larry Gritz <lg@larrygritz.com>
As part of the recent big python wheel merge, compiler.cmake got changed
in a subtle way in how it sets CMAKE_INSTALL_RPATH.
This broken iv for me on a Mac with dynamic libraries. Strangely (and
I'm too lazy to chase down exactly why), oiiotool seems fine. So this
was missed entirely by the CI, which never runs 'iv', since it's an
interactive app. (Note to self: we should find some way to at least
verify that it can execute as part of the testsuite.)
Anyway, I believe that this change adds the install lib area back to the
path, and it does seem to repair my ability to run iv on my end.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The implementation of IB::pixeltype() was:
- Make sure at least the spec had been read (lazy read).
- If there are "local pixels", return the pixel type from the spec,
otherwise return the cache pixel type.
But there is a flaw in this logic: If only the metadata has been read
but not the pixels, the local pixels might not be allocated yet, but
also there is not a cache and no cached pixel type, so pixeltype() will
return UNKNOWN even though we do already know what type of pixels will
be stored when they are eventually read.
The correct logic is:
- Make sure at least the spec had been read (lazy read).
- If it's a cached image, return the cache pixel type, otherwise return
the pixel type from the spec (which will be correct because we
definitely have read the spec at this point).
Also, for legit cached images, ensure that cachedpixeltype is set upon
read_spec() and doesn't have to wait for a full read().
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We were only running the wheel workflow on pushes of TAGS, but this
change also will run it when we push to release, and dev-3.* branches.
My usual release workflow is to first push the "release" branch, watch
for it to pass CI, then add the tag and push the tag and draft the
release. If a problem with the wheel workflow isn't discovered until
we've already pushed the tag, then a fix will require a new tag and new
release (we never re-tag a release). So this change will allow us to
catch any problems with the wheel worflow BEFORE we actually irrevocably
make the tag, since it's no big deal to add corrections to the branch
before tagging. It still won't upload the wheels until the tag itself is
pushed.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Use the publicly-available ARM-native linux runners for CI checks and
building Python wheels
---------
Signed-off-by: Zach Lewis <zachcanbereached@gmail.com>
Quite often I find myself debugging code or CI specific to just one
platform. It makes for much longer iteration for a huge CI job that
reruns lots of tests that won't be affected by the changes I'm
experimenting with. Sometime I make a commit that places `if: 0` in
strategic places to exclude irrelevant batches of tests. But this is a
pain to add, and has to be removed again to re-enable the tests prior to
turning the experiment into a proper PR.
This PR, then, adds some logic to ci.yml that lets me do the same just
with special branch names, rather than needing to edit ci.yml. Any
branch you push containing the substring "windows-only" will not run any
Linux or MacOS CI tests. And the analogous behavior is added for
"macos-only" and "linux-only".
This shouldn't affect any CI jobs in which people are not purposely
using those special branch names. You should NOT use these special names
for branches from which you submit PRs, since obviously we want every PR
to validate on all CI combinations. But it is very helpful for topic
branches on your own fork as you are developing and iterating over many
CI runs but only need results from one platform.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This got lost with the 2-level namespace changes in #4567, but is used
by e.g. OpenUSD. Might as well not break the API here.
Signed-off-by: Brecht Van Lommel <brecht@blender.org>
Implement FR from #4546
Add a new `--layersplit` command to `oiiotool` to split an image into
its channel-name-based layers onto the stack.
The extracted layer names are then stored in the `oiio:subimagename`
metadata (in case it is needed for later use, e.g merging these "layers"
as sub-images), and the extracted channel names replace the old channel
names in the new images.
Example:
an image with channels `R, G, B, A, diffuse.R, diffuse.G, diffuse.B`
will be split into two images, the first one with channels `R, G, B, A`
and the second one named `diffuse` with channels `R, G, B`.
> Note: we did not implement the `--layertosi` command suggested in the
FR as it can already be done with a combination of `--layersplit` and
`--siappendall`
We add a test in the test suite that takes a "multi-layer" image, splits
the layers and merges them as sub-images. The final image should have 3
sub-images.
---------
Signed-off-by: Loïc Vital <mugulmotion@gmail.com>
### Summary
This PR is the spiritual successor to #4011. It implements a
`scikit-build-core`-based python build-backend, making it possible to
use `pip install .` to build from source; and it adds a Github workflow
for building with `cibuildwheel` and publishing to pypi.org binary
distributions (bdists) of the Python module / extensions / CLI tools for
cpython 3.8-3.13, across major operating systems and architectures.
When you `pip install OpenImageIO`, pip attempts to retrieve an
OpenImageIO bdist from pypi.org for the host's platform / architecture /
Python interpreter. If it can't find something appropriate, pip will
attempt to build locally from the OpenImageIO source distribution
(sdist), downloading and temporarily installing cmake and ninja if
necessary.
### PEP-Compliant Packaging: `pyproject.toml`
The `pyproject.toml` file is organized in three parts:
1. **Package metadata**: standard attributes identifying and describing
the Python package, its run-time and build-time requirements,
entry-points to executable scripts, and so forth.
2. **scikit-build-core options**: governs how `pip install ...`
interacts with `cmake`.
3. **cibuildwheel options**: additional steps and considerations for
building, packaging, and testing relocatable wheel build artifacts in
isolated environments.
### Additions to `__ init __.py`
Previously, we were using a custom OpenImageIO/__ init__.py file to help
Python-3.8+ on Windows load the shared libraries linked by the Python
module (i.e., the .dll files that live alongside oiiotool.exe under
$PATH).
This PR adds an additional method for loading the DLL path, necessitated
by differences between pip-based and traditional CMake-based installs.
It also adds a mechanism for invoking binary executables found in the
.../site-packages/OpenImageIO/bin directory. This provides a means for
exposing Python script "shims" for each CLI tool, installed to
platform-specific locations under $PATH, while keeping the actual
binaries in a static location relative to the dynamic libraries. Upshot
is, in `pyproject.toml`,
each item under `[project.scripts]` is turned into a Python script upon
installation that behaves indistinguishably to the end user to the CLI
binary executable of the same name.
### Relocatable Binary Distributions with `cibuildwheel` + `repairwheel`
[cibuildwheel](https://github.com/pypa/cibuildwheel) is a widely-used
tool for drastically streamlining the process of building, repairing,
and testing Python wheels across platforms, architectures, interpreters,
and interpreter versions.
Additionally, the cibuildwheel-based builds set CMAKE_BUILD_TYPE to
"MinSizeRel" to optimize for size (instead of speed) -- this seems to
shave ~1.5MB off each .whl's size, compared to "Release"
### "Wheels" Github workflow
I straight-up copied `.github/workflows/wheel.yml` from OpenColorIO and
made a few OIIO-specific modifications. When pushing a commit tagged
v3*, the workflow will invoke a platform-agnostic "build sdist" (source
distribution) task, followed by a series of tasks for building OIIO
wheels for cpython-3.8-3.13 on Windows, Linux (x86_64 + aarch64, new
libstdc++), and MacOS (x86_64 + arm64) and persisting build artifacts;
followed finally by a task for publishing the build artifacts to
pypi.org
Note: For the sake of simplicity and troubleshooting, I've made as few
changes to OpenColorIO's wheel.yml as I could get away with; but in the
future, we can also build wheels for the PyPy interpreter, and possibly
pyodide.
Note: A "trusted publisher" must be set up on pypi.org. See
https://docs.pypi.org/trusted-publishers/creating-a-project-through-oidc/
### Other Changes
I made some minor adjustments to `pythonutils.cmake` and
`fancy_add_executable.cmake` that only affect scikit-build-core-based
installs:
- -- namely, on Linux and macOS, I'm setting the INSTALL_RPATH property
to point to the relative path to the dynamic libraries, for the Python
module and CLI tools, respectively. This helps ensure that pip-based
builds and installs from source (as opposed to installs from repaired,
pre-built wheels) yield relocatable, importable packages, without
needing to mess with $LD_LIBRARY_PATH etc.
## Tests
`cibuildwheel` tests if `oiiotool --buildinfo` runs. If that command
elicits code zero, it means the "oiiotool" Python script installed by
the wheel is able to `import OpenImageIO`; that the actual binary
executable `oiiotool` is properly packaged and exists in the expected
location (e.g., at `.../site-packages/OpenImageIO/bin`); and that all
runtime dependencies are found.
## Inspiration, Credit, Prior Art
- @aclark4life's and @JeanChristopheMorinPerso's efforts + direction +
discussion + advice. See
[#3249](https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/3249),
and
[#4011](https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/4011),
as well as JCM's
[python_wheels](https://github.com/JeanChristopheMorinPerso/oiio/tree/python_wheels)
and
[python_wheels_windows](https://github.com/JeanChristopheMorinPerso/oiio/tree/python_wheels_windows)
branches. This PR is an attempt to leverage OIIO-2.6+
self-building-dependency features with # 4011's minimalist and modern
approach to packaging.
- OpenColorIO -- I tried to copy as much as I could from @remia et al's
fantastic work with all things wheels-related. The __init __.py
modifications, the way we're wrapping the CLI tools, and the github
Wheels workflow are lifted almost-verbatim from OCIO. Insert pun about
reinventing the wheel here.
- @joaovbs96's help and patience with testing stuff on Windows.
---------
Signed-off-by: Zach Lewis <zachcanbereached@gmail.com>
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Signed-off-by: Anton Dukhovnikov <antond@wetafx.co.nz>
Signed-off-by: Basile Fraboni <basile.fraboni@gmail.com>
Signed-off-by: Vlad (Kuzmin) Erium <libalias@gmail.com>
Signed-off-by: Joseph Goldstone <joseph.goldstone@mac.com>
Signed-off-by: Darby Johnston <darbyjohnston@yahoo.com>
Signed-off-by: Jeremy Retailleau <jeremy.retailleau@gmail.com>
Signed-off-by: zachlewis <zachlewis@users.noreply.github.com>
Co-authored-by: Larry Gritz <lg@larrygritz.com>
Co-authored-by: Anton Dukhovnikov <antond@wetafx.co.nz>
Co-authored-by: Basile Fraboni <basile.fraboni@gmail.com>
Co-authored-by: Vlad (Kuzmin) Erium <libalias@gmail.com>
Co-authored-by: Joseph Goldstone <joseph.goldstone@mac.com>
Co-authored-by: Darby Johnston <darbyjohnston@yahoo.com>
Co-authored-by: Jeremy Retailleau <buddly27@users.noreply.github.com>
Co-authored-by: Jean-Christophe Morin <38703886+JeanChristopheMorinPerso@users.noreply.github.com>
Adds attributes to the input's ImageSpec for the black level and bit depth
when the raw bayer buffer is requested.
Signed-off-by: Anton Dukhovnikov <antond@wetafx.co.nz>
Fixes a bug in demosaicing where the code ignores the ROI channels.
The old behaviour was to always decode into the first 3 channels.
Signed-off-by: Anton Dukhovnikov <antond@wetafx.co.nz>
Perform the endian byteswap on the correct buffer area for PSD RLE
decoding. The original code used `dst` but that pointer is modified from
its original value as it progresses through decoding.
Fixes#4599
---------
Signed-off-by: Jesse Yurkovich <jesse.y@gmail.com>
Also some very minor code cleanup that was helpful in setting up the
test cases and getting better code coverage.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
TL;DR: In main, change the enclosing namespace from `OpenImageIO_v3_1`
to a 2-level scheme: `OpenImageIO::v3_1`. This can't be backported to
the current release for obvious reasons; it is a forward-looking
change only.
More ponderous explanation follows:
We have for some time had a single namespace that contains the
version, defaulting to `OpenImageIO_v3_0` in the current release, for
example. A namespace alias, `OIIO`, always aliases the current
namespace, so client applications can just say `OIIO::foo` without
needing to change code for every minor release (let alone if a
build-time option sets up a custom namespace).
By bumping the namespace for every minor release, we make the symbol
names themselves enforce a rigid ABI compatibility test, so you can't
accidentally compile against one minor release and link against an
older minor release. This gives us the freedom to break the ABI (link
compatibility) for each minor release, without subtle user errors.
(Doing it wrong makes a total failure to link, which is hard to miss.)
But being able to introduce ABI changes annually by changing the
enclosing namespace comes at the cost of a *complete* ABI compatibility
break with every minor release. Even classes or functions that haven't
change at all will be incompatible by virtue of their changed symbol
names.
This is an unfortunate limitation, and in an ideal world, we would
like downstream users to be able to upgrade to a newer minor release
more painlessly, and confident that they could even relink or perhaps
compile with a request to use an old ABI.
I haven't fully worked out all the details, or even if this is going
to be worth the trouble, but I think that a change we can introduce
now that will allow more flexibiity in the future is to switch to a
2-level namespace scheme, for example, `OpenImageIO::v3_1` instead of
the current `OpenImageIO_v3_1`.
The reason this might be helpful in the future is that we can use
"inline" namespaces that default to finding things that are in, say,
OpenImageIO::v3_2 without needing to specify it explicitly, while
giving the ability to explicitly give an alternate inner versioned
namespace. I haven't implemented that part here, it's reserved for
future expansion (though I have experimented with it, it does work,
but I want to discuss separately whether or not to actually do it).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Bump the 'latest' test to test the actual latest, 1.15.
* Be sure that the 'oldest' tests actually test the oldest we support,
1.8.
* Sprinkle around a couple other versions that we don't test currently.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Users can enable the legacy behavior by setting env var
`OIIO_PYTHON_LOAD_DLLS_FROM_PATH=1`.
This commit also changes the environment variable name from
`OIIO_LOAD_DLLS_FROM_PATH` to `OIIO_PYTHON_LOAD_DLLS_FROM_PATH` to match
the convention used by OCIO.
This PR addresses issue #4519
---------
Signed-off-by: Zach Lewis <zachcanbereached@gmail.com>
Implements X-Trans pattern demosaicing using a simple bilinear
algorithm. Adds the Xtrans pattern layout to the ImageBuf attributes in
the raw reader plugin.
I have implemented unit tests in imagebufalgo_test.cpp which cover all
permutations of the Bayer and X-Trans layouts, as the X-Trans pattern is
a bit too complicated to generate using oiiotool.
---------
Signed-off-by: Anton Dukhovnikov <antond@wetafx.co.nz>
For reasons that are unclear (maybe a change to which python versions
are installed where on the GH runners?) this seems to have broken within
the past couple days, in a way that did not correspond to any change we
made on the OIIO side.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* When the script was run on aarch64, it was actually hard-coded to
retrieve a version older than we currently require as a minimum. Use the
CMAKE_VERSION hint just like we do for x86_64.
* Bump the default version that this script makes to the latest, 3.31.3.
(This does not change OIIO's minimum required cmake version, it merely
changes the default if this hepler script that we use for CI is run
without any specific version requested.)
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This improves the mediocre testing line coverage in nullimageio.cpp from
50% to 95%.
* Test writing
* Test reading of tiled and MIP-mapped null images
* Test setting and reading of attributes
* Simplify some code
* Along the way, identified and fixed a nullimage bug -- it was
answering that it supported "rectangles" but had no write_rectangle
method. Testing works!
Signed-off-by: Larry Gritz <lg@larrygritz.com>
See discussion
https://github.com/AcademySoftwareFoundation/OpenImageIO/discussions/4054
and PR
https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/4315
In PR 4315, we fixed PNG read/write to do the required premultiplication
in linear space (that is, linearize, multiply by alpha, then go back to
the sRGB or gamma space). Which we really believe is "correct." Except
that maybe there are a ton of incorrectly made PNG files out there
(maybe most of them?) where partial alpha pixels had their
premultiplication occur on the sRGB or gamma values directly.
In this patch, we partly revert, allowing both potential behaviors,
controlled by an attribute "png:linear_premult", which instructs the PNG
driver to do any premultiplication in linear space if it's set to
nonzero. It can be set globally (via `OIIO::attribute()`), as well as
set/overridden on any individual file by setting the attribute in the
configuration hints for an ImageInput or in the spec for an ImageOutput.
As presented in this patch, we're setting the default to 0, meaning that
by default we are reverting back to the old behavior of doing the
premultiply of partial alpha pixels on the direct values intead of in a
linear space. Applications or sites that are confident that any PNG
files they encounter are "correct" can set the attribute to do the
multiplication linearly.
I'm not 100% confident about the default, and so am very happy to
entertain arguments for keeping the default set to do the multiplication
in linear space.
I had to change an internal spin mutex to a recursive mutex in order to
address a latent misdesign that was made symptomatic by this change.
Basically, asking for an attribute inside an ImageInput::init could make
a deadlock because certain attribute queries (that catalogue the list of
plugins) might instantiate those plugins, thus causing their init
functions to run, leading to recursive dependence on the mutex that
guards attribute queries.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Co-authored-by: Brecht Van Lommel <brecht@blender.org>
* Ptex testsuite entry had been accidentally disabled somewhat recently
when we switched to using Ptex exported cmake config instead of our own
find module, because the "found" variable changed from PTEX_FOUND to
Ptex_FOUND.
* Change the test to print stats, thus forcing the ptex test to read
pixels instead of just header info, which improves the testing coverage
of ptexinput.cpp.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Upgrade upload-artifact action, old one is being deprecated.
* Update checkout and cache actions to latest.
* Fix workflow warning: save-always is not really an input to
actions/cache.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Chipping away at minor warnings and areas identified by static analysis
as potentially problematic or hard to analyze.
* benchmark.h: suppress warnings about uninitialized Benchmarker
members.
* dpxoutput.cpp: change some raw pointers to spans
* dead code removal (color_ocio.cpp, filesystem.cpp, sysutil.cpp)
* exclude build-scripts from static analysis
* some misc warning suppression
Signed-off-by: Larry Gritz <lg@larrygritz.com>
It escaped our notice before that the IPTC spec dictates length limits
for many fields. Getting this wrong can confuse some software, including
Photoshop, apparently.
This patch enforces length limits wherever we could figure them out from
the IPTC spec -- it will simply truncate those strings that are too long
before writing them to a binary IPTC tag.
For this reason, we also are ending the practice of automatically
translating several IPTC fields to and from what we figured were the
equivalent generic metadata names. This was perhaps a dubious practice
to begin with, but now the enforcement of length limits for IPTC (but
not the generic metadata) makes it even more frought. There is probably
nobody depending on this behavior (and maybe few OIIO users depending on
IPTC support at all?), so now anybody purposely using IPTC metadata is
fully responsible for setting it and dealing with any issues of whether
it's "out of sync" with any other metadata that OIIO stores in or reads
from a file.
Fixes#4342
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
It was syntactically correct, but without an actual variable, there was
no RAII scope and so it destructs immediately, causing the timers to
capture 0 time.
Found by Sonar warnings.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Fix missing secrets that accidentally disabled the analysis workflow a
few commits back. After splitting into reusable steps, we needed the
main analysis workflow to let the steps workflow inherit secrets.
* Run analysis on TOT dev branches.
* For analysis, only build OIIO as debug, not dependencies.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The logic for the GHA steps was repeated across multiple workflows and
platforms. Consolidate it all into a common "ci-steps" workflow, and
have all the other workflows reference that when possible. This should
hopefully make maintenance of the CI scripts easier in the future, with
less redundancy of the step logic.
The only major place where we deviate is that we have a separeately
specified set of steps for the "old" VFX Platform 2021 and 2022 based
tests (because we need to specify different container arguments to it).
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We don't do anything with this at the present time, but this PR reserves
and documents this attribute for future use.
The intent is to be able to set whether we want image readers to try
being as tolerant as possible when reading a file with flaws (press on
and see if the rest of the file is ok?), or be more conservative and
abandon reading any file as soon as a corruption or invalid data is
encountered (because that might be a clue that the file is arbitrarily
corrupted or even maliciously constructed).
I documented it as defaulting to 0 (err on the side of being permissive
of bad input), with high-security applications being responsible for
setting it to 1. But it's open for debate if people think that a better
default is to be strict and let applications who want to be more
tolerant be responsible for accepting the risk and switching the mode.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Fixes#4552
Caught during fuzzing with address sanitizer.
The source of the problem was a corrupted/truncated pgm file. Several
minor modifications in this PR shore up various cascading errors that
followed. Not all were directly causal to the sanitizer trigger, in some
cases I fixed what appeared to be related areas.
* In imagebuf.cpp, any time we free the local pixel memory m_pixels,
also explicitly clear the m_bufspan that has a span representation of
the usable memory and its bounds.
* An extra check related to oiiotool --printstats to make sure that the
image is valid before passing along to stats collection.
* In pnminput.cpp, a better error message when we hit a premature end of
file.
With these fixes in place, we seem to get a graceful error message and
exit when running the POC that was provided with the bug report.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Fixes#4553
Caught during fuzzing with address sanitizer. The file appeared to have
a resolution so big it would not be able to satisfy the memory
allocation.
Solution: add the check_open to take an early abort if resolutions are
bigger than could possibly be valid.
Also have Strutil::stoi hande 32 bit overflow without UB overflow that
the sanitizer complains about (that was the other cascading error that
this same test case encountered in the sanitizer after the bad
allocation).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Affects jpeg, tiff, and other formats with embedded ICC profiles.
Corrupted/invalid tag offset and/or length values were causing us to
touch memory outside the memory allocated for the profile.
Fixes#4551
Not sure if this is wise or not, but for JPEGs, I also enabled some
previously commented-out code that has the effect of treating a file
with a verifiably corrupted/nonsensical ICC profile block as a failed
file. If people ever complain about this in the future, maybe we'll come
back and add some kind of global option that lets the app express how
tolerant OIIO readers should be about corruptions in files -- do you
assume that the whole file is bogus or malicious upon discovering the
first invalidity, or do you press on and hope you can get something else
useful from the pixels? Let's be cautious for now.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
When using pkgconfig the value of `<NAME>_CFLAGS_OTHER` can contain
compiler flags along with preprocessor definitions. If there is a
compiler flag and its passed to `target_compile_definitions` then it
will be treated as if it is a preprocessor definition.
Found a case where a statically compiled LibRaw with a statically
compiled Little-CMS caused `LibRaw_DEFINITIONS` to have `-pthread` in
its listing which was passed to the compiler as a definition,
`-D-pthread`. This caused a build failure with `auto-moc`.
Add an additional parameter for the `add_oiio_plugin` macro to take
`COMPILE_OPTIONS` and pass those to `target_compile_options` which can
differentiate between compiler flags and preprocessor definitions. Pass
`LibRaw_DEFINITIONS` to the CMake macro to prevent the above error.
Issue found while updating Little-CMS in
https://github.com/microsoft/vcpkg/pull/42187 and the change fixes the
build issue found there. This is an attempt to upstream it so the issue
is fixed.
Signed-off-by: Don Olmstead <don.j.olmstead@gmail.com>
Implements IBA::scale, an algo which takes two images, one of which is
required to be a single channel image, each channel of the other image
gets multiplied by that single channel, per pixel.
Note that this is subtly different from IBA::mul(), which is a
channel-by-channel multiplication, and so doesn't have a provision for
allowing you to scale all channels of one image by the single channel of
a second image.
I have added tests for the C++ code, python code, oiiotool, and the
documentation examples
Signed-off-by: Anton Dukhovnikov <antond@wetafx.co.nz>
OIIO 2.3.13 with PR #3307 changed
MultiByteToWideChar/WideCharToMultiByte usage to C++11 <codecvt>
functionality, but that has two issues:
1) it is *way* slower, primarily due to locale object access (on Visual
C++ STL implementation in VS2022 at least). Since primary use case of
these conversions is on Windows, maybe it is better to use a fast code
path.
2) whole of <codecvt> machinery is deprecated with C++17 accross the
board, and will be removed in C++26. I've kept the existing functions in
there since otherwise it would have been an API break, but really maybe
with OIIO 3.0 they should have been un-exposed. Too late now though :(
## Tests
Performance numbers: doing ImageInput::create() on 1138 files where they
are not images at all (so OIIO in turns tries all the input plugins on
them). Ryzen 5950X, VS2022, Windows:
- utf8_to_utf16 3851ms -> 21ms
- utf16_to_utf8 1055ms -> 4ms
Signed-off-by: Aras Pranckevicius <aras@nesnausk.org>
Primary cost of detecting whether a given file is valid jpeg2000 file is
the thread pool initialization and shutdown that `Jpeg2000Input::open`
does (the actual thread pool part is inside OpenJpeg code).
So add a dedicated `valid_file()` implementation that only needs to
check 12 bytes of the header. While at it, I changed already existing
`isJp2File()` to `is_jp2_header()` to better match naming conventions
used elsewhere, and instead of trying to handle both little and big
endian cases by manual repetition of two sets of magic integers, let's
do just byte comparisons with `memcmp` instead.
On my PC (Ryzen 5950X, SSD, Windows VS2022), doing
`ImageInput::create()` on 1138 files where they are not images at all
(so OIIO in turns tries all the input plugins on them):
- Before: **3.4 seconds** spent in `Jpeg2000Input::open` (1.9s
`opj_thread_pool_create`, 1.3s `opj_thread_pool_destroy`)
- After: **33 milliseconds** spent in `Jpeg2000Input::valid_file`
No new tests. I checked behavior on the official Jpeg2000 conformance
data set
(https://github.com/uclouvain/openjpeg-data/tree/master/input/conformance),
seems to work.
Signed-off-by: Aras Pranckevicius <aras@nesnausk.org>
The background here takes some explaining...
The ASWF's prepared Docker containers that reproduce a CentOS7-based
environment corresponding to VFX Platform 2021 and 2022 years contain a
glibc that is too old to run a newer version of "node" -- which GitHub
Actions themselves use. The old one is no longer receiving support, and
GitHub finally changed the policy that was allowing its GHA runners to
still run the old version that worked with those ASWF containers. The
new node version that is happy on the GHA runners requires a glibc newer
than the one in the containers. So that breaks all our CI runs in 2021
and 2022 VFX Platform configurations.
See this thread, if you dare:
https://academysoftwarefdn.slack.com/archives/C0169RX7MMK/p1732574400981949
Anyway, Jean-Francois Panisset proposed and tested a fix, which lets us
continue working (for now) by downloading an old node implemenation
(which is happy with the old glibc) and installing it atop the
conainers, so it's this old one that the GH actions will use. It feels
like it's all held together by chewing gum and duct tape, but it
unbreaks our CI and hopefully it will keep working until all the studios
have switched fully to RHEL/Alma/Rocky 9.x and we no longer care about
testing on CentOS-7 based containers or supporting the VFX Platform
years that are too old to be relevant to the new OS that the studios are
using.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Having such a document is one of the remaining steps to getting our
OpenSSF Best Practices silver level badge.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Having such a document is one of the remaining steps to getting our
OpenSSF Best Practices silver level badge.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Get rid of the FORTIFY options, which were too specific to one compiler
feature. Add OIIO_HARDENING that takes a level meaning:
- 0 : do nothing, not recommended.
- 1 : enable features that have no (or nearly no) performance impact,
recommended default for optimized, shipping code.
- 2 : enable features that trade off performance for security,
recommended for debugging or deploying in security-sensitive
environments.
- 3 : enable features that have a significant performance impact, only
recommended for debugging.
Default to 1 for optimized builds, 3 for debug builds (so will be
thoroughly tested by our sanitizer and other CI tests). Users that have
more stringent security requirements may choose to build with 2 even for
shipping code (they should benchmark to see if that is acceptable to
them).
These levels turn on a variety of compiler options that are recommended
for additional safety. We will add more as they are developed.
There are also some warning suppressions that needed to be added to code
in a few areas where it was unavoidable to use some constructs that
trigger the elevated safety checks.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
These commands, which are intended to alter the top image on the stack,
were each allocating and copying an image unnecessarily for each
operation. By changing the implementation to the right calls that know
it doesn't need to make a copy (already a feature I'd implemented in the
OiiotoolOp helper class) greatly speeds them up.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
When the exr "name" metadata (used for the names of subimages/parts)
existed but was the empty string, we were treating it like it didn't
exist (and upon write substituted our own "subimageXY" name, as we are
wont to do for unnamed parts). But now we've seen multi-part exrs where
the name is "" (exists, but empty string) and we want a read-write cycle
to preserve that. Even though it's not a very sensible or helpful part
name, it's technically legal, so we should honor it.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This is needed to port Blender's current JPEG IO code to using OIIO, but
is also a useful feature to have in general.
For reading, the code tries to parse comments as colon-separated
key-value pairs and sets metadata accordingly. For writing, this needs
to be explicitly enabled by setting jpeg:com_attributes to 1 in order to
avoid accidentally bloating files for existing applications.
Tests:
I've added a small (~10KB) JPEG file containing Blender metadata and a
basic test that parses it, checks that the metadata was read correctly,
writes it twice (once with and once without `jpeg:com_attributes`), and
then checks that those files are also parsed as expected.
In case you're wondering why the info for "no-attribs.jpg" still
contains one Blender attribute - that's because the first COM field is
still put into `ImageDescription` just like before, so even without
`jpeg:com_attributes` it ends up being written to the output file and
recognized during parsing.
---------
Signed-off-by: Lukas Stockner <lukas@lukasstockner.de>
The function libraw_filter_to_str() converts the `imgdata.idata.filters`
field to human readable form, by mapping a few hardcoded values. I found
the issue by converting images with disabled demosaicing, which writes
the Bayer pattern into the ImageBuf metadata. The Panasonic GX9 didn't
have any pattern recognised; The Canon R5 was incorrectly recognised as
GMCY.
The `imgdata.idata.filters` actually contains a sequence of 2-bit
values, where each represents an index in a 4-character long string
`imgdata.idata.cdesc`. I have modified the code to extract the channels
from `imgdata.idata.cdesc` as expected.
I have also disabled the normalisation code, which I believe was
incorrect, and never executed due to the issue above.
The current tests pass, however, I'll start a discussion on the piece of
code I have concerns with.
---------
Signed-off-by: Anton Dukhovnikov <antond@wetafx.co.nz>
* Add build recipes for certain dependencies needed for linking static
OpenColorIO libs: expat, minizip-ng, pystring, and yaml-cpp
* Add SOURCE_SUBDIR option to `build_dependency_with_cmake` that mimics
the behavior of the FetchContent option of the same name
* Tweak OpenColorIO build recipe to always build static libraries by
default.
---------
Signed-off-by: Zach Lewis <zachcanbereached@gmail.com>
Add an API to query available fonts based on family and style name. The
API can be used to build a UI where users can select a font from a list.
They consist of new queries for `OIIO::getattribute()`:
- `string font_family_list`
A semicolon-separated list of all the font family names that
OpenImageIO can find.
- `string font_style_list:family`
A semicolon-separated list of all the font style names that
belong to the given font family.
- `string font_filename:family:style`
The font file (with full path) that defines the given font
family and style.
Fixes#4503.
---------
Signed-off-by: peter.horvath <peter.horvath@autodesk.com>
Replacing 3x IBA + Helper function that generate 4 fulls size image
buffers with single unsharp_mask_impl() that use parallel_image() to
compute unsharp:
src + contr * (((src - blur) < threshold) ? 0.0 : (src - blur))
Added two pass 1D convolution for a kernels higher than 3x3
## Tests
```
ImageBuf sharped(input.spec());
const int repeats = 50;
std::cout << "Start sharpening\n";
auto start = std::chrono::high_resolution_clock::now();
for (int i = 0; i < repeats; i++)
{
//ok = ImageBufAlgo::unsharp_mask(sharped, input, "gaussian", 15.0f, 10.0f, 0.01f);
ok = ImageBufAlgo::unsharp_mask(sharped, input, "gaussian", 5.0f, 2.0f, 0.05f);
std::cout << ".";
}
std::cout << "\n";
auto part1 = std::chrono::high_resolution_clock::now();
std::chrono::duration<double> elapsed_part1 = part1 - start;
std::cout << "Elapsed time: " << elapsed_part1.count() << " s\n";
```
both single threaded (one IB at time) and multithreaded (multiply IB at
time) show pretty good speedup:
~30-40% with less memory use.
for 5x5 gaussian kernels two pass mode should add at least 20% speedup.
(if someone can do independent benchmark, will be great. As soon as I
had a big differences on them depend on real or synthetic use)
---------
Signed-off-by: Vlad (Kuzmin) Erium <libalias@gmail.com>
Initial feature request: https://github.com/AcademySoftwareFoundation/OpenImageIO/issues/4424
Add support in the `jpeg` plugin for reading Ultra HDR images using the reference codec `libultrahdr`: https://github.com/google/libultrahdr
In short, "ultra hdr" images are a clever extension of JPEG where the image file really is an old school JPEG file and will be interpreted correctly as such by old readers not aware of ultra hdr, but readers that are aware will see an extra piece of metadata that contains a gain map, that when applied to the base layer then yields an HDR image. Pretty clever approach!
Images used for testing during development:
https://github.com/MishaalRahmanGH/Ultra_HDR_Samples
---------
Signed-off-by: loicvital <mugulmotion@gmail.com>
The latest libheif patch release changed something very slightly that
requires us to update some saved reference output.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Doing some benchmarking and adding extra LoggedTimer instrumentation
(included in another PR), I discovered that quite frequently, the
in-memory flavor of ImageBuf was reading its image multiple times --
once upon construction, and then again upon later calls to read(), even
though the latter should have taken early outs when the read was
unnecessary.
So I put some additional mechanism for the IB to internally know if it
read the pixels from disk into the buffer, and more effectively make
calls to IB.read() be a no-op. To help facilitate this, I also clarified
in the docs that read() should not be used to move an already-read image
to a different subimage or MIP level or a different data type -- reset()
really should be used for that purpose. (I don't know that anybody ever
relied on read() to do reset's job, but assuming that they won't helps
eliminate some tricky cases.)
Also, I noticed that inside ImageInput::read_image(), depending on
whether or not a data type conversion is needed, it can either read
directly into the user's buffer, or must read the native data into a
temp buffer and subsequently do a copy-and-type-convert into the user's
buffer. The latter is obviously more expensive. The direct read approach
was used when the user asked for no type conversion (just to receive the
native data type in the file), but it neglected to take the same
shortcut if the user asked for an explicit type conversion, but it just
so happened that the requested type was the type in the file.
Additional changes:
The evaluation of whether the buffer was contiguous (for various
shortcuts) only applied to LOCALBUFFER style storage, not APPBUFFER.
Fixing these two problems dramatically speeds up some reads. A benchmark
I was looking at where it reads a 10k x 10k uint8 uncompressed TIFF file
into an ImageBuf while requesting uint8 data went from (timings on my
laptop):
function calls total
IB::read 2 0.505s (avg 0.25s)
II::read_image 2 0.478s (avg 0.24s)
to:
IB::read 1 0.172s (avg 0.17s)
II::read_image 1 0.172s (avg 0.17s)
So almost 3x faster. Most of which was due doing 2 reads instead of 1,
but still each individual read improved from 0.25s to 0.17s because of
the fix that lets us avoid the unnecessary buffer copies.
Grain of salt: this was a very large image, and it was stored
uncompressed (so NO decompression that can sometimes be a significant
amount of overhead), so you may not see this much improvement in real
world scenarios.
This is only scratching the surface, I believe there are lots of other
areas to optimize more carefully.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Fixes#4502.
This PR lets you specify fonts (for `IBA::render_text()` or `oiiotool --text`)
by family and style name, rather than only by filename of the font file.
The implementation is to add `init_font_families()` function, that iterates
over the available font files, reads family and style name via the freetype
API and caches them. The function is called from `resolve_font()` on
demand.
Added a new test to the oiiotool-text folder, that loads a font by the
name "Droid Serif Bold".
---------
Signed-off-by: peter.horvath <peter.horvath@autodesk.com>
The GHA macos-12 runner is deprecated and soon to be retired. It's
already flaky and they're doing purposeful outages to remind people (the
cloud equivalent of turning the lights on and off right before the bar
closes).
We still want to test earlier MacOS-12 compatibility to match VFX
Reference Platform, so use the macos-13 runner as the minmum but set
MACOSX_DEPLOYMENT_TARGET=12.0 to test that configurations.
Also, the MacOS-15 runner is in beta, so add that to the mix. (Along the
way, that also verifies that we run fine against python 3.13.)
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Because it's not in std::string_view.
If you must use this functionality (with caution about when it is safe),
then use the freestanding `OIIO::c_str(OIIO::string_view)` function
instead.
I would like to include this warning in 3.0.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Fixes#4504
The PR removes the code that looks for font folders based on a prefix
and suffix list. Instead, adds all known system font folders per each
platform. The global "font_searchpath" attribute, `OPENIMAGEIO_FONTS`
and `OpenImageIO_ROOT` environment variables are still searched as
before, such as fonts directories relative to the currently running
binary. However $HOME and $SystemRoot folders (with all previous
suffixes) are not searched anymore, thus this is a breaking change, and
the documentation might need to be updated accordingly.
The search order did not change. It's still "font_searchpath" attribute,
environment variables, system font folders, folders relative to binary.
---------
Signed-off-by: peter.horvath <peter.horvath@autodesk.com>
I don't understand why, but for certain combinations of gcc, fmt, etc.,
we were getting exceptions from fmt. I finally caught it in the act! It
seems related to formatting specs that looks like `{:-20s}`. Maybe this
used to be allowed but isn't officially supported? Anyway, left
justification is default anyway, so there's no point to the `-`.
Removing it makes fmt happy, no more exceptions.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Newline in strings on Windows is encoded as '\r\n', that is currently
not handled by the text render code and so carriage return (\r) shows up
in the rendered text. As a fix we simply ignore this character.
We came across this issue while exporting user specified text from DCCs
to render on Windows.
I'm not sure how to make a proper test case for this. Current tests
specify a command as a string that is passed to oiiotool as arguments
and executed in the command line. But adding a newline to the text
breaks the command.
Signed-off-by: Peter Horvath <peter.horvath@autodesk.com>
Signed-off-by: peter.horvath <peter.horvath@autodesk.com>
The `y` argument in `OIIO::ImageBufAlgo::render_text()` should represent
the top and bottom edge of the text if vertical alignment is set to
`TextAlignY::Top` and `TextAlignY::Bottom`. However currently the
rendered text is off. The current code adjusts y to point to the top of
the text box before rendering the text, but it has to be adjusted to
point to the baseline of the text.
Tests
There is a existing test case, testing vertical alignment, but the
reference image seems wrong and has now been updated. See the "Topleft"
and "Rightbot" texts.
| Current | Fixed |
| -------- | ------- |
|
|
|
Note: The testsuite contains a reference image for freetype v2.4.11
(aligned-freetype2.4.11.tif). I don't have that version built, and
haven't been able to update the image. I'm wondering if this is still
used or could be removed. I guess CI will tell.
---------
Signed-off-by: peter.horvath <peter.horvath@autodesk.com>
Address a few small issues encountered during CMake configuration step
and during build using Makefile `BUILD_MISSING_DEPS` option enabled.
- Mismatch between expected libraw versions specified in
`build_libraw.bash` 0.21.3 and `externalpackages.cmake` 0.20.0
- OpenEXR link error in libutil tests
- Update top level Makefile to accept `BUILD_MISSING_DEPS` as string
instead of bool. Pass the string option down to CMake
Verified build completes with no errors: `BUILD_MISSING_DEPS=all make`
Signed-off-by: kuba <kuba456@gmail.com>
Detect libbz2 only if necessary, i.e., FFmpeg or FreeType is enabled.
This avoids an unnecessary dependency on libbz2 if these features are
disabled.
In particular, with vcpkg, OpenImageIO might pick up a dependency on
libbz2.so from the host system since bzip2 from vcpkg is not listed as a
dependency of OpenImageIO (if the FFmpeg and FreeImage features are
disabled, and bzip2 is not already installed for other reasons).
Signed-off-by: Joachim Reichel <43646584+jreichel-nvidia@users.noreply.github.com>
A logical error was causing us to not correctly cycle through the
successive MIP levels. Somehow, we missed this all these years.
You can tell that there was something wrong all along by examining the
testsuite/iinfo test output (which we have updated here) -- in the
old/wrong one, the statistics "FiniteCount" suspiciously stayed the same
for all MIP levels, which is clearly wrong. Now you can see that it does
correctly reflect the reduced resolution of each MIP level.
This bug was limited to 'iinfo' itself. There wasn't any inherent
problem with reading MIP levels directly with ImageInput or ImageBuf.
The line with the bug has been unchanged for 14 years! We were trying to
be clever about not double-reading, except that the only context in
which this was called, that would never have happened anyway, so that
cleverness (which was incorrectly implemented) was unnecessary.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Times we DON'T need to test rebuilding the docs:
* When just the ci.yml workflow changes (like when we bump what versions
of dependencies ci tests use).
* When tests changes only by altering their run.py.
But DO try rebuilding docs:
* When cpp files change that are outside of src/, such as the cpp files
in the testsuite that are used to supply code snippets that appear in
the docs.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
I got part way through the OSL side changes to accommodate what we did
recently with TextureOpt and TextureOptBatch. Ugh, it's a lot harder
than
I thought and I don't know how to finish the job without postponing the
3.0 release even more, which is unacceptable at this point. So here I do
to TextureOptBatch to get out of this mess:
* "Version" TextureOptBatch like we did for TextureOpt (should have done
that all along).
* Revert "v1" to the same byte-for-byte layout we used to have, which
involves using `int` for some fields as it implicitly did before,
because
the new enum classes totally change the field sizes and struct layout.
* Sketch out in comments what v2 might look like, to more fully match
the
TextureOpt. The introduction of struct versioning means we can make that
change for 3.1, it doesn't need to wait years for 4.0.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
It's failing to get Mesa packages for some reason. Disable Qt and OpenCV
on this one test to work around it for now.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
These are all things long marked as deprecated (often since before the
official launch of 2.0) that we've waited for 3.0 to finally remove. Now
is the time.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Version the class for easier future expansion! Let's call it
`TextureOpt_v2` and we make a `using` alias so that the name
`TextureOpt` means this. We can then revise the struct in the future by
not losing link compatibility.
* The macro OIIO_TEXTUREOPT_VERSION (now 2) can be used to test which is
the current version.
* Get rid of TextureOpt-local enums by finally switching to the
freestanding ones in the Tex namespace. (We've wanted to do this for a
long time, but couldn't break compatibility.) Preserve API compatibility
for now with clever use of `static constexpr` and `using`.
* Make more efficient use of space in the TextureOpt by: field order
rearrangement to eliminate empty space, switching a couple of the enums
that needed only a few values to be `class enum xxx : uint8_t` so they
take only 1 byte instead of 4, switch `anisotropic` to be 16 bits (do we
really need an anisotropic ratio > 65535?), elimiate the time and
samples fields which have never been used (they were reserved for shadow
maps, which we never implemented).
I think this set of changes is minimally onerous to downstream projects
for the 3.0 release, while laying a path forward (via the versioning) to
allow future changes to the struct to happen with a lower pain
threshold.
I don't claim that this perfectly preserves downstream source
compatibility, but the fact that outside of these changes to
delcarations in texture.h, only 8 lines in the entire rest of the OIIO
codebase needed to change to accommodate them. So I think that the
impact downstream will be minimal. (OSL might be the exception, since
its LLVM code generation is exquisitely sensitive to the exact layout of
the TextureOpt struct, but I'm working on those changes, too.)
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This finally switches Strutil::format to use std::format conventions.
You can still access the old printf convention via
`Strutil::old::format()`, and can explicitly access the new convention
uniformly on both 2.x and 3.x via `Strutil::fmt::format()`.
It also removes a long-deprecated vformat function.
We've been waiting for 3.0 to make this non-back-compatible change.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
A few minor changes were needed to keep a working build, due to things
long ago rigged to be disabled as soon as the version bumped.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
A round of color management improvements (I hope).
We were pretty squirrelly about what "linear" meant. Get more exact. Now
we will treat "linear" as a legacy synonym for "lin_rec709" (sRGB
primaries, but linear response). Change file format readers to say
"lin_rec709" when they mean it, and not use the more generic alias
"linear".
"scene_linear", on the other hand, is the internal working color space
for math. Which concrete color space it is an alias for is determined by
the OCIO config in use.
Use "g18_rec709" and "g22_rec709" for rec709 + gamma 1.8 or 2.2,
respectvely.
A few new API calls help out: equivalent_colorspace(),
set_colorspace(spec,name), set_colorspace_rec709_gamma().
OpenEXR always tagged files as linear by default. It was too aggressive
about this, so with this PR, now it defaults to lin_rec709 only for
files that appear to be RGB and not have any other clues about the color
space.
Similarly, the raw reader was too eager to set the Exif:ColorSpace to
look like sRGB even when it wasn't.
Lots of reference outputs needed to be updated.
This is a potentially big behavior change. But I think it's for the
better.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
I was profiling the builds and saw that modules with lots of template
expansion dominate the compile time. For example,
imagebufalgo_pixelmath.cpp alone took 290s to compile on my 2020
MacbookPro (!), and imagebufalgo_addsub.cpp took 84 seconds.
This is all due to the combnatorics of expanding IBA templates via the
DISPATCH macros in imagebufalgo_util.h separately for every type that
the arguments can be. But I claim that most combinations are rarely if
ever used. I mean, how often does anybody need IBA::add() to add an int8
image to an int16 image? So this PR rewrites those macros to simplify
the cases as follows:
* The common pixel data types are float, half, unint8, and uint16.
* Specialized versions are fully expanded only when the result and input
images are one of these types. Images not of one of those types are
first automatically converted to float to make them reduce to a common
case. That makes uncommon pixel data types like (signed) int16 not
expand the template, but rather convert to and from float and use the
float specializations.
* For binary and ternary operations (those with 2 or 3 image inputs), if
the pixel types of the inputs doesn't match, we make sure they both are
converted to float. So, for example, we don't need a specialized version
that adds a half image to a uint16 image -- just convert them to float
and use the common case. But we do specalize if the two inputs are both
the same common case, such as adding two uint16 images.
* Assume that commonly, the result image will either be float, or will
be the same pixel data type as the inputs. Other combinations trigger
assignment to a temporary float IB, then copying with convertion to the
uncommon use-supplied result buffer.
* Additionally, we cut down on a little bit more templating by moving
some "deep" methods from the type-templated ImageBuf::Iterator to its
type-generic non-templated base class IteraterBase.
The net result of all this is an awful lot less template expansion. With
this in place, my laptop compiles imagebufalgo_pixelmath.cpp in 97s (vs
290 before) and imagebufalgo_addsub.cpp in 26s (from 84). It takes a big
bite out of all the iba files, and reduces project-wide compile time by
over 10%, around 30s out of 300 for a fresh, uncached, optimized build
with 16 threads.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
I refactored the companion project full of testing images, putting all
the clutter in the main directory into a subdirectory for each file
type. These are the main OpenImageIO changes needed as a result.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
When building OIIO as a subdirectory of a super-project (with
`add_subdirectory`), it is better to honor the output directories
specified by the parent project.
In our project, it was causing issues because OIIO libs were ending in a
different folder from USD plugins, which thereby failed to load
correctly in the build tree.
Signed-off-by: Luc Touraille <touraill@adobe.com>
Starting with OpenEXR 3.1.3, the preferred API call for setting
DWAA/DWAB compression level has changed. We never changed the OIIO side.
Luckily, OpenEXR seems to have kept respecting the old API calls we were
making (passing as an attribute). But this modernizes the approach, now
that we don't have OpenEXR 2.x support to maintain.
But in the process, we also fixed a bug! Turns out we weren't
propagating the compression properly -- if the compression name
was plain "dwaa", we ignored any "openexr:dwaCompressionLevel"
attribute, effectively.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Adds new feature: floating point exceptions, to imageinout_test binary.
Running `imageinout_test --enable-fpe` will run the test with FPEs enabled.
Currently it is catching a FP overflow in libjxl.
Once that is fixed, it may find other FPEs.
For now, this is a Linux-only feature.
```
$ ./imageinout_test --enable-fpe
```
Signed-off-by: Bram Stolk <b.stolk@gmail.com>
* Move RELEASING.md from the top directory to docs/dev
* Move 0.x and 1.x CHANGES documents from src/doc to docs
* A couple master->main renames that were MIA
Signed-off-by: Larry Gritz <lg@larrygritz.com>
OpenEXR 3.3 was just released and its main branch has bumped its
advertised version to 4.0 (the latter breaking our "bleeding edge" CI
test).
Raise our "latest releases" test to use 3.3.
Our "bleeding edge" test picking up the now-called-4.0 OpenEXR main
branch required some minor logic changes to not reject the version as
too new to be compatible.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
It's an old flavor not well supported by homebrew any longer, so what's
happening is that some of the dependency packages we ask to install have
to build from source, which means it takes a long long time.
By disabling OpenCV and Qt homebrew installation for it, we can make it
fast again. Neither the minimal OpenCV interop, nor osltoy (the only
thing that needs Qt) are exercised by the automated testsuite anyway, so
I don't think there's any loss of testing functionality in practice.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Some behavior changed for fmt error conditions, an in a way that
requires us to remove a test, because there is no reproducible output
to compare against.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
When writing a partial tile, the unused pixels still go through float
conversion.
This means, that floating point operations are done on uninitialized
data.
This can easily lead to NaN and to floating point exceptions, if those
were to be enabled.
This change will set the shared buffer used for all partial tiles to all
zero pixels.
Tested by running valgrind before and after the change
FIXES: #4461
Signed-off-by: Bram Stolk <b.stolk@gmail.com>
Adding data and display window overlay in iv.
Dashed line: Display window
Solid line: Data window
Off by default, select "Show display and data window borders" from the View menu.
---------
Signed-off-by: tkchanat <andychancse@gmail.com>
First draft of front end changes for issue #4436.
The end goal is to remove redundant ImageSpec from ImageCache internals (issue #4436).
But that necessarily changes some assumptions about what is accessible through the public
APIs for ImageCache and TextureSystem. This PR contains the API-side changes, but without
the internals overhaul that will come later.
* `get_imagespec()` and `imagespec()` lose their miplevel and native parameters. We are
now assuming now that there is no arbitrary metadata varying per-mip-level. Also, loss
of native param means that it's always copying an ImageSpec that reflects the file.
* New `get_cache_dimensions()` is a lighter-weight function that can retrieve the limited
items that really do vary between MIP levels, and may differ between the file and the
in-cache pixels: resolution, tile size, pixel data format.
* Harmonize the order of parameters in the analogous functions exposed by IC and TS
(they confusingly differed before).
* For now, API-backward-compatible wrappers for the old functions, which will eventually
be deprecated and then removed.
---------
Signed-off-by: Basile Fraboni <basile.fraboni@gmail.com>
cumentation to testsuite conversion for cpp and python for the
ImageBufAlgo Chapter, Section: Image Arithmetic up to the div function.
The PR also fixes a python testing image file reference in
..docs-examples-python/ref/out.txt, where reference images "simple.tif"
and "scanlines.tif" should have the docs-examples-cpp prefix (see
..docs-examples-python/run.py line 11 for details).
---------
Signed-off-by: Lydia Zheng <lydiamzheng@gmail.com>
This fixes OIIO's CI "bleeding edge" test that broke when OCIO pushed
to their main the new config files that will be in OCIO 2.4. (Yay!)
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This addresses:
```
==126424== Conditional jump or move depends on uninitialised value(s)
==126424== at 0x16A24E: IvGL::mouseMoveEvent(QMouseEvent*) (src/iv/ivgl.cpp:1208)
==126424== by 0x5FCD5F6: QWidget::event(QEvent*) (in /usr/lib/x86_64-linux-gnu/libQt6Widgets.so.6.4.2)
==126424== by 0x5F843AF: QApplicationPrivate::notify_helper(QObject*, QEvent*) (in /usr/lib/x86_64-linux-gnu/libQt6Widgets.so.6.4.2)
==126424== by 0x5F7D2C6: QApplication::notify(QObject*, QEvent*) (in /usr/lib/x86_64-linux-gnu/libQt6Widgets.so.6.4.2)
==126424== by 0x6E35447: QCoreApplication::notifyInternal2(QObject*, QEvent*) (in /usr/lib/x86_64-linux-gnu/libQt6Core.so.6.4.2)
==126424== by 0x5F79D7D: QApplicationPrivate::sendMouseEvent(QWidget*, QMouseEvent*, QWidget*, QWidget*, QWidget**, QPointer<QWidget>&, bool, bool) (in /usr/lib/x86_64-linux-gnu/libQt6Widgets.so.6.4.2)
==126424== by 0x5FDBAD6: ??? (in /usr/lib/x86_64-linux-gnu/libQt6Widgets.so.6.4.2)
==126424== by 0x5FDCE74: ??? (in /usr/lib/x86_64-linux-gnu/libQt6Widgets.so.6.4.2)
==126424== by 0x5F843AF: QApplicationPrivate::notify_helper(QObject*, QEvent*) (in /usr/lib/x86_64-linux-gnu/libQt6Widgets.so.6.4.2)
==126424== by 0x6E35447: QCoreApplication::notifyInternal2(QObject*, QEvent*) (in /usr/lib/x86_64-linux-gnu/libQt6Core.so.6.4.2)
==126424== by 0x6762DE6: QGuiApplicationPrivate::processMouseEvent(QWindowSystemInterfacePrivate::MouseEvent*) (in /usr/lib/x86_64-linux-gnu/libQt6Gui.so.6.4.2)
==126424== by 0x67AD5CB: QWindowSystemInterface::sendWindowSystemEvents(QFlags<QEventLoop::ProcessEventsFlag>) (in /usr/lib/x86_64-linux-gnu/libQt6Gui.so.6.4.2)
==126424== Uninitialised value was created by a heap allocation
==126424== at 0x4846FA3: operator new(unsigned long) (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==126424== by 0x11F509: ImageViewer::ImageViewer(bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (src/iv/imageviewer.cpp:140)
==126424== by 0x17D248: main (src/iv/ivmain.cpp:127)
```
With this patch, this particular valgrind error goes away.
Tested by running valgrind on iv before and after change.
Signed-off-by: Bram Stolk <b.stolk@gmail.com>
Fixes#4445
This avoids UB by initializing a variable.
## Tests
I ran valgrind after the fix: no longer issues error.
```
==77256==
==77256== HEAP SUMMARY:
==77256== in use at exit: 25,167,390 bytes in 8,232 blocks
==77256== total heap usage: 183,524 allocs, 175,292 frees, 4,415,783,216 bytes allocated
==77256==
==77256== LEAK SUMMARY:
==77256== definitely lost: 0 bytes in 0 blocks
==77256== indirectly lost: 0 bytes in 0 blocks
==77256== possibly lost: 0 bytes in 0 blocks
==77256== still reachable: 25,167,390 bytes in 8,232 blocks
==77256== suppressed: 0 bytes in 0 blocks
==77256== Rerun with --leak-check=full to see details of leaked memory
==77256==
==77256== For lists of detected and suppressed errors, rerun with: -s
==77256== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
```
---
Signed-off-by: Bram Stolk <b.stolk@gmail.com>
This fixes a problem seen on RHEL 9.2 Linux when launching an image in
iv, eg:
iv testsuite/texture-overscan/ref/out-exact.exr
The image displayed as a constant brown and the terminal showed:
GL error create program 1281 - Invalid value
GL error After attach vertex shader. 1281 - Invalid value
The fix is to initialize m_shader_program and m_vertex_shader in the
class constructor.
ERROR ANALYSIS:
The bug was introduced in 5fe44415 "feat(iv): OCIO color managed display
(#4031)".
Before that commit, create_shaders() initialized those members to 0
before they were used.
After that commit:
* uninitialized m_shader_program is passed to glDeleteProgram() (near
the top of create_shaders) ), causing the
"create program" warning (there isn't an print_error call after
glDeleteProgram so the error is reported as "create program").
* ununitialized m_vertex_shader is non-0 when checked by if
(!m_vertex_shader), causing the vertex shader not to be created and
causing the later call to glAttachShader() with uninitialized
m_vertex_shader to fail. That causes the second error message, and lack
of a vertex shader apparently causes the image to display as constant
brown.
Signed-off-by: David Adler <david1.pub@gmail.com>
This adds a null pointer check in the raw input plugin when the user
requested raw bayer data for a file format which doesn't have it. With
this fix OIIO shows an error instead of crashing in such case.
This fixes the issue mentioned in the comments of #4361. The title issue
of #4361 has been resolved in LibRaw 0.22.0, so we can close that issue,
I guess.
This also fixes the crash on 5 out of 6 examples in #3115. The last one
is unrelated, and I don't have a good fix for that at this time (crash
due to the raw buffer having half resolution, with the pixel aspect
ratio of 0.5). Perhaps we could create a separate issue and close#3115.
I have not added any tests for this change specifically, as it is fairly
trivial.
I have started working on a more comprehensive test suite for the raw
plugin / rawtoaces, but it's not ready yet, and I'm not sure where it
should live at this stage.
Signed-off-by: Anton Dukhovnikov <antond@wetafx.co.nz>
For old versions of fmt library, some error output was slightly
different. Address it with an additional acceptable reference
output for the oiiotool test.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Basically, this switches to creating the colorconfig that oiiotool uses
from statically constructed to build-on-demand, which should slightly
lower runtime overhead for oiiotool invocations that don't need to do
any color transformations.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Next round of making safer interfaces. Primarily, this adds new
span-baesd versions of ImageBuf::get_pixels, set_pixels, setpixel,
getpixel, interppixel, interppixel_NDC, interppixel_bucibuc,
interppixel_bicubic_NDC.
The old ones took raw pointers and implied sizes -- these still exist
but are considered "unsafe" and aimed at experts and special cases. (If
a downstream project wants to be sure not to use them, they can define
symbol "OIIO_IMAGEBUF_DEPRECATE_RAW_PTR" before including imagebuf.h to
force deprecation warnings.)
The new additions instead take a span -- a combined pointer and length
that can be bounds-checked (in debug mode). I strongly encourage people
to use the new interface, as it forces the caller to think hard about
the exact region of memory that's ok to read from or write to, and then
for the OIIO side to honor that and enfoce/debug with bounds checks
against that if we add such plumbing.
Along the way, I added some interesting helper functions:
In span.h:
* `span_within()` checks that one byte span lies entirely within
another.
* `check_span()` checks if a pointer and length lies entirely within a
span.
* `OIIO_ALLOCA_SPAN` is a macro much like OIIO_ALLOCA, except what it
returns is a span rather than a raw pointer to the stack-allocated
region.
In imageio.h:
* `span_from_buffer()`/`cspan_from_bufffer()` : Given a raw pointer,
pixel data type (TypeDesc), and image dimensions and strides, returns a
byte stride describing the extent of memory that encompasses all pixels
laid out by that description.
I hope the switch to spans is not very onerous for people. I really
think that this will help us write more solid and deliberate code to
begin with, and when things do go wrong, the various debug-mode checks
and assertions that spans provide should help us more quickly identify
exactly where buffer overruns and the like occurred.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Right now, the only option is gperftools, because that's what I happened
to be trying. But this can be expanded later.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Oops.
I didn't really have time to add an "example" right now. I hope to
return to beef this up (or that somebody else takes it on).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This test was running very slowly, taking too long in all circumstances,
but shockly causing GHA CI Mac runners to hit the test timeout
sometimes.
It turns out we were doing a lot of redundant work. There was no need to
make separate pattern and bayer images for every data type. Instead,
hoist the test image generation out of the loop and just make one
(float), then use `-i:type=<type>` to read it into an ImageBuf of the
appropriate data type.
This cuts the time for this test to run by about 3x.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We no longer supply a FindOpenColorIO.cmake, so it makes no difference.
But it was getting in the way of a weird case where a wrapping project
wanted to supply one.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
With 3.0 approaching, it seems like the only time to make these changes
that I made note of a while back:
* It seems to me that `tell()` should be const because it isn't going to
modify or change the state of the IOProxy.
* It seems to me that `flush()` should NOT be const, because it probably
does change internal state.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
When I added perpixel_op recently, I didn't get the function signature
right. It only would work for plain old function pointers, and would not
work properly for "functors" or lambdas with capture. I should have used
function_view which can do a light-weight wrapper of all three, so now I
do. Also modified imagebufalgo_test to explicitly test all three forms
to be sure they work.
Also update function_view ever so slightly.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Fixes the bug where IBA::demosaic only works with float images.
Fixes#4416
I've extended the unit tests to run on float, half, uint16, uint8
images.
Signed-off-by: Anton Dukhovnikov <antond@wetafx.co.nz>
openexr_copy and openexr-decreasingy
These recently added test were writing into the ref/ directory, which
they certainly should not do. Write into the build test area, compare to
what's in the ref area.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Convert C++ and Python examples from the "Image transformations and data
movement" section of the "imagebufalgo" chapter into tests within the
"docs-examples" testsuites (related to issue #3992).
Update 'oiiotool' tabs to use a simple 'tab' instead of 'code-tab' to
prevent synchronized tab selection, which is inconsistent with the other
tabs using the 'literalinclude' directive.
---------
Signed-off-by: Jeremy Retailleau <jeremy.retailleau@gmail.com>
This allows somebody who is enhancing the docs to check the results
entirely through CI without having to set up the dependencies necessary
for docs on their end. (Perfect for Dev Days.) It also ensures that any
changes we make to the code base don't break the documentation somehow,
because now a successful docs build will be part of the CI.
Note that the docs building job leaves behind an artifact, which will be
a zip file of the directly containing the HTML docs, just like one would
find on RTD, so you can download that zip file and view the docs it made
to verify that everything looks right.
I made the docs builder a separate workflow, but I also changed some
things in the CI workflow for the sake of making them more similar.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The flavor of ImageBuf that wraps caller-owned memory is constructed or
reset by merely passing a raw pointer and it assumes that the range of
memory implied by the strides is entirely safe to read from or write to.
And even if correct, there is not much within ImageBuf internals to
prevent bugs from buffer overruns.
This PR introduces a new constructor and reset() method to ImageBuf
where instead of taking a raw pointer, it takes a span describing the
extent of the safely addressible memory (and also an optional "pointer
to first pixel" within the buffer, which is necessary if you're using
negative strides or for some other reason are describing safe memory
that starts before the location of pixel 0). That span is stored in the
ImageBuf, and also it remembers if it was a cspan (which should indicate
a read-only IB) or a non-const span (which should wrap writable memory).
At this moment, I'm just adding this API and will start using and
documenting it as the perferred method, with the raw pointer versions
being regarded as unsafe for now (and maybe some day will be
deprecated?). But we're not yet using the span internally for safety
checks.
In the future, my intent is to gradually add safeguards to do bounds
checking against the span for IB operations that touch the wrapped
memory. Because we'll most likely bounds check only in debug mode, it
shouldn't ever impact performance, but it will make debugging,
sanitizer, and fuzz tests more robust to catch and zero in on buffer
overruns. I also like how it forces the caller, at the call site, to
think hard about exactly what range of memory the IB is wrapping, and
not merely pass a single pointer and hope it works out.
I also changed all ImageBuf ctrs and resets within the OIIO codebase to
use the new span-based versions.
This is part of an overall trend to migrate from raw pointer based API
calls to span-based API design as much as possible.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
span_util.h is relatively new. I'm honestly no longer sure why I thought
it was necessary to make a second file, but so little is in it that I
now believe its contents should be folded into span.h to keep all the
span-related definitions in once place. It's just not big enough to need
to be divided into several headers.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
When fmt arguments don't match the format string, fmt will throw an
exception, or terminate if we disable exceptions (which we do). For
gcc11+, we tried intercepting these, but let's do it for all platforms,
and let's not terminate ourselves, but insted print and log the error.
But, oof, to do this properly, I needed to move some error recording
functionality from libOpenImageIO to libOpenImageIO_Util and make it
owned more properly by strutil.cpp, so that this all works even when
only using the util library. The logic isn't changing, it's just moving
over to the other library.
This all helps to address #4388
Unfortunately, the exception thrown by fmt doesn't tell us the bad
format string itself. That would have really allowed to probably zero
right in on it. But at least we know it's occurring, and one could put a
breakpoint on pvt::log_fmt_error to catch it in the act and see where
it's being called, revealing the bad line.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
- ColorProcCacheKey: Update definition, signature, and usage to support
caching NamedTransform processor handles.
(Note: **ABI-breaking**)
- ColorConfig: add NamedTransform convenience functions:
`getNumNamedTransforms`, `getNamedTransformNameByIndex`,
`getNamedTransformNames`, `getNamedTransformAliases`
- ColorConfig: add `createNamedTransform` function returning new
ColorProcessorHandle
- Add `ImageBufAlgo::ocionamedtransform` functions
- oiiotool: add `--ocionamedtransform` operator
- python: add NamedTransform convenience function bindings
- python: add `ImageBufAlgo.ocionamedtransform(...)` bindings
- Update `oiiotool` and `python` documentation with examples
## Description
OCIO-2 config authors may define a set of stand-alone "source-agnostic"
`NamedTransforms`. Unlike the `colorconvert`, `ociodisplay`, and
`ociolook` IBAs, which explicitly convert _from_ a color space to
something else, the `ocionamedtransform` IBA behaves more like the
`ociofiletransform` IBA, where the transform itself can be applied
either in the forwards or inverse direction, and does not take into
consideration the input encoding or image state.
Quoting from the OCIO [NamedTransform
documentation](https://opencolorio.readthedocs.io/en/latest/guides/authoring/authoring.html#named-transforms):
> Sometimes it is helpful to include one or more transforms in a config
that are essentially stand-alone transforms that do not have a fixed
relationship to a reference space or a process space. An example would
be a “utility curve” transform where the intent is to simply apply a
LUT1D without any conversion to a reference space. In these cases, a
named_transforms section may be added to the config with one or more
named transforms. [...] This feature may be used to emulate older
methods of color management that ignored the RGB primaries and simply
applied one-dimensional transformations.
Notably, the built-in OCIO-2 Studio config
(`ocio://studio-config-latest`) contains NamedTransforms "sRGB - Curve"
and "Rec709 - Curve", which we could implement as drop-in replacements
for the existing `linear_to_sRGB`, `sRGB_to_linear`, `linear_to_Rec709`,
and `Rec709_to_linear` functions in `color.h` (unless those functions
are deprecated...).
There are several other applications for NamedTransforms as well. They
can be extremely useful as workflow-oriented transforms implemented at
various points in facility pipelines (e.g., for applying shot-specific
conversions, grades, gamut-compression, etc.); they can be used to
provide diagnostic transforms for visualizing "exposure zones", clipped
areas, NaNs, etc; for visually "tagging" problematic clips; for applying
parts of ACES Metadata Files parsed by OCIO's forthcoming AMF parser;
for conversion to non-RGB color models... all sorts of things.
## Tests
Tests are forthcoming. I've tested everything locally, but have not yet
implemented proper tests in the testsuite.
---------
Signed-off-by: Zach Lewis <zachcanbereached@gmail.com>
Background: The ImageCache and TextureSystem had historically been
implemented as pure virtual abstract base classe defining an interface,
then a hidden concrete subclass. The flexibility this was intended to
provide was never really used; instead the imagined main client for the
flexibility, OSL, has a different mechanism for customizing the texture
system (via their RendererServices class). Meanwhile, this has made the
interfaces to IC and TS impossible to change without breaking the ABI.
So this PR does the following:
* ImageCache and TextureSytem are fullly de-virtualized, and are now
concrete classes, no hidden subclassing, no virtual methods. As such,
removing or changing existing methods will be an ABI break, but adding
new methods is not.
* Each now use a PIMPL idiom, where all the data and internal methods
are hidden from the public interface and do not affect the ABI.
* Roughly speaking, we added the PIMPL pointers and made the
ImageCacheImpl/TextureSystemImpl -- what used to be the hidden concrete
subclasses -- into the hidden PIMPL classes.
* Moved a lot of classes that used to be in a "pvt" namespace into the
main namespace. If they are opaque types not exposed in public headers
or as symbols exported from the library, it doesn't matter, so this just
removes some arbitrary clutter.
* Had to rearrange some of the recently added heapsize/footprint code,
moved some inline functions in the headers into methods of the classes,
defined in the cpp files.
This itself is a huge ABI change, so will only be merged into master, to
become part of the 3.0 release. This is not an API change, though! This
is all about the internals, and should not require any client software
to change a single line of code.
I may do further simplification or refactoring of this code in the
future, but that will all be smaller and have no further API/ABI changes
that break compatibility.
Although the structure of the class hierarchy has changed around, none
of the logic about how IC and TS actually *work* has changed, so there
should be no change in observable behavior.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Added as_bytes and as_writable_bytes that convert spans of arbitrary
type to spans of std::byte. (This mimics C++20 utilities of the same
name.)
* spancopy, spanset, spanzero: add parameter defaults for the common
case of wanting to do the operation on the whole span, also simplify the
bound-limiting logic in those functions a bit (but still performing
precisely the same task).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Because IBA::transpose changes resolution, it doesn't pass the input
image to IBAPrep(), and there was a special case there that always set
destination to float if there were no source images.
So change transpose to specifically request a format, to be the same as
the input image.
And that required a tweak to IBAPrep allowing us to specify such a thing
(there was already an option that meant "force it to float", but not the
general case of an arbitrary type).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This change adds additional FFmpeg metadata from the video and data
streams. These streams can hold the start timecode which is very useful
in OpenTimelineIO applications.
I also added `av_guess_frame_rate()` which might be more robust than
just checking `stream->avg_frame_rate`, but I can also create a separate
PR if it is better to break it up.
It doesn't look like there are any FFmpeg tests in `testsuite`?
I tested with the DPEL ALab trailer and some Netflix Open Content
movies, and was able to see the timecode metadata in the OIIO image
attributes.
---------
Signed-off-by: Darby Johnston <darbyjohnston@yahoo.com>
Added IBA::demosaic().
Currently supports 4 Bayer pattern layouts, "RGGB", "GRBG", "GBRG", "BGGR", and 2 algorithms: "linear", "MHC".
Should be expandable in the future.
---------
Signed-off-by: Anton Dukhovnikov <antond@wetafx.co.nz>
In trying to avoid doing the color conversion pointlessly for pixels
with alpha=0, I accidentally also managed to skip the multiplication by
alpha for those pixels. Oops.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Don't read or store full metadata on every MIP level for every face.
That can really add up! Ptex can only have one set of metadata, the
format itself doesn't allow it to vary from face to face.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
It was disabled to make prettier formatting for getargs, but neglected
to turn it back on for the rest of the file.
I also changed a few old `<<` stream output to `print()`.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Nominally, fix four places where "it's" was used when "its" was
appropriate. Realistically, force the EasyCLA dialog so that I can get
an individual CLA in place, my employment with ARRI having terminated on
15 July of this year.
Signed-off-by: Joseph Goldstone <joseph.goldstone@mac.com>
Change IC::create() and TS::create() to return shared_ptr instead of a
raw pointer. This cleans up a lot of lingering lifetime management issue
of these classes. The switch to 3.0 is an opportunity to make breaking
changes to these APIs.
For the sake of downstream users, we define preprocessor symbols
OIIO_IMAGECACHE_CREATE_SHARED and OIIO_TEXTURESYSTEM_CREATE_SHARED to
signal that the new APIs are present. Some very minor changes may be
needed at the call site.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Fully linking OpenCV against libOpenImageIO is burdensome for people
downstream, especially those who want a static version of
libOpenImageIO.
There were only three IBA functions that use OpenCV: to_OpenCV(),
from_OpenCV(), and capture_image(), and all three are fairly short. Move
them to a separate header, imagebufalgo_opencv.h, make them fully INLINE
(so they don't actually produce object code in libOpenImageIO, which
therefore does not need to link against OpenCV).
Applications wanting these three functions should include this header
and will be responsible themselves for ensuring that the include paths
and linkage of their application makes provisions for finding and using
OpenCV. All other applications don't need to deal with OpenCV dependency
at all.
Our own oiiotool (for --capture) and our python bindings still link
against OpenCV to support this functionality (again, still as an
optional dependency, enabled only if OpenCV is found at build time). But
downstream apps using libOpenImageIO and who do not themselves need our
OpenCV functioality now no longer need to link against OpenCV.
We have removed capture_image() from the Python bindings -- Python
scripts that need to capture live camera images can use OpenCV or any
other capture API of their choice without going through OIIO.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Microsoft's runtime library doesn't support the standard setenv()
function. _putenv_s() should be used instead.
Without fix iv do not compiling:
`E:\GH\OpenImageIO\src\iv\ivmain.cpp(119,9): error C3861: 'setenv':
identifier not found [E:\GH\OpenImageIO\build\src\iv\iv.vcxproj]`
---------
Signed-off-by: Vlad (Kuzmin) Erium <libalias@gmail.com>
This version dates from early 2021 (so still supporting back 3.5 years).
The reason for the bump is to simplify a bit on our part and avoid older
versions known to be broken or lacking certain features.
This is for master/2.6/3.0 only, will not be backported to 2.5, since we
never raise minimum dependencies in already-released branches.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
CI sanitizers in master have been failing lately. A different PR
addresses the root cause, but this patch contains some improvements I
made along the way while investigating it:
* CMake: Change the SANIIZE build variables to use set_cache, thus
allowing easier override by env variable.
* Bumped the CI sanitizer test to use the 2024 container and latest
compilers.
* Suppress UB warnings in STL function.h, where it's claiming undefined
behavior in the std::function guts called by ArgParse internals. Only
seen failing on my Mac, wasn't actually failing in the GHA CI. I'm
pretty sure it's a false positive.
* Also, only on my Mac building against OpenVDB 11, it found some UB in
OpenVDB, which I'm suppressing (I think it's also false positive).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Fix a bug in oiiotool wherein `-i:ch=...` (input only a subset of
channels when reading from a file) was not adjusting the spec's
alpha_channel and z_channel -- that is, they still contained the OLD
channel numbers of those things, which not only could be wrong due to
the reordering, but they might be out of range of the new number of
channels.
Also a related fix in the TIFF reader, to fix a possible dereference of
an empty vector. This was being triggered by that out-of-range
alpha_channel. Maybe that can't happen anymore with the above bug fix,
but it still feels like the safer way to proceed.
This was all found by the sanitizers in CI -- and only recently, despite
this code having this bug for a long time. So yay for sanitizer CI!
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Make OCIO a hard dependency, at least version 2.2.
Revise and improve the OCIO "auto-build" scripts.
Remove OCIO 1.x testsuite reference output.
No longer need FindOpenColorIO.cmake module
As part of this, I split the CI "oldest/hobbled" test into two different
tests:
* oldest -- test against the oldest supported versions of all
dependencies.
* hobbled -- test against most optional dependencies missing and
optional features disabled.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
First PR of two, following @lgritz and I discussions on memory tracking
in the OIIO::ImageCache.
- Add two template methods and their specializations for types various
to help memory tracking:
```c++
// return the total heap allocated memory held by the object and its members
template<typename T> inline size_t heapsize(const T& t);
// return the total memory footprint including the size of the structure itself
template<typename T> inline size_t footprint(const T& t);
```
- Specialized for: ParamValue, ParamValueList, ImageSpec, ImageInput,
ImageOutput, std::vector, std::shared_ptr, std::unique_ptr,
oiio::intrusive_ptr, ImageCacheImpl related objects.
- New files:
- `include/memory.h` : adds base and few types specialization of
heapsize and footprint.
- `libtexture/imagecache_memory_pvt.h` : adds specilializations for
ImageCacheImpl and related objects.
- `libtexture/imagecache_memory_print.h` : adds a helper function that
print a breakdown of the ImageCacheImpl total memory usage as well, as
well as per image format. Note: this is slow, but gives accurate memory
estimation.
Related PR from Larry :
https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/4317
Signed-off-by: Basile Fraboni <basile.fraboni@gmail.com>
The main purpose of this change is to make it possible to use OIIO for
reading raw files in rawtoaces instead of calling LibRaw directly. There
are the changes:
- add the missing hints needed to implement all combinations of
white-balancing methods and matrix methods provided by rawtoaces.
- add the DNG-specific attributes
The change adds this functionality:
- new "raw:user_black" hint to override the default black point
- new "raw:use_auto_wb" hint to force LibRaw to white balance by
averaging over the whole image.
- new "raw:grey_box" hint to make LibRaw to white balance by averaging
over the given rectange.
- new "raw:dng:XXX" attributes added to the output ImageBuf if the input
image is a DNG file. The attributes consist of 2 sets of [calibration
illuminant; calibration matrix, XYZ to camera RGB matrix]. Note, the
current DNG standard supports up to 3 calibration illuminants, but both
LibRaw and rawtoaces only use 2 currently.
I have manually tested all permutations of white-balancing modes and
matrix methods which are currently supported by raw to aces. The images
match up to a rounding error.
The current unit tests pass, but they only seem to use the default
conversion settings. We may want to extend those. I'm not clear on how
to do that, there are multiple reference images for different versions
of LibRaw, not sure if I will have to re-generate all of them. I intend
to make more changes to the raw plugin soon, may come back to updating
tests during/after that.
There are currently no tests using DNG files, so the new DNG-specific
attributes are not covered. The code relying on those in rawtoaces works
fine.
I have also updated the documentation to add the new hints, however, I
haven't been able to build the documentation.
Signed-off-by: Anton Dukhovnikov <antond@wetafx.co.nz>
JPEG output configuration hint "jpeg:iptc" (default: 1), if set to 0,
will suppress IPTC block output to the file.
In the process, we changed the return type of utility function
encode_iptc_iim() to return true if anything was successfully encoded,
false otherwise.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
It's not just in oiiotool. This seems clearer and adheres to the env
variable naming convention we chose.
Reminder: This controls whether command line history gets written to
output image metadata by default by oiiotool and maketx. We historically
did it, but recently stopped because of security concerns.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Additional stack manipulation commands:
* `--popbottom` discards the bottom-of-stack image
* `--stackreverse` reverses the order of the whole stack
* `--stackclear` fully empties the stack
* `--stackextract <index>` moves the indexed item from the stack (index
0 means the top) to the top.
Make `--for` work correctly in both directions:
* Correct behavior if `--for` has a negative step value.
* If the end value is less than the begin value and no step is supplied,
assume -1 (analogous to how we usually assueme step=1 under ordinary
circumstances).
* Error if step is 0 (presume it will make an infinite loop).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Fixes a simple copy/paste error in a copy constructor where the y
coordinate gets initialised twice instead of y and z.
Signed-off-by: Anton Dukhovnikov <antond@wetafx.co.nz>
* Get rid of some obsolete cmake code.
* Movement (but no change) to some parts of CMakeLists.txt, primarily to
make it closer to the corresponding file in OSL to make it easy for me
to diff them and port innovations back and forth between them.
* Some typo/etc fixes
* Remove unused OIIO_UNUSED_OK macro that's been deprecated since 2.0,
and OIIO_CONSTEXPR and OIIO_CONSTEXPR14, neither of which have been
needed for years.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Even that is quite old -- released in 2012! But it's the first version
that guaranteed thread safety for error reporting. So we're finally
making that our minimum and taking out the locking we did on our side.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Raised from 1.5.13 (maybe? we never carefully checked) to 1.6.0 minimum,
which was released in 2013 (so absolutely nobody should be bothered by
this).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This new version dates to 2020. Previously, we had supported
(purportedly) WebP 0.6.1, but we didn't test that configuration. By
bumping to 1.1 minimum, we can use WebP's own exported cmake config,
rather than maintaining a FindWebP.cmake mmodule on our side.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Improvements to expression evaluation of oiiotool command line arguments
surrounded by `{braces}`.
* Just like `TOP` refers to the top image on the stack, now `BOTTOM`
refers to the bottom image.
* Referring to images with `IMG[id]` previously worked if `id` was a
literal integer or an image label or an image filepath. But now it can
be any expression that evaluates to any of those things. So, for
example, you could say `{IMG[i].filename}` to get the filename of the
i-th image down the stack (where `i` is a variable), or
`{IMG[NIMAGES-2].width}` to get the x resolution of the 2nd-from-bottom
image on the stack.
* The `--label` and `--set` commands check that the name you use for an
image label or user variable follow the "C identifier" lexical rules:
must be alphanumeric + underscore, but not start with number. It was
previously possibly to do completely chaotic things like `--set 1 2`.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* The old versions (deprecated since 2.0) of IBA::compare() and
computePixelStats() that took a reference to the CompareResults or
PixelStats structure now have deprecation warnings. Use the versions
that return the structure instead.
* The deprecated (often since as far back as 2.0) versions of functions
that took raw pointers to color values now have deprecations warnings.
Use the versions that take `span<>` or `cspan<>` instead. These include
versions of: isConstantColor, isConstantChannel, isMonochrome,
isConstantChannel, colorconvert, fill, checker, add, sub, absdiff, mul,
div, mad, pow, channel_sum, channels, clamp, color_count,
color_range_check, render_text.
* The `histogram()` function that takes a reference to a result vector
(deprecated since 2.0 and previously warned) has been removed. Use the
version that returns the vector instead. The `histogram_draw()`
(deprecated since 2.0 and previously warned) has been removed and has no
replacement since it was always silly.
* The versions of ociodisplay that lacked an `inverse` parameter, which
were marked as deprecated since 2.5, now have deprecation warnings. Use
the version that takes an `inverse` bool.
* The OpenCV-related functions that take old-style IplImage pointers
(deprecated since 2.0) have been removed. Use the modern ones that use
cv::Mat.
* The pre-KWArgs versions of resize, warp, and fit now have deprecation
warnings. Use the versions that take KWArgs instead.
* IBA::type_merge, deprecated since 2.3, now has a deprecation warning.
Instead, use TypeDesc::basetype_merge().
All of the IBA functions with deprecation warnings are additionally
implemented inline, so that they can be easily removed later if
necessary without an ABI break at that time.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Reduces the memory footprint of the `LevelInfo` struct. Three changes
are here, in order of most to least impactful:
- Makes one `ImageSpec` optional, most of the time `spec` and
`nativespec` are the same, it appears they are only different when
autotile is enabled, this saves about 160 bytes.
- Uses `std::unique_ptr<float[]>` (8 bytes) instead of
`std::vector<float>` (24 bytes).
- Reorders members to remove excess padding.
We noticed this memory usage on scenes with per-face textures with
multiple mips per face. For a scene with 12.5 million mips, this saves
about 2GB.
Note, it would probably also be possible to remove the
`polecolorcomputed` member, instead relying on the `polecolor` pointer.
But since this bool occupies existing padding in the struct it wouldn't
save any memory currently.
---------
Signed-off-by: Curtis Black <curtis.w.black@gmail.com>
* Add deprecation warnings to the varieties of ImageBuf constructor and
`reset()` that don't take subimage and miplevel parameters. The
equivalent is to just pass `0` for both of those parameters. The warned
constructors have been marked as deprecated since OIIO 2.2.
* The misspelled `ImageBuf::make_writeable()` has been given deprecation
warnings. Since OIIO 2.2, the correct spelling is `make_writable`.
* The `ImageBuf::error()` method that uses printf-style formatting now
has deprecation warnings. Use `ImageBuf::errorfmt()` instead.
* The `ImageBuf::interppixel_NDC_full()` method, which has been marked
as deprecated since OIIO 1.5, now has deprecation warnings. Use
`interppixel_NDC()` instead.
Undeprecate the name+spec varieties of constructors. They were
documented as deprecated but warnings were never added. In doing this
set of changes, I realized that they were used for something useful
after all -- it makes it easy to remember the name of a new buffer
you're making not from a file, so you can later `write()` it without
needing to know the filename at that time. I also added a new
`set_name()` method to do it directly for times when you don't want to
rely on the constructor or want to change the name.
By the way, I'm being careful also to convert all the warned or marked
deprecated functions to inline, if at all possible, so that in the
future removing them will not be an ABI break.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
It's silly to copy the full metadata from the source (which could
include big things like ICC profiles) to EVERY level of a MIPmap.
This mostly applies to TIFF textures, since OpenEXR MIPmaps don't store
attributes separately for each MIP level anyway.
This can be disabled (that is, make it copy full metadata to all levels
of the MIPmap) by setting a new make_texture configuration hint,
"maketx:mipmap_metadata" to nonzero.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
With 3.0 coming soon, it's time to break compatibility by switching
OIIO::span::size() to return an unsigned value, matching C++20
std::span.
The history is that when we first added OIIO::span, at that time, the
C++ draft proposal for std::span included a signed size() result, but by
the time it got finalized into C++20, it was changed to unsigned to
match all the other std containers, but it was too late for us to switch
without breaking API compatibility rather seriously.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Remove long deprecated and empty header missing_math.h
* paramlist.h: Deprecate the old form of ParamValue constructors.
People should be using the versions with strongly typed params, not
raw bools.
* parallel.h: Remove deprecated parallel_for variants.
* strutil.h: Add deprecation warnings to old-style format.
* sysutil.h: Deprecate Sysutil::physical_concurrency.
Also, I noticed that all the external declarations in this header were
incorrectly OIIO_API but should have been OIIO_UTIL_API to reflect that
they are in the OpenImageIO_Util library.
* thread.h: mark deprecated yield() and this_thread_is_in_pool().
* tiffutils.h: removed deprecated redundant functions.
* ustring.h: Remove deprecated ustringHash.
* type_traits.h: Remove deprecated OIIO::void_t, use std::void_t instead.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Old version of TextureOptions we haven't used since OIIO 1.x.
* Texture methods that were based on the VaryingRef helper. These are
actually remnants of early OSL needs when it was more Reyes-like, BEFORE
we used LLVM!
* Get rid of the varyingref.h header, no longer needed by anything.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* array_view.h: Remove this long-deprecated header.
* benchmark.h: Attach DEPRECATED warning to external use of
benchmark.h's time_trial().
* bit.h: Remove bit_cast that's had deprecation warnings for several
releases, and add deprecation warnings to the old rotl32 and rotl64
* color.h: add deprecation warnings and make the deprecated things
inline so they can be removed later without an ABI break.
* errorhandler.h: remove deprecated old printf style methods
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Initial Exif and XML metadata support for JPEG XL.
- EXIF - works and enabled by default
- XML - works and enabled by default
- IPTC - that part need some help. JPEG XL support XML based IPTC, but Ido not get what IPTC create an OIIO
- JBIG - not sure
new options:
- `jpegxl:use_boxes` - export EXIF, XMP, IPTC, JBIG
- `jpegxl:compress_boxes` - compress meta boxes (disabled for this moment by default)
- jpegxl:exif_box = 1 (enabled)
- jpegxl:xmp_box = 1 (enabled)
- jpegxl:jumb_box = 0 (disabled)
- jpegxl:iptc_box = 0 (disabled)
No tests yet, but if we have I can add.
---------
Signed-off-by: Vlad (Kuzmin) Erium <libalias@gmail.com>
The old name, OIIO_LIBRARY_PATH, was inconsistently named in two ways:
First, it was not clear that it was specific to searching for plugins,
and since it is related to the to the global attribute
"plugin_searchpath", I thought making it mention plugins specifically
would be more clear.
Second, I noticed that all of the publicly documented environment
variables that are "for users" started with OPENIMAGEIO_, except for
this one. There are a bunch of environment variables we check that begin
with OIIO_, but all but this one are intended for developers when
testing or debugging.
So I'm using a new name that is consistent with both principles.
Henceforth, the naming rule about environment variables is:
* `OPENIMAGEIO_FOO` for public interfaces, are fully documented and
intended for users/sites to use to customize behavior.
* `OIIO_FOO` for developers to use when testing and debugging, are
undocumented (except in the code itself), and are not considered part of
the public API/behavior of OpenImageIO -- their meaning may change, they
may disappear without warning, or they may only be functional for debug
builds.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
I should not have hidden virtual functions behind
OIIO_DISABLE_DEPRECATED checks. Never make the vtbl possibly vary for
clients of the library from the way we built it!
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Enable the ImageCache for the `iv` tool. It more or less expects it to
be present and certain features, like loading the Image's mip-levels,
requires the ImageCache to be used.
This was breaking CI for us. Hard break for icc/icx tests, since those
REQUIRED a yum install of the intel compilers. Softer undetected break
for all the ASWF <= 2022 containers based on CentOS 7, which were not
failing outright but were failing to install certain optional packages.
The solution (for now) is to configure yum to exclude the missing repo.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
For the last year or so, I have been using git-cliff to generate the
first draft of release notes. It gets me 90% of the way there for the
monthly patch releases, perhaps more like 30% (but a very good starting
point) for the big annual major releases.
In this PR, I document my process and check in the toml file that
provides the git cliff configuration that gets mostly what we need.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Inspired by a question by Vlad Erium, I have added a simpler way for C++
users of OIIO to construct IBA-like functions for simple unary and
binary operations on ImageBufs where each pixel is independent and based
only on the corresponding pixel of the input(s).
The user only needs to supply the contents of the inner loop, i.e. just
doing one pixel's work, and only needs to work for float values. All
format conversion, sizing and allocation of the destination buffer,
looping over pixels, and multithreading is automatic.
If the actual buffers in question are not float-based, conversions will
happen automatically, at about a 2x slowdown compared to everything
being in float all along, which seems reasonable for the extreme
simplicity, especially for use cases where the buffers are fairly likely
to be float anyway.
What you pass is a function or lambda that takes spans for the output
and input pixel values. Here's an example that adds two images channel
by channel, producing a sum image:
// Assume ImageBuf A, B are the inputs, ImageBuf R is the output
R = ImageBufAlgo::perpixel_op(A, B,
[](span<float> r, cspan<float> a, cspan<float> b) {
for (size_t c = 0, nc = size_t(r.size()); c < nc; ++c)
r[c] = a[c] + b[c];
return true;
});
This is exactly equivalent to calling
R = ImageBufAlgo::add(A, B);
and for float IB's, it's just as fast.
To make the not-float case fast and not require the DISPATCH macro
magic, I needed to change the ImageBuf::Iterator just a bit to add
store() and load() method templates to the iterators, and add a field
that holds the buffer type. That might make a slight ABI tweak, so I am
thinking that I will make this for the upcoming OIIO 3.0, and not
backport to the release branch.
I think this is ready to introduce at this time, but I'm also studying
whether more varieties of this approach are needed, whether the
non-float case can be sped up even more, and whether some of the
existing IBA functions should switch to using this internally (good
candidates would be those that are almost always performed on float
buffers, but for which the heavy template expansion of the DISPATCH
approach to handling the full type zoo currently makes them very bloated
and expensive to compile, for very little real-world gain).
We should probably consider this to be experimental for a little while,
just in case the function signature for this changes as I think about it
more or add functionality.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
In master, since we have raised the minimum Imath/OpenEXR to 3.x, no
more 2.x, we no longer need generate our Imath.h and half.h from
templates, they can be turned into regular header files.
Additionally, define IMATH_HALF_NO_LOOKUP_TABLE in half.h before
including Imath/half.h, which ensures that the Imath code we include
converts float <-> half using actual inline code and not the lookup
table. This can reduce or sometimes eliminate the need to link agains
Imath only for the sake of accessing that lookup table.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Track down and fix many places internal to OIIO where we were still
using deprecated API calls.
Add deprecation warnings to some things that have been nominally
deprecated, but weren't already tagged as such.
Guard deprecated things behind OIIO_DISABLE_DEPRECATED so that
downstream projects can hide them to help them find and change any use
of the deprecated calls, and behind !defined(OIIO_INTERNAL) so they are
hidden from the rest of the internals of our code base.
Make some of the deprecated things inline, so that their final removal
later doesn't constitute a link ABI compatibility change. (We do absorb
an ABI compatibility boundary with this PR, so I've bumped the version
number appropriately.) Move all the OIIO-namespace level deprecated
items to the end of imageio.h so they're as out of the way as possible.
A lot of the things that are still present here, just guarded by the
deprecation warnings, will be removed entirely before the final OIIO 3.0
release.
Please note some particular things in imageio that have been deprecated
for a while and that we expect to disappear entirely by 3.0:
* The old style `ImageInput::read_*` functions that don't take explicit
subimage and miplevel paramters, because they aren't thread-safe due to
their implied statefulness. You are expected to use the ones that take
those parameters and are stateless.
* The variety of seek_subimage that takes an `ImageSpec&` to fill in.
* Some obsolete varieties of ImageInput and ImageOutput create() static
methods.
* The `destroy` methods that are no longer needed (plain old `delete` is
fine).
* The various error-setting functions that use the old printf-like
formatting conventions. Use `errorfmt()` and friends that use the
std::format conventions.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
TIFF export with EXIF metadata (e.g. oiio::ColorSpace) creates invalid
TIFF files in the presence of an I/O proxy ("TIFF directory is missing
required "ImageLength" field"). The reasons is that TIFFSetDirectory()
requires to read some of the previously written data, but neither
writer_readproc() nor the available proxy implementations support that.
Note that TIFF output does not yet use this new scheme for I/O proxy
that was used for many other plugins.
I think writer_readproc() should not take any shortcuts and just forward
the call to the I/O proxy.
Is there a reason why IOFile (in write mode) and IOVecOutput do not
allow to re-read written data? This looks like an artificial restriction
of the underlying resources. An alternative not modifiying these two
implementations would be to let the TIFF output use a variant of
IOVecOutput that allows re-reading and forward all data at the very end
to the passed-in proxy.
Is there some policy to add assertions in all methods that are not
supposed to be called? This would have helped here a lot to reduce the
test case.
I did not add an explicit test. I guess the proper way would be to
convert TIFF ouput to use the same I/O proxy scheme that is used by many
other plugins. Small repro case is attached.
[repro.cpp.txt](https://github.com/user-attachments/files/15886829/repro.cpp.txt)
---------
Signed-off-by: Joachim Reichel <43646584+jreichel-nvidia@users.noreply.github.com>
Like the existing Vector2i, but, you know, 3.
Along the way, fixed a bug in the way the Vector2i type is represented
as a string.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Fully remove some things that have had deprecation warnings for a long,
long time (mostly since 1.x), so we think it can't possibly break
anyone.
Add deprecation warnings to some things that have been nominally
deprecated for some time. The build warnings will serve as final notice
that they will fully disappear by OIIO 3.0 release.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This continues the adventure from #4082 where the processing of fmt
headers, both locally built and externally referenced, causes large
rebuilds to occur when making small or even zero changes between builds.
Since that previous PR is stuck I'll try again here.
This makes two sets of changes:
- Check if our generated files exist before writing to them (in the
externally referenced case)
- Use `copy_if_different` to prevent redundant copies from occuring, and
triggering timestamp updates (in the local build case)
Note that while this seems to fix the problem, the build is still not
"clean" wrt fmt processing. The following will be be seen when
triggering a build, with zero changes since the last build, within VS:
```
1>------ Build started: Project: ZERO_CHECK, Configuration: Debug x64 ------
1>1>Checking Build System
2>------ Build started: Project: fmt_internal_target, Configuration: Debug x64 ------
3>------ Build started: Project: CopyFiles, Configuration: Debug x64 ------
2>Generating ../../include/OpenImageIO/detail/fmt
2>Building Custom Rule C:/Users/jesse/source/oiio/src/include/CMakeLists.txt
2>Generating ../../include/OpenImageIO/detail/fmt/core.h, ../../include/OpenImageIO/detail/fmt/format-inl.h, ../../include/OpenImageIO/detail/fmt/format.h, ../../include/OpenImageIO/detail/fmt/ostream.h, ../../include/OpenImageIO/detail/fmt/printf.h, ../../include/OpenImageIO/detail/fmt/std.h, ../../include/OpenImageIO/detail/fmt/chrono.h
2>C:\Program Files\Microsoft Visual Studio\2022\Preview\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(254,5): warning MSB8065: Custom build for item "C:\Users\jesse\source\oiio\build\CMakeFiles\b19a1b09c2e7eaa53ca90801840dfdd0\fmt.rule" succeeded, but specified output "c:\users\jesse\source\oiio\build\include\openimageio\detail\fmt" has not been created. This may cause incremental build to work incorrectly.
2>C:\Program Files\Microsoft Visual Studio\2022\Preview\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(254,5): warning MSB8064: Custom build for item "C:\Users\jesse\source\oiio\build\CMakeFiles\b253238471e54ff57d14ae397110f40e\core.h.rule" succeeded, but specified dependency "c:\users\jesse\source\oiio\build\include\openimageio\detail\fmt" does not exist. This may cause incremental build to work incorrectly.
2>Done building project "fmt_internal_target.vcxproj".
4>------ Build started: Project: OpenImageIO_Util, Configuration: Debug x64 ------
```
Whatever it is, the VS generator doesn't much like the way we're
handling the processing but I don't know enough cmake-fu to ascertain
why that might be.
Signed-off-by: Jesse Yurkovich <jesse.y@gmail.com>
TL;DR: when turning unassociated alpha into associated for
gamma-corrected PNG images, we got the exponent wrong for the
linearization step, and when doing the same for sRGB images, we didn't
linearize at all.
WARNING: will change appearance of PNG files with partial alpha.
Details:
Ugh, two separate problems related to how we associate alpha (i.e.
premultiply the colors) for PNG pixels with partial alpha. The correct
thing to do is linearize the unassociated pixel value first, then
associate, then go back to the nonlinear space.
First problem: PNGs have three possible transfer functions: gamma
correction (with a particular gamma), no gamma correction / linear, and
explicitly sRGB. Guess what? We were neglecting the case of pngs tagged
as sRGB and not doing the linearize/delinearize round trip for those
images.
But if that's not enough, also for the gamma case, we were, ugh,
swapping the gamma and 1/gamma, resulting in those partial alpha pixels
ending up a whole lot darker than they should have been.
None of this affected most ordinary PNGs with no alpha channel, or where
alpha was 1.0. It only affected "edge" or "partially transparent" pixels
with 0 < alpha < 1. But it was definitely wrong before, for which I
apologize and hope you'll understand why those pixels are going to
change now (hopefully, always always for the better).
While I was at it, I also made the color space handling a little more
robust -- instead of just a straight string compare for color space
names, use the ColorConfig to check `equivalent`, which should make us a
lot more robust against aliases and whatnot.
Fixes#4314Closes#4054
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
In several places, we still used OIIO versions of isnan/isinf/isfinite
and is_same. All of there for some time have been aliased to the std
versions (having our own was mostly a pre-C++11 artifact). Also,
there are a couple obsolete function names (pow2roundup and
pow2rounddown were replaced with ceil2 and floor2 to match later
C++ standard names).
Change those OIIO::foo to std::foo or use the modern names
throughout the codebase, and hide the old declarations themselves
behind some guards that:
1. Will eliminate them completely as soon as the version is higher than
2.6.x.
2. Makes them invisible now internally to OIIO (so we aren't
accidentally using them ourselves), and to external projects (like OSL)
that define OIIO_DISABLE_DEPRECATED to hide deprecated things.
3. Add deprecation warnings if they are not hidden by the logic of (2)
above.
Before 3.0 release is final, we will truly remove them entirely, but
these staged warnings will help people with the transition.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Fully remove some things that have had deprecation warnings for a long,
long time (mostly since 1.x), so we think it can't possibly break
anyone.
Add deprecation warnings to some things that have been nominally
deprecated for some time. The build warnings will serve as final notice
that they will fully disappear by OIIO 3.0 release.
This is part of a series of small steps to fully deprecate a whole lot
of cruft between now and the 3.0 release.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Co-authored-by: Thiago Ize <ThiagoIze@users.noreply.github.com>
The primary issue seemed to be that the ChannelInfo width and height
were not filled in for some of the layers, notably the merged image data
layer in most(all?) cases.
A second issue was that there was incorrect nesting of the switch
statement in `read_channel_row`. The last 3 case statements were all
nested under the `if (!bigendian()) {` conditional from the 1st case.
Fixes#4298
## Tests
Use `iconvert` on all PSD files contained in the attached .zip.
Converting to both .exr and .png is a good test to ensure the data is
flowing out to those formats correctly.
[psd.zip](https://github.com/user-attachments/files/15905814/psd.zip)
---------
Signed-off-by: Jesse Yurkovich <jesse.y@gmail.com>
This is the continuation of [4305
](https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/4305)(Add
extra step to Makefile config target to force CMake to rescan).
The error occurs when configuring with `make
OpenImageIO_BUILD_MISSING_DEPS=all` option which fails the build as
`libuitil` does not have `./dep/dist/include` on the path.
The original solution was to rerun cmake and let it rescan paths. This
step was only required to run once - after the initial CMake
configuration step. This kind off worked but I'm still unsure why. So
instead of that we can just add the missing location to CMakeLists.
Signed-off-by: kuba <kuba456@gmail.com>
Build-time check for Cuda toolkit as optional dependency. This can be
explicitly disabled with OIIO_USE_CUDA=OFF (though currently, it
defaults to off, you have to turn it on at build time because this is
all still experimental). When this is enabled at build time, C++
preprocessor symbol OIIO_USE_CUDA will be defined within the OIIO
codebase.
OIIO global `attribute("gpu:device", "CUDA")` can be used to enable CUDA
functionality, which at this point doesn't do anything other than
initialize the CUDA device and find out information about it. Several
global `getattribute("cuda:*")` queries let you find out some
information about any device found.
I'm trying to keep a lot of the approaches fairly generic, so that the
interface won't need to change much to allow OpenCL or Metal or whatever
else comes along, though for now I'm trying to flesh out the
implementation mostly with CUDA.
oiiotool adds a new `--cuda` option, reserved for enabling CUDA
functionality. Currently the only way this is used is to print
information about the CUDA device in the `oiiotool --help` messages. An
example of that is:
```
$ oiiotool --cuda --help
oiiotool -- simple image processing operations
...
OIIO 2.6.2.0spi | Linux/x86_64
Build compiler: gcc 9.3 | C++17/201703
HW features enabled at build: sse2,sse3,ssse3,sse41,sse42
CUDA 11.6.0 support enabled at build time
...
Running on 32 cores 187.3GB sse2,sse3,ssse3,sse41,sse42,avx,avx2,avx512f,
avx512dq,avx512cd,avx512bw,avx512vl,fma,f16c,popcnt,rdrand
Compute hardware available: CUDA on Quadro RTX 6000, driver 12020,
runtime 11060, compat 705, memory 23.6 GB
```
Note that at this point, we aren't actually *using* CUDA for anything.
This is just setting up some basics for future expansion. It's all
hidden in the pvt namespace for internal use only, and any of the
functions or nomenclature may change as we continue to add
functionality. I want to iterate on it a bit before it's in any way
exposed in public headers or interfaces.
I'm a big fan of introducing even big strategic changes in the form of a
series of MVP easy to review discrete steps. This step's only goals are:
(a) allowing the build system to find and link against the CUDA Toolkit;
(b) allowing runtime initialization and querying of a few basic facts
about the device found; (c) start noodling around just a bit with
function and attribute interfaces to gain a little experience that will
guide what we eventually want.
LONG TERM goals of this initiative include:
* A full TextureSystem work-alike that can be used from CUDA (and
possibly other compute device APIs) based renderers and with OSL's OptiX
back and, having an analogous (ideally identical) interface and as close
as possible to feature parity with CPU TextureSystem.
* Augment ImageBuf with the ability to store its images in GPU (or
unified) memory and for IBA functions to be able to operate on those
GPU-side buffers with GPU compute kernels, somewhat analogously to how
libraries like PyTorch provides optional GPU support for tensor
operations. (The hope is that this improves impage processing
performance, though it's possible that it'll turn out that typical OIIO
image processing workflows are so I/O dominated that there's not a lot
of potential upside. I think we'll have to try before we really find out
if it's going to be worth it.)
* Allow image reading/writing plugins (or other parts of OIIO
functionality) to leverage a GPU if available, and if there is something
productive they can do with it.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Simplifies the clearing of the tile cache so it no longer allocates a
vector of tile IDs prior to erasing them all.
---------
Signed-off-by: Curtis Black <curtis.w.black@gmail.com>
* List external dependencies found, as well as those not found.
* Sort and deduplicate the lists before printing.
* Print versions of everything
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Split the CI "oldest/hobbled" test into two different tests:
* oldest -- test against the oldest supported versions of all
dependencies.
* hobbled -- test against most optional dependencies missing and
optional feature disabled (including no SSE, no OCIO, etc.).
And add an additional test:
* local-builds -- a test that builds and uses every dependency that we
know how to build locally/automatically.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
freetype 2.10.0 dates from 2019, so we're still supporting 5 years back.
Previously, we enforced no minimum, so we're not sure how far back it
would really work, but the earliest version we tested in CI was 2.8,
which dated from 2018.
Add a build_Freetype.bash for easy setup for CI (and other users).
Add build_freetype.cmake to allow automatic local builds if it can't
be found at build time.
CI many cases need to build freetype, because the old CentOS 7 era
freetype is now too old.
Remove some reference output that was only needed for older freetype.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We're seeing nvcc compilation warnings like the following:
```
ustring.h(869): warning #20012-D: __device__ annotation is ignored on a function("operator=") that is explicitly defaulted on its first declaration
__attribute__((host)) __attribute__((device))
```
According to nvidia's documentation
(https://docs.nvidia.com/cuda/archive/10.1/cuda-c-programming-guide/index.html#compiler-generated-functions),
functions that are explicitly-defaulted should not have execution space
specifiers like \_\_device\_\_ or \_\_host\_\_. Instead the compiler
will determine the annotations from the functions that invoke it.
This patch goes through the ustring and typedesc headers and removes
OIIO_HOSTDEVICE from all defaulted functions.
Signed-off-by: Chris Hellmuth <chellmuth@gmail.com>
Latest versions of cmake are warning about FindPython being a deprecated
module, they recommend FindPython2 or FindPython3 instead. In master,
we're already python3 minimum, so go ahead and switch this to the modern
method.
This also seems to fully fix our recently broken CI on one of the Windows tests!
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
High level TLDR:
* If checked_find_package doesn't find a dependency (or it is not an
acceptable version), it looks for `src/cmake/build_<pkg>.cmake` and
if that exists, includes it. It can do anything, but is expected to
somehow provide the dependency so that a second find_package will
find it and then proceed as if it were a system install.
* I've implemented these scripts so far for Imath, OpenEXR,
OpenColorIO, fmt, and robin-map, that download, build, install the
package in OIIO's build area. More to come later?
* This is really simple with a new build_dependency_with_cmake macro,
much simpler than ExternalProject_Add, as I've seen it used
elsewhere.
* Just look at any of the new build_blah.cmake files to see how simple
it is for each new dependency we set up this way.
* By default, pre-installed packages it can find always take precedent
over building locally. So if you have all the dependencies already
installed, none of this should behave any differntly than before.
But there are variables that let you override on a package by package
basis, giving the option of never building locally, building locally
if the package is missing, or forcing a local build to always happen.
----
Various details:
A bunch of cmake things (including checked_find_package) have been
moved into a new file, dependency_utils.cmake.
build_Imath.cmake, build_OpenColorIO.cmake, build_OpenEXR.cmake,
build_Robinmap.cmake, and build_fmt.cmake implement local builds of
those packages. They're very simple, and lean heavily on common
infrastructure of build_dependency_with_cmake, which also can be found
in dependency_utils.cmake.
Robinmap and fmt are extra simple because we use them as header-only
libraries.
For Imath and OpenEXR, I build them as static libraries, so they will
be incorporated into libOpenImageIO (and/or _Util) libraries to be
totally internal to them, there should be no symbols exposed outside
our libraries. This should mean that the resulting libOpenImageIO
should be perfectly safe to link in an application that also links
against OpenEXR, Imath, or OpenColorIO, even different versions
thereof, without any interference. Note that none of those packages
are used in our public APIs, only internally.
OpenColorIO was a little trickier. It builds its own dependencies as
static libraries that are internalized, but OCIO itself is a dynamic
library. So we end up having to make it part of our install, but I use
OCIO's build system to make a custom symbol namespace and a custom
library name, so it still should not interfere with any other OCIO
linked into the application.
We'll see how it goes for furture dependencies we want to add. The
header only, static libraries incorporated and hidden, and dynamic
library but renamed and with custom namespace, are all techniques that
work well. I'm not sure I'd advocate doing local builds of any
dependency that we can't incorporate in one of these ways, but I guess
we'll cross that bridge when we get to it.
New option_utils.cmake has two handy new utilities: set_cache() is
much like the built-in set() when making a cache variable, and
set_option() is much like build-in option(). The biggest difference is
that both allow an environment variable of the same name, if it
exists, to supply the default value. This is something that cmake does
with many of its own controls, like CMAKE_BUILD_TYPE, but does not
make any provision for built-in set() or option() let users do it.
checked_find_package() has moved to dependency_utils.cmake, and has
been enhanced to take several new options, and also so that if the
enclosed find_package() fails and there is a
src/cmake/build_PKG.cmake, it will run it to build the dependency
itself in the build area. If that build_PKG sets a variable called
PKG_REFIND, it will try find_package again to find the one just built.
build_dependency_with_cmake() is given a git repo and tag, and
basically clones the repo, checks out the tag, configures, builds, and
installs it (all in our own build area).
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This cuts down CI time, as before we were building both release and
debug versions of the packages.
The dependency building step of our CI typically took around 6.5-7
minutes previously, and with this patch, now typically is about 5
minutes and sometimes slightly less.
Thanks so much to @cedrik-fuoco who pointed me at this feature of vcpkg
which had eluded me for a long time!
Signed-off-by: Larry Gritz <lg@larrygritz.com>
A few minor fixes for things I noticed as I started to actually use
these new utilities in other branches I'm working on.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
While making a CI to build python wheels for OIIO on windows I spotted a
few issues in the instructions, these edits made it work for me.
Small adjustment to expected paths for the manual instructions.
Signed-off-by: Mel Massadian <melmassadian@gmail.com>
This file imageio_pvt.h.in was used as the input template for CMake to
create imageio_pvt.h, doing a couple build variable substitutions along
the way. But this was inconvenient (because it doesn't look like a real
.h file to editors or clang-format) and it turns out that there were
only 2 lines that needed the substitution.
So in this patch, we split the file, making 99% of it just plain old
imageio_pvt.h that needs no rewriting, and the couple lines that need
variable substitions becomes buildopts.h.in and is included only from
the one source file that actually cares, imageio.cpp.
Other than cleaving those two definitions off to another header and some
file renaming, I haven't changed any of the other content of the
headers, so there should be no change in code the compiler sees or other
behavior.
The reformatting you see is because now that it's a legit .h file,
clang-format wants its way with it.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
I misunderstood all along -- the add_compile_definitions,
add_compile_options, etc., aren't truly global, they are restricted to
targets in the same directory or below. They aren't really prone to
polluting "surrounding" projects when OIIO is used as a subproject.
So for simplicity, reverting that part and using the usual built-ins.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
New set_utils.cmake has some handy new utilities: set_cache() is much
like the built-in set() when making a cache variable, and set_option()
is much like build-in option(). The biggest difference is that both
allow an environment variable of the same name, if it exists, to supply
the default value. This is something that cmake does with many of its
own controls, like CMAKE_BUILD_TYPE, but does not make any provision for
built-in set() or option() let users do it.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
In PR #4268, I accidentally changed the clause addressing what is
seemingly a bug with older MSVS. I'm just putting things back the way
they were. It's not a new change, just restoring this line closer to the
way it was about 3 commits back.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Switch runner type for bleeding edge, use gcc14 and python 3.12
* Various minor gcc-14 + C++20 + Python 3.12 fixes
Notable: gcc14 warns about std::atomic_load for share_ptr as deprecated,
since C++20 provides a true atomic shared pointer.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We've been plagued on and off for a long time with spurious wrong
results in our SIMD versions of exp and log. It was really squirrelly,
came and went with changes to seemingly unrelated code, and had long
periods of time with no symptoms or ability to reproduce it. It just
started popping up again in CI in the dev-2.5 branch, consistently.
Today I think I finally tracked it down for real. I believe the problem
was actually in the SIMD bitcast_to_int and bitcast_to_float functions,
for the variety that was 8-wide when running on 4-wide hardware, and the
16-wide one when running on 8-wide (or less) hardware.
Background: When real 16-wide HW is not available, we define the 16-wide
vectors as an array of two 8-wide vectors. (Same for 8 vs 4, I'll spare
you explaining it all twice.) So most of the 16 wide functions boil down
to:
vfloat16 op (vfloat16 arg, ...)
{
#if 16 wide HW is available
return real_16_wide_intrinsic(arg);
#else /* recursively define in terms of 8-wide halves */
return vfloat16(op(arg.lo()), op(arg.hi()));
#endif
}
But for the bitcast operators, we did something different, a shade too
clever: we just did a pointer cast, dereference and return:
vint16 bitcast_to_int (const vfloat16& x)
{
#if OIIO_SIMD_AVX >= 512
return _mm512_castps_si512 (x.simd());
#else
return *(vint16 *)&x;
#endif
}
I believe that the problem is that this cast is undefined behavior for
the case where the vector x was broken into two halves, and by being in
the middle of a long code sequence of SIMD ops, the two halves happened
to be in registers at that moment. Basically, we are taking the address,
and it just doesn't seem to always understand that it needs to
materialize the whole thing in memory for the pointer cast to work
properly.
I removed the pointer cast and replaced with
return vfloat16(bitcast_to_float(x.lo()), bitcast_to_float(x.hi()));
basically falling back on the usual recursive definition of 16-wide
operations to being in terms of 8-wide operations when 16-wide HW is not
available. This totally clears up the problem since we're no longer
potentially relying on needing the address of something that has no
address at that moment.
A couple other spots (andnot) also had dubious casts that I fixed in the
same way.
N.B.: The 16-wide cast from bool to int had to do something slightly
different, because it's represented differently and isn't a true
bitcast.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This should address the following 2 issues when using MSVC:
- MSVC does not set defines like `__SSE2__` or `__SSE4_2__` etc. we must
do that ourselves
- MSVC uses the `/arch` flag differently than other compilers and it
should be set only once
The ramifications of not having those defines set basically means that
OIIO won't notice any SIMD is possible.
For MSVC the /arch flag only controls which instruction set the compiler
may use when generating code "on its own". For example, the x64 compiler
by default will use SSE2 whenever possible when its generating code but
when using `/arch:AVX` it will produce AVX code as it sees fit. NOTE:
case matters here too! "avx" does not work but "AVX" does.
However, if the source code uses intrinsics, MSVC will happily use those
instead. In fact it's possible to use `avx512` intrinsics without
specifying any /arch flag at all and it will happily generate
corresponding avx512 code. This is much different than say GCC where, in
order to use intrinsics at all, you must specify the corresponding flags
allowing you to do so for the arch in question.
Fixes#4265
## Tests
Existing simd_test.cpp should be sufficient I think. In addition to
those, I've verified the below locally:
Currently: when using `-DUSE_SIMD="sse4.2,avx2"` to build, we get the
wrong `build:simd` flags:
`sse2`
After: using the same flags as above:
`sse2,sse3,ssse3,sse41,sse42,avx,avx2`
---------
Signed-off-by: Jesse Yurkovich <jesse.y@gmail.com>
A small piece of the ongoing switch to std::format/print style text
output. This part is for error and debugging output when reading/writing
exr images.
I also switched conditional compilation to the DBG macro idiom that I've
used elsewhere, where it takes a combination of DEBUG compile and
setting of environment variable OIIO_DEBUG_OPENEXR to trigger the printf
style debugging (the first use case of this was in our color management
code, which I also modify slightly here to conform).
Also opportunistically simplify gcd declaration now that we're C++17
minimum.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Add proxy support for EXR multipart output by taking the setup code from
the other open() overload and calling the constructor accepting a stream
instead of a filename.
---------
Signed-off-by: Joachim Reichel <43646584+jreichel-nvidia@users.noreply.github.com>
Recent libtiff versions export cmake configs that, if you built libtiff
as a static library, will reference targets it set up when it built, but
which are nonstandard names and won't exist when you build and consume
their config exports. Ugh!
Some can be swept under the rug by turning off some libtiff optional
dependencies, but I can't seem to get rid of a target CMath::CMath that
static libtiff target wants. It just points to the math library. So I'm
just setting that up so it will be found when we do the
find_package(tiff). Oh, the hoops we jump through for badly constructed
packges.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Some weird combination of old libheif library gives a different
oiio:OriginalOrientation result. Need this ref because that's how it
behaves in that version (which I have to build against at work), but
it's not worth tracking down a "fix" because it's so old and works fine
on newer versions.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This is just a code move. There no changes to the code logic, though in
the process, I do change old "local" uses of eval_as_bool to the new
Strutil one.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Background: Some formats that support multi-image files know as soon as
they are opened and read the header what the total subimage count is.
Others (I'm looking at you, TIFF) can't know without reading each
subimage in succession until you hit the end of the file. Format readers
who can easily figure it out open first opening are supposed to set
attribute "oiio:subimages" to the number of subimages, or 0 / unset if
it can't be cheaply or immediately determined.
ImageBuf has a `nsubimages()` method that you'd think would give you the
right count, and it did when all IBs were read through the ImageCache
(which fully inventories the subimages of any file it reads). But when
OIIO 2.5 changed IB behavior to only be backed by IC when explicitly
requested, we effectively lost this functionality most of the time, and
instead returned 0 ("don't know yet"). This was confusing. And also
silly, because it reported 0 even for files whose formats don't permit
multiple images (so you know the count is 1).
This patch does the following:
* Changes the behavior and documentation of `ImageBuf::nsubimages()` to
the following return values:
- 1, for formats that are always a single subimage. (Or for ImageBufs
that are not direct reads of a file.)
- n>=1 for files that permit multiple subimages and the count is known.
- 0 for files that permit multiple subimages, but the count is not known
because it's one of the formats where it can't easily be determined upon
opening the file.
* Add a new ImageInput::supports("multiimage") that says if the input
file format permits multiple images, and ensure that it returns true for
all formats where this is the case.
* Ensure that formats permitting multiimage and can easily know the
count up front set the "oiio:subimages" attribute (not all did).
* Document in the built-in format section of the docs the formats that
are expected to set this attribute.
Fixes#4225
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This is a continuation of PR 4246, had one more spot where we needed to
disable warnings for unity builds involving clang18 and deprecated
unicode conversion functions.
There's some changes included to ci.yml and install_homebrew_deps.bash
that are part of work I did to make it easier for me to force it to use
clang-18 for certain tests while working on this.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Remove some long-ago deprecated printf-like error functions. Mark as
deprecated some other ones that we will remove when it gets closer to
3.0 release. I think these are the very last uses internally of the old
style that we want to remove.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The nullptr at the end of default_font_name was left over from a time
that we iterated by going through array elements until we hit the
nullptr that signalled we had passed the last valid element. At some
point we switched to a range-for, which iterates over every element of
the array without needing an "end sentinel", and we should have
eliminated the nullptr but failed to do so.
Fixes#4248
Signed-off-by: Larry Gritz <lg@larrygritz.com>
clang/llvm 18 (and probably other compilers soon) has made noisy
warnings about the C++17-deprecated unicode conversions. Actually
replacing them enirely with something else is rather hairy, we will
tackle that later.
In the mean time, this patch seems to be the minimal change to disable
the warnings that should be totally easy to backport.
These seem to be the only changes necessary to make a clean build with
clang 18.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Only use the OIIO_LIBRARY_PATH environment variable by default.
Searching through such system paths has some risk loading plugins
installed for another purpose or that may be incompatible.
If the previous behavior is desired, applications can add these
environment variables to the plugin search path manually.
Signed-off-by: Brecht Van Lommel <brecht@blender.org>
The recent switch from boost thread_specific_ptr left a bug: if an
ImageInput (say) is deleted without the caller clearing its error state,
and then later another ImageInput is allocated and happens to get the
same address as the first one, it can "inherit" the unretrieved error
state left behind by the other.
Fix by using a unique ID (initialized by an incrementing atomic
counter) to index the object in the per-thread error store.
I feel like we still don't have a foolproof replacement for the really
nice lifetime management that thread_specific_ptr was doing for us. One
could argue that unretrieved errors in an object are a "bug" in the
user's program, but unretrieved bugs currently just accumulate in the
map, even after the ImageInput is destroyed, until the thread terminates.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
These are build-breaking changes that impose a new minimum OpenEXR/Imath
version supported of 3.1 or higher.
Needless to say, this will not be backported to any release branches,
which guarantee that they will never raise the minimum supported
versions of any dependencies. This is just for future OIIO. There is
already a "developer preview" branch dev-2.6.1 marking the last spot in
master in which the old dependencies are supported.
Fixes#4156
Signed-off-by: Larry Gritz <lg@larrygritz.com>
It was pointed out that oiiotool and maketx's habit of embedding their
exact command line arguments in the Software and ImageHistory metadata
of output images was potentially problematic in that it might
inadvertently leak important internal data (for example, file paths may
reveal secret projects, etc.).
This patch changes the behavior of oiiotool and maketx as follows:
* The default behavior is now to use `"OpenImageIO <version> <hash>"`
where `<hash>` is the SHA-1 hash of the command line arguments (which
allows you to tell if two files use the same command line args, but
doesn't let you know what they were).
* A new `--history` argument (or setting OIIOTOOL_METADATA_HISTORY env
variable to a nonzero integer) instead uses the full command line
arguments with no obfuscation. (This used to be the default behavior but
is now deemed unsafe.) When the env variable is in effect, a
`--no-history` argument explicitly disables it.
* For oiiotool, the existing `--nosoftwareattrib`, as before, causes the
Software and ImageHistory metadata to not be modified at all.
Here is an example of the behavior:
```
$ oiiotool tmp.tif -o tmp.exr && iinfo -v tmp.exr
tmp.exr : 640 x 480, 3 channel, half openexr
channel list: R, G, B
Software: "OpenImageIO 2.6.2.0dev : B897973B86EC13E3F1858AAE5E057EFCCF99889A"
...
$ oiiotool --history tmp.tif -o tmp.exr && iinfo -v tmp.exr
tmp.exr : 640 x 480, 3 channel, half openexr
channel list: R, G, B
Software: "OpenImageIO 2.6.2.0dev : oiiotool --history tmp.tif -o tmp.exr"
Exif:ImageHistory: "oiiotool --history tmp.tif -o tmp.exr"
...
```
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This is an initial code for the RED Digital Cinema support.
CPU only so far.
See the RED section in the docs builtinplugins.rst for more details on what currently works and current limitations.
---------
Signed-off-by: Peter Kovář <peter.kovar@reflexion.tv>
This is just the first step in many for removing deprecated things along
the way to a 3.0 release. I'm doing a little at a time.
I believe that here I'm only removing things that (a) have been
deprecated since 2.0 or earlier, (b) most or maybe all were marked with
OIIO_DEPRECATED and so should have been triggering warnings for a long
time if used downstream (thus, almost certainly nobody is using it), and
(c) are inline definitions, so this change should not appear to break
link ABIs.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
As Marie Kondo would say, Boost no longer sparks joy for us, so we take
one last look, thank it for its many years of service to our project,
and then send it on its way.
Fixes#4158
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Remove the dependency on boost's thread_specific_ptr by using hashmap's
of static duration between the owning object and the target datatype.
After digging a bit into how boost's pointer works behind the scenes, it
is actually a very similar mechanism, with the lookup being into an
std::map. I don't have an easy way to measure performance here, but none
of these are on a hot path (with the possible exception of the
per_thread_info for the image cache, but this one can be managed
externally if you want the best performance.
---------
Signed-off-by: Chris Kulla <ckulla@gmail.com>
We disable fmt execptions and force it to do one of our ASSERT's
instead. But fmt in their main branch recently added `[[noreturn]]`
annotations in a way that cause this to be compiler errors for our
overrides. Needed an extra std::terminate() call to guarantee the
noreturn promise.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The doc comment said it should initialize, and the analogous structure
for scalar texturing, TextureOpt, initialized. Missed this one.
Fixes#4219
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This change fixes out-of-bounds reads and incorrect scanline ordering
when OpenEXR's decreasingY lineOrder mode is used.
OpenEXR expects to process scanlines in decreasing order when the
decreasingY lineOrder option is enabled. This change fixes the code that
provides the chunks of scanlines to OpenEXR so that the proper scanlines
are available when it accesses the FrameBuffer OIIO created. The old
code was providing incorrect scanlines AND setting up the FrameBuffer
with incorrect pointers so out-of-bounds reads were occurring.
The key things to keep in mind are:
- Chunks of scanlines need to be sent to
OpenEXROutput::write_scanlines() from the bottom of the image to the top
when decreasingY is enabled. If the chunk's scanline range does not
contain the scanlines OpenEXR is expecting to fetch, then it can cause
out-of-bound reads because the wrong information is used to construct
the pointers for the Slices created when constructing a FrameBuffer.
- OpenEXROutput::write_scanlines() does its own chunking of scanlines
and this logic must also properly start from the bottom of the chunk
data passed into this function when decreasingY is enabled.
- The to_native_rectangle() call in OpenEXROutput::write_scanlines() may
return a pointer to m_scratch if a format conversion is needed. This
means the pointers passed to OpenEXR, in this situation, _WILL NOT_ be a
full frame buffer. This caused OpenEXR to do out-of-bounds reads because
the computations used to create the Slices were using the wrong scanline
number to adjust the pointer.
- The progress callback computation needed to be modified to handle
traversing scanlines in decreasing order.
I added an openexr-decreasingY test to verify that the code no longer
crashes because of the out-of-bound reads and it also verifies that the
decreasingY images match the increasingY images.
---------
Signed-off-by: Aaron Colwell <300262+acolwell@users.noreply.github.com>
It seems that two targets `libOpenImageIO`, `libOpenImageIO_Util` were left out
during the [switch to target-based definitions]
(https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/4193)
refactor and have no `-ftest-coverage -fprofile-arcs` options added in
the CODECOV=1 mode.
This was due to #4193 failing to propagate the link options to the dynamic libraries.
Adding the missing options resolves build errors.
Signed-off-by: kuba <kuba456@gmail.com>
To shed the boost dependency, we are disabling stacktrace for now. It's
not vital and probably not worth keeping boost just for that.
C++23 has std::stacktrace, so as compilers add support for this, we will
phase it back in.
We can also add it back if somebody has an alternate suggestion for a
3rd party solution that is a simply drop-in replacement, portable,
lightweight dependency, with compatible license. A good compromise might
be this one: https://github.com/jeremy-rifkin/cpptrace But I think what
I'm proposing is that we accept this PR first, get rid of boost, then we
can come back to looking at cpptrace as potentially a way to restore the
functionality prior to C++23.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
LibRaw 0.20 dates from mid-2020 and in any case, this version or newer
is necesary for use with C++17 which is our new minimum.
Bump the version built by build_scripts/build_LibRaw.bash to the latest.
Remove some old libraw testsuite output references that were only
applicable to the older versions.
This change will not be backported to release branches, the bump will
only be for OIIO 3.0 and later.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Adds support for the following DXGI formats when using the DX10 extended
header on DDS files:
- Compressed BCn and its sRGB variants.
- BGRA8/BGRX8/RGBA8 and its sRGB variants.
- RGB10A2.
- R16.
On the sRGB formats, the attribute `oiio:ColorSpace` is also set to
`sRGB`.
Closes#4045.
Added new DDS test images to the OpenImageIO-images repo and updated the
testsuite expected output.
OpenImageIO-images PR:
https://github.com/AcademySoftwareFoundation/OpenImageIO-images/pull/7
Signed-off-by: Alejandro Aguirre <alexguirre.aa@gmail.com>
Soon we'll raise the minimum OpenEXR to 3.1, and OpenVDB 9.0 is the
first version to support that. And because OpenVDB has Imath types in
its public APIs, it's really hard to mix and match our using an
OpenEXR/Imath 3 internal to OIIO with an OpenVDB that uses an
OpenEXR/IlmBase 2.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
A tiny bit more progress towards #4158.
None of the code was actually dependent on the fact that these
containers are sorted by key. Switching to the robin containers instead
removes one less use of boost in the project.
Signed-off-by: Chris Kulla <ckulla@gmail.com>
These are build-breaking changes that impose a variety of new minimum
dependencies for the toolchain.
The new minimum Python version supported is now 3.7.
Get rid of 'from future' lines in python scripts in testsuite.
Remove python 2.7 specific ref images from testsuite.
Needless to say, this will not be backported to any release branches,
which guarantee that they will never raise the minimum supported
versions of any dependencies. This is just for future OIIO.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
These are build-breaking changes that impose a variety of new minimum
dependencies for the toolchain.
C++17 is the new minimum. All hail C++17.
Bump minimum compiler versions to the respective releases of each one
that is requred to fully support C++17 for us. This means new minimum
changes of gcc 6.1 -> 9.3, clang 3.4 -> 5.0, icc 17 -> 19, and MSVS
v19.14 or higher now.
Remove CI tests for too-old platforms, and adjust some others that
needed particular components upgraded. We now will no longer attempt to
test on "VFX Platform 2019" or 2020. The 2021 platforms are the earliest
we claim to support (reflecting VFXP's guidance about trying to support
up to 3 years back).
By removing the old test cases, we're also removing the last platforms
that test Python 2.7. So while this PR isn't the one that officially
removes the last vestiges of Python 2, it does mean that once this is
merged, we will no longer be testing less than Python 3.7. (A separate
PR removing all Python 2 support will follow and presumably be merged
around the same time.)
With C++17 min, remove traces of boost::filesystem. We can rely on
std::filesystem now.
Clean up pre-C++17 macros in platform.h
Needless to say, this will not be backported to any release branches,
which guarantee that they will never raise the minimum supported
versions of any dependencies. This is just for future OIIO. I won't
merge this until after April 1, at which point I will also tag the last
spot in master that is known to work with C++14/gcc6.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
As mentioned in #4205, #3900 and
https://github.com/AcademySoftwareFoundation/OpenImageIO/discussions/3910#discussioncomment-8913737
OpenImageIO previously did not support reading of 16- and 32-bit files
beyond their merged image data section. I have now implemented this by:
1. Adding support for reading layer info from the "Lr16" and "Lr32"
Tagged Blocks as well as adding additional checks to the regular layer
info section
2. Implemented the Zip and ZipPrediction compression codec which is
required to read these files as that is photoshops preferred compression
codec for those bitdepths
Due to the way the Zip data is stored (per channel but not per scanline)
the current implementation reads the whole channel into a buffer which
is then memcpy'd from by calls to `read_native_scanline()`. The buffer
gets cleared on deletion of the `PSDInput` object
---------
Signed-off-by: EmilDohne <86836589+EmilDohne@users.noreply.github.com>
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Co-authored-by: Larry Gritz <lg@larrygritz.com>
Windows CI was failing because it couldn't get a proper libtiff from
VcPkg, which in turn is because lzma is a dependency of the libtiff
build in VcPkg, and for the time being, it looks like VcPgk is failing
to get lzma which is part of the big xz debacle of 2024.
So work around for now by just building our own libtiff rather than
relying on VcPkg. Seems to work fine.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Add another optional flag to our checked_find_package cmake function --
CONFIG means only accept a config for the package, never use a
FindBlah.cmake module.
(Reminder: this is in contrast to the existing PREFER_CONFIG option,
which tries a config first, then if that fails, will try the module, and
also different from the default if neither is used, which is to prefer
the module over the exported config.)
And rig our search for DCMTK to use the new CONFIG option. There is a
FindDCMTK.cmake modules that comes with cmake, and for at least some
versions of cmake, it seems subtly broken in such a way that actually
wrecks our finding of other packages properly.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
For OSL, we want to be able to decode a wrapmode directly from a
ustringhash without having to expensively convert it to ustring.
Signed-off-by: Chris Hellmuth <chellmuth@gmail.com>
We had lots of stray usage of add_defintions, add_compile_options,
etc. In modern cmake, this is considered bad form, since it makes
subprojects pollute the global option lists. A better way is to add
all these to specific targets, so now that's what we do in this PR.
This should make all the definitions and options we want to set for
compilng OIIO's code not have any interference with subprojects, or
with higher-level surrounding projects.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
... in the OCIO 1.x case where we relied on our own Find module rather
than an exported cmake config from OCIO itself. This is desired for
uniformity of naming with the exported configs of the later versions.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
In this PR, we convert nearly all `sprintf` uses to `format`.
It should not change any behavior or compatibility. Nonetheless, I think
this is destined for master only and won't be backported, to reduce
risk.
Along the way, we add format_to and format_to_n to Strutil.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Minor simplification. Also add a few `noexcept` in some places that we
didn't have them already.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Add a `OpenImageIO_BUILD_STATIC_UTIL_LIBRARY` CMake variable that, if
enabled, will build a separately named static version of the
OpenImageIO_Util library, in addition to the dynamic one. It has a
separate name, libOpenImageIO_Util_static, and a separate exported cmake
target, `OpenImgaeIO::OpenImageIO_Util_static.`
This can be a help to applications that only need OpenImageIO_Util, and
not the main OpenImageIO library, and furthermore would prefer a static
library so they don't need a runtime dependency on our dynamic
libraries.
Cavat emptor: Beware that there are singletons inside ustring, the oiio
thread pool, and possibly others I'm not remembering at the moment. It
could be a bad thing to have an application that contains both the
static and dynamic versions of libOpenImageIO_Util, or to have multiple
copies of the static library inside the same executable (including
transitively through dependency libraries that were built
independently). You have been warned!
Note also that this is swimming upstream in cmake land. For reasons I
never quite understood, cmake assumes that a project wants to build
either static or dynamic libraries, but provides little direct support
for doing both in the same build. So we're taking the matter into our
own hands, perhaps. If somebody thinks the way I'm proposing is very
wrong, please do speak up.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Base on the [OIIO slack channel
discussion](https://academysoftwarefdn.slack.com/archives/C05782U3806/p1689031780561469),
this PR implements the `copy_image` function for the OpenEXR format. The
main motivation of this PR was due to a recompression issue where
copying a lossy compressed EXR image (pixels are decoded and then
encoded again), which can cause the image quality to downgrade on each
iteration.
- `OpenEXROutput` is now a friend class of `OpenEXRInput`. Since
`OpenEXROutput::copy_image()` needs to access some of the private
members in `OpenEXRInput` (This approach is referencing the
implementation in `JpegInput` and `JpegOutput`).
- The `OpenEXRInputStream` and `OpenEXRInput` class have been moved from
`exrinput.cpp` to `exr_pvt.h`. As `OpenEXRInput` depends on
`OpenEXRInputStream`, so both have to be moved together. Such relocation
is necessary due to the above reason, i.e. `OpenEXROutput::copy_image()`
needs to access some of the private members in `OpenEXRInput`.
- And the implementation itself, `OpenEXROutput::copy_image()` overrides
the base class method just like `JpegOutput` does.
The `OpenEXROutput::copy_image()` implementation took the advantages of
the existing OpenEXR function `Imf::OutputFile::copyPixels`, quoting the
comment from the function:
> //--------------------------------------------------------------
// Shortcut to copy all pixels from an InputFile into this file,
// without uncompressing and then recompressing the pixel data.
// This file's header must be compatible with the InputFile's
// header: The two header's "dataWindow", "compression",
// "lineOrder" and "channels" attributes must be the same.
//--------------------------------------------------------------
IMF_EXPORT void copyPixels (InputFile &in);
When the output image is copied from an input image, the pixel data can
only be located in one of the input parts. So it checks if both input
and output part is initialized (non-null), then invoke the `copyPixels`
function in place.
All the error handling is done within the OpenEXR `copyPixels` call.
There is a try-catch block in OIIO side that emits an error message and
fallback to the default image copy routine, e.g.
`ImageOutput::copy_image`.
Note that currently, this doesn't do anything optimized for the
"core library" code path.
---------
Signed-off-by: Andy Chan <andychancse@gmail.com>
Preliminary support for JPEG XL.
A variety of additions are needed, but this is a good start.
Top items on our to-do list (not necessarily in priority order):
- [ ] Document in builtinformats.rst
- [ ] Handle different pixel types -- right now, always reads/writes float.
- [ ] Compression options
- [ ] EXIF
- [ ] Test cases in the testsuite
---------
Signed-off-by: Peter Kovář <peter.kovar@reflexion.tv>
simd.h improvements and refactorization for consistency:
* Use `= default` for all the uninitialized constructors, copy
constructors, and assignment from the same type.
* Fix the `operator=` to be more standard by returning a `vec&` instead
of `const vec&`, which apparently is unusual.
* Test that these all result in C++ understanding that these classes are
trivially copyable.
There is a bit of awkwardness where for the vint types, we have to
define the copy constructors for the Intel icc compiler. It just seems
to do the wrong thing for `=default` and fails lots of our tests. I
don't understand why, and I'm not inclined to investigate too deeply,
since it's a discontinued compiler we support only for legacy reasons,
so it doesn't bother me too much that we special-case it. One reason I
think it's a compiler bug is that it has no problem with the floating
point vector types. It's confused about something with the simd ints.
Disable SIMD acceleration of fast_exp2 when using MSVS prior to 2022.
It seems to sporadically generate bad code in this function and I
can't seem to restructure the function in a way that consistently is
correct. Doesn't seem to happen with newer MSVS, nor on any other
platform or compiler. I'm assuming this is a compiler bug and it's not
worth pursuing for an old version of one compiler, for this one
function that isn't really used all that often or in
performance-critical paths.
Some extra slosh in the simd fast_exp test.
Rewrite OIIO_CHECK_SIMD_EQUAL_THRESH to print the differences and use
more digits of precision when it fails.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We inadvertently started using libheif functions that only became
available in their version 1.16.0. Guard against that and now we
recommend at least that version (but we aren't yet changing the minimum
requirement). This fixes a build break introduced in #4142 (but only
breaks for older libheif).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The operation it's supposed to be performing is to merge in the gathered
value where the mask is set, and leave mask=0 lanes unchanged. But for
our "no SIMD" fallback (i.e., no AVX2 for 4- and 8-wide), we incorrectly
implemented it as setting the unmasked lanes to 0!
This was quite wrong, and unfortunately masked by the fact that our test
of the function initialized a variable to 0. Change to a better number.
The best number.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Remove boost from `Strutil`. Use existing mechanisms and STL features
instead.
Partially addresses #4158
Signed-off-by: Jesse Yurkovich <jesse.y@gmail.com>
We had picked a version that is too new to run on the ASWF docker
containers, so wasn't actually working properly when CI failed and we
wanted the artifact uploaded.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Remove some SPI-specific make and cmake customizations from long ago, we
haven't used those makefiles to build for a long time. Also remove the
detectplatform.mk file, we haven't used the results of what it does for
a while. This just declutters.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
A variable was declared unconditionally but only used when Freetype is
enabled. Move its declaration to be within the right conditional, on
order to avoid compiler warnings about unused static variables.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Automatically recognize some additional color space name synonyms:
"srgb_texture", "lin_rec709" and "lin_ap1" are names that MaterialX and
USD use as canonical names.
Also add common permutation "srgb_tx" and "srgb texture" as additional
aliases for srgb.
Fixes#4165
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Added a new bit of pseudo-metadata that can be retrieved by oiiotool's
expression substitution: `{TOP.NONFINITE_COUNT}` will be replaced by the
count of inf and nan values present in the top image.
Also, the `-fixnan` command now has the side effect of setting a user
variable called `NONFINITE_COUNT` to the number of nonfinite values that
were detected or repaired.
Between the two of these, it's now easy to solve the following problem
that actually came up in production for us.
Sometimes, nans creep into images, and it's easier to repair them than
to fix the complicated process that produced them in the first place.
For some process in the studio, we had a script that was doing this
preemptively on every image it encountered:
oiiotool image.exr -fixnan black -o image.exr
It was very expensive to repair every image whether it needed it or not,
when actually it's only needed once every
many-thousands-of-images. Most of the expense was in outputting the
repaired file overtop the original. Now we can do this
oiiotool image.exr -if "{TOP.NONFINITE_COUNT}" -fixnan black -o
image.exr -endif
It still has to read the image and look for NaN values, but since there
are almost never any actual NaN's, now for the usual case, it can skip
writing the output -- which was the vast majority of time.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* exif.cpp: use spans to avoid several possible buffer overruns
* paramlist_test.cpp: use spans to be sure we're ok on memory bounds
* jpeg: use spans in jpeg icc profile manipulation for memory safety
* psd output: switch cmyk conversion to use spans for memory safety
I've been cobbling these together bit by bit over the last several
weeks, tracking down the memory safety issues identified by the Sonar
static analysis. I think that *mostly* they are not real bugs, but the
code is confusing enough that it's hard to verify. This refactor makes
it safer, and even if the code was previously correct but merely hard to
analyze, the bounds checking in span (for debug mode) gives the static
analyzer the clues it needs to know this is safe.
As further explanation, these situations generally involve something
that looks like this, schematically:
caller () {
T buffer[known_size];
call function(buffer, ...params...);
}
function(T* buffer, ...params...) {
i, j = ...very complex things, hard to analyze...
foo = buffer[i];
buffer[j] = bar;
// Did we do a buffer overrun??? ¯\_(ツ)_/¯
// This function doesn't necessarily have knowldege of the
// true bounds, even if it wanted to check every access.
}
By rewriting this function as follows:
function(span<T> buffer) {
// all indexed access to buffer will be bounds checked in debug
}
we are passing the bounds as well, and at least for debug builds, are
also checking bounds with an assertion on every indexed access into
buffer, without the function needing a lot of clutter. Fire and forget.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Many IBA functions take lots of optional parameters, which are from time
to time augmented, leaving us with a clutter of semi-deprecated
versions.
This PR prototypes a new approach, which is to make IBA functions have
explicit parameters for their core/necessary controls, and the minor
optional behavioral control parameters are passed as keyword/value
argument lists. I think this has a few really big advantages over
explicit parameter lists with ever-growing optional parameters:
1. It makes the call sites very readable, since the keywords document
what the parameters do.
2. It allows us to add (or deprecate) optional parameters without ABI
breaking changes, and without proliferation of many versions of each
function.
A new class is added: ParamValueSpan, which as the name implies, is
simply a `span` of ParamValue structs (actually, it's a subclass that
inherits from `span<const ParamValue>`, and adds a bunch of helper
methods). Within the IBA namespace, this is aliased to KWArgs, to make a
short and self-documenting name to pass this collection of optional
parameters. This leads to IBA declarations that look like this:
ImageBuf OIIO_API resize(const ImageBuf &src, KWArgs options = {},
ROI roi = {}, int nthreads = 0);
and calls that looks like this:
resize(dst, src, { { "filtername", "lanczos3" },
{ "filterwidth", 6.0f } });
In this first stab, I have converted IBA::resize(), fit(), and warp().
That seems like enough to evaluate whether we like the direction this is
going. If so, we'll tackle more IBA functions over time. I believe I've
done this all in a way that preserves API and ABI compatibility for now
A few other odds and ends that come along for the ride:
* Filter1D and Filter2D have a new create_shared static method that
returns a shared_ptr instead of raw pointer.
* Make all filter ctrs ok with 0 default filter width value, which means
to use the natural default width for that filter.
* IBA::make_kernel is extended so that if you pass 0 for width or hight,
it selects the a resolution using the default size for the named filter
kernel. This way, the caller doesn't need to know the correct extents of
the kernel, just the name.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Co-authored-by: Thiago Ize <ThiagoIze@users.noreply.github.com>
We've been using C++14 minimum for years, and in fact will move to 17
for next year's major release.
Also use error reporting correctly if a sufficient C++ verion is not
found
Signed-off-by: Larry Gritz <lg@larrygritz.com>
It seems that unlike most image formats where we merely note the camera
rotation but don't rotate the pixels, the heic reader automatically
rotated.
The trick is that using the C++ API wrapper of libheif, there is no way
to pass the decoder options struct that is where you need to say not to
auto-transform. Needed to drop down to the lower level C API for this
one spot.
We want this format to behave like the others -- the orientation is
advisory, and it's up to the app how to deal with it. But we have back
compatibility to deal with.
If a rotated image is aecountered, OIIO 2.5 and earlier will by default
still auto-rotate to preserve compatibility, but OIIO 2.6+ will not
auto-rotate.
When not rotating, the Orientation metadata will reflect the desired
display orientation of the image, according to what was in the file.
When auto-rotating, Orientation will be 1 (canonical display
orientation), and a new attribute "heif:Orientation" will reflect what
was originally in the file.
When opening a file for input, the special configuration metadata hint
"heif:reorient" can express a preference for auto-rotation, overriding
the default. Thus, setting this hint to 0 for OIIO 2.5 will turn off
auto-rotation, and setting it to 1 for OIIO 2.6+ will make it
auto-rotate like it used to do.
I also added some additional tests for heif files.
Along the way I fixed a rather embarrassing documentation problem as
well, where all the orientation codes were "off by 1", oh boy.
Fixes#4123
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This PR is for allowing the TypeDesc type to be used directly in a C API
without indirection, and therefore also in a Rust API without
indirection.
---------
Signed-off-by: Scott Wilson <scott@propersquid.com>
* Make sure clang variants are before clang and things that might
emulate gcc are before gcc.
* Use the right symbol for detecting ICC
* Capitalize `__linux__` correctly, oops.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
It somehow escaped our notice, but not only does DCMTK have an exported
config, but also CMake itself has a FindDCMTK.cmake module. So get rid
of ours, and prefer the config if available. This is hoped to solve some
edge cases for finding and using this library properly.
Fixes#4146
(I hope?)
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Static analysis has been yelling about code in iffoutput.cpp possibly
overrunning buffers, and frankly the code in this module is so confusing
that I can't tell if it's correct or not. (It's 13 years old, hasn't
been touched for a long time.)
In an ideal world, this would all be rewritten to use spans so the
buffer lengths are known and checked. But I don't really have the time
or inclination to rewrite it -- after all, iff is not a very important
file format, though I do think it's used enough that we can't drop it
entirely. The payoff from a full rewrite is marginal. So I came up with
the following compromise, embodied by this PR:
I'm making spans of the regions we're ultimately reading from and
writing to, passing those down the chain of function calls, and so even
though the actual operations are total pointer arithmetic spaghetti, we
can use those spans to verify that the pointers are still within the
span bounds any time we read or write through them. We do it with
assertions that are only active for debug builds, so there shouldn't be
any measurable performance hit. But the assertions will be active in CI
for both the static and dynamic analysis tests, so should catch any
latent bugs.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This is Update 7, the original PR is ongoing and we will continue to
update as additional people relicense.
Per PR #3905 (https://github.com/OpenImageIO/oiio/pull/3905), update the
RELICENSING document with the names of people and companies who have
signed on to the relicensing thus far, and update the SPDX license
notices on source files whose extant authorship (per 'git blame')
comprises only authors who have relicensed.
To the best of our estimation, we now have 99.85% of lines of code
relicensed, and 93.2% of all source files now consist of ONLY
Apache-2.0 licensed code with no traces of BSD remaining.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Add testsuite/filters that verifies that the filter shapes are correct
and don't change if we make future changes to the filter code.
Along the way, add a new global getattribute query 'filter_list', that
returns a semicolon-separated list of all the 2D filters it knows about.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We did this separately each place we needed it. Do it just once, in a
public header, and give it a name that won't clash with other packages.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Remember that ImageBuf only stores one pixel data type per buffer, but
of course they may have differing-per-channel types in image files. So
how does oiiotool know what to write? There is a series of heuristics:
-d is for explicit user control, if not, then try to deduce it from the
input files.
A bedrock precept is that `oiiotool in.exr -o out.exr` ought to write
the same channel types that it read in.
The logic to implement this with the various image operations and
possible overrides can be hard to get right. Indeed, I did not get it
right. It made mistakes even in simple cases like the above. And the
code was complicated and hard to understand, as well.
This PR rewrites a couple functions that implement this chunk of this
logic, both correcting several errors as well as (I hope) being clearer
and better commented so that we can more easily reason about it and
modify it in the future.
I also added some additional tests to testsuite/perchannel to verify
some of the cases that were previously failing.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Primarily, recent changes (PR #4071) to vint4::store for the NEON case
appear to have some type mismatches, which apple clang on ARM-based Mac
(including our CI) seems ok with, but which is generating type errors on
other ARM Linux platforms.
I think the types were weird here, so I tightened it up to get the types
right for temporary variables in that function. That's the primary fix
here.
Secondarily, I modified simd.h and the CMake setup so that build option
USE_SIMD=0 will disable NEON in the same way that it disables SSE. (I
realized that USE_SIMD=0 was not disabling NEON, so there was no way for
a NEON platform to completely disable SIMD if they needed to.)
Fixes#4111
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* jpeg output: Extra memory check
* TS: batch texture call fixed to handle any number of channels
* TS: assert to guard against possible null pointer dereference
* oiiotool printinfo: clarify assertion for the sake of static analysis
* jpeg2000 reader: assertion to ensure we don't dereference a null
pointer
Signed-off-by: Larry Gritz <lg@larrygritz.com>
No more need for the paid macos13-arm-py311 runner and only use it on
the ASWF repo selectively. An ARM-based MacOS-14 runner is now available
for open source projects, so can run free on every PR and on user forks.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* New header span_util.h with span-related helpers:
- make_span/make_cspan helps for converting vectors & arrays to spans
when passing to templated functions.
- spancpy, spanset, spanzero are memory-safe, range-checking
substitutions for memcpy, memset when working with spans.
* Add range check DASSERT to span access in debug mode. There were a
couple spots where we were taking the address of an out-of-range value
that needed to be adjusted to use a better idiom.
* ParamValue: extra OIIO_DASSERT range check for get() method.
* Simplify constructors by replacing `&x[0]` with using `data()`, which
is always well-defined.
* Fix typos in span.h comments
The range checking imposes no extra cost for release/optimized builds.
But it might help us for CI analysis and any time we're debugging.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We had all along omitted an important 'int' conversion case for an
internal to_floats() helper function that's used by the
TypeDesc::convert_type utility. Which is in turn used by
ParamValue::get_float() and ParamValueList::get_float().
All of these would fail to return the proper float if the underlying
data was an int, despite these functions saying that they would have no
trouble returning the float equivalent of int data when asked.
I added a test that verifies that it's wrking.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
New OIIO global attributes (read only):
- "build:platform" retrieves the OS and CPU type that this build of OIIO
is fof (e.g., "Linux/x86_64").
- "build:compiler" retrieves the compiler name and version used to build
OIIO itself (e.g. "gcc 9.3").
- "build:dependencies" retrieves a semicolon-separated list of library
dependencies -- both for image format support (like libtiff) but also
general (boost, TBB, Python, fmt, OCIO).
- "build:simd" is a new (preferred) synonym for the SIMD and other
hardware capabilities selected at build time. The old name, "oiio:simd"
is hereby softly deprecated as confusing.
`oiiotool --buildinfo` is a new command that prints this information.
Example output:
```
$ oiiotool --buildinfo
OIIO 2.6.0.2spi | MacOS/x86_64
Build compiler: Apple clang 15.0 | C++17/201703
HW features enabled at build: sse2,sse3,ssse3,sse41,sse42
Dependencies: OpenEXR 3.2.1, LIBTIFF Version 4.6.0, jpeg-turbo
3.0.1/jp80, dcmtk 3.6.8, FFMpeg 6.0 (Lavf60.16.100), gif_lib 5.2.1,
libheif 1.17.6, OpenJpeg 2.5.0, null 1.0, OpenVDB 11.0.0abi11, libpng
1.6.40, Ptex 2.4, libraw 0.21.2-Release, Webp 1.3.2, Boost 1.83,
OpenColorIO 2.3.1, Python 3.11.7, TBB 2021.11.0, fmt 10.2.1
```
This is all a convenience utility to make it easy to ask users to run a
simple command that will tell us a lot of information relevant to
resolving their issues in cases where it's related to some of their
build-time compiler or dependencies.
Also cleaned up the "bug report issue template" to clarify the
instructions and specifically ask people to run this command and include
the information with any bug report.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
A better default for users is probably to assume a 2.2 value if the
GAMMA env variable is not set. If you want something different, set
GAMMA proactively or use a real color managed workflow.
Fixes#318
Signed-off-by: Larry Gritz <lg@larrygritz.com>
OpenVDB >= 10.1 is incompatible with building OIIO using C++14. We had
tried to disable this disallowed combination, but a typo prevented it
from working properly.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
With GitHub runners with more cores becoming available, adjust our CI
scripts to do parallel builds with nprocs+2 instead of hard-coding. We
use slightly more than the number of available cores because these
builds and tests tend to be I/O bound, so they can be a little
over-threaded to get maximum utilization.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Upon reading, the subsampled Y/BY/RY(/A) channels of luminance-chroma
images are automatically converted to RGB(A) channels. These images will
set a metadata "openexr::luminancechroma" to 1 in the ImageSpec, to indicate
that the original image was luminance/chroma.
Subsampled channels are not supported with the exception of reading
luminance-chroma images with vertical and horizontal sampling rates of 2.
This limited support does not work when OpenEXR's C Core API in used, only
when OpenEXR's C++ API is used. Furthermore, it does not work in
combination with tiles, multiple subimages, mipmapping, or deep pixels.
The test images from the OpenEXR testsuite have been added and work with
the C++ variant of the test.
Fixes#4051.
---------
Signed-off-by: Joachim Reichel <43646584+jreichel-nvidia@users.noreply.github.com>
These things were scattered around fmath.h. This grouping corresponds to
things in C++20 <bit> (but of course works for older C++ and for Cuda)
and lets modules use those without needing the full fmath.h.
Switch our swap_endian implementation to the "byteswap" name and
semantics to match C++20. The old swap_endian just calls those. Also add
a version of byteswap (in fmath) that takes an OIIO::span, since we
generally are leaning toward preferring spans rather than raw pointers
and lengths as separate parameters.
None of this should break source or binary compatibility for
applications.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
There are two different IBA::fit() entry points -- one that takes a
Filter2D* and one that takes a name of the filter. In both cases, if the
supplied filter is empty, fit will make its best guess (which, roughly
speaking, is supposed to be lanczos3 when shrinking and
blackman-harris when zooming).
You might expect that both, for the "pick my filter" case, would end up
with the same filters, but they did not! And the "wrong" one was -- ick
-- using an awful triangle filter. So when they were unified to both use
the same logic, it was easy to see that the one being used by oiiotool
was using the triangle and definitely over-blurring! So I think the
images look much nicer now that it's using the lanczos3 in appropriate
circumstances.
Sometimes lanczos3 can over-sharpen and you see some ringing. That was
easy to see at the edges of images that were being letterboxed, so I
also put a little extra guard in filtered_sample() so that when it's
asked to do edge clamping, it doesn't try to apply the filter too far
off the edge of the image boundary.
You *still* may see a little over-ringing in some cases, and if you do,
that's a sign that maybe you should be selecting a different, slightly
less harsh, filter rather than use its default. Or, does anybody want to
argue that we should switch the default resizing filter to be
blackman-harris, which is a lot less likely to have unsightly ringing,
and if you want something even sharper, you should have to ask for
lanczos3?
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Some of the ways we were using unions for type punning in the vbool16
class were unnecessary, as both union members were basically the same
type. Simplify by getting rid of the unnecessary m_bits union member.
This also makes Sonar static analysis happier about these dubious uses
of unions for type punning.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Bleeding edge CI test failures due to more changes in fmt's trunk.
Solutions:
* Accommodate some recent header refactoring in fmt.
* Straighten up some const issues that worked before, but break under
the latest fmt. Making the custom formatter's 'format' method be const
fixes the latest, but breaks some of the oldest versions of fmt we
support, so we add the const only from fmt 9.0 on.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Use default implementations of copy ctr, destructor, and assignment
from ustring. This allows ustring & ustringhash to be recognized as
being a variety of flavors of trivially constructible, copyable, and
destructible.
* It seems that the assignment operators were returning const &
whereas it's more typical to return &, so now we do.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* pngoutput.cpp: Catch possible exceptions.
* printinfo.cpp: assertions to assure we don't dereference a null
pointer.
* testtex.cpp: make sure allocated chunk is initialized
* WriterInternal.h: Catch condition that could lead to buffer underflow.
* imagebufalgo_mad.cpp: Simplify pointless clause. The prior test was
redundant. Since a few lines above, if B was an image but A was not, we
swapped to ensure that A was always an image. So here, we can just test
A to know for sure that at least one of them is an image. That makes the
test here simpler, but it also makes it more clear to static analysis
that from this point forward, A can't be nullptr.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The function signatures in the header didn't match those in the docs,
after some previous changes. It's probably been broken in the docs for
some time. Nobody noticed?
Signed-off-by: Larry Gritz <lg@larrygritz.com>
If you are looping i over range [0,width), and want index to go the
other direction, you need `index = width - 1 - i`, not `index = width - i`.
Identified by Sonar static analysis.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Bump our clang-format standard from llvm 12 to llvm 17, and our
C++ standard (for formatting) to 17.
Fixes/improvements along the way:
* Moved the clang-format CI test from an aswf container to bare Ubuntu
to make it run faster -- it doesn't need most of the dependencies,
so there's no point wasting 2 minutes or so downloading a container.
* build_llvm.bash: Make sure llvm installs when needed, even if not
using clang.
* gh-installdeps.bash: Setting SKIP_SYSTEM_DEPS_INSTALL=1 skips
installation of many non-essential dependencies.
There is some reformatting here and there as 12->17 changed some formatting
choices somewhat, but it's fairly minor. Most changes are improvements,
some are neutral, a few are inexplicable but not less readable than before.
We've long since accepted the tradeoff that using clang-format makes some
things formatted differently/worse than we would have preferred, but we deem
it worthy of the benefit of having it all 10% automatic and never fighting
over formatting in PRs.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Remove duplicate dependecy for OpenImageIO_Util to fmt.
OpenImageIO target already depends on fmt through OpenImangeIO_Util
library.
Signed-off-by: Dominik Wójt <domin144@o2.pl>
Verified that the memcpy trick works fine (even when used in vectorized
loops) for gcc, clang, MSVS, icx. So change the specialized versions using
intrinsics so they are only used for icc, which fails to vectorize loops when
the memcpy trick is used.
Also get rid of unused double specializations.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
fmt provides cmake config files since at least version 7.0.0, which is
the minimum version supported.
Fixed some variable names, so the provided config can be used.
The previous variable names were missing, if the library was installed
in a custom location,
like in case of manual build or provided by conan.
---------
Signed-off-by: Dominik Wójt <domin144@o2.pl>
This patch adds a new oiiotool command: `--cryptomatte-colors`, which
takes the name of a cryptomatte set of channels, and produces a
color-coded matte in which each ID gets a distinct color in the image.
This can be useful for visualizing the matte, among other things.
Maybe this is the first cryptomatte related functionality of many?
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We did `import OpenImageIO as oiio` then instead of using `oiio`, we
used `OpenImageIO`.
This was failing sometimes. Frankly, I'm not sure why it ever worked.
Maybe it works on some versions of Python but not others?
I dunno. But this fixes it everywhere and is clearly correct.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We had an awkward situation of the simd matrix44::inverse() only being
defined for Intel SSE, and in all other cases falling back on
Imath::M44f::inverse. The problem with that is that it made simd.h
depend either directly on ImathMatrix.h, or assume that it would be
included prior to inclusion of simd.h. This was not good.
This patch rewrites inverse() to eliminate all of the direct SSE
intrinsics in terms of other simd wrapper functions, which means that
they will be defined one way or another on all architectures.
Along the way I added a new vfloat4 shuffle that combines two vectors
(corresponding to `_mm_shuffle_ps` in SSE land), and a vfloat4
constructor from 2 float*', where the first two elements come from the
first ptr and the second two elements come from the second ptr.
Ultimately, the final implementation I settled on for the new inverse
didn't use these (an earlier version did), but I want to keep them
around anyway in case they are useful in the future. I did add a test
for them.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Move a bunch of specific tests from testsuite/oiiotool to
testsuite/oiiotool-copy to group similar tests together, and also to
load balance better between the tests.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This broke some time back, probably partly related to our change in
repository home. This seems to get it mostly working? The reports are
still not showing up, but I think the solution might be to make sure
this is checked into master, so we need to merge this before assessing
whether there is more work to be done in this area.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
For image with more than 4 channels, we were misallocating the OpenGL
buffer by using the total channels instead of the 4 we wanted to create
the texture out of. Also, make sure the calculation usees wide types to
avoid possible integer overflow.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
It's a little involved to explain why, but at work a combination of
libraries and some trickery with static linkage led me to really need a
way to inject another library into the mix, just for iv linkage.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
For the in-place case (where src and dst refer to the same IB), we were
doing an unnecessary full-image copy.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Unity builds (which we do for CI to speed it up) combine source files,
sometimes in ways that are a bit non-deterministic, and can run into
violations of the ODR that were not a problem when separate definitions
of the same thing (a macro in this case) were in different compilation
units but are in the same one as a unity build.
In this case, a private OIIO_DISPATCH_TYPES macro in printinfo.cpp
conflicted with a different macro of the same name in
imagebufalgo_util.h.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
## Description
Minimal change to update link to OpenEXR's new Test Images page. I poked
around a bit to find a "friendlier" variant of the link but could find
none.
Note that the section containing this link implies that folks need to
download these files themselves and should be "placed in a sibling
directory to oiio-testimages". The problem is that not even OpenEXR
lists the repository for the test images nor does the page linked above
offer a download.
Should we list out the openexr-images repo address too and say to just
clone that?
Fixes#4042
---------
Signed-off-by: Jesse Yurkovich <jesse.y@gmail.com>
Fix for #4084. Will now check if the given filename exists before
checking for REST parameters.
Before:
```
bin % ./oiiotool \?.png -o \?-output.png
oiiotool ERROR: read : "?.png": ImageInput::create() called with malformed filename
Full command line was:
> oiiotool ?.png -o ?-output.png
```
After:
```
bin % ./oiiotool \?.png -o \?-output.png
```
No output, `?-output.png` created.
Signed-off-by: AdamMainsTL <adam.mains@topazlabs.com>
Please look this over. If you are one of the duplicated names and I
haven't chosen your preferred canonical email (I made my best guess),
please comment here with corrections.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Historically, ImageBuf has provided Iterator for writing to mutable IB's
and ConstIterator for reading into IBs. If you initialized an Iterator
to an IB that was ImageCache-backed (and therefore by definition not
mutable), it would convert the IB to the mutable kind (basically by
allocating the memory and reading the file through the IC, thus freeing
the IB of its dependence on the cache.
An important bug was fixed by PR 3997, which kinda proved that people
were not depending on this behavior, since it would tend to fail. But it
also got me thinking about to what degree the whole Iterator vs
ConstIterator was really necessary, or did it just make things harder
for users.
In this proposed patch, we change the Iterator behavior so that instead
of immediately making the IB writable as soon as the Iterator was
instantiated and associated with the IB, we instead treat it as a
ConstIterator UNTIL it actually tries to write a value, at which point
we ensure that the IB is writable.
The bottom line is that this seems to work. If you don't want to have to
think about Iterator vs ConstIterator, you don't have to. Iterator is
fine. If you don't actually write into any of the pixels, it behaves
just like a ConstIterator, including being perfectly happy with an
IC-backed IB. Performance metrics indicate that there is NO penalty for
doing so -- if you are only reading pixel values, traversing an IB with
Iterator is the same speed as doing it with ConstIterator.
I had one sleepless night after implementing this when suddenly I
realized that this was totally not thread-safe, that somehow multiple
iterators traversing the same IB would screw each other up if one of
them wrote to a pixel and converted the image. So I put in a test to
have many threads concurrently traversing the same IB, some with
ConstIterator, some with Iterator but only reading, and some with
Iterator and writing. I have not been able to trigger any failures or
crashes. It seems fine. (I did use the mutex a little more aggressively
than before.)
Why is this not as dangerous as I thought? Mainly because once an
iterator (of either variety) thinks it's dealing with an IC-backed
ImageBuf, basically, it will just happily keep using the IC all on its
own, even if the IB that it's supposedly traversing has, because another
thread's iterator has written to a pixel, "localized" the image and
severed the IC dependence. Then if it also needs to write, it will
(safely, because of a mutex) see that the image is already
localized/writable, and start using that representation.
So herein lies a very important caveat about using IB iterators: If you
have multiple iterators traversing the same IB and any of them are
*writing* to the image, the iterators may not see a globally temporally
consistent version of the image. Put in plain English: iterators that
are strictly reading may see an IC-backed disk image as it was on disk,
and not as it has since been modified by a different active iterator. So
if you are modifying an image in place, beware of using multiple
iterators, because a reading iterator may or may not see the changes
that a writing iterator made to a pixel value.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This change adds OCIO functionality to iv.
To enable OCIO, first set the $OCIO environment variable to point with a
path to the OCIO config. "ocio://default" should work as well.
If a valid config provided, a new menu gets added under "View/OCIO" with
an on/off toggle and submenus to select the image color space and
display/view and optional look.
If an image color space, display and view are provided as command line
parameters ("image-color-space", "display", "view"), then the OCIO mode
is switched on automatically.
If no image color space is provided as command line parameters, the top
most color space of the config get pre-selected in the menus.
If no display or view is provided as command line parameters, the
default values get pre-selected in the menus.
In OCIO mode the exposure and gamma adjustments are done by OCIO. The
exposure is done in scene linear, gamma is still in display space. The
exposure adjustments don't match between the OCIO and non-OCIO modes.
When OCIO fails to initialise the pipeline for any reason, an error
message is shown in the status bar, and the non-OCIO path gets used.
TODO:
- All channel shuffling is done by the existing shader and happens
before the OCIO step. We may want to discuss/change that?
- The initial image color space gets selected from the command line
parameter, or the first color space in the config. There is no attempt
being made to use the image metadata.
- The exposure/gamma values get baked into the shader, which gets
recompiled on every change. We may want to use uniforms instead. - DONE
---------
Signed-off-by: Anton Dukhovnikov <antond@wetafx.co.nz>
I didn't have the condition right -- the `if` isn't respected in the
`pull_request` section, so this job was running on every PR instead of
just the ones with "macarm" in the branch name. Refactor to get the
conditional on the job instead.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
There was a flaw in PNGOutput::write_scanlines when the y range of
scanlines to write consisted of less than the whole image, and alpha
de-association was needed -- we used the whole-image size instead of the
y range of scanlines size.
Fixes#4044
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Without getting the version properly, we weren't disabling use of a
deprecated field for OpenJpeg >= 2.5, as we thought we were.
This fixes recently broken CI when building against the current openjpeg
master.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
In some environments, only one kind of the library could be available.
Choose proper fallback in such cases. Prefer static library, if
LINKSTATIC is ON and dynamic otherwise.
Signed-off-by: Dominik Wójt <domin144@o2.pl>
## simd.h improvements:
vint4::load from unsigned char pointer got pre-SSE4 code path. Testing
on Ryzen 5950X / VS2022 (with only SSE2 enabled in the build):
- vint4 load from unsigned char[]: 946.1 -> 4232.8 Mvals/sec
vint4::store to unsigned char pointer got simpler/faster SSE code path,
and a NEON code path. Additionally, it got test correctness coverage,
including what happens to values outside of unsigned char range (current
behavior just masks lowest byte, i.e. does not clamp the integer lanes).
- vint4 store to unsigned char[]: 3489.8 -> 3979.3 Mvals/sec
- vint8 store to unsigned char[]: 5516.9 -> 7325.3 Mvals/sec
NEON code path as tested on Mac M1 Max (clang 15):
- vint4 store to unsigned char[]: 4137.2 -> 6074.8 Mvals/sec
## Tests
vint4 store to unsigned char pointer got actual correctness checking
test, which seemingly was lacking before (only had a benchmark test).
Slightly different output to match for the combo of OCIO >= 2.2, but
simultaneously Python 2.7 -- which was not in our CI test matrix, so I
missed it, but ran into it at work.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Add a few more assertion checks and error messages in imagebufalgo_test.
I observe that if somebody runs the testsuite with `$OCIO` set to a
config that doesn't contain the required color space names for this
test, it will fail. As a backstop, use the default built-in config if
the initial request fails. That will always give a working processor
between these spaces if OCIO >= 2.2 is being used.
Also one more tiny SPI-specific enhancement to hacky identification of
nonconforming color space names in our old configs.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
- Fix perceptual test false positive by removing the initialization of
`cr` to avoid losing the information when leaving block scope
Included additional test with parameter `-fail 1`, which was giving a
false positive pass for a perceptual difference.
This was due to the fact that the result variable `cr` was getting out
of scope for the perceptual calculation and the first `if (cr.nfail <=
imagesize_t(allowfailures)) {` was testing for the result of the
absolute difference instead of the perceptual difference.
Before:
```
Comparing "img1.exr" and "img2.exr"
PASS
```
After:
```
Comparing "img1.exr" and "img2.exr"
64 x 64, 3 channels
Mean error = 0
RMS error = 0
Peak SNR = 0
Max error = 1 @ (5, 17, R) values are 0.1, 0.1, 0.1 vs 0.1, 0.6, 0.1
0 pixels (0%) over 0.008
121 pixels (2.95%) over 1
121 pixels (2.9541%) failed the perceptual test
FAILURE
```
Signed-off-by: Aura Munoz <aura.munoz@autodesk.com>
A recent change (PR 4026) tried to include Imath.h in cases where there
was no SSE support (to provide a fallback implementation of matrix
invert, because simd.h's native one requires SSE), but it had a mistake
-- it used `#if !OIIO_SIMD_SSE` prior to that symbol being defined! This
caused the header to always be included, which is not what we wanted, as
we desired for that header to not be exposed in the public headers.
This patch offers a temporary/partial fix by simply moving that guarded
include later in the file to after OIIO_SIMD_SSE is defined. For
architectures that support SSE, at least, this achieves our goal of not
exposing Imath.h in public headers.
I need this patch right away to solve a particular problem I'm running
into.
Longer term, we note that this isn't a full solution -- on non-SSE
architectures (like ARM), it still ends up exposing Imath.h against our
desire.
For today, so be it. A better fix would be to rewrite our current
SSE-requiring invert to change the SSE intrinsics therein to the wrapped
kind we use with our simd classes, which will let it work fine with ARM.
But I feel like that needs to be done carefully and with more testing
than I can afford at this minute when I need this fix. I will return to
it shortly, but need this patch now.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The important bug fix is in oiiotool image input when autocc is used,
where we reversed the sense of a test and did the autocc-upon-input if
the file's color space was equivalent to the scene_linear space
(pointlessly), and skipped the conversion if they were different (oh no,
big bug!).
Along the way:
* Add missing try/catch around OCIO call that should have had it.
* Some very ugly SPI-specific recognize-by-name color space heuristics.
I hate this, sorry, but it solves a really important problem for me.
* A bunch of additional debugging messages during color space inventory,
enabled only when debugging, so nobody should see these but me.
* A couple places where we canonicalize the spelling of
"oiio:ColorSpace".
Note that there is still an issue with the color space classification
using OCIO 2.3+'s identification of built-in equivalents by name. These
are shockingly expensive. I will return to this later.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Used to be no problem, but after a recent VSCode update, this seems to
have been causing errors when asking VSCode to format. No idea why this
recently became problematic.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
It's started failing on GHA and is old enough that it's not worth trying
to track down what's broken. We still have MacOS 12 and 13 tests in the
matrix.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Everything works, no code changes needed. But with some libtiff builds,
there is some variation in exactly which error messages bubble up when
reading certain corrupted files. This fixes some recently broken CI.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This is Update 6, the original PR is ongoing and we will continue to
update as additional people relicense.
Per PR #3905 (https://github.com/OpenImageIO/oiio/pull/3905), update the
RELICENSING document with the names of people and companies who have
signed on to the relicensing thus far, and update the SPDX license
notices on source files whose extant authorship (per 'git blame')
comprises only authors who have relicensed.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
When we bumped to cmake 3.15 minimum for 2.5, we changed lots of
constructs like this:
if (VERBOSE)
message (STATUS "blah")
endif ()
to this:
message (VERBOSE "blah")
but I didn't realize that this was not enough, and to do it right
requires CMAKE_MESSAGE_LOG_LEVEL. So even when VERBOSE=ON, we weren't
actually getting all the verbose messages. Here, we fix it.
Also, when verbose, print the list of C++ compile features that cmake
discovers.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
During certain creation tasks, open may be called twice on the
ImageInput. If the first open call moves the ioproxy forward, then the
second call to open may fail.
DDS and PSD were affected so they've been fixed to mirror what other
formats (png, hdr, tga, etc.) do by always explicitly seeking to
position 0.
Fixes#4043
Signed-off-by: Jesse Yurkovich <jesse.y@gmail.com>
It is a implemented by simply wrapping boost::physical_concurrency(). So
this is one more step in eliminating boost from our requirements.
Until it's fully removed, just make it be a synonym for
hardware_concurrency().
We don't seem to use physical_concurrency anywhere, and I'm not sure
what it's good for. The underlying boost impementation just falls back
to hardware_concurrency on many platforms or circumstances.
Note for the record that Sysutil::hardware_concurrency() is also
redundant, and merely wraps the std::thread function of the same name.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
To help with some other investigation I'm doing, amend imagebuf_test
with benchmarks to compare the relative speeds of traversing a big
ImageBuf with Iterator, ConstIterator, or just using pointers on an
equivalently large array.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This change adds a menu item "Move to new window" under the menu "File" which
creates a new window and initialises it with the currently displayed
image.
I have tested the UI changes on Mac and linux, I don't have a Windows
machine currently.
No other functionality should be affected by this change.
This was one item from issue #2058
---------
Signed-off-by: Anton Dukhovnikov <antond@wetafx.co.nz>
Takes the make_texture section of the imagebufalgo doc and turns it into
a unit test in both python and c++.
This adds a make_texture section to the `docs-examples-cpp` test and the
`docs-examples-python` test.
Signed-off-by: Danny Greenstein <danny.gr@gmail.com>
- ``raw:threshold``
- float
- Libraw parameter for noise reduction through wavelet denoising..
The best threshold should be somewhere between 100 and 1000.
- ``raw:fbdd_noiserd``
- int
- Controls FBDD noise reduction before demosaic. 0 - do not use FBDD
noise reduction, 1 - light FBDD reduction, 2 (and more) - full FBDD reduction
---------
Signed-off-by: ssh4net <libalias@gmail.com>
Revision of PR #4023
I have since determined that although OpenVDB raised their own build
requirements to C++17 starting with 10.0, it wasn't until 10.1 that
C++17-isms crept into their header files and force downstream projects
to need C++17 to consume OpenVDB's headers. So we can go back to
allowing use of OpenVDB 10.0 even when we are building with C++14.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
There's a new GitHub runner in town!
Set up CI to run on Mac ARM. But not all the time -- it's too expensive!
This runner is only available for paid accounts, and it's 20x more
expensive than paying for a Linux runner. The ASWF can't afford to run
this job on every iteration of every PR, and again upon merge.
We split it into a separate workflow so we can direct it to run only for
tagged releases, branches with names containng "-alpha", "-beta",
"release", or contain the substring "macarm" (this last one is so we can
force a particular PR that we think is directly relevant to ARM to test,
even though it's not one of those other release-related cases). We also
run it once a month, on the 27th -- a few days before each monthly
release -- to try to catch any ARM problems we've accidentally
introduced by PRs that didn't test.
It's also rigged to run only on the ASWF repo, so you don't need to
worry that you will get charged for pushes to your own fork. (On the
other hand, it also means you aren't testing against ARM when you push
to your fork prior to making a PR.)
In the process of testing this, of course I found some minor things that
needed to be fixed on ARM -- a couple compile issues, and several
testsuite reference outputs that needed to be updated because they
differ ever so slightly in some math calculations on the different CPU
architecture.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This is for master only! Will not be backported to any release branch.
We are assuming the 2.5 release family will be the last to support
C++14, and the next year's 2.6 family will raise the minimum to C++17.
We're not ready to throw that switch yet, but as an intermediate step,
let's raise the *default* level to C++17. It's still possible to compile
for C++14, but to do so, one must proactively `-DCMAKE_CXX_STANDARD=14`.
This has two benefits:
1. The majority of people who are already using a C++17-capable compiler
will take advantage of those features by default, instead of needlessly
restricting to C++14.
2. The people using a compiler that tops out at C++14 (e.g., gcc 6.3)
are going to have to set `-DCMAKE_CXX_STANDARD=14`, thus making them
aware that (in master anyway) they're swimming against the current and
have a limited time in which that will continue to work.
I hope that some time around the new year, we'll be able to fully raise
the minimum to 17? But it's going to depend on my place and other major
users finally retiring the DCC versions that require a gcc6/C++14
ecosystem, and we're not quite ready to do that at this moment. Soon,
soon.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Looks like OpenVDB >= 10.0 requires a minimum of C++17. Warn at build
time if trying to use an OpenVDB too new for the C++ compiler being
used.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We were using parallel_convert_image but not realizing that asking for
the IB::localpixels() would give us nullptr for an IC-backed image.
Instead, "wrap" the cv::Mat with an IB and then use IBA::copy(). This
handles both in-memory and IC-backed ImageBuf's, and also has many other
advantages (such as format conversion) we may wish to take advantage of
in the future.
Fixes#3800
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Convert C++ and Python examples from the "imagebufalgo" chapter into
tests within the "docs-examples" testsuites (#3992).
Signed-off-by: Jeremy Retailleau <jeremy.retailleau@gmail.com>
Solves the issue "Load a directory: iv dirname ought to be the same as
specifying all the image files in that directory" from the Great Big iv
Wish List #2058
This PR addresses the request from the Master list of iv feature
requests to allow loading a directory as an argument, which in turn
loads all images within the specified directory. The code modifications
include checks to validate the file or directory existence, and if a
directory is provided, iterates through its files to load all valid
images into iv. This enhancement simplifies the process of viewing
multiple images collectively located in a single directory.
---------
Signed-off-by: Chaitanya Sharma <39400946+CheeksTheGeek@users.noreply.github.com>
Teach `idiff` to accept a directory as the 2nd argument.
Fixes#4009
## Tests
Functional tests for `idiff` don't currently exist. Manual testing
verifies it works as advertised.
---------
Signed-off-by: David Aguilar <davvid@gmail.com>
Slightly different set of issues, so I wanted to provide a full example
for anyone who wants to work on this chapter.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Our exr writer uses some special hints starting with "openexr:" to
control aspects of the writing, and the reader sets these to
communicate some things about the file it's reading.
But we identified a case where the actual exr file had metadata named
"openexr:levelmode" -- which is of no particular meaning to openexr
itself, but it is the name we use to communicate hints to our writer
by make_texture. We just need to make sure that if we don't set it
proactivesly, we make sure to clear it if there is a stray bit of
metadata with that same name coincidentally.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
In the ongoing effort to help newcomers to OIIO diagnose their own
problems, this uses a trick to print any global error messages that
never got retrieved by OIIO::geterror() at the time that the program
exits. (This is all similar to a change we made recently that adds an
analogous behavior to ImageBuf, but in this case it's for those error
messages that don't have an object like an ImageBuf to attach to.)
This should have no effect on anybody who diligently checks
OIIO::has_error() or OIIO::geterror() after any failed calls that would
send error messages to that global store. Also, it is possible to
disable the behavior by setting the new global attribute
"oiio:print_uncaught_errors" to 0.
But for users who don't know to check errors and are experiencing
baffling behavior with OIIO, maybe seing those error messages will
prompt them to understand what's going wrong before they feel the need
to file an issue against OIIO or need to ask us for help.
Doing this helped me to find two places where our own code in OIIO
failed to clear the global error at times that it should, and these are
also fixed as part of this PR.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The following sequence would have a floating point exception:
// Readable IB backed by ImageCache
ImageBuf buf("file.exr"), 0, 0, ImageCache::create());
// Make a mutable iterator, even though it's an image file reference.
ImageBuf::Iterator<float> it(buf); <----- CRASHES INSIDE HERE!
The problem was in ImageBuf::IteratorBase::init_ib, several steps taken
if it's a mutable IB looking at a "read-only" IC-backed file, including
calling make_writable() on the buffer. But we neglected to set
m_localpixels to true.
Fixes#3770
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
A static_assert was added recently as part of PR #3898, which assures
that the size of a TileID is a multiple of 8 bytes. But on architectures
with 32 bit pointers, this assertion fails. Which is not quite what we
want, because the code isn't wrong if the struct isn't a multiple of 8
bytes; it's just less efficient. And we don't want to fail for 32 bit
arch.
Bottom line: only do this check to ensure we're in the most efficient
case if we're 64 bits. Nobody's expecting highest performance on a 32
bit machine anyway.
Fixes#4001
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Add a GHA workflow that is triggered up on drafting a release, which
will echo a notice to the ASWF slack #release-announcements channel.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
First, some background: In the course of working toward OIO 2.5, we
added some heuristics that used OpenColorIO 2.2's new
isColorSpaceLinear, and also using its new "built-in config" to figure
out if an arbitrarily named color spaces in a user's config were
equivalent to some built-in ones that are commonly used.
All seemed well until... we recently discovered that it was moderately
expensive (~0.1s per color space!) on the OCIO side, which wasn't
particularly noticeable when I was testing, but can add up to several
seconds of needless startup time when inventorying large configs with
dozens and dozens of color spaces. And even worse, it was exceptionally
expensive for color spaces with LUT3Ds because of a bug on the OCIO side
that would invert the 3D LUT needlessly (taking >10s per LUT3D
inversion), leaving to MINUTES of overhead for some configs with many
such spaces.
So this PR attempts to work around all of this with several big changes
(which are amplified by the fact that I've really shuffled around the
code between methods and within the file, sorry):
1. Per suggestion from Doug Walker, there's a new heuristic to try to
detect if a color space from or to the reference space appears to
contain a 3D LUT, and if so, skip some of the more expensive heuristics
that appear to cause the LUT to be inverted. It's not perfect, but we're
making a bet that no LUT3D-involving color spaces are going to be one of
the few canonical spaces we're trying to detect. This fixes the "10s of
seconds of delays" problem for those particular color spaces.
2. When using OCIO 2.3+, use its newly introduced utility
IdentifyBuiltinColorSpace to have it tell us which color space in the
user's config matches some of the canonical ones we care about in the
built-in config.
3. Whew, switch to a kind of lazy evaluation of the propererties of each
color space. Whereas before, we fully inventoried the config and for
each cs we found in it, do all the heuristics, now we only do the
inexpensive tests up front (silly rules of thumb where we guess based on
the *names* of the color spaces). We save the potentially more expensive
but thorough ones (like transforming some sentinel colors around to see
if the behavior of the cs matches a color space we care about, and
asking OCIO what it thinks about isColorSpaceLinear) only if we are
asked to do certain things with *that* color space. In other words, a cs
in the config that OIIO doesn't need to *use* in some way will now not
undergo the expense of the trickier investigation about what it might be
equivalent to.
These add up to making the overhead for ordinary use be significantly
sub-second, no longer a problem. Nature is healing.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
It was possible for `bit_range_convert` to be called with a 0 for the
`FROM_BITS` parameter leading to an infinite loop in its processing.
Fixes#3993: OIIO library freezes when loading R16G16 uncompressed DDS files
The test image from the above bug is now able to be read correctly.
Signed-off-by: Jesse Yurkovich <jesse.y@gmail.com>
For the upcoming DevDays, I wanted to spruce up our main README to make
it easier to find info, and I wrote a new guide for how to locally build
the documentation.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
libjpeg-turbo has exported cmake configs since its version 2.1. Relying
on this simplifies the build and allows us to completely remove the
troublesome Findlibjpeg-turbo.cmake. This implies raising the floor for
libjpeg-turbo support to 2.1 (the current release is 3.0) which was
released in April 2021 (beta in Nov 2020).
That's cutting it a little close (we usually support dependencies going
back at least 3 years at a minimum, and often significantly longer). But
since this is an optional dependency anyway and simplifies the build,
I'm tempted to do so.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The behavior of ImageBuf has always been that an IB referencing from a
file is backed by ImageCache unless you carefully use read() with the
"force" parameter set to true. The IC can be really helpful if you only
need *part* of the file, or if you have so many large IB's at once that
you want IC to manage the memory for you to keep the total reasonable.
Over the years, I've started to wonder if this was the right approach
for the default. It's much more typical that IBs aren't big enough or
numerous enough to gain much from the IC, and it's usually worse
performance to have it IC-backed. Plus, now the properties of the
ImageCache (size, behavior for nontiled images, etc.) can affect the
performane of ImageBuf without users understanding why.
So this patch tries on a different approach: ImageBuf's only use an
ImageCache underneath if explicitly passed a pointer to one in the IB
constructor or reset() call. Otherwise, it just uses local memory owned
by the IB. It's much simpler to reason about, and should be higher perf
for most common cases.
I did add a global OIIO option, "imagebuf:use_imagecache" that can
revert to the old behavior of having all IB's by default be backed by IC
when reading from a file.
Changed oiiotool to match: By default, IB's own their local memory.
Using the --cache option (which previously existed to set the size of
the cache) also enables use of the cache, so it's still possible to use
IC-backed IBs if the user needs it.
For the reader of this PR: What do you think? Is this a change you'd
like to see? Is it something we should try to squeeze into 2.5? Or are
we not sure enough about whether it's the right thing that we should not
be merging it close to a major release?
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We found that PNG output (which is required to be unassociated alpha)
had precision problems with low alpha values because it quantized to the
final bit depth before doing the alpha deassociation.
Explanation:
PNG requires associated alpha to be stored in the file. So if you have
(unassociated) pixel value [0.00235, 0.00106, 0.00117, 0.0025], that
means that we need to divide the colors by alpha before storing in the
file, so that's [0.94, 0.424, 0.468, 0.0025] if you have full precision.
But you don't, because PNG only supports uint8 and uint16. For uint8,
this should end up as [240, 108, 119, 1].
But it turns out that our png writer does the integer conversion before
the un-premultiplication, so actually we first quantize from [0.00235,
0.00106, 0.00117, 0.0025] to [1 0 0 1], and THEN we un-premultiply,
giving us the [255, 0, 0, 1] that made you think "Why is that red value
clipped? And where did green and blue go?"
The solution, then, is to deassociate as floats first, then quantize to
the final integer data type.
Also add a test for this, as well as a test that verifies correctly
writing of 16 bit files that I had some problems with in the last png
PR.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
I had previously prototyped an instance of this for the ImageOutput
chapter.
This patch makes empty placeholders for all of the other API chapters.
They don't contain any actual tests or generate any output now, but they
do build and run, so people can drop new examples into them without
having to worry about how to set up new files for each chapter.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
`oiiotool --mosaic` was always allocating a `float` buffer for the
result. Instead, make it use the "widest" data type of the input images.
So, for example, if you are making a mosaic of uint8 images, the result
buffer should be uint8, not have to allocate 4x bigger nor do
uint8->float conversions as it pastes the images.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This is Update 5, the original PR is ongoing and we will continue to
update as additional people relicense.
Per PR #3905 (https://github.com/OpenImageIO/oiio/pull/3905), update the
RELICENSING document with the names of people and companies who have
signed on to the relicensing thus far, and update the SPDX license
notices on source files whose extant authorship (per 'git blame')
comprises only authors who have relicensed.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This patch adds a new test to the CI matrix: verifying that we do not
break the ABI during releases.
Basically, in addition to building the current checkout, it also builds
a checkout of the designated "ABI standard" (generally, the first
non-beta release of a major version), then generates ABI dumps of both
versions and runs the ABI compliance checker using the tools from
https://github.com/lvc
Making two builds sounds expensive, but both builds are rigged to be
minimal, since only the main two libraries are needed for ABI checks (we
can turn off the binary tools, unit tests, python bindings, and support
for many formats), and of course the test suite need not run. So
actually, it's less expensive than most of the test matrix entries, and
it helps ensure that we don't accidentally break ABI when we don't mean
to.
I'm not sure how many false positives we'll have. We might need to
adjust the methodology slightly as we gain experience with this.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
When writing out EXIF headers in PNG files, the `tiff_version` field
wasn't handling endianness properly.
This lead to some tools like `exiv2` refusing to process the file at
all[1]
[1] https://github.com/Exiv2/exiv2/issues/2745
---------
Signed-off-by: Jesse Yurkovich <jesse.y@gmail.com>
* Implement PNGOutput::write_scanlines, previously we only had
write_scanline. Sadly, being able to write multiple scanlines at once
reduces overhead by much less than I had hoped. But there's a bit of
savings, so I'll keep it.
* New compression recognized: "pngfast", which translates to the
Z_BEST_SPEED setting. I don't particularly recommend this, but it's
useful for benchmarking.
* New compression recognized: "none", which turns off compression. Also
not recommended, as it makes the files much larger. Mostly done for
benchmarking and other comparisons.
* Don't unconditionally copy user's data buffer to m_scratch. It's only
necessary if there's a data type conversion, stride shuffling, alpha
deassociation, or endian swapping. When none of those are needed, we can
avoid the extra allocatin and copy, and so now we do. (Sigh, in real
use, we're almost always doing the alpha deassociation, so this rarely
is a savings.)
* Better documentation on the custom output attribute hint
`"png:filter"` noting that a non-default value can make it write PNG
files dramatically faster, but with the tradeoff of having much larger
files.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
New theory of code examples in the docs: every one should live in the
testsuite somewhere, and merely be included by reference in the docs.
This ensures that every code example works, stays current with the
evolution of the APIs, and never breaks. It also beefs up our test cases
for more thorough code overage in the testsuite.
I've done this with just two examples here, but hopefully it serves as
an example for others to -- bit by bit -- convert all the other code
examples in the docs into tests in a similar manner.
Doing this for any one example is a perfect "good first issue," since
it's bite sized and can easily be done in a day, it doesn't require any
knowledge of deep OIIO internals, and in the process it also teaches you
something about how both the docs and the testsuite are set up.
A few notes:
* I'm structuring it (for now?) as two testsuite entry for each chapter
of the documentation (one for C++ and one for Python).
* Note how I can have many code snippets in a test source file, and use
the Sphinx `:start-after:` and `:end-before:` modifiers of
`literalinclude` to clip particular subsections of the file, using
special comment markers in the code.
While I was there, I also fixed some typos I noticed in the text of the
chapter, oops.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
To my eyes, these two constructors look like they should be treated as
doing exactly the same thing:
MyType()
: mymember(...)
{ }
and
MyType()
{
mymember = ...;
}
But nvcc in Cuda mode says no, if the constructor is constexpr, the
former is okey dokey, but latter gives a warning. Fine, Cuda, whatver
you say.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
While testing out the 2.5 beta for use in Blender, I hit an odd build
error[1] apparently related to include order. Apparently `windows.h`
should be included before `shellapi.h`[2]. Swapping the order does fix
things, however, it turns out neither is used in this particular file so
just remove them both. Unsure why we didn't see it until now.
[1]
```
20>[9/146] Building CXX object src\libutil\CMakeFiles\OpenImageIO_Util.dir\strutil.cpp.obj
20>FAILED: src/libutil/CMakeFiles/OpenImageIO_Util.dir/strutil.cpp.obj
20>cl : Command line warning D9002: ignoring unknown option '/arch:sse2'
20>C:\Program Files (x86)\Windows Kits\10\Include\10.0.22621.0\um\shellapi.h(68): error C2146: syntax error: missing ';' before identifier 'DECLSPEC_IMPORT'
20>C:\Program Files (x86)\Windows Kits\10\Include\10.0.22621.0\um\shellapi.h(79): error C2065: 'HDROP': undeclared identifier
20>C:\Program Files (x86)\Windows Kits\10\Include\10.0.22621.0\um\shellapi.h(82): error C2086: 'int EXTERN_C': redefinition
20>C:\Program Files (x86)\Windows Kits\10\Include\10.0.22621.0\um\shellapi.h(68): note: see declaration of 'EXTERN_C'
```
[2]
https://stackoverflow.com/questions/2953704/vs2010-lots-of-errors-when-including-standard-librarieshttps://social.msdn.microsoft.com/forums/vstudio/en-us/ed93ae48-969a-4280-8fd5-ef57a332022b/problems-with-shellexecute-and-shellapih?forum=vcgeneral
## Checklist:
<!-- Put an 'x' in the boxes as you complete the checklist items -->
- [X] I have read the [contribution
guidelines](https://github.com/OpenImageIO/oiio/blob/master/CONTRIBUTING.md).
- [ ] I have updated the documentation, if applicable.
- [ ] I have ensured that the change is tested somewhere in the
testsuite
(adding new test cases if necessary).
- [ ] If I added or modified a C++ API call, I have also amended the
corresponding Python bindings (and if altering ImageBufAlgo functions,
also
exposed the new functionality as oiiotool options).
- [X] My code follows the prevailing code style of this project. If I
haven't
already run clang-format before submitting, I definitely will look at
the CI
test that runs clang-format and fix anything that it highlights as being
nonconforming.
Signed-off-by: Jesse Yurkovich <jesse.y@gmail.com>
Bump fmt minimum to 7.0 for OIIO 2.5. fmt 7.0 is 3 years old, so let's
stop doing extra work to support 6.x. It's not that we depend on newer
features per se, but I'd like to no longer need separate reference
output for some tests whose formatting behavior changed slightly between
6.x and 7.0.
Rig CI to properly test "latest" and "bleeding edge" fmt. It turns out
we had not yet been testing 10.1 or keeping up with their master.
In the process, I discovered that the combo of fmt >= 10.1 and gcc >= 11
results in a mangled AVX math heisenbug, symptomatic in our simd_test. I
struggled with this on and off for many days, having great difficulty
reproducing (though it does fail every time and deterministically, I
just can't seem to narrow it to a smaller example). It only happens with
gcc, only gcc >= 11, only when we use FMT_EXCEPTIONS=0 to disable true
exceptions in fmt. The change happens specifically at fmt commit 9a034b0
(midway between 10.0 and 10.1), which changes the definition of
FMT_THROW when exceptions are disabled. Why this should affect SIMD math
is a total mystery, and currently I am suspecting either a gcc compiler
error or that it's exposing undefined behavior. I found that redefining
FMT_THROW on our side to something innocuous is a good workaround. If I
no longer disable fmt exceptions, that also makes it work, but I don't
know what that might break for us or downstream.
My report of this to the fmt project is
[here](https://github.com/fmtlib/fmt/issues/3620), maybe they'll uncover
the true cause.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Advanced users know to check error returns and retrieve errors.
New users often just try OIIO calls with no error checks when they don't
get behavior they want, ask questions to the dev team. We hypothesize
that if they only saw the text of the error messages that they never
retrieved, they could probably self-diagnose many of their problems.
This patch makes an ImageBuf, when it is destryed, print any pending
error messages that were never retrieved. It happens by default, but can
be disabled by setting the new global attribute
"imagebuf:print_uncaught_errors" to 0 (it defaults to 1).
This doesn't cover everything, but for ImageBuf and ImageBufAlgo, maybe
it will help some users? It could annoy advanced users who intend to not
retrieve error messages (why?) but still don't want this printed. But
that's who the new attribute is for.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Typo introduced in #3894 broke our ability to build against libheif <
1.13. A conditional compilation that should have kicked in for >=1.16
was mistakenly written as 1.6.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Add documentation for oiiotool --normalize
* Add tests for oiiotool --normalize
* Revamp existing normalize tests in testsuite/python-imagebofalgo: Get
rid of old test images and refs, because their values were already
normalized, so was probably not testing the math. Make new tests that
are much smaller, 3x3 images that contain specific (non-normalized)
values to verify they all get normalized corectly. They'll run a lot
faster, too.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Not unlike the ImageOutput::open_check we added a few months back, this
creates an ImageInput utility function that centralizes a number of
validation checks. Currently, this is primarily a check on allowed
resolution and total size, aimed at making a guess at files that are
corrupted or malicious.
Also adds a new `supports("noimage")` query.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This is Update 4, the original PR is ongoing and we will continue to
update as additional people relicense.
Per PR #3905 (https://github.com/OpenImageIO/oiio/pull/3905), update the
RELICENSING document with the names of people and companies who have
signed on to the relicensing thus far, and update the SPDX license
notices on source files whose extant authorship (per 'git blame')
comprises only authors who have relicensed.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Proposal from Thiago to add Jesse Yurkovich, representing Blender, to
the TSC.
Vote passed by majority of TSC members voting yes on the PR itself.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Avoid possible double define of NOMINMAX in platform.h
* Move other Windows symbol definitions into the cmake file instead of
repeating them in certain .cpp files.
Fixes#3964
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Normalize 3D vectors (i.e., divide by their length) textures.
inCenter and outCenter define 0.0 coordinate in input and output
3D vectors encodings (e.g. 0.0f for [-1,1] or 0.5f for [0,1] encoding).
scale defines the scale factor to apply to the normalized vectors
(i.e. 1.0f for a full range normalization, 0.5f for 0.0-1.0 range
encoding).
Default values are for [-1,1] encoding and full range normalization.
Functions required 3 or 4 channels. If destination have no alpha
channel,
but source have, alpha channel will be dropped.
## Tests
```
int main(int argc, const char* argv[])
{
ImageBuf input(argv[1]);
ImageBuf output;
bool ok = ImageBufAlgo::normalize(output, input, 0.0f, 0.0f, 1.0f);
if (ok)
{
ok = output.write(argv[2]);
}
else {
std::cerr << "Could not normalize image: " << argv[1] << std::endl;
return -1;
}
return 0;
}
```
---------
Signed-off-by: ssh4net <libalias@gmail.com>
Signed-off-by: Anton Dukhovnikov <antond@wetafx.co.nz>
Signed-off-by: Vlad Erium <libalias@gmail.com>
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Co-authored-by: Anton Dukhovnikov <131838425+antond-weta@users.noreply.github.com>
Co-authored-by: Larry Gritz <lg@larrygritz.com>
Fixes#3950
Fix the div-by-0 when handling unassociated alpha during a DXT4 DDS
load. The code mirrors what other formats do by first checking to see if
the alpha value is not 0 before continuing.
Added the 218-byte file from the bug to the local test suite.
---------
Signed-off-by: Jesse Yurkovich <jesse.y@gmail.com>
Enable set of Boost_NO_BOOST_CMAKE through make using
MY_CMAKE_FLAGS=-DBoost_NO_BOOST_CMAKE:BOOL=OFF.
Signed-off-by: Mikael Sundell <mikael.sundell@gmail.com>
The incoming length of the RLE data could be 0. In this case we would
attempt to take the address of the 1st element of an empty vector.
Added a new 1004-byte file to the local test suite.
Fixes#3951
---------
Signed-off-by: Jesse Yurkovich <jesse.y@gmail.com>
We had disabled the use of the "openexr core" library by default, even
for OpenEXR versions that nominally support it, when it was discovered
that it stil didn't support the DWA compression types.
That has finally been added to the OpenEXR trunk as well as the OpenEXR
3.1.10 release. When we are building against a sufficiently new OpenEXR
to fully support all the compression types for exr core, then re-enable
it by default.
Bump the "latest" CI tests to OpenEXR 3.1.11.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Fixes#3952
Several interrelated fixes to guard against corrupted files:
* Extend the `limits:imagesize_MB` sanity check to Targa files to
address #3952 specifically, and check in that failing example file into
the testsuite. (N.B.: Only a couple readers use this, TIFF and now
Targa. it should be extended to the others. Mackerel!)
* Improve the error message used for the above, both for Targa and TIFF.
* For low-memory situations, perhaps the default of 32GB is too large.
Make the default be the minimum of 32GB and 1/2 the size of physical
memory on the machine. (An app that legit expects humongous files is
expected to raise the limit appropriately.)
* The generic "Read error: hit end of file" we sometimes issued from
`ImageInput::ioread()` was hard to discern which reader it came from, in
cases where the file extension was absent or misleading and a succession
of files were tried. By changing this error message to say which format
reader triggered the error, it might make these errors easier to track
down in the future. This also changed the reference output of a few test
involving truncated or corrupted files.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
On overloaded VMs, the timer_test is ureliable and fails spuriously too
often.
Two strategies in this patch, hope some combination does the trick:
1. Increase the absolute error allowed for the timing tests when doing
CI runs.
2. Add just a bit more relative error (via a new unit test macro,
OIIO_CHECK_EQUAL_THRESH_REL).
Hopefully this will cut down on CI failures for unit_timer test case.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Disable GCC warnings around mismatched new/delete for shared_texturesys
creation
Signed-off-by: Shootfast <markboo99@gmail.com>
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Co-authored-by: Larry Gritz <lg@larrygritz.com>
* Add string literal operators for ustring and ustringhash so you can
write: ``` "foo"_us "bar"_ush ``` Note that on Cuda (but not on the
host), this ustringhash construction is constexpr.
* Add a new `ustringhash(const char*, size_t)` constructor, which is
also constexpr on the device side.
* Mark the existing `ustringhash(string_view)` constructor as constexpr
on the device side.
We can mark those things as constexpr on the device side because on the
Cuda side, the implementation only takes the hash (which is constexpr),
and does not actually add a ustring to the table (which cannot be
constexpr). To make such declarations easy, add a new macro to
platform.h, `OIIO_DEVICE_CONSTEXPR` which is constexpr for the device,
merely inline for the host.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The most important fix is to remove links to the old CLAs and update
references to the mail list to the new one.
Also spruced up some language while I was there, hopefully to coax out
improved PRs and Issues.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
When building the code with Xcode ("cmake -G Xcode"), there are 1000+
warnings about implicit 64 to 32 bit conversion.
This change adds a compile option to suppress such warnings when
generating an Xcode project.
Signed-off-by: Anton Dukhovnikov <antond@wetafx.co.nz>
This is Update 3, the original PR is ongoing and we will continue to
update as additional people relicense.
Per PR #3905 (https://github.com/OpenImageIO/oiio/pull/3905), update the
RELICENSING document with the names of people and companies who have
signed on to the relicensing thus far, and update the SPDX license
notices on source files whose extant authorship (per 'git blame')
comprises only authors who have relicensed.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Text rendering doesn't work with utf-8...
I can't render chinese, arabic... or special chars as éèûîà...
It seems that the string function to convert utf-8 to unicode is quiet
simple.
Fixes#3933
[Note]: OpenImageIO is used in the [gpx2video
project](https://github.com/progweb/gpx2video).
---------
Signed-off-by: Nicolas VIVIEN <nicolas@vivien.fr>
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Co-authored-by: Larry Gritz <lg@larrygritz.com>
I think it will be handy to be able to have ParamValue contain
ustringhashes and have a TypeDesc that can describe it explicitly.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Rationale:
* LibRaw 0.18 was released in last 2016, so we are stil supporting all
libraw from up to 6 years back.
* LibRaw 0.18 is necessary for ACES, plus it improves a lot of metadata
handling.
* LibRaw 0.20 is the minimum that can be used when compiling with C++17,
and we will almost certainly move to C++17 as a minimum in OIIO 2.6, so
it seems like LibRaw 0.18 is reasonable minimum for the upcmoming 2.5.
Remove some cruft that's no longer necessary if we drop support for
0.15-0.17.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This bump brings access to the following interesting features (not a
comprehensive list):
- 3.13 Nov 21, 2018
- -B, -S let you specify source and build directories without the "cd" dance
- target_link_options()
- ctest --progress / CTEST_PROGRESS_OUTPUT
- 3.14 Mar 14, 2019
- cmake --build now supports -v/--verbose
- file(CREATE_LINK )
- FindGIF now has imported targets
- 3.15 Jul 17, 2019
- env CMAKE_GENERATOR can be used instead of -G
- when using --build, the --target can now list multiple targets (and also can be `-t`)
- --install can be used separately without running through the whole build
- message() now supports NOTICE, VERBOSE, DEBUG and TRACE.
- CMAKE_FIND_PACKAGE_PREFER_CONFIG variable
- Enable C++20 for Apple Clang
Make sure all CI cases have a new enough cmake
Use GIF targets, now enabled with CMake 3.14
Use message(VERBOSE ...) feature to simplify lots of `if` blocks.
Clean up code for older cmake versions
Note that this will NOT be backported to the current release branch,
but it will be part of the upcoming 2.5 family of reseases.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
When an incoming 1-dimensional, buffer-like, object was used in Python,
the corresponding check inside OIIO would overflow if its length
exceeded a signed 32bit value.
An error similar to `Pixel data array error: Python array shape is
[2304000000] but expecting h=24000, w=24000, ch=4` would be raised.
I modified a few similar calculations and tried to follow the
predominant pattern already used.
Tested using the following snippet which would previously fail:
```
import OpenImageIO as oiio
import numpy as np
filename = "t:/foo.png"
xres = 24000
yres = 24000
channels = 4 # RGB
pixels = np.zeros(xres * yres * channels, dtype=np.uint8)
spec = oiio.ImageSpec(xres, yres, channels, "uint8")
image = oiio.ImageOutput.create(filename)
image.open(filename, spec)
if not image.write_image(pixels):
print("error = ", image.geterror())
image.close()
```
---------
Signed-off-by: Jesse Yurkovich <jesse.y@gmail.com>
Per PR #3905 (https://github.com/OpenImageIO/oiio/pull/3905), update the
RELICENSING document with the names of people and companies who have
signed on to the relicensing thus far, and update the SPDX license
notices on source files whose extant authorship (per 'git blame')
comprises only authors who have relicensed.
This is part 2, the original PR is ongoing and we will continue to
update as additional people relicense.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Intel's servers have again updated the default icx that you get when you
install from their yum repository, and the new one seems incompatible
with our libstdc++ toolchain. So lock down on 2023.1.0 for now so that
our CI can pass.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Optimises the code in the tile hash function. During a render this
function is called billions of times so this small improvement adds up,
in some test scenes it reduces total render time by an average of about
2%.
See https://godbolt.org/z/jMzz4PTce for the exact details. A summary
would be:
- Reduced instruction count from 75 to 61, this is due to the redundant
add and shift operations that can be implicitely handled by taking
advantage of the member layout in the class.
- Halves the number of memory fetch instructions because all accesses
are now 64-bit.
- Removes branching instructions and extra memory indirection inside
`ustring::hash()`.
There is a slight change in behaviour, which is that the hash is now
computed from the `m_file` pointer rather than from
`m_file->filename().hash()`, so runtime reproducibility is lost. However
the performance gain seems worth it.
Signed-off-by: Curtis Black <curtis.w.black@gmail.com>
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Co-authored-by: Larry Gritz <lg@larrygritz.com>
This updates ICOInput to use the IOProxy model so it can read from files
or the IOMemReader
Signed-off-by: Jason Baumeister <jason@jasonbaumeister.com>
During the CMYK to RGB conversion in PSDInput, the alpha channel was
never copied to the RGBA output. Converting a PSD to JPEG worked because
it ignored all the 0 alpha values, but converting to WebP would result
in a fully transparent image.
I added a cmyk-with-alpha.psd test to the psd-colormodes test to check
this
Signed-off-by: Jason Baumeister <jason@jasonbaumeister.com>
Clean up the `#if` tests that disable Intel SIMD when not on Intel
architectures, which incorrectly disabled ARM NEON when on ARM.
Fixes#3909
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Update the ImageInput documentation to not use the deprecated
`read_image` APIs.
Based on the discussion in #3902
---------
Signed-off-by: Jesse Yurkovich <jesse.y@gmail.com>
Per PR #3905 (https://github.com/OpenImageIO/oiio/pull/3905), update the
RELICENSING document with the names of people and companies who have
signed on to the relicensing thus far, and update the SPDX license
notices on source files whose extant authorship (per 'git blame')
comprises only authors who have relicensed.
This is part 1, the original PR is ongoing and we will continue to
update as additional people relicense.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Our own FindWebP.cmake was setting up targets that didn't
match the capitalization of the cmake configs that come with WebP itself.
Signed-off-by: Benjamin Buch <benjamin.buch@technoteam.de>
This batch is for files that LG (relicensed!) has contributed, in
addition to >2 other authors. Also add appropriate license notices to
some that had no posted license before due to oversight.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Add copyright/license notices to some mixed-authorship files that
lacked them before (all related to tests)
* Get rid of the pointless year in many copyright notices (change to the
current LF recommended wording)
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Since LG has re-licensed all his old code under Apache-2.0, we can
change the license notice from BSD to Apache-2.0 for all files in which
LG is the only extant author. Also change all files that LG co-authored
with other people to specify "BSD-3-Clause and Apache-2.0".
Some files (mostly test scripts) had no license notice, so in those
cases, we added one.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
By majority vote of the TSC, we are changing OpenImageIO's main
incoming/outgoing open source license from BSD-3-Clause to Apache-2.0.
New code submitted from July 1, 2023 will be contributed under the
Apache 2.0 license.
Old code submitted prior to July 1 2023 is under the original BSD
3-clause license, unless relicensed per the process described in the new
RELICENSING.md file. We hope to get the vast majority of the code base
relicensed quickly and we will document which set of files contain a
mixture of licenses.
Instructions to everybody reading this:
* If you are contributing code under the authorization of an employer,
please let them know we are changing to Apache 2.0 and make sure that
you are authorized to make all future contributions to under that
license.
* If you made prior contributions to OpenImageIO over the years and are
willing to relicense your code under the Apache 2.0 license, please
submit a PR adding your name to the list in RELICENSING.md. If you were
working for a company that might own your IP output at the time you made
those contributions, please also ask somebody there (who has the
authority to do so) to submit a PR adding the company name as well,
indicating that they are re-submitting all of the old contributions
under the new license.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
By unanimous vote of the OpenImageIO TSC, as of July 1, 2023, we
adopt the technical charter, Code of Conduct, new CLAs, and GOVERNANCE
document.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The underlying farmhash algorithm (which is the basis for
Strutil::strhash, and thus ustring's hash representation) turned out to
not be properly constexpr under certain circumstances and needed to be
expressed a different way to be fully constexpr on all platforms and for
all strings.
New implementation suggested by Alex Wells.
Also, make sure string_view ctr from char* can be constexpr, even for
C++14. This is related for the sake of the strhash calls that, when
taking string literals, actually choose the variety that accepts a
string_view.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
repology has a handy tool that will show all the package distros
and which version of openimageio they supply.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Remove leftover logic from when we used TravisCI.
* Pick a default for LLVM_DISTRO_NAME so it works when the script is run
standalone when not on GitHub Actions CI. (But it's still really only
meant to run on x86 Ubuntu, and not designed for anything useful outside
of our CI.)
* Eliminate all the complexity with old LLVM versions -- in practice,
our CI only uses this to download fairly modern clang versions.
Fixes#3891
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The strukturag repo for libheif recently upgraded the version it was
serving from 1.14 to 1.16, and that caused our tests to fail because (a)
some of the codec support has been moved to plugins that are loaded at
runtime and those are now distributed in packages we need to install
separately; (b) you must call heif_init() or it won't know about the
optional plugins.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
If colorconvert, ociolook, ociodisplay, ociofiletransform fail
because OCIO support is not enabled, make sure the error message
makes that clear.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Make sure we can input exif tags as strings containing numbers when
number are required.
E.G. if my input is "Orientation"=>"8", i would like the jpeg output to
contains the value 8 for the exif tag orientation.
PR #2147, addressing issue #1844, is related to an apparent OpenEXR bug
wherein single channel tiled images where the tile dimensions were not a
power of 2 and >= 16 would not work properly with dwa compression. We
work around the problem by automatically substituting zip compression in
those cases. But it was implemented with a subtle bug, forgetting that
tile_size == 0 for untiled images, and thus we were silently switching
from dwa to zip compression for all *untiled* single channel images of
any size (which presumably were never problematic). This patch fixes the
test so we are not changing the compression for the untiled files.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
J2K output was defaulting to UINT8 when asked to output an unsupported
type. It should have been UINT16 to try to minimize the amount of
precision loss.
Fixes#3876
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Everybody please go to https://lists.aswf.io/g/oiio-dev and sign up for
the new mail list as soon as possible. The old list is on its last legs
and having trouble, let's get this done.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This change adds a `shutdown()` method to the API that can be used to
perform shutdown of the thread-pool(s) (or other clean-up activities).
Applications that utilize OpenImageIO are advised to call it before exiting their
main function to prevent possible deadlocks during application shutdown
(see #3851)
This change also updates the documentation and all oiio-tools to call
the new `shutdown` method.
This is the final change for fixing #3851
Part of solving #3851.
When decreasing the number of threads in `thread_pool::Impl::resize` it
was possible for the threads marked for termination to still be fully
alive at process exit time if the call to `resize` and the exit of the
main function were close enough to each other.
This change makes `resize` wait for the threads to actually join
guaranteeing the threads have been shut down in a orderly manner when
the `resize` function exits.
This cleanly terminates all worker threads of the default thread pool
by resizing the pool to 0 threads... but only if the pool has been
created in the first place. Note that the alternate idiom
default_thread_pool()->resize(0);
seems like it should accomplish the same thing, but in fact will start
by creating a pool and its threads, if it has not done so before, only
to immediately terminate them.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
When reading a PSD file, if it contains a thumbnail, we read it from the
in-memory blob with a combination of an IOMemReader and an ImageBuf.
But... we always named it "thumbnail.jpg" without considering that
multiple simultaneous PSD files reading identically named images (but
which are in fact different) could cause a weird kind of clashing in any
underlying use of ImageCache!
The simple fix is to use ordinary ImageInput approach to reading the
thumbnail from the IOMemReader, but just store it in the ImageBuf.
Fixes#3824
Signed-off-by: Larry Gritz <lg@larrygritz.com>
I got the test backwards, and it was incorrectly hiding the definition
for the default case of OIIO_DISABLE_DEPRECATED == 0.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
A variety of minor documentation fixups:
- Use Python 3 style 'print' statements consistently everywhere (some
snippets had a mix of both)
- Correct the usage of `ImageSpec::channels` that should be
`ImageSpec::nchannels`
- Template syntax errors in two C++ snippets
- Document the new default for `max_memory_MB`
Background info: oiiotool multithreads *within* any individual image
operation (such as `--over`), but a series of operations comprising the
oiiotool command are done serially -- each command finishes before the
next starts. And similarly, for multi-frame operations using frame
ranges, frame number wildcards, or view wildcards, the set of commands
for each frame runs to completion before the next begins. For complex
oiiotool commands, especially when running over frame ranges, this is
not a particularly efficient way of parallelizing, since any serialized
operation (which often includes the expensive writing of the result
image from each frame to disk) can substantially serialize the entire
run.
This PR adds a new oiiotool --parallel-frames option that assumes all
frames in the range can be computed independently and in any order, and
can therefore be run in parallel. As long as there are a similar number
(or more) of frames versus cores, and you have enough RAM to handle that
many image pipelines at once, this is a more efficient way of
parallelizing the operations than multithreading individual image
processing operations.
Benchmarking: On my 32-core workstation, I tried the following oiiotool
command, contrived but representative of a reasonable oiiotool command
complexity that might be a production task:
oiiotool --runstats -v --frames 1-100 tmp/noise.exr -cut
2880x1620+960+540 -resize 1920x1080 -colorconvert lin_srgb srgb -d uint8
-o "tmp/test.#.tif"
The noise.exr file was a previously created 4k x 3channel 'half' exr
image. The command loads the file, cuts out a 3k section of it, resizes
that to HD res, does a color space transformation, then outputs as an 8
bit TIFF file, for each of 100 sequential frames.
threads real time user sys peak memory
1 4:44 4:23 0:21 122 MB
2 2:58 4:24 0:29 122 MB
4 1:40 4:36 0:29 122 MB
8 1:10 5:00 0:30 127 MB
16 0:56 5:28 0:37 131 MB
all(32) 0:55 8:12 0:54 127 MB
parallel 0:25 8:13 1:04 3.4 GB
Here we can see the diminishing returns of using multiple threads within
each frame, where for low thread counts we are improving steadily, but
we pretty much peak at 16 threads, seeing virtually no further gain at
32, and maxing out at about 5x the single threaded performance. This is
due to serialization of the I/O and increasingly bumping into various
locking at higher thread levels.
However, using `--parallel-frames` gives us another doubling of total
throughput (giving us a total of 11.3x speedup versus single threaded),
though at the expense of 28x the memory use! That memory explosion is
acceptable in this case, even 3.4 GB is really nothing on a modern
machine, but we can imagine that other possible oiiotool command lines
might be more memory intensive and having dozens of frame pipelines
computing simultaneously would have problematic memory use. So use this
feature carefully and with purpose, always checking that your own
workflows experience sufficient speedup and don't use more memory than
you have available.
There's some code I put in not long ago to automatically call exr's
thread pool setNumThreads(0) upon exit, as an attempt to work around
some exr thread pool deadlock conditions. But... we've seen some
weirdness from this workaround as well, so try to limit its application
by not bothering when using OpenEXR >= 3.1.8 (at which point we believe
the exr side bug was fixed) and also to do so only on Windows.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Various improvements to OpenCV support:
* IBA::from_OpenCV and to_OpenCV now set error messages for every type
of failure they encounter, instead of returning false with no
explanation. For from_OpenCV, the error message can be retrieved from
the ImageBuf returned; for to_OpenCV, the error message can be retrieved
from the global OIIO::geterror(). Several assertions were removed,
instead relying on error return codes and messages.
* Add global OIIO attribute "opencv_version" which allows you to find
out what version of OpenCV OIIO was built against, which was previously
impossible to know without seeing the build-time status messages. This
can be retrieved via
OIIO::get_int_attribute("opencv_version")
and also you can find out from the command line with
oiiotool --echo "{getattribute(opencv_version)}"
The return value is an integer encoded as `10000*major + 100*minor +
patch`, and it will return 0 if no OpenCV support was enabled when
OpenImageIO was built.
* For OpenCV >= 4.0, to_OpenCV can now correctly construct a 'half'
matrix. For OpenCV 3.x, we turn it into a float matrix as before, since
CV_16F was only supported starting in 4.0.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Just a routine update of these files as I proofread them in light of the
move to ASWF. This patch is addressing the "old" project, there will be
further ASWF-specific edits for changes that will take effect after the
project is fully moved.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
In `jpegoutput.cpp` when computing the density fields the relationship
between `XRes`, `YRes` and `aspect` is `aspect = XRes / YRes`.
However this relationship was not respected in one specific case, when
`aspect >= 1.0` and `XRes` and `YRes` are not set.
This PR solves this problem by enforcing the correct relationship.
Related to: https://github.com/OpenImageIO/oiio/issues/2822
The Intel yum repo renamed this package a couple weeks ago, and it broke
our CI. Update to the new name.
Adjust fmt use to solve icc warning.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
When reading a raw image, setting the `user_flip` to 1 meant ignoring
the `flip` value (to avoid buffer rotation from LibRaw).
However the value 1 for the `user_flip` is **not neutral**: values lie
between 0 and 7 (see
https://www.libraw.org/docs/API-datastruct-eng.html#libraw_image_sizes_t),
and 1 corresponds to a horizontal mirror (in my understanding of the
source code).
The proposed fix is to use 0 as `user_flip` value for ignoring the
`flip` value.
This modification invalidates the changes made in this PR:
https://github.com/OpenImageIO/oiio/pull/3847.
Therefore, flip to orientation conversion has been adapted to match
exactly LibRaw's documentation, which handles only rotations (+90, +180,
-90 degrees).
The previous PR was only tested on images of landscape and human faces,
hence the mirroring issue remained unnoticed.
When reading images containing text however the problem is clearly
visible.
This PR fixes this mirroring issue on the following 4 orientations: 0,
+90, +180 and -90 degrees rotation.
Other orientations could not be tested as they don't seem to ever appear
on raw images datasets.
Make all oiiotool actions take ot as a parameter.
This is a necessary first step to being able to parallelize over frame
ranges.
Also finally convert the last of the `int argc, const char**argv` idioms
to `cspan<const char*>`.
When reading a raw image while having a `user_flip` set to 1, we set the
orientation Exif tag to LibRaw's flip value (so that LibRaw doesn't
rotate and/or mirror the buffer but the image still gets displayed
correctly).
However LibRaw's convention for flip values differ from Exif orientation
values: https://www.libraw.org/comment/5176#comment-5176.
In this PR we fix this issue by converting LibRaw flip values to Exif
orientation values. This conversion is based on:
- https://www.libraw.org/comment/5178#comment-5178 (conversion proposed
by LibRaw's maintainer)
-
https://github.com/LibRaw/LibRaw/blob/6fffd414bfda63dfef2276ae07f7ca36660b8888/src/write/file_write.cpp#L22
(LibRaw's function for applying the flip).
Tested on:
- CR2 images with flip values 1, 5 and 6
- ARW images with flip values 1 and 5
(unfortunately I could not find any image covering other flip values).
Since the conversion is `[1, 2, 3, 4, 5, 6, 7, 8] -> [1, 2, 4, 3, 5, 8,
6, 7]`, the only test that came out with different results before and
after the change is the one with flip value 6:
in that case, the image orientation got corrected by the fix.
Following up #3826, this implements efficient `valid_file` support for
DDS, DPX, PSD, HDR, and WEBP.
The quick magic/signature checks were abstracted into small helper
functions where necessary to eliminate duplicate code.
For WEBP, some redundant checks were also removed and there's also a fix
for a `uint64_t` vs `size_t` check which would have caused issues on
32-bit builds.
The PSD changes exposed an issue with array attributes. During a normal
ImageInput create+open sequence, an ImageInput may be opened then closed
and then opened again.
Array attributes are not cleared as part of close and so their contents
will be doubled up next time they are read into the existing `m_spec`.
This PR does not fix that general problem but now that PSD has a proper
valid_file which doesn't do an open, the array attributes are now
correct:
```
This PR
2023-05-06T02:32:05.8774584Z - photoshop:ColorMode: 3
2023-05-06T02:32:05.8774888Z - photoshop:ICCProfile: "sRGB IEC61966-2.1"
2023-05-06T02:32:05.8775194Z - photoshop:LayerName: 1, 2, 3
2023-05-06T02:32:05.8775436Z - photoshop:LayerText: 1, 2, 3
vs. the current
2023-05-06T02:32:05.8775725Z + photoshop:ColorMode: 3, 3
2023-05-06T02:32:05.8776023Z + photoshop:ICCProfile: "sRGB IEC61966-2.1"
2023-05-06T02:32:05.8776494Z + photoshop:LayerName: 1, 2, 3, 1, 2, 3
2023-05-06T02:32:05.8776746Z + photoshop:LayerText: 1, 2, 3, 1, 2, 3
```
Fixes#3837
oiiotool already attempted to trim extra channels when outputting to a
format that can't support it. It correctly handled the two common cases:
* More than 4 channels for formats that don't have
`supports("nchannels")` will drop after the first 4 channels.
* More than 3 channels for formats that don't have `supports("alpha")`
will drop after the first 3 channels.
But this omits an interesting (but rare) corner case: a 4 channel file
where the last channel is NOT alpha, and the output file supports an
alpha channel but not arbitrary channels. A concrete example might be an
input file that is exr with channels R, G, B, Z, writing to a PNG file
(which can support RGB or RGBA). We wouldn't want to write what appears
to be an RGBA file if that last channel isn't really alpha.
Also, noticed that oiiotool --chnames didn't properly clear the
alpha_channel and z_channel flags, which is important if the new channel
lists drop one of those channels.
fmt 10.0 was recently released, and it breaks a lot for us because they
have fully removed legacy support for automatically being able to format
anything that has an iostream `<<` operator. This facility was deprecatd
in 8.1 but could be re-enabled with FMT_DEPRECATED_OSTREAM, but even
that was removed in 10.0.
So this means that we had to write several explicit format handlers for
some types for use with format() and print().
Also, they are even more strict with sprintf/printf which don't honor
the custom formatters as far as I can tell, so in several cases I also
switched those to the std::format/print style.
Fixes#3835
As Fabrice Macagno points out on the mail list, we weren't setting the
correct variable name CMAKE_CXX_VISIBILITY_PRESET.
Fixing this -- and finally hiding all the symbols correctly -- uncovered
an instance where testtex was relying on some classes that should have
been hidden in libOpenImageIO. For symbol visibility reasons, move the
functionality of testtex's test_hash() function to be in TextureSystem
itself. This is a much cleaner separation.
We have some OIIO 1.x era type aliases float4, int4, etc., for the
simd types that have long since been renamed vfloat4, vint4, etc.
Hide those aliases when buiding OIIO itself, to be sure we aren't
inadvertently using them. And indeed, in one spot, we were! (Fix it.)
Introuce a new preprocessor symbol, `OIIO_DISABLE_DEPRECATED`, which
if set to an encoded version number (such as 20100 for version 2.1) by
a downstream package, indicates that we are permitted to hide
declarations that were considered deprecated from that version
on. This is an aid to downstream projects who wish to enforce that
they are not inadvertently using any deprecated OIIO calls which may
disappear in a future release.
So back to the deprecated simd names: Also hide them if
`OIIO_DISABLE_DEPRECATED` is defined to exclude things that were
deprecated as of OIIO 1.9+, which apparently is when we changed this
definition.
I'm honestly not sure why some GHA CI jobs suddenly started warning
about this in the past day or so. Something must have updated on their
end. But switching to the safer Strutil equivalents fixes it.
`ssize_t` is a posix type, msvc seemingly doesn't know about leading to
a build error with MSVC 2019
usage elsewhere in this files used the `py::ssize_t` type, which seems
to fix this
`ssize_t` is a posix type, msvc seemingly doesn't know about leading to
a build error with MSVC 2019
usage elsewhere in this files used the `py::ssize_t` type, which seems
to fix this
`make_texture()` already took a configuration option "maketx:forcefloat"
(defaults to 1), which if set to 0 disables the conversion to a float
buffer for intermediate computation.
For `oiiotool -otex`, there was no way to specify this. This patch
allows it to be disabled with the new optional modifier
`-otex:forcefloat=0`.
Also, in make_texture, this option only controlled the initial
conversion of the whole buffer to float, but the downsized levels all
were float either way. This patch now uses the forcefloat config option
to also control the buffers used for downsizing.
Disabling forcefloat like this has a speed penalty (the downsizing math
on float buffers is faster than going in and out of float for every
indvidual value) and loses some precision in the process of repeatedly
downsizing to generate the MIP levels. But it can sure save a lot of
memory if you have a ginormous uint8 image and want to avod the 4x
balooning of memory. I don't recommend disabling forcefloat except for
images that can't fit into memory for the texture creation without it.
But when this is the right hammer to use, it can be the difference
between being able to create the texture, or not.
Some other improvements:
* `oiiotool -otex` wasn't outputting ANY status output during texture
output as maketx does, and for that matter, -v and -debug were not being
influencing the behavior of make_texture at all despite it having such
support. So fix to pass down the right flags. Now there are proper
verbose status and debug messages when requested.
* Eliminate useless call to is_monochrome by calling it only when the
previously-computed pixel stats indicate that the average value of all
the channels is the same (if they don't have the same average, there's
no way the pixel-by-pixel values will be the same in all channels for
all pixels). This saves an extra traversal of all the pixel memory
during texture creation.
* Use synchronized/flushed status messages so users can see them as they
are issued for very long maketx tasks. This also included some
conversion from old iostream stuff to fmt-based print.
* For verbose/debug, more fine-grained timing for make_texture, in
particular, print the time spent downsizing and writing each MIP level.
ABI-BREAKING CHANGE!
Every format that has an existing `valid_file(std::string)`
implementation, and which supports an
ioproxy, now also has a `valid_file(IOProxy)` implementation for
completeness.
The existing std::string overloads have been implemented in terms of the
IOProxy overload where
possible to cut down on the boilerplate. This is done at the base of the
hierarchy in ImageInput.
The remaining formats that support ioproxy, but do not currently have a
`valid_file` method, will be
submitted in a follow-up PR as the additional logic required for those
will require closer review.
The PSD reader was written 12 years ago by a GSOC student and has
changed little since. The following is not a knock on that student, who
clearly did a good job if his code has survived mostly unomdified for
over a decade. In chasing down problems with handling extremely large
psd files, I've noticed some oddities and inefficiencies that need
fixing. It's not an emergency, so I'll handle it in pieces.
The item addressed by this patch is memory usage and thread concurrency.
It turns out that the reader allocates memory for the scanline reads as
it goes... and basically keeps it in the ImageInput until it's
destroyed, even though it will never actually be used again. I guess
it's just trying to save the next allocation if it reads the same
scanline again or on the next subimage? But this means that if you read
a really big psd image, it's basically doubling the memory use -- it
reads one copy into the user's buffer, but also holds on to this extra
copy for no reason. Also, a bunch of the decoding helper functions
implicitly use this memory, which means we lock a mutex for the duration
of read_native_scanline (which in theory should not be necessary since
all I/O is happening through thread-safe IOProxy calls).
So this patch does the following:
1. Remove both this retained scanline memory as well as some other
scratch memory from the ImageInput, and make it allocated and freed
local to read_native_scanline.
2. Change all the helper methods that used it implicitly so that they
are passed buffers to use. In many cases these have been changed to
`const` methods and in a few cases to `static`, making it much easier to
assure ourselves that they don't modify (or sometimes use at all)
anything in the PSDInput to do their task.
3. The result of these changes means that read_native_scanline doen't
need to modify any of the data methods of the class (though it does use,
without modifcation, some things set up in the course of open). Because
it's now non-mutable, we can also remove the mutex, now allowing
multiple threads to read scanlines from the same file simultaneously
without blocking each other.
4. Some other miscellaneous cleanup, including that some scratch memory
was allocated as std::string (!), whereas it should really be a
`vector<unsigned char>` or `unique_ptr<unsigned char>`, and also
changing some computations from double to float.
There is still some more work to do after this is merged. In particular,
I think now that scanline read is stateless / non-mutable with respect
to the PSDInput object, this is a candidate for adding a multi-scanline
read_native_scanlines method (replacing the default implementation of
looping and calling read_native_scanline serially for each scanline),
which should also be able to read multiple scanlines with some overhead
and allocations done once per block, and with blocks of scanlines
running in parallel using the thread pool. This should be a huge
efficiency gain when reading psd files. But it will wait until the next
patch.
The latest version of libfmt began using the `__float128` type but this
is [not supported in the NVPTX backend for
LLVM](https://discourse.llvm.org/t/compiling-cuda-code-fails/61240/13).
A way to detect this particular platform seems to be to check both
`__CUDA_ARCH__` and `__clang__`.
I am not disabling it entirely on any CUDA target because NVCC 10+
understands `_float128` and generates instructions for it when tested on
godbolt.
Following up from https://github.com/OpenImageIO/oiio/issues/3809.
Co-authored-by: Edoardo Dominici <edoardod@sy1wse0003.al.com.au>
Windows isn't smart about symbolic links. There were a few places in the
testsuite where we used symlinks to essentially de-duplicate test
reference images that were expected to be identical. But then the
resulting archive you get when you ask GitHub for a zip file of the
whole repo (which some people download to get OIIO source rather than
git clone) can't be unzipped correctly on Windows (at least not by
cmake's unzip command).
OK, so just change those links to full copies.
Fixes#3811
The `openexr-images` repo was never updated after the default branch was
renamed to `main`. This will fix the following silent errors in CI:
```
Cloning into '/__w/oiio/oiio/build/testsuite/openexr-images'...
warning: Could not find remote branch master to clone.
fatal: Remote branch master not found in upstream origin
```
Additionally, the cmake attempted to symbolically link local repos like
`oiio-images` if they were found. Unfortunately this type of link will
fail unless running as admin on Windows (which is rare) and the
COPY_ON_ERROR directive did _not_ work here (cmake 3.24; weird;
sigh...). The output will now look something like the following:
```
-- Linking oiio-images from C:/Users/jesse/source/oiio-images
-- Link result failed to create symbolic link 'C:/Users/jesse/source/oiio/build/testsuite/oiio-images': A required privilege is not held by the client.
-- Copying oiio-images from C:/Users/jesse/source/oiio-images
```
We know that there are some corner cases on Windows when it can be
unsafe to exit because the Ilm thread pool (which is used implicitly by
OpenEXR) hangs. The exact conditions that cause this are not well
understood.
Some time back, in PR 3582, we added an explicit shutdown in oiiotool
right before we exit, to alleviate the problem. But this doesn't help
for other apps like idiff, nor for apps that use libOpenImageIO and
aren't aware of the problem, so wouldn't know to do this.
This patch changes to a different strategy: In set_exr_threads(), which
will always be called any time we use openexr files, use `std::atexit()`
to ensure that when the program terminates -- any program using
OpenImageIO in cases when openexr was used -- we set the IlmThread
global thread pool size to 0 to shut it down cleanly. Hopefully, that
will cover even more cases where this OpenEXR bug might be tickled.
When decoding RLE data, some arguments to decompress_packbits() were
inexplicably declared uint16_t (they were passed uint32_t arguments, and
I honestly don't know why we didn't get a warning all along). This
prevented us from correctly reading PSD images whose width was more than
64k, which I suppose are rare enough that it's not hard to understand
why this went unreported for so long.
But anyway, just changing those parameters to uint32_t does the trick.
I ran into the problem described in #2929 and did a little digging to
see if I could find the source of the bad color conversions. I tracked
it down to `fast_exp2()`
https://github.com/OpenImageIO/oiio/blob/a77e36250fadfb4c391688eb1df4b3dd88d5e367/src/include/OpenImageIO/fmath.h#L2050-L2080
and in particular the return statement on L2071. In the process of
trying to figure out what part of that return statement was going awry,
I found that separating the bitcasts to separate variables as in this
PR, the problem went away. All testing was done with the `color_test`
added in #3023.
It also succeeded when adding various `std::cout` lines to check what
the values of each computation were, so it does sound like a compiler
optimization that does or does not get triggered is the problem.
Unfortunately I don't know why it's not failing on the CI builds, I
double checked to make sure I'm using the same settings and flags. I
also tested on two different machines and get the same failures, and
success with this patch, on both.
One strange wrinkle is that even without this patch, color conversions
worked fine when using AVX2 for the SIMD mode.
Summary :
Previous results (without this patch)
- USE_SIMD=0 - PASS
- USE_SIMD=SSE4.2 - FAIL
- USE_SIMD=AVX - FAIL
- USE_SIMD=AVX2 - PASS
With this patch
- USE_SIMD=0 - PASS
- USE_SIMD=SSE4.2 - PASS
- USE_SIMD=AVX - PASS
- USE_SIMD=AVX2 - PASS
Visual Studio 2019 versions 16.11.32228.343, 16.11.33328.57 and
16.11.33423.256(same as current Github build runners) in RelWithDebInfo
mode.
Chris Gloor points out that after my latest change, the "font_dir_list"
got loaded with every directory underneath that it discovered through
that one level of recursion. Not harmful, but very cluttered. My
original intention was for font_dir_list to only have the search path
directories.
So this prompted me to do a bigger refactor to fix that up and clean up
some other things. Now all the searching is done once when it builds up
the initial inventory the first time a font is needed, no more potential
directory searching or touching non-essential files for each font-using
operation.
Not long ago, we removed recursion from the font searching, for
performance reasons. But now I see at there are some systems where none
of the font directories likely in the searchpath have any of the font
files directly there, but instead they are in subdirectories one level
further down.
To compromise, then, I'm adding ONE level of recurson to our font
search. If you need to search deeper than that, you should directly add
the paths to the OPENIMAGEIO_FONTS searchpath.
For cases of calls like `ImageOutput.write_image(array)`, this improves
the error messages when the array dimensions don't match the size of the
image we're writing, in two ways:
1. For the case of 2D images where the dims don't match, print what the
array shape actually is and what was expected. That's more helpful than
just saying "can't figure out array shape" and printing the number of
dimensions (but not the shape).
2. A mistake caused a number of error messages to be substituted by a
cryptic "can't handle numpy array with noncontiguous channels", but that
was only supposed to be for one particular case; fix that.
We are not ready to throw the switch quite yet, because the exr core
library still cannot read dwaa/dwab compressed images. It's only safe
to use if you are sure you will not encounter those files.
I feel really stupid for not catching this earlier. Inadequate testing!
Add a new "openexr-compression" test that writes one of every compression
methods available to openexr and ensures it can be read back. We have to
disable it for openexr:core=1 because it will fail on the dwa cases.
Currently, if you run oiiotool with --bumpslopes and --cdf at the same time (or -obump:cdf=1), the CDF is only produced for the first four (of six) channels. The techniques described in Burley (2019) actually do apply reasonably to the squared derivative channels, so it's useful to allow all six channels to have CDFs here.
maketx and oiiotool-maketx tests updated with new relevant tests.
When we build against OpenEXR >= 3.1, default to using the new
"exrcore" APIs unless disabled at build time with
`-DOIIO_USE_EXR_C_API=OFF` or at runtime through the global openexr:core
attribute.
TIFF output using JPEG compression seems unreliable, and after several
hours trying to chase it down, I'm not making much progress. As far
as I can tell, this has always (or at least for a long time) been
broken, and nobody noticed, which is strong circumstantial evidence
that it's not something people need or is trying to use, and therefore
it doesn't seem like a good use of more time to fix a feature that
no one uses.
So I'm hiding all the TIFF output with jpeg compression support behind
`#if` guards and disabling them for now. This will cause requests to
use jpeg compression for TIFF output to just revert to the default zip
compression.
Reading TIFF files that are jpeg-compressed seems fine, this doesn't
disable our ability to read the files, just to produce them.
It is also worth noting: It's one thing to write an actual .jpg file
as *final* output for web display, viewing, or other "terminal" uses.
But JPEG-compressed image data should never ever be used as input to
further image computation operations, since each read/write/read cycle
with jpeg compression can lead to further data loss. So I think for
this reason also, this patch should help avoid the temptation or
inadvertent ability to use this lossy compression method when writing
TIFFs.
There were still some bolded (indicating newly changed) dependency
minimums that actually were bumped in 2.4 or even 2.3 and should have
already been unbolded.
* Don't use deprecated ubuntu-18.04 runner. Switch to running on
ubuntu-20.04, but use docker containers that supply ubundu-18.04.
* Bump 'latest' version of openexr, webp, pugixml.
There turns out to be a way to get wedged into a mutual deadlock on
the file handle cache, with how get_texture_info on a udim pattern
checks to see if the attribute requested is present and equally valued
in all constituent files of the udim set. This hits the handle set
pretty hard, and if multiple threads are hammering the same sets and
rapidly cycling textures out of the cache, there can be trouble, it
seems.
Remedies in this patch:
* When it's trying to close files because the number of handles is too
high, we already had it so that if we only exceeded the max_files
limit by a little bit, we would simply try to get the lock, and
abandon the attempt if somebody else is holding it, but if we are
quite a bit over, then stay and wait for the lock to be released.
In this patch, we make the that default behavior -- if we are over
the limit and need to free some handles, simply try to get the lock,
and if somebody else is holding it, leave it all up to them and
return immediately. The pros of this are (a) snappier response by
never waiting at this point, and (b) removing the possibility of a
deadlock, because if you can't acquire the lock, you just move
on. But the con is that lots of threads adding new files may run
faster than the one thread who has the file and is trying to release
files, so it's possible that the number of handles will exceed the
limit by a wider margin now, but hopefully only momentarily. (But
note that this can be overridden with a new ImageCache attribute
"max_open_files_strict", which if set to nonzero revert to the old
behavior. At least for now, in case I am neglecting to understand
some bad consequence of this change.)
* The recursive_mutex controlling access to the ImageInput in each
tile cache record is changed to a recursive_timed_mutex, and when
closing a handle by calling release(), instead of an unconditional
lock, try acquiring a lock for 100ms. If it fails to do so, it may
be in a possible deadlock, so just give up and move on. The only
consequence is that maybe that handle isn't freed after all. It will
then move on to the next one, not the end of the world, but at least
it can't get stuck there.
* In find_file, eliminate the call to check_max_files -- it's not
needed here, since it's also called by open(), which is the only
time that new file handles might be opened. We just don't also need
it again simply because we're adding a file to the cache.
Setting the cmake option OIIO_DISABLE_BOOST_STACKTRACE to ON disables the use of
Boost stacktrace at compile time. This avoids dependencies on helper libraries
like dbgeng.dll.
* test: Add a corrupted test file
* fix(IC): upon tile read error, mark the file as broken
The idea is to keep fruitlessly reading from a corrupted file.
* fix: race condition in TIFF reader
TIFFInput::read_native_tiles had a very subtle race condition. There
is a section where it has a task_set of decompression tasks that are
sent to the thread queue. When the method ends, the task_set goes out
of scope, and its destructor will wait for all the remaining tasks to
finish. All fine, right? Except that other variables in the function
include unique_ptr's with allocated memory that the tasks need. So
what can happen is that if the compiler generates the code such that
the destructor of the unique_ptr happens before the destructor of the
task_set, the memory that the unique_ptr held may be deallocated
before some of the tasks still running in threads have finished.
In practice, we seem only to hit this race condition for particular
cases involving corrupted files, because the early `return` seems
quite likely to run those destructors before all the threads are done,
but it was possible to encounter it for valid files as well, if the
timing is just right. Shocking that we never had it reported.
The solution is just to have an explitit tasks.wait(), which will
block until all the tasks have finished before hitting the actual end
of the method and letting the other destructors to run. And for the
"early return" case, change it to a break so that we go through the
wait call at the end.
I believe that this fix addresses TALOS-2023-1709 / CVE-2023-24472.
There are two ways to fail the FITS magic number check: fail to read
the bytes, or the bytes don't match the magic number. It seemed
suspicious that we call close() for the latter but not the
former. Combine them into a single error response (they had the same
message, anyway).
Also, make sure FitsInput::init() fully clears some other data
structures. Oversight?
I don't know if any of these things are real bugs, but they look
sloppy and potentially problematic, so fixing them up.
* Make lookups of const unordered_maps thread-safe
* added back deleted end brace
* clang-format
---------
Co-authored-by: Larry Gritz <lg@larrygritz.com>
For non-0-origin images, we were checking the resolution range
incorrectly (`>=` vs `>`).
Also were printing the error message with the wrong notation, should
have shown a half-open range as `[beg,end)` and not `(beg,end]`.
* Be more careful constructing strings from what we read in the header;
corrupted files may make them not be zero-terminated.
Fixes TALOS-2023-1707 / CVE-2023-24473
* Watch out for pixels with palette indices that exceed the size we
allocated for the palette.
Fixes TALOS-2023-1708 / CVE-2023-22845
* Check alpha_type for valid range.
This isn't 100% final -- I'm sure there are some corner cases I
missed, maybe things that will eventually need some performance work,
but enough works and passes tests that I'm confident in the API and so
I want to get this merged and we can iterate on internals later.
The use case is as follows: User or shader knows that a particular
texture is in something other than the default working color space,
this is not necessarily apparent from the file itself and so cannot be
fully automatic, but it is presumed that it can be communicated in the
texture call. We aim to be efficient, fully converting each tile upon
input. Note that we are sweeping under the rug the fact that a
non-default color space lookup is not mathematically correct with
respect to the MIP-map downsizing math if that was not considered at
texture creation time. But by converting texels as they are read,
rather than the shader itself trying to do a color transformation on
the final texture result, at least we are doing the right math for the
filtering within and among MIP levels.
The new interface consists of communicating the presumed working color
space, finding the colortransform ID of a given from/to (with the "to"
defaulting to the working space, if not specified), and then on each
texture call, supplying that colortransform ID as one of the settings
in TextureOpt (or TextureOptBatch).
The outline of changes are as follows:
* BREAKING CHANGE: TextureOpt and TextureOptBatch now contain an
additional int `colortransformid`. The default, 0, is the old (and,
frankly, preferred) behavior. Nonzero values indicate that we want
the texels to go through a color transformation as the tiles are
read into the cache.
* BREAKING CHANGE: New public methods have been added to TextureSystem
`get_colortransform_id(fromspace, tospace)` that return a transform
ID given the from and to color spaces (ustring or ustringhash). This
is the value that goes in the TextureOpt to request a color
transformation on the pixels as they are read.
* BREAKING CHANGE: IC::get_image_handle / TS::get_texture_handle now
take an additional optinal parameter that is a TextureOpt*. This is
currently unused, but is reserved for possible future use and/or for
alternate TextureSystem implementations that choose a different
implementation strategy wherein an entirely different TextureHandle
may correspond to each color transformation. That's not how I've
done it here, but it's a valid approach (and maybe one we'll switch
to in the future). We presume that it's ill-defined to ask for a
handle with one explicit color transformation, then use that handle
with a TextureOpt that implies a different color
transformation. Using those consistently ensure that either TS
strategy will work equally well.
* New TextureSystem attribute "colorspace" gives the working color
space name (defaulting to "scene_linear"), and "colorconfig" can
optionally point to a color config file (though this is not yet
functional -- it just uses the default color config).
* Internally, the tile cache hash has been adjusted so that the same
tile in the disk file will appear as a separate tile in the cache
for each color transform that is applied to it. Thus, there is no
conflict if some different texture calls ask for different color
space interpretations of the same texture file. (That said, it's not
recommended to be throwing color spaces around willy nilly -- it
still takes multiple cache entries, plus extra color conversion math
as the tiles are read in.)
* A new preprocessor symbol OIIO_TEXTURESYSTEM_SUPPORTS_COLORSPACE in
texture.h allows client software can easily tell if this is a
version new enough to support color management and has the additional
fields in the texture option structures.
Because there are breaking changes here to ABI, this is destined only
for master (future 2.5) and will not be backported to OIIO 2.4.
2023-02-04 14:53:30 -08:00
1188 changed files with 97142 additions and 146162 deletions
description:Generate or update release notes for a patch, minor, or major release, or just to update main. Run git-cliff, organize and edit the output per project conventions, and insert into CHANGES.md.
argument-hint:[patch|minor|major|main] [prev-tag]
---
Generate release notes for an OpenImageIO release.
Arguments: `$ARGUMENTS`
- First argument (optional): release type — `patch` (default), `minor`, `major`, or `main`.
- Second argument (optional): previous release tag (e.g. `v3.1.2.0`). If omitted, find the most recent tag automatically. If the release type is `main`, instead of a tag, look for the last commit at which the CHANGES.md file was updated.
## Steps
1.**Determine the previous tag** if not provided:
```
git describe --tags --abbrev=0
```
or look at recent tags: `git tag --sort=-version:refname | head -10`.
However, if the "release type" is `main`, instead of a tag, just find
### New minimum dependencies and compatibility changes:
### ⛰️ Major new features and public API changes:
* *New image file format support:*
* *oiiotool new features and major improvements*:
* *Command line utilities*:
* *API changes*
* Other notable new feature:
### 🚀 Performance improvements:
### 🐛 Fixes and feature enhancements:
### 🔧 Internals and developer goodies
### 🏗 Build/test/CI and platform ports:
* CMake build system and scripts:
* Dependency and platform support:
* Testing and Continuous integration (CI) systems:
### 📚 Notable documentation changes:
### 🏢 Project Administration
### 🤝 Contributors
```
Note that the section outline may already be present, in which case you
only need to fit items into the existing category outline.
Rules:
- Group commits into sections; within each section, cluster related items together.
- When needed, expand terse one-liners into enough prose that users understand what changed and why it matters.
- For `feat:` commits, make sure the feature is explained sufficiently — don't just copy the commit subject.
- For `api:` or `api!:` commits, clearly call out what changed in the public API.
- Include PR links and author attribution for every entry, in the combined
`(by @githubid / Full Name)` form from step 3.
- The notes should "tell the story" of the release, not just be a dump of commit subjects.
- We aim to make major/minor releases approximately in October 1 of each year. If the anticipted release date is already in the file, don't change it. If it is not present, ask for confirmation of the planned release date.
### For updating release notes in main:
Rules:
- Generally, follow the rules for "major/minor releases", except as noted in other items below.
- Don't apply a new skeleton of category listings unless they are not present for the upcoming release.
5. **Insert the formatted notes** into `CHANGES.md` in the appropriate place (as detailed below). Leave the existing content intact.
- When updating `main` or preparing a `major` or `minor` release, insert the updates in the top section for the upcoming major/minor release. Insert a new set of category sections only if it's not already present.
- When doing a `patch` release, insert the changes immediately above the last patch release of that branch, so that CHANGES.md lists releases in descending numerical (version) order.
- When porting a set of release notes from a release branch into main, or from an older (obsolete) release branch to the current release branch, insert it into the right place to maintain overall descending order.
6. **Double check that the notes are adequately descriptive.** (See more
detailed description of this step below.)
7. **Forward-port release notes from release branches if needed**
8. **Show a summary** of what was inserted and ask the user to review before finalizing.
## Forward-porting release notes
Release notes may have been generated independently in main, and release
branches. When updating one branch, we ensure that any changes from older
branches have been incorporated.
- When preparing `patch` release notes, check the CHANGES.md file in the
"dev-X.(Y-1)" branch for the previous minor release family to identify any
X.(Y-1).Z patch release notes that are not reflected in the current release
notes that we are updating.
- When updating `main` or doing a `major` or `minor` release, check the
CHANGES.md file in both the "dev-X.(Y-1)" and "dev-X.(Y-2)" branches.
Check whichever is newer of the local and remote copy of that branch, since
the local one may have had release notes or its version in CMakeLists.txt
updated, but not yet pushed to GitHub.
- If any patch releases are present in the older dev branches checked, insert
the release notes for those patch releases into the right positions in the
current release notes that we are updating.
- If a change is forward-ported in this manner and the same PR is an update in
the current set of changes we are updating as our main task, document that
in the current set of notes using the following convention: in the line of
the notes, the explanation of the version where it appears should reflect
the first version of all branches where it appeared, for example, `(3.2.0.0,
3.1.3.0, 3.0.8.0)` to indicate that the patch was added to each of those
versions. The versions should be listed in descending order.
## Useful abbreviations for category labels
| Abbrev | Meaning |
|--------|---------|
| IB | ImageBuf |
| IBA | ImageBufAlgo |
| IC | ImageCache |
| TS | TextureSystem |
| oiiotool | the oiiotool command |
| build | CMake/build system |
| deps | Changes to accommodate dependency or toolchain changes |
| ci | CI/GitHub Actions |
| docs | documentation |
| int | internal/refactor |
| test | testsuite or unit tests |
| HEADER.h | Developer utilities in a public header file |
## Combining PRs into single entries
To be more concise and easier to read, within a release's notes, related
PRs/commits can be combined into a single bullet-point line, which would look
**PLEASE DO NOT ASK USAGE QUESTIONS USING GITHUB "ISSUES"**
If you are pretty sure that you have identified a **BUG** in OpenImageIO,
please file a [bug report issue](https://github.com/OpenImageIO/oiio/issues/new?template=bug_report.md).
please file a [bug report issue](https://github.com/AcademySoftwareFoundation/OpenImageIO/issues/new?template=bug_report.md).
But if you are just asking a question:
* How do I...?
@@ -18,10 +18,10 @@ But if you are just asking a question:
* Am I doing something wrong?
* I can't build OpenImageIO
For anything of this nature, the best way to get help using OpenImageIO is
to ask a question on the [oiio-dev developer mail list](http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org).
For anything of this nature, the best way to get help using OpenImageIO is to
ask a question on the [oiio-dev developer mail list](https://lists.aswf.io/g/oiio-dev) or the [ASWF Slack](https://slack.aswf.io) `#openimageio` channel.
The [documentation](https://openimageio.readthedocs.org)
The [documentation](https://docs.openimageio.org)
is pretty comprehensive, so please check there first; you may find the answer
more quickly than you can get your question answered on the mail list.
- [ ]I have updated the documentation, if applicable.
- [ ] I have ensured that the change is tested somewhere in the testsuite
(adding new test cases if necessary).
- [ ] My code follows the prevailing code style of this project.
- [ ]**I have read the guidelines** on [contributions](https://github.com/AcademySoftwareFoundation/OpenImageIO/blob/main/CONTRIBUTING.md) and [code review procedures](https://github.com/AcademySoftwareFoundation/OpenImageIO/blob/main/docs/dev/CodeReview.md).
- [ ]**I have read the [Policy on AI Coding Assistants](https://github.com/AcademySoftwareFoundation/OpenImageIO/blob/main/docs/dev/AI_Policy.md)**
and if I used AI coding assistants, I have an `Assisted-by: TOOL / MODEL`
line in the pull request description above.
- [ ]**I have updated the documentation** if my PR adds features or changes
behavior.
- [ ]**I am sure that this PR's changes are tested in the testsuite**.
- [ ]**I have run and passed the testsuite in CI***before* submitting the
PR, by pushing the changes to my fork and seeing that the automated CI
passed there. (Exceptions: If most tests pass and you can't figure out why
the remaining ones fail, it's ok to submit the PR and ask for help. Or if
any failures seem entirely unrelated to your change; sometimes things break
on the GitHub runners.)
- [ ]**My code follows the prevailing code style of this project** and I
fixed any problems reported by the clang-format CI test.
- [ ] If I added or modified a public C++ API call, I have also amended the
corresponding Python bindings. If altering ImageBufAlgo functions, I also
exposed the new functionality as oiiotool options.
The OpenImageIO project grows and thrives from assistance from Contributors.
Contributors include anyone in the community who contributes code,
documentation, or other technical artifacts that have been incorporated into
the project repository.
Anyone can be a Contributor. You need no formal approval from the project,
beyond the legal forms.
### How to Become a Contributor
* Review the coding standards and [contributing guidelines](CONTRIBUTING.md)
to ensure your contribution is in line with the project's coding and styling
guidelines.
* Sign the Individual CLA, or if you are employed by an organization that
might have any claim to IP you create, have your organization sign the
Corporate CLA.
* Submit your code as a PR with the appropriate DCO sign-off on each commit.
The easiest way to do this is to ensure that you commit your code with `git
commit -s`.
## Committers
Project Committers have merge access on the OpenImageIO GitHub repository
and are responsible for approving submissions by Contributors.
### Committer Responsibilities
Typical activities of a Committer include:
* Helping users and novice contributors.
* Ensuring a response to questions posted to the
[Oiio-dev developer mail list](https://lists.aswf.io/g/oiio-dev)
* Contributing code and documentation changes that improve the project.
* Reviewing and commenting on issues and pull requests.
* Ensuring that changes and new code meet acceptable standards and are in
the long-term interest of the project.
* Participation in working groups.
* Merging pull requests.
### How to Become a Committer
Any member of the OpenImageIO community (though typically an existing
Committer or TSC member) may nominate an individual making significant and
valuable contributions to the OpenImageIO project to become a new Committer.
To nominate a new Committer, open an issue in the OpenImageIO repository, send
mail to the TSC mail list, or raise the issue at a TSC meeting.
The TSC may periodically review the Committer list to identify inactive
Committers. Past Committers are typically given Emeritus status. Emeriti may
request that the TSC restore them to active Committer status.
## Technical Steering Committee
The Technical Steering Committee (TSC) has final authority over this project.
As defined in the project [technical charter](https://github.com/AcademySoftwareFoundation/foundation/blob/main/project_charters/openimageio_charter.pdf), in
addition to committer activities, TSC responsibilities also include, but are
not limited to:
* Coordinating technical direction of the Project.
* Project governance and contribution policy.
* GitHub repository administration.
* Maintaining the list of additional Committers
* Appointing representatives to work with other open source or open
standards communities.
* Discussions, seeking consensus, and where necessary, voting on technical
matters relating to the code base that affect multiple projects.
* Coordinating any marketing, events, or communications regarding the
project.
Within the TSC are two elected leadership roles to be held by its members
and voted on annually. Any TSC member can express interest in serving in a
role, or nominate another member to serve. There are no term limits, and one
person may hold multiple roles simultaneously. Should a TSC member resign
from a leadership role before their term is complete, a successor shall be
elected through the standard nomination and voting process to complete the
remainder of the term. The leadership roles are:
* **Chair**: This position acts as the project manager, organizing meetings
and providing oversight to project administration.
* **Chief Architect**: This position makes all the final calls on design and
technical decisions, and is responsible for avoiding "design by committee"
pitfalls.
The chair role is assumed to rotate annually (though there are no term
limits, so the TSC may reelect an existing chair). The chief architect
position should be a source of stability and coherent design vision, so the
TSC is encouraged to choose an architect who can serve for many years and
only change architects when it is necessary for the health of the project
and its community.
At the time of election, the TSC will also agree upon which of these two
leaders will serve as the OpenImageIO ASWF (Academy Software Foundation) TAC
(Technical Advisory Council) representative for the term. This member
represents the project at all ASWF TAC meetings.
### TSC Members
Current voting members of the TSC are:
* **Chair and TAC representative**: Daniel Greenstein - Sony Pictures Imageworks
* **Chief Architect**: Larry Gritz - Sony Pictures Imageworks
* Anton Dukhovnikov - Wētā Digital / Unity
* Jesse Yurkovich - Blender
* Joseph Goldstone - ARRI
* Luke Emrose - Animal Logic
* Martin Davies - LAIKA
* Robin Rowe - Heroic Robots
* Thiago Ize - Autodesk
### TSC Nomination and Succession
Any proposal for additional members of the TSC may be submitted by Committers,
TSC members, or other major stakeholders of the OpenImageIO community by
opening an issue outlining their case or raising the issue at a TSC meeting.
New TSC members are accepted or rejected by majority vote of the TSC.
If a TSC member is for an extended period not regularly participating or
performing the responsibilities expected of TSC members, the TSC may by
majority vote request an alternate TSC member be submitted by that
organization, or remove the inactive member from the TSC.
A voting member of the TSC may nominate a successor in the event that such
voting member decides to leave the TSC, and the TSC, including the departing
member, shall confirm or reject such nomination by a vote. In the event that
the departing member's nomination for successor is rejected by vote of the
TSC, the departing member shall be entitled to continue nominating successors
until one such successor is confirmed by vote of the TSC. If the departing
member fails or is unable to nominate a successor, the TSC may nominate one on
the departing member's behalf.
TSC membership is presumed to be retained by the individual even if they
change employers, provided they remain active in the project. The TSC may take
action to ensure that organizational stakeholder representation not become
severely disproportionate, for example by urging an organization that loses
its sole TSC representative to nominate a new member, or by limiting the total
number of voting members from any one organization if too many members all
move to the same organization.
### TSC Meetings
Any meetings of the TSC are intended to be open to the public, except where
there is a reasonable need for privacy. The TSC meets regularly in a voice
conference call, at a cadence deemed appropriate by the TSC. The TSC Chair
moderates the meeting, or appoints another TSC member to moderate in his or
her absence. Meetings may also be streamed online where appropriate;
connection details will be posted to the oiio-dev mail list in advance of the
scheduled meeting.
The biweekly TSC meetings can be found on the [ASWF Calendar](https://calendar.openimageio.org),
just click on the OpenImageIO TSC meeting entries to get the Zoom link.
Items are added to the TSC agenda which are considered contentious or are
modifications of governance, contribution policy, TSC membership, or release
process, in addition to topics involving the high-level technical direction
of the project.
The intention of the agenda is not to approve or review all patches. That
should happen continuously on GitHub and be handled by the larger group of
Committers.
Any community member or Contributor can ask that something be reviewed by
the TSC at the meeting by logging a GitHub issue. Any Committer, TSC member,
or the meeting chair can bring the issue to the TSC's attention by applying
the `TSC` label.
Prior to each TSC meeting, the meeting chair will share the agenda with
members of the TSC. TSC members can also add items to the agenda at the
beginning of each meeting. The meeting chair and the TSC cannot veto or
remove items.
The TSC may invite additional persons to participate in a non-voting
capacity.
The meeting chair is responsible for ensuring that minutes are taken and
archived in the project repository or other designated accessible location.
Due to the challenges of scheduling a global meeting with participants in
several time zones, the TSC will seek to resolve as many agenda items as
possible outside of meetings on the public mailing list or through GitHub
issues, discussions, and pull requests.
Meeting notes (usually very brief) can be found at: https://docs.google.com/document/d/1quSh1K5U2jUDi3cB1eX84i9Y2p7r58w1fZMXM8rnH-w/
| make VERBOSE=1 ... | Show all compilation commands |
| make STOP_ON_WARNING=0 | Do not stop building if compiler warns |
| make EMBEDPLUGINS=0 ... | Don't compile the plugins into libOpenImageIO |
| make USE_OPENGL=0 ... | Skip anything that needs OpenGL |
| make USE_QT=0 ... | Skip anything that needs Qt |
| make MYCC=xx MYCXX=yy ... | Use custom compilers |
| make USE_PYTHON=0 ... | Don't build the Python binding |
| make OIIO_PYTHON_BINDINGS_BACKEND=both ... | For source/CMake builds, build the existing pybind11 bindings and the nanobind (WIP) module |
| make BUILD_SHARED_LIBS=0 | Build static library instead of shared |
| make IGNORE_HOMEBREWED_DEPS=1 | Ignore homebrew-managed dependencies |
| make LINKSTATIC=1 ...| Link with static external libraries when possible |
| make SOVERSION=nn ... | Include the specified major version number in the shared object metadata |
| make NAMESPACE=name | Wrap everything in another namespace |
The command 'make help' will list all possible options.
You can also ignore the top level Makefile wrapper, and instead use
CMake directly:
mkdir build
cd build
cmake ..
cmake -B build -S .
cmake --build build --target install
If the compile stops because of warnings, try again with
@@ -257,8 +281,9 @@ If the compile stops because of warnings, try again with
or, if you are using CMake directly,
cd build
cmake -DSTOP_ON_WARNING=0 ..
rm -rf build
cmake -B build -S . -DSTOP_ON_WARNING=0
cmake --build build --target install
@@ -270,73 +295,71 @@ Building on Windows
You will need to have Git, CMake and Visual Studio installed.
The minimal set of dependencies for OIIO is: Boost, zlib, libTIFF, OpenEXR, and libjpeg or libjpeg-turbo. If you have them built somewhere then you skip
the section below, and will only have to point OIIO build process so their locations.
The minimal set of dependencies for OIIO is: zlib, libTIFF, Imath, OpenEXR,
OpenColorIO, and libjpeg or libjpeg-turbo. If you have them built somewhere
then if you set the environment variable `CMAKE_PREFIX_PATH` to include those
areas so CMake will find them, the OpenImageIO build will find those pre-built
packages and use them. If you don't have them build, or are unsure, the
`OpenImageIO_BUILD_MISSING_DEPS=all` CMake option will cause the build process
to download the sources of those dependencies, build them, and use them.
* Boost: get the boost source archive, extract into `{BOOST_ROOT}`.
```
cd {BOOST_ROOT}
bootstrap
b2
```
* zlib: this will build it, and then delete the non-static library, so they don't get picked up:
* OpenEXR: you'll have to point it to your `{ZLIB_ROOT}` location from the above. If copy-pasting the multi-line command (with lines ending in `^`) into
cmd.exe prompt, make sure to copy all the lines at once.
and that will build the package and install it into the `{OIIO_ROOT/dist`
area. (If you instead wanted the install to end up elsewhere, the easiest way
is on the earlier config step, to add the option
`-DCMAKE_INSTALL_PREFIX=somewhere_else`.)
On the other hand, if you would prefer to open the generated Visual Studio
solution, the "cmake configure" will have produced
`{OIIO_ROOT}/build/OpenImageIO.sln` that can be opened in Visual Studio IDE.
Note that the solution will be only for the Intel x64 architecture only; and
will only target "min-spec" (SSE2) SIMD instruction set.
Optional packages that OIIO can use (e.g. libpng, Qt) can be build and pointed to OIIO build process in a similar way.
@@ -363,29 +386,81 @@ produce a dynamic-linked version.
3. Execute the PowerShell command from where vcpkg is located in directory. ``vcpkg install openimageio``
**Note: Importing the OpenImageIO Python Module**
As of OpenImageIO 3.0.3.0, the default DLL-loading behavior for Python 3.8+ has changed.
If you've built OIIO from source and ``import OpenImageIO`` is throwing a ModuleNotFound exception, revert to the legacy DLL-loading behavior by setting environment variable
``OPENIMAGEIO_PYTHON_LOAD_DLLS_FROM_PATH=1``.
Python-based Builds and Installs
--------------------------------
**Installing from prebuilt binary distributions**
If you're only interested in the Python module and the CLI tools, you can install with `pip` or `uv`:
> ```pip install OpenImageIO```
**Building and installing from source**
If you have a C++ compiler installed, you can also use the Python build-backend to compile and install
from source by navigating to the root of the repository and running: ```pip install .```
This will download and install CMake and Ninja if necessary, and invoke the CMake build system; which,
in turn, will build missing dependencies, compile OIIO, and install the Python module, the libraries,
the headers, and the CLI tools to a platform-specific, Python-specific location.
See the [scikit-build-core docs](https://scikit-build-core.readthedocs.io/en/latest/configuration.html#configuring-cmake-arguments-and-defines)
for more information on customizing and overriding build-tool options and CMake arguments.
This repo contains python type stubs which are generated from `pybind11` signatures.
The workflow for releasing new stubs is as follows:
- Install [`uv`](https://docs.astral.sh/uv/getting-started/installation/) and `docker`
- Run `make pystubs` locally to generate updated stubs in `src/python/stubs/__init__.pyi`
- Run `make test-pystubs` locally to use mypy to test the stubs against the code in
the python testsuite.
- Commit the new stubs and push to Github
- In CI, the stubs will be included in the wheels built by `cibuildwheel`, as defined in `.github/wheel.yml`
- In CI, one of the `cibuildwheel` Github actions will rebuild the stubs to a
temp location and verify that they match what has been committed to the repo.
This step ensures that if changes to the C++ source code and bindings results
in a change to the stubs, developers are notified of the need to regenerate
the stubs, so that changes can be reviewed and the rules in `generate_stubs.py`
can be updated, if necessary.
Note that if you can't (or don't want to) build the stubs locally, you can
download an artifact containing the wheel and `__init__.pyi` file from any job
that fails the stub validation.
Test Images
-----------
We have yet another project containing a set of sample images for testing
OpenImageIO. We split test images into a separate project in order to make
the main source code tree smaller and simpler for people who don't need the
test suite.
There are several projects containing sets of sample images for testing
If you build with `EMBEDPLUGINS=0`, remember that you need to set the
environment variable `OIIO_LIBRARY_PATH` to point to the 'lib' directory
where OpenImageIO is installed, or else it will not be able to find the
plugins.
Additional resources:
- [User quick start](docs/QuickStart.md) is a quick example of using
OpenImageIO in Python, C++, and the command line.
- [Releases](https://github.com/AcademySoftwareFoundation/OpenImageIO/releases) -- see this for the latest release notes, also found in the [CHANGES.md](CHANGES.md) file.
Documentation
-------------
The primary user and programmer documentation can be found
on [openimageio.readthedocs.io](https://openimageio.readthedocs.io/en/latest/)
as html or as [PDF](https://readthedocs.org/projects/openimageio/downloads/pdf/latest/).
👷 Building and installing OpenImageIO
--------------------------------------
- [Build and installation instructions](INSTALL.md) for OpenImageIO. Such
as it is. This could use some work, particularly for Windows.
Contact & reporting problems
----------------------------
🚑 Contact & reporting problems
-------------------------------
Simple "how do I...", "I'm having trouble", or "is this a bug" questions are
| master | :white_check_mark: :construction: ALL fixes immediately, but this is a branch under development with a frequently unstable ABI and occasionally unstable API. |
| 2.3.x | :white_check_mark: All fixes that can be backported without breaking ABI compatibility. New tagged releases monthly. |
| 2.2.x | :x: Only receives occasional critical fixes, upon request. |
| <= 2.1.x | :x: No longer receiving patches of any kind. |
| main | :white_check_mark: :construction: ALL fixes immediately, but this is a branch under development with a frequently unstable ABI and occasionally unstable API. |
| 3.1.x | :white_check_mark: All fixes that can be backported without breaking ABI compatibility. New tagged releases monthly. |
| 3.0.x | :warning: Important fixes that can be easily backported without breaking ABI compatibility. New tagged releases as needed, and becoming less frequent over time. |
| <= 2.5.x | :x: No longer receiving patches of any kind. |
## Reporting a Vulnerability
If you think you've found a potential vulnerability in OpenImageIO, please
report it by emailing security@openimageio.org. Only the project administrator
has access to these messages. Include detailed steps to reproduce the issue,
and any other information that could aid an investigation. Our policy is to
respond to vulnerability reports within 14 days.
report it to the maintainers. Include detailed steps to reproduce the issue,
and any other information that could aid an investigation.
Our policy is to address critical security vulnerabilities rapidly and post
patches as quickly as possible.
The best way to report a vulnerability is to file a GitHub [security
If that is not possible, it is also fine to email your report to
security@openimageio.org. Only the project administrators have access to these
reports.
Our policy is to respond to vulnerability reports within 14 days, and to
address critical security vulnerabilities rapidly and post patches quickly,
usually at the next scheduled patch release at the beginning of every month.
For especially severe vulnerabilities, or when a flaw is believed to be
exploited in the wild, we will try to make a special patch release as soon as
possible.
## What do we consider a vulnerability?
A true vulnerability, by our definition, requires the following:
- It allows arbitrary code execution, privilege escalation, exfiltration of
secrets beyond network boundaries, destruction of data or equipment, or
similarly serious outcome.
- It can be plausibly exploited by an untrusted party through normal product
inputs (for example, a maliciously crafted input image).
We would like to hear about these through the confidential GitHub
vulnerability reporting mechanism described above.
The following are examples of problems that we DO NOT consider
vulnerabilities:
- A crash or program termination that does not directly present the
opportunity for further exploits. This includes hardening assertions that
terminate the program rather than allow undefined behavior.
- Out-of-memory errors, as long as the program terminates in a reasonable
amount of time. (But if a tiny crafted input can make it loop infinitely or
spin for minutes, then it might qualify as a DOS vulnerability. Use your
best judgment.)
- Misuse or incorrect use of the APIs. We consider all programs directly using
OIIO's APIs to be trusted.
- Flaws whose root cause lies in a dependency. These should be reported and
fixed upstream; the upstream project owns the CVE when one is warranted.
Please do not use the GitHub vulnerability reporting mechanism for these, and
instead file ordinary Issues and PRs. We do not support requesting a CVE for
these kinds of problems.
Users operating in an environment in which it might encounter untrusted inputs
that could be malicious, and are be running in a situation where real damage
could occur are expected to:
- Enable all the OpenImageIO build-time options for safety checks, hardening,
etc. A problem that only occurs when these are disabled is not considered a
vulnerability.
- Ensure that all the dependencies OIIO builds against are on their latest
patched versions, and not rely on the OS distribution default, or even
OIIO's own "dependency auto-build" scripts. A problem that depends on
having anything but the lastest dependencies is not an OIIO vulnerbility.
## Other security features
### Signed tags
Starting with OpenImageIO 3.0, we cryptographically sign release tags.
To verify a tag, you can use the `git tag -v` command, which will check
the signature against the public key that is included in the repository.
For example,
```bash
git tag -v v3.0.0.3
```
## Outstanding Security Issues
None known
## History of CVE Fixes
Most recent fixes listed first, more or less:
- CVE-2026-63638: Cineon invalid bit depth heap out-of-bounds write [advisory](https://github.com/AcademySoftwareFoundation/OpenImageIO/security/advisories/GHSA-9hxv-jvgr-3x8g) / [Fix: PR #5283](https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/5283) (fixed in 3.1.16.0)
- CVE-2026-63635: PSD RawColor invalid color mode causes global out-of-bounds read and allocation DoS [advisory](https://github.com/AcademySoftwareFoundation/OpenImageIO/security/advisories/GHSA-3c8w-9xvm-r6gf) / [Fix: PR #5282](https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/5282) (fixed in 3.1.16.0)
- CVE-2026-63635: PSD RawColor invalid color mode causes global out-of-bounds read and allocation DoS [advisory](https://github.com/AcademySoftwareFoundation/OpenImageIO/security/advisories/GHSA-3c8w-9xvm-r6gf) / [Fix: PR #5282, 3.1.16.0](https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/5282) (fixed in 3.1.16.0)
- CVE-2026-63420: PSD RawColor indexed image out-of-bounds read in `interleave_row` [advisory](https://github.com/AcademySoftwareFoundation/OpenImageIO/security/advisories/GHSA-x877-h4xx-5m5j) / [Fix: PR #5307](https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/5307) (fixed in 3.1.16.0)
- CVE-2026-63419: OpenImageIO IFF ZBUFFER tile read writes past caller tile buffer [advisory](https://github.com/AcademySoftwareFoundation/OpenImageIO/security/advisories/GHSA-w6wc-gcf4-5pj2) / [Fix: PR #5268](https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/5268) (fixed in 3.1.16.0)
- CVE-2026-59956: Heap-buffer-overread in IffInput::readimg() when ZBUFFER flag is set [advisory](https://github.com/AcademySoftwareFoundation/OpenImageIO/security/advisories/GHSA-hjfv-gvxc-qgvh) / [Fix: PR #5251](https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/5251) (fixed in 3.0.20.0, 3.1.15.0)
- CVE-2026-59181: Stack buffer overflow in OpenImageIO Cineon reader via unchecked numberOfElements [advisory](https://github.com/AcademySoftwareFoundation/OpenImageIO/security/advisories/GHSA-xh8r-vmqq-56pp) / [Fix: PR #5250](https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/5250) (fixed in 3.0.20.0, 3.1.15.0)
- CVE-2026-59156: Unbounded recursion in FITS header parser leads to stack overflow [advisory](https://github.com/AcademySoftwareFoundation/OpenImageIO/security/advisories/GHSA-xvwr-x6ch-v2fq) / [Fix: PR #5248](https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/5248) (fixed in 3.0.20.0, >= 3.1.15.0)
- CVE-2026-50291: Segmentation Fault in BmpInput::read_native_scanline / [advisory](https://github.com/AcademySoftwareFoundation/OpenImageIO/security/advisories/GHSA-q3c7-3225-66h7) / [Fix: PR #5030](https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/5030) (Fixed in 3.0.16.0, 3.1.11.0)
- CVE-2026-43909: Signed integer overflow in SwapRGBABytes loop index leads to out-of-bounds read/write in DPX ABGR decoder / [advisory](https://github.com/AcademySoftwareFoundation/OpenImageIO/security/advisories/GHSA-g267-j53j-5258) / [Fix: PR5170](https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/5170) (Fixed in 3.0.18.1, 3.1.13.1)
- CVE-2026-43908: Signed integer overflow in ConvertCbYCrYToRGB leads to heap out-of-bounds write in DPX 4:2:2 decoder / [advisory](https://github.com/AcademySoftwareFoundation/OpenImageIO/security/advisories/GHSA-2jr5-q49v-3858) / [Fix: PR5170](https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/5170) (Fixed in 3.0.18.1, 3.1.13.1)
- CVE-2026-43907: Integer overflow in QueryRGBBufferSizeInternal leads to heap out-of-bounds write in DPX decoder (kCbYCr and kABGR) / [advisory](https://github.com/AcademySoftwareFoundation/OpenImageIO/security/advisories/GHSA-cq46-hp4h-cvfr) / [Fix: PR5170](https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/5170) (Fixed in 3.0.18.1, 3.1.13.1)
- CVE-2026-43996: Integer wraparound in bounds check of decode_pixel leads to out-of-bounds read in TGA paletted image decoder / [advisory](https://github.com/AcademySoftwareFoundation/OpenImageIO/security/advisories/GHSA-mq8j-73c4-cr55) / [Fix: PR5165](https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/5165) (Fixed in 3.2.0.1, 3.1.13.0, 3.0.18.0)
- CVE-2024-55194: Broken pgm had memory access error leading to heap overflow / [advisory](https://github.com/AcademySoftwareFoundation/OpenImageIO/security/advisories/GHSA-583r-43f7-cw8w) / [#4559](https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/4559) (Fixed in 3.0.2.0, 3.1.4.0)
- CVE-2024-40630: Fixed incorrect image size for certain HEIC files.
[advisory](https://github.com/AcademySoftwareFoundation/OpenImageIO/security/advisories/GHSA-jjm9-9m4m-c8p2) (Fixed in 2.5.13.1)
- CVE-2023-42295: Fix signed integer overflow when computing total number of pixels while reading BMP files. [#3948](https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/3948) (by xiaoxiaoafeifei) (Fixed in 2.5.3.0/2.6.0.1)
- CVE-2023-36183: Heap-buffer-overflow while reading ICO files [#3872](https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/3872) (by xiaoxiaoafeifei)
- TALOS-2022-1627, CVE-2022-41977: Guard TIFF reads against corrupt files with buffer overflows. [#3628](https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/3628) (2.4.5/2.5.0.0)
- TALOS-2022-1626 / CVE-2022-41794, TALOS-2022-1632 / CVE-2022-41684, TALOS-2022-1636 / CVE-2022-41837: Exif (all formats that support it, TIFF/JPEG/PSD) fix bugs where corrupted Exif blocks could overrun memory. [#3627](https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/3627) (2.4.5/2.5.0.0)
- TALOS-2022-1634 / CVE-2022-41838, TALOS-2022-1635 / CVE-2022-41999: Fix DDS reading crashes for cubemap files when a cube face was not present, and check for invalid bits per pixel. [#3625](https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/3625) (2.4.5/2.5.0.0) #3625 (2.4.5/2.5.0.0)
- [x] libheif 1.11 (from 1.3), which supports many additional features of that format. [#4380](https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/4380)
- [x] WebP 1.2 (from 0.6) which lets us use their exported CMake configs and retire FindWebP.cmake. [#4354](https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/4354)
- [x] pybind11 2.6 or 2.7 (from 2.4). [#4297](https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/4297)
- [x] [#4158](https://github.com/AcademySoftwareFoundation/OpenImageIO/issues/4158) Eliminate the last few places where we use Boost and eliminate it as a dependency.
<br>
## API Modernization
- [ ] [#4159](https://github.com/AcademySoftwareFoundation/OpenImageIO/issues/4159) Using span and string_view instead of raw pointers
Scrub the APIs to find where we have API calls that take a pointer and size
(or worse -- just a pointer and an assumption about the size), and instead
use `span<T>` / `cspan<T>` for things that are like arrays, and
Using named keyword/value lists more extensively in imagebufalgo.h
- [ ] Deprecate as much as possible of the old printf-style string formatting we use internally, instead using the fmt/std::format style everywhere. This includes changing Strutil::format to alias to Strutil::fmt::format (currently it aliases to Strutil::old::format, which uses printf style).
- [ ] Remove deprecated API elements
Hunt for things marked DEPRECATED and try to get rid of them where possible.
If we can't remove them, at least make sure they are marked as
`OIIO_DEPRECATED` (or `[[deprecated]]` in modern C++).
## The rest
Other initiatives we hope to have completed by the time of this next
major release:
- [ ] [#4164](https://github.com/AcademySoftwareFoundation/OpenImageIO/issues/4164) Better color management
Regardless of OCIO availability, version, or contents of any configs, have universal support for the [canonical color spaces](https://github.com/AcademySoftwareFoundation/MaterialX/blob/main/documents/Specification/MaterialX.Specification.md#color-spaces-and-color-management-systems) that seem to be the common consensus of ACES, MaterialX, and USD.
- [ ] Preliminary Rust bindings for most of the OpenImageIO APIs.
- [x] Python wheel construction so `pip install openimageio` will be an easy way
for users to install the whole banana. [#4428](https://github.com/AcademySoftwareFoundation/OpenImageIO/issues/4428)
If there is something you think should be on the roadmap for the next major
release but is not, please open an issue or discussion to propose it, or
bring it up at a TSC meeting, or on the mail list or Slack channel.
## Parking
Here is where we will put things that definitely should be on the roadmap, but
that need not be completed in time for the fall 2026 release of OIIO 3.2.
Use of "AI coding assistants" is permitted on this project, with the following
guidelines and principles.
Summary of our core values:
- Human must always be in the loop, and is the responsible party for
the contents of a PR (including fully understanding and being able
to explain, defend, and modify it in response to review comments).
- Interact with the project and community yourself, not by agent.
- Disclose what tools you used and how. At a minimum, we require an
"Assisted-by: TOOL/MODEL" line in the commit comments and PR description.
- Don't waste maintainer's time with low quality PRs.
The long version:
### Human authorship and interaction with the project
**The human making a PR is considered the author** and is fully responsible
for the code they submit.
**No PRs should be submitted, reviewed/merged, or deployed without a human
fully in the loop.** You may use agents locally to help you write code (and
docs, tests, etc.), but you may not use agents to autonomously interact with
the project community. Submit your own PRs, write your own discussion
comments, respond to community members in your own words, approve or reject
code in review using your own brain.
Code autonomously submitted without an identifiable human author and
responsible party will be immediately closed, and the account associated with
the submission may be banned from future project participation.
**DO NOT "vibe code."** This term refers to a pure prompt-to-behavior cycle
where the human accepts un-inspected code based on the program behavior alone.
We don't want this. Code that must be maintained by the project should be
truly understood. Project code is ideally the result of discussion / pair
programming between the author and assistant, but even if the machine is doing
most of the work, all code should be approved and fully understood by the
author prior to PR submission.
### Professionalism and quality
**The usual high quality level of PRs should be maintained regardless of tools
used.** This includes not only code, but also the design, testing, PR
preparation, and the author's ability to explain and defend the code and
respond to questions and requests for changes. We never want to hear "but the
AI..." as an excuse for anything.
**PRs should always be [reviewed](CodeReview.md) and approved by someone other
than their author if at all possible.** This is especially true of code with
machine-generated components, because of the additional risk that the
submitter/operator may not fully understand what they did not write
themselves. This is part of the reason for AI tool disclosure -- to ensure
that another pair of eyes is on it. AI code review may be requested for
additional input, but the AI cannot give "approval", nor merge code itself.
Even when using a coding assistant to write the code, you are strongly
encouraged to write your own PR description, in your own words, if for no
other reason than to force yourself to understand the code well enough to
describe it to others.
### Disclosure
**Disclosure is required.** Patches that were written with the aid of AI
coding assistants must have, at a minimum, the following line in the commit
comment and PR description body:
Assisted-by: TOOL / MODEL
Ideally, to the extent practical, the PR description should also have a brief
description of how the tool was used, including the gist of key prompts or
summarizing the direction of the dialog. A full log is not necessary; a short
summary will do.
There are several reasons for this disclosure/documentation, even though it
will sometimes be inconvenient:
- Reproducibility: We want even automated work to be reproducible by others in
principle, just like a scientific paper.
- Education: We want to actively teach each other what tools, prompts, or
methods are successful at making high quality results for the project.
- Metrics: We would like the ability to look back and compare assisted vs
non-assisted code for things like defect rate (how often were new bugs
introduced or commits needed to be reverted), whether people are more likely
to write comprehensive tests when using assistants, etc.
- Compliance: Many key users and developers are in companies where they need
to disclose how AI tools were used in their work, and in other software that
they use. Let's make it easy for them.
- Insurance: Should future legal decisions radically change the IP status of
AI-generated code, or if particular models are implicated in copyright
violations, we want a way to find out which contributions might need to be
revisited in order to bring the project back into compliance.
### Intellectual Property
Regardless of how the code came to be -- from your head alone, from a friend,
from Stack Overflow, from a blog post, from reading other code bases, or from
a coding assistant -- we expect the terms of the
[DCO](https://developercertificate.org/) and CLA to apply, and for the author
to take reasonable care that code is not copied from a source with an
incompatible license.
You may find that your confidence about complying with the DCO+CLA depends on:
- If you provided a detailed specification and how much you edited or guided
the results yourself.
- Whether you are modifying or extending existing code versus writing large
amounts of entirely new code.
- The degree to which the generated code seems to fit into our structure and
idiomatic style, and therefore seems unlikely to be copied from elsewhere.
- Whether your tool has guardrails to prevent answers that are too similar to
existing code, for example as claimed by [Claude](https://privacy.claude.com/en/articles/10023638-why-am-i-receiving-an-output-blocked-by-content-filtering-policy-error) and [Copilot](https://docs.github.com/en/copilot/how-tos/manage-your-account/).
### Extractive submissions
**Maintainer time and attention are precious commodities**, and use of coding
assistants is not an excuse to submit poor PRs or to externalize costs onto
maintainers/reviewers (such as responsibility for understanding, testing, or
fixing PRs that the human author does not have a full understanding of).
Maintainers are free to take counter-measures against submitters of sub-par
PRs, or other violations of this policy, up to and including banning habitual
abusers from future participation in the project.
**We discourage use of AI tools to fix GitHub issues labeled as "good first
issue" or for "Dev Days" work.** Cultivating and educating new contributors is
important, and as such, we do not want people to swoop in and use automated
tools to trivially solve tasks that were curated specifically for somebody to
actually learn from. We expect each developer to implement at most one "good
first issue" per Dev Days occasion, and we encourage you to solve the problem
with your own brain to maximize the learning experience.
### Exceptions
This AI tool use policy is not meant to encompass cases such as:
- "Smart auto-complete", spell-checking, grammar checking, or other uses that
aren't really contributing substantively to authorship.
- Use of LLMs to explain code or learn about the codebase, answer basic
programming questions, help with background research, or audit the code
for bugs that are subsequently confirmed and fixed by people.
- Language translation for non-fluent English speakers or other accessibility
accommodations.
- Trivial or de-minimis fixes such as fixing a typo, obviously wrong variable
use, etc.
- Reviewing your own code for mistakes prior to submitting a PR (as long as it
isn't making the fixes for you).
### Creative contributions -- out of scope
This project repository contains code for tools, and their documentation. We
don't really directly host any artistically creative work here, so we have not
addressed additional policy issues that might come up in such a context.
If you are representing a different project that is using this document for
guidance or inspiration for your own policies, please be aware that you may
have needs that are not covered by these policies but deserve well-crafted
guidance.
### References and inspiration
Our policy has been informed and inspired by the following efforts in other communities:
- [LLVM AI Tool Policy](https://llvm.org/docs/AIToolPolicy.html)
- [Fedora Project policy on AI-assisted contributions](https://communityblog.fedoraproject.org/council-policy-proposal-policy-on-ai-assisted-contributions/)
- [Linux Foundation policy on Generative AI](https://www.linuxfoundation.org/legal/generative-ai)
- [Rust policy on rejecting burdensome PRs](https://github.com/rust-lang/compiler-team/issues/893)
- The METR paper [Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/)
- [GitHub Blog: Rethinking open source mentorship in the AI era](https://github.blog/open-source/maintainers/rethinking-open-source-mentorship-in-the-ai-era/)
For monthly patch releases, we don't have any formal alpha/beta/RC stages.
For the annual minor or major releases we have a staged release:
- `alpha` is everything after we have branched, in which we think all the big
new features have been added, and while we will make continued changes and
fixes, we are aiming for increased stability and to minimize additional
breaking changes prior to the release.
- `beta` should be about one month prior to the final release date. We create
a tag `v1.2.3.4-beta` and announce the beta (see below "Making the
release"). Once we tag a beta, we try very hard not to allow any further
changes that change the API or are not ABI-compatible (i.e., safe fixes
only).
- `RC` or release candidates around one week prior to the final release date.
This is a trial run, hopefully identical to the final release (except for
removing the RC label and making a new tag). The only changes allowed after
RC are fixes to the most critical bugs or build breaks that would be bad for
users to encounter. If critical problems are found with RC1, they should be
fixed and immediately re-tagged as RC2, etc. All other fixes should be
postponed until the next month's patch release.
### Making the release
A final release is made on the scheduled date, provided the latest RC has
survived for at least a few days of testing without finding any critical
problems. Don't make the release final until you are sure there are no
truly critical bugs or build breaks that users will encounter.
The following are the steps for making the release:
1. Double check the top-level CMakeLists.txt to ensure the right version
number and type of version designation (i.e.,
`PROJECT_VERSION_RELEASE_TYPE` should be set appropriately).
2. Edit CHANGES.md to reflect the correct date of the release and ensure it
includes any last-minute changes that were made during beta or release
candidate stages. Please see the section "Generating release notes" below.
3. Push it to **your** GitHub fork, make sure it passes CI.
4. Tag the release with a signed tag:
$ git tag -s v1.2.3.4 # add a signed tag
$ git tag -v v1.2.3.4 # verify the tag
Be sure that the tag also reflects if it's an alpha (v1.2.3.4-alpha),
beta (v1.2.3.4-beta), release candidate (v1.2.3.4-RC1), or developer
preview (v1.2.3.4-dev). If it's a final release, just use the plain
version number (v1.2.3.4).
Note that the tag signing requires you to have a GPG key set up and
associated with your GitHub account. Please see [GitHub's signed tagging instructions](https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-tags) for more information.
If you get errors messages "gpg: signing failed: Inappropriate ioctl for device", then the solution is to first:
$ export GPG_TTY=$(tty)
5. If this will now be the recommended stable release, move the `release`
branch marker to the same position.
6. Push it to GitHub: `git push OpenImageIO release --tags`
(This example assumes "OpenImageIO" is the name of the remote for the
Loaded 100 of 1188 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Larry Gritz
