Typo introduced in #3894 broke our ability to build against libheif <
1.13. A conditional compilation that should have kicked in for >=1.16
was mistakenly written as 1.6.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This is Update 4, the original PR is ongoing and we will continue to
update as additional people relicense.
Per PR #3905 (https://github.com/OpenImageIO/oiio/pull/3905), update the
RELICENSING document with the names of people and companies who have
signed on to the relicensing thus far, and update the SPDX license
notices on source files whose extant authorship (per 'git blame')
comprises only authors who have relicensed.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Proposal from Thiago to add Jesse Yurkovich, representing Blender, to
the TSC.
Vote passed by majority of TSC members voting yes on the PR itself.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Fixes#3950
Fix the div-by-0 when handling unassociated alpha during a DXT4 DDS
load. The code mirrors what other formats do by first checking to see if
the alpha value is not 0 before continuing.
Added the 218-byte file from the bug to the local test suite.
---------
Signed-off-by: Jesse Yurkovich <jesse.y@gmail.com>
Enable set of Boost_NO_BOOST_CMAKE through make using
MY_CMAKE_FLAGS=-DBoost_NO_BOOST_CMAKE:BOOL=OFF.
Signed-off-by: Mikael Sundell <mikael.sundell@gmail.com>
The incoming length of the RLE data could be 0. In this case we would
attempt to take the address of the 1st element of an empty vector.
Added a new 1004-byte file to the local test suite.
Fixes#3951
---------
Signed-off-by: Jesse Yurkovich <jesse.y@gmail.com>
Fixes#3952
Several interrelated fixes to guard against corrupted files:
* Extend the `limits:imagesize_MB` sanity check to Targa files to
address #3952 specifically, and check in that failing example file into
the testsuite. (N.B.: Only a couple readers use this, TIFF and now
Targa. it should be extended to the others. Mackerel!)
* Improve the error message used for the above, both for Targa and TIFF.
* For low-memory situations, perhaps the default of 32GB is too large.
Make the default be the minimum of 32GB and 1/2 the size of physical
memory on the machine. (An app that legit expects humongous files is
expected to raise the limit appropriately.)
* The generic "Read error: hit end of file" we sometimes issued from
`ImageInput::ioread()` was hard to discern which reader it came from, in
cases where the file extension was absent or misleading and a succession
of files were tried. By changing this error message to say which format
reader triggered the error, it might make these errors easier to track
down in the future. This also changed the reference output of a few test
involving truncated or corrupted files.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
On overloaded VMs, the timer_test is ureliable and fails spuriously too
often.
Two strategies in this patch, hope some combination does the trick:
1. Increase the absolute error allowed for the timing tests when doing
CI runs.
2. Add just a bit more relative error (via a new unit test macro,
OIIO_CHECK_EQUAL_THRESH_REL).
Hopefully this will cut down on CI failures for unit_timer test case.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Disable GCC warnings around mismatched new/delete for shared_texturesys
creation
Signed-off-by: Shootfast <markboo99@gmail.com>
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Co-authored-by: Larry Gritz <lg@larrygritz.com>
* Add string literal operators for ustring and ustringhash so you can
write: ``` "foo"_us "bar"_ush ``` Note that on Cuda (but not on the
host), this ustringhash construction is constexpr.
* Add a new `ustringhash(const char*, size_t)` constructor, which is
also constexpr on the device side.
* Mark the existing `ustringhash(string_view)` constructor as constexpr
on the device side.
We can mark those things as constexpr on the device side because on the
Cuda side, the implementation only takes the hash (which is constexpr),
and does not actually add a ustring to the table (which cannot be
constexpr). To make such declarations easy, add a new macro to
platform.h, `OIIO_DEVICE_CONSTEXPR` which is constexpr for the device,
merely inline for the host.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The most important fix is to remove links to the old CLAs and update
references to the mail list to the new one.
Also spruced up some language while I was there, hopefully to coax out
improved PRs and Issues.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
When building the code with Xcode ("cmake -G Xcode"), there are 1000+
warnings about implicit 64 to 32 bit conversion.
This change adds a compile option to suppress such warnings when
generating an Xcode project.
Signed-off-by: Anton Dukhovnikov <antond@wetafx.co.nz>
This is Update 3, the original PR is ongoing and we will continue to
update as additional people relicense.
Per PR #3905 (https://github.com/OpenImageIO/oiio/pull/3905), update the
RELICENSING document with the names of people and companies who have
signed on to the relicensing thus far, and update the SPDX license
notices on source files whose extant authorship (per 'git blame')
comprises only authors who have relicensed.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Text rendering doesn't work with utf-8...
I can't render chinese, arabic... or special chars as éèûîà...
It seems that the string function to convert utf-8 to unicode is quiet
simple.
Fixes#3933
[Note]: OpenImageIO is used in the [gpx2video
project](https://github.com/progweb/gpx2video).
---------
Signed-off-by: Nicolas VIVIEN <nicolas@vivien.fr>
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Co-authored-by: Larry Gritz <lg@larrygritz.com>
When an incoming 1-dimensional, buffer-like, object was used in Python,
the corresponding check inside OIIO would overflow if its length
exceeded a signed 32bit value.
An error similar to `Pixel data array error: Python array shape is
[2304000000] but expecting h=24000, w=24000, ch=4` would be raised.
I modified a few similar calculations and tried to follow the
predominant pattern already used.
Tested using the following snippet which would previously fail:
```
import OpenImageIO as oiio
import numpy as np
filename = "t:/foo.png"
xres = 24000
yres = 24000
channels = 4 # RGB
pixels = np.zeros(xres * yres * channels, dtype=np.uint8)
spec = oiio.ImageSpec(xres, yres, channels, "uint8")
image = oiio.ImageOutput.create(filename)
image.open(filename, spec)
if not image.write_image(pixels):
print("error = ", image.geterror())
image.close()
```
---------
Signed-off-by: Jesse Yurkovich <jesse.y@gmail.com>
Per PR #3905 (https://github.com/OpenImageIO/oiio/pull/3905), update the
RELICENSING document with the names of people and companies who have
signed on to the relicensing thus far, and update the SPDX license
notices on source files whose extant authorship (per 'git blame')
comprises only authors who have relicensed.
This is part 2, the original PR is ongoing and we will continue to
update as additional people relicense.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Intel's servers have again updated the default icx that you get when you
install from their yum repository, and the new one seems incompatible
with our libstdc++ toolchain. So lock down on 2023.1.0 for now so that
our CI can pass.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This updates ICOInput to use the IOProxy model so it can read from files
or the IOMemReader
Signed-off-by: Jason Baumeister <jason@jasonbaumeister.com>
During the CMYK to RGB conversion in PSDInput, the alpha channel was
never copied to the RGBA output. Converting a PSD to JPEG worked because
it ignored all the 0 alpha values, but converting to WebP would result
in a fully transparent image.
I added a cmyk-with-alpha.psd test to the psd-colormodes test to check
this
Signed-off-by: Jason Baumeister <jason@jasonbaumeister.com>
Clean up the `#if` tests that disable Intel SIMD when not on Intel
architectures, which incorrectly disabled ARM NEON when on ARM.
Fixes#3909
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Update the ImageInput documentation to not use the deprecated
`read_image` APIs.
Based on the discussion in #3902
---------
Signed-off-by: Jesse Yurkovich <jesse.y@gmail.com>
Per PR #3905 (https://github.com/OpenImageIO/oiio/pull/3905), update the
RELICENSING document with the names of people and companies who have
signed on to the relicensing thus far, and update the SPDX license
notices on source files whose extant authorship (per 'git blame')
comprises only authors who have relicensed.
This is part 1, the original PR is ongoing and we will continue to
update as additional people relicense.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Our own FindWebP.cmake was setting up targets that didn't
match the capitalization of the cmake configs that come with WebP itself.
Signed-off-by: Benjamin Buch <benjamin.buch@technoteam.de>
This batch is for files that LG (relicensed!) has contributed, in
addition to >2 other authors. Also add appropriate license notices to
some that had no posted license before due to oversight.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Add copyright/license notices to some mixed-authorship files that
lacked them before (all related to tests)
* Get rid of the pointless year in many copyright notices (change to the
current LF recommended wording)
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Since LG has re-licensed all his old code under Apache-2.0, we can
change the license notice from BSD to Apache-2.0 for all files in which
LG is the only extant author. Also change all files that LG co-authored
with other people to specify "BSD-3-Clause and Apache-2.0".
Some files (mostly test scripts) had no license notice, so in those
cases, we added one.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
By majority vote of the TSC, we are changing OpenImageIO's main
incoming/outgoing open source license from BSD-3-Clause to Apache-2.0.
New code submitted from July 1, 2023 will be contributed under the
Apache 2.0 license.
Old code submitted prior to July 1 2023 is under the original BSD
3-clause license, unless relicensed per the process described in the new
RELICENSING.md file. We hope to get the vast majority of the code base
relicensed quickly and we will document which set of files contain a
mixture of licenses.
Instructions to everybody reading this:
* If you are contributing code under the authorization of an employer,
please let them know we are changing to Apache 2.0 and make sure that
you are authorized to make all future contributions to under that
license.
* If you made prior contributions to OpenImageIO over the years and are
willing to relicense your code under the Apache 2.0 license, please
submit a PR adding your name to the list in RELICENSING.md. If you were
working for a company that might own your IP output at the time you made
those contributions, please also ask somebody there (who has the
authority to do so) to submit a PR adding the company name as well,
indicating that they are re-submitting all of the old contributions
under the new license.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
By unanimous vote of the OpenImageIO TSC, as of July 1, 2023, we
adopt the technical charter, Code of Conduct, new CLAs, and GOVERNANCE
document.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The underlying farmhash algorithm (which is the basis for
Strutil::strhash, and thus ustring's hash representation) turned out to
not be properly constexpr under certain circumstances and needed to be
expressed a different way to be fully constexpr on all platforms and for
all strings.
New implementation suggested by Alex Wells.
Also, make sure string_view ctr from char* can be constexpr, even for
C++14. This is related for the sake of the strhash calls that, when
taking string literals, actually choose the variety that accepts a
string_view.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Remove leftover logic from when we used TravisCI.
* Pick a default for LLVM_DISTRO_NAME so it works when the script is run
standalone when not on GitHub Actions CI. (But it's still really only
meant to run on x86 Ubuntu, and not designed for anything useful outside
of our CI.)
* Eliminate all the complexity with old LLVM versions -- in practice,
our CI only uses this to download fairly modern clang versions.
Fixes#3891
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The strukturag repo for libheif recently upgraded the version it was
serving from 1.14 to 1.16, and that caused our tests to fail because (a)
some of the codec support has been moved to plugins that are loaded at
runtime and those are now distributed in packages we need to install
separately; (b) you must call heif_init() or it won't know about the
optional plugins.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
If colorconvert, ociolook, ociodisplay, ociofiletransform fail
because OCIO support is not enabled, make sure the error message
makes that clear.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Make sure we can input exif tags as strings containing numbers when
number are required.
E.G. if my input is "Orientation"=>"8", i would like the jpeg output to
contains the value 8 for the exif tag orientation.
PR #2147, addressing issue #1844, is related to an apparent OpenEXR bug
wherein single channel tiled images where the tile dimensions were not a
power of 2 and >= 16 would not work properly with dwa compression. We
work around the problem by automatically substituting zip compression in
those cases. But it was implemented with a subtle bug, forgetting that
tile_size == 0 for untiled images, and thus we were silently switching
from dwa to zip compression for all *untiled* single channel images of
any size (which presumably were never problematic). This patch fixes the
test so we are not changing the compression for the untiled files.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
J2K output was defaulting to UINT8 when asked to output an unsupported
type. It should have been UINT16 to try to minimize the amount of
precision loss.
Fixes#3876
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Everybody please go to https://lists.aswf.io/g/oiio-dev and sign up for
the new mail list as soon as possible. The old list is on its last legs
and having trouble, let's get this done.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
When reading a PSD file, if it contains a thumbnail, we read it from the
in-memory blob with a combination of an IOMemReader and an ImageBuf.
But... we always named it "thumbnail.jpg" without considering that
multiple simultaneous PSD files reading identically named images (but
which are in fact different) could cause a weird kind of clashing in any
underlying use of ImageCache!
The simple fix is to use ordinary ImageInput approach to reading the
thumbnail from the IOMemReader, but just store it in the ImageBuf.
Fixes#3824
Signed-off-by: Larry Gritz <lg@larrygritz.com>
A variety of minor documentation fixups:
- Use Python 3 style 'print' statements consistently everywhere (some
snippets had a mix of both)
- Correct the usage of `ImageSpec::channels` that should be
`ImageSpec::nchannels`
- Template syntax errors in two C++ snippets
- Document the new default for `max_memory_MB`
Various improvements to OpenCV support:
* IBA::from_OpenCV and to_OpenCV now set error messages for every type
of failure they encounter, instead of returning false with no
explanation. For from_OpenCV, the error message can be retrieved from
the ImageBuf returned; for to_OpenCV, the error message can be retrieved
from the global OIIO::geterror(). Several assertions were removed,
instead relying on error return codes and messages.
* Add global OIIO attribute "opencv_version" which allows you to find
out what version of OpenCV OIIO was built against, which was previously
impossible to know without seeing the build-time status messages. This
can be retrieved via
OIIO::get_int_attribute("opencv_version")
and also you can find out from the command line with
oiiotool --echo "{getattribute(opencv_version)}"
The return value is an integer encoded as `10000*major + 100*minor +
patch`, and it will return 0 if no OpenCV support was enabled when
OpenImageIO was built.
* For OpenCV >= 4.0, to_OpenCV can now correctly construct a 'half'
matrix. For OpenCV 3.x, we turn it into a float matrix as before, since
CV_16F was only supported starting in 4.0.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Just a routine update of these files as I proofread them in light of the
move to ASWF. This patch is addressing the "old" project, there will be
further ASWF-specific edits for changes that will take effect after the
project is fully moved.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
In `jpegoutput.cpp` when computing the density fields the relationship
between `XRes`, `YRes` and `aspect` is `aspect = XRes / YRes`.
However this relationship was not respected in one specific case, when
`aspect >= 1.0` and `XRes` and `YRes` are not set.
This PR solves this problem by enforcing the correct relationship.
Related to: https://github.com/OpenImageIO/oiio/issues/2822
The Intel yum repo renamed this package a couple weeks ago, and it broke
our CI. Update to the new name.
Adjust fmt use to solve icc warning.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
When reading a raw image, setting the `user_flip` to 1 meant ignoring
the `flip` value (to avoid buffer rotation from LibRaw).
However the value 1 for the `user_flip` is **not neutral**: values lie
between 0 and 7 (see
https://www.libraw.org/docs/API-datastruct-eng.html#libraw_image_sizes_t),
and 1 corresponds to a horizontal mirror (in my understanding of the
source code).
The proposed fix is to use 0 as `user_flip` value for ignoring the
`flip` value.
This modification invalidates the changes made in this PR:
https://github.com/OpenImageIO/oiio/pull/3847.
Therefore, flip to orientation conversion has been adapted to match
exactly LibRaw's documentation, which handles only rotations (+90, +180,
-90 degrees).
The previous PR was only tested on images of landscape and human faces,
hence the mirroring issue remained unnoticed.
When reading images containing text however the problem is clearly
visible.
This PR fixes this mirroring issue on the following 4 orientations: 0,
+90, +180 and -90 degrees rotation.
Other orientations could not be tested as they don't seem to ever appear
on raw images datasets.
When reading a raw image while having a `user_flip` set to 1, we set the
orientation Exif tag to LibRaw's flip value (so that LibRaw doesn't
rotate and/or mirror the buffer but the image still gets displayed
correctly).
However LibRaw's convention for flip values differ from Exif orientation
values: https://www.libraw.org/comment/5176#comment-5176.
In this PR we fix this issue by converting LibRaw flip values to Exif
orientation values. This conversion is based on:
- https://www.libraw.org/comment/5178#comment-5178 (conversion proposed
by LibRaw's maintainer)
-
https://github.com/LibRaw/LibRaw/blob/6fffd414bfda63dfef2276ae07f7ca36660b8888/src/write/file_write.cpp#L22
(LibRaw's function for applying the flip).
Tested on:
- CR2 images with flip values 1, 5 and 6
- ARW images with flip values 1 and 5
(unfortunately I could not find any image covering other flip values).
Since the conversion is `[1, 2, 3, 4, 5, 6, 7, 8] -> [1, 2, 4, 3, 5, 8,
6, 7]`, the only test that came out with different results before and
after the change is the one with flip value 6:
in that case, the image orientation got corrected by the fix.
Fixes#3837
oiiotool already attempted to trim extra channels when outputting to a
format that can't support it. It correctly handled the two common cases:
* More than 4 channels for formats that don't have
`supports("nchannels")` will drop after the first 4 channels.
* More than 3 channels for formats that don't have `supports("alpha")`
will drop after the first 3 channels.
But this omits an interesting (but rare) corner case: a 4 channel file
where the last channel is NOT alpha, and the output file supports an
alpha channel but not arbitrary channels. A concrete example might be an
input file that is exr with channels R, G, B, Z, writing to a PNG file
(which can support RGB or RGBA). We wouldn't want to write what appears
to be an RGBA file if that last channel isn't really alpha.
Also, noticed that oiiotool --chnames didn't properly clear the
alpha_channel and z_channel flags, which is important if the new channel
lists drop one of those channels.
`make_texture()` already took a configuration option "maketx:forcefloat"
(defaults to 1), which if set to 0 disables the conversion to a float
buffer for intermediate computation.
For `oiiotool -otex`, there was no way to specify this. This patch
allows it to be disabled with the new optional modifier
`-otex:forcefloat=0`.
Also, in make_texture, this option only controlled the initial
conversion of the whole buffer to float, but the downsized levels all
were float either way. This patch now uses the forcefloat config option
to also control the buffers used for downsizing.
Disabling forcefloat like this has a speed penalty (the downsizing math
on float buffers is faster than going in and out of float for every
indvidual value) and loses some precision in the process of repeatedly
downsizing to generate the MIP levels. But it can sure save a lot of
memory if you have a ginormous uint8 image and want to avod the 4x
balooning of memory. I don't recommend disabling forcefloat except for
images that can't fit into memory for the texture creation without it.
But when this is the right hammer to use, it can be the difference
between being able to create the texture, or not.
Some other improvements:
* `oiiotool -otex` wasn't outputting ANY status output during texture
output as maketx does, and for that matter, -v and -debug were not being
influencing the behavior of make_texture at all despite it having such
support. So fix to pass down the right flags. Now there are proper
verbose status and debug messages when requested.
* Eliminate useless call to is_monochrome by calling it only when the
previously-computed pixel stats indicate that the average value of all
the channels is the same (if they don't have the same average, there's
no way the pixel-by-pixel values will be the same in all channels for
all pixels). This saves an extra traversal of all the pixel memory
during texture creation.
* Use synchronized/flushed status messages so users can see them as they
are issued for very long maketx tasks. This also included some
conversion from old iostream stuff to fmt-based print.
* For verbose/debug, more fine-grained timing for make_texture, in
particular, print the time spent downsizing and writing each MIP level.
I'm honestly not sure why some GHA CI jobs suddenly started warning
about this in the past day or so. Something must have updated on their
end. But switching to the safer Strutil equivalents fixes it.
`ssize_t` is a posix type, msvc seemingly doesn't know about leading to
a build error with MSVC 2019
usage elsewhere in this files used the `py::ssize_t` type, which seems
to fix this
fmt 10.0 was recently released, and it breaks a lot for us because they
have fully removed legacy support for automatically being able to format
anything that has an iostream `<<` operator. This facility was deprecatd
in 8.1 but could be re-enabled with FMT_DEPRECATED_OSTREAM, but even
that was removed in 10.0.
So this means that we had to write several explicit format handlers for
some types for use with format() and print().
Also, they are even more strict with sprintf/printf which don't honor
the custom formatters as far as I can tell, so in several cases I also
switched those to the std::format/print style.
Fixes#3835
The PSD reader was written 12 years ago by a GSOC student and has
changed little since. The following is not a knock on that student, who
clearly did a good job if his code has survived mostly unomdified for
over a decade. In chasing down problems with handling extremely large
psd files, I've noticed some oddities and inefficiencies that need
fixing. It's not an emergency, so I'll handle it in pieces.
The item addressed by this patch is memory usage and thread concurrency.
It turns out that the reader allocates memory for the scanline reads as
it goes... and basically keeps it in the ImageInput until it's
destroyed, even though it will never actually be used again. I guess
it's just trying to save the next allocation if it reads the same
scanline again or on the next subimage? But this means that if you read
a really big psd image, it's basically doubling the memory use -- it
reads one copy into the user's buffer, but also holds on to this extra
copy for no reason. Also, a bunch of the decoding helper functions
implicitly use this memory, which means we lock a mutex for the duration
of read_native_scanline (which in theory should not be necessary since
all I/O is happening through thread-safe IOProxy calls).
So this patch does the following:
1. Remove both this retained scanline memory as well as some other
scratch memory from the ImageInput, and make it allocated and freed
local to read_native_scanline.
2. Change all the helper methods that used it implicitly so that they
are passed buffers to use. In many cases these have been changed to
`const` methods and in a few cases to `static`, making it much easier to
assure ourselves that they don't modify (or sometimes use at all)
anything in the PSDInput to do their task.
3. The result of these changes means that read_native_scanline doen't
need to modify any of the data methods of the class (though it does use,
without modifcation, some things set up in the course of open). Because
it's now non-mutable, we can also remove the mutex, now allowing
multiple threads to read scanlines from the same file simultaneously
without blocking each other.
4. Some other miscellaneous cleanup, including that some scratch memory
was allocated as std::string (!), whereas it should really be a
`vector<unsigned char>` or `unique_ptr<unsigned char>`, and also
changing some computations from double to float.
There is still some more work to do after this is merged. In particular,
I think now that scanline read is stateless / non-mutable with respect
to the PSDInput object, this is a candidate for adding a multi-scanline
read_native_scanlines method (replacing the default implementation of
looping and calling read_native_scanline serially for each scanline),
which should also be able to read multiple scanlines with some overhead
and allocations done once per block, and with blocks of scanlines
running in parallel using the thread pool. This should be a huge
efficiency gain when reading psd files. But it will wait until the next
patch.
The latest version of libfmt began using the `__float128` type but this
is [not supported in the NVPTX backend for
LLVM](https://discourse.llvm.org/t/compiling-cuda-code-fails/61240/13).
A way to detect this particular platform seems to be to check both
`__CUDA_ARCH__` and `__clang__`.
I am not disabling it entirely on any CUDA target because NVCC 10+
understands `_float128` and generates instructions for it when tested on
godbolt.
Following up from https://github.com/OpenImageIO/oiio/issues/3809.
Co-authored-by: Edoardo Dominici <edoardod@sy1wse0003.al.com.au>
Windows isn't smart about symbolic links. There were a few places in the
testsuite where we used symlinks to essentially de-duplicate test
reference images that were expected to be identical. But then the
resulting archive you get when you ask GitHub for a zip file of the
whole repo (which some people download to get OIIO source rather than
git clone) can't be unzipped correctly on Windows (at least not by
cmake's unzip command).
OK, so just change those links to full copies.
Fixes#3811
The `openexr-images` repo was never updated after the default branch was
renamed to `main`. This will fix the following silent errors in CI:
```
Cloning into '/__w/oiio/oiio/build/testsuite/openexr-images'...
warning: Could not find remote branch master to clone.
fatal: Remote branch master not found in upstream origin
```
Additionally, the cmake attempted to symbolically link local repos like
`oiio-images` if they were found. Unfortunately this type of link will
fail unless running as admin on Windows (which is rare) and the
COPY_ON_ERROR directive did _not_ work here (cmake 3.24; weird;
sigh...). The output will now look something like the following:
```
-- Linking oiio-images from C:/Users/jesse/source/oiio-images
-- Link result failed to create symbolic link 'C:/Users/jesse/source/oiio/build/testsuite/oiio-images': A required privilege is not held by the client.
-- Copying oiio-images from C:/Users/jesse/source/oiio-images
```
When decoding RLE data, some arguments to decompress_packbits() were
inexplicably declared uint16_t (they were passed uint32_t arguments, and
I honestly don't know why we didn't get a warning all along). This
prevented us from correctly reading PSD images whose width was more than
64k, which I suppose are rare enough that it's not hard to understand
why this went unreported for so long.
But anyway, just changing those parameters to uint32_t does the trick.
I ran into the problem described in #2929 and did a little digging to
see if I could find the source of the bad color conversions. I tracked
it down to `fast_exp2()`
https://github.com/OpenImageIO/oiio/blob/a77e36250fadfb4c391688eb1df4b3dd88d5e367/src/include/OpenImageIO/fmath.h#L2050-L2080
and in particular the return statement on L2071. In the process of
trying to figure out what part of that return statement was going awry,
I found that separating the bitcasts to separate variables as in this
PR, the problem went away. All testing was done with the `color_test`
added in #3023.
It also succeeded when adding various `std::cout` lines to check what
the values of each computation were, so it does sound like a compiler
optimization that does or does not get triggered is the problem.
Unfortunately I don't know why it's not failing on the CI builds, I
double checked to make sure I'm using the same settings and flags. I
also tested on two different machines and get the same failures, and
success with this patch, on both.
One strange wrinkle is that even without this patch, color conversions
worked fine when using AVX2 for the SIMD mode.
Summary :
Previous results (without this patch)
- USE_SIMD=0 - PASS
- USE_SIMD=SSE4.2 - FAIL
- USE_SIMD=AVX - FAIL
- USE_SIMD=AVX2 - PASS
With this patch
- USE_SIMD=0 - PASS
- USE_SIMD=SSE4.2 - PASS
- USE_SIMD=AVX - PASS
- USE_SIMD=AVX2 - PASS
Visual Studio 2019 versions 16.11.32228.343, 16.11.33328.57 and
16.11.33423.256(same as current Github build runners) in RelWithDebInfo
mode.
For cases of calls like `ImageOutput.write_image(array)`, this improves
the error messages when the array dimensions don't match the size of the
image we're writing, in two ways:
1. For the case of 2D images where the dims don't match, print what the
array shape actually is and what was expected. That's more helpful than
just saying "can't figure out array shape" and printing the number of
dimensions (but not the shape).
2. A mistake caused a number of error messages to be substituted by a
cryptic "can't handle numpy array with noncontiguous channels", but that
was only supposed to be for one particular case; fix that.
Currently, if you run oiiotool with --bumpslopes and --cdf at the same time (or -obump:cdf=1), the CDF is only produced for the first four (of six) channels. The techniques described in Burley (2019) actually do apply reasonably to the squared derivative channels, so it's useful to allow all six channels to have CDFs here.
maketx and oiiotool-maketx tests updated with new relevant tests.
* Tests for maketx -u, --attrib, --sattrib, --cdf
* maketx was documeted to take --cdf as a simple flag, but was
incorrectly implemented as requiring a (pointless) nonzero argument.
Fix it and hope nobody was depending on the undocumented behavior.
* maketx -v arg never had any effect. Add it back.
* Don't use deprecated ubuntu-18.04 runner. Switch to running on
ubuntu-20.04, but use docker containers that supply ubundu-18.04.
* Bump 'latest' version of openexr, webp, pugixml.
There turns out to be a way to get wedged into a mutual deadlock on
the file handle cache, with how get_texture_info on a udim pattern
checks to see if the attribute requested is present and equally valued
in all constituent files of the udim set. This hits the handle set
pretty hard, and if multiple threads are hammering the same sets and
rapidly cycling textures out of the cache, there can be trouble, it
seems.
Remedies in this patch:
* When it's trying to close files because the number of handles is too
high, we already had it so that if we only exceeded the max_files
limit by a little bit, we would simply try to get the lock, and
abandon the attempt if somebody else is holding it, but if we are
quite a bit over, then stay and wait for the lock to be released.
In this patch, we make the that default behavior -- if we are over
the limit and need to free some handles, simply try to get the lock,
and if somebody else is holding it, leave it all up to them and
return immediately. The pros of this are (a) snappier response by
never waiting at this point, and (b) removing the possibility of a
deadlock, because if you can't acquire the lock, you just move
on. But the con is that lots of threads adding new files may run
faster than the one thread who has the file and is trying to release
files, so it's possible that the number of handles will exceed the
limit by a wider margin now, but hopefully only momentarily. (But
note that this can be overridden with a new ImageCache attribute
"max_open_files_strict", which if set to nonzero revert to the old
behavior. At least for now, in case I am neglecting to understand
some bad consequence of this change.)
* The recursive_mutex controlling access to the ImageInput in each
tile cache record is changed to a recursive_timed_mutex, and when
closing a handle by calling release(), instead of an unconditional
lock, try acquiring a lock for 100ms. If it fails to do so, it may
be in a possible deadlock, so just give up and move on. The only
consequence is that maybe that handle isn't freed after all. It will
then move on to the next one, not the end of the world, but at least
it can't get stuck there.
* In find_file, eliminate the call to check_max_files -- it's not
needed here, since it's also called by open(), which is the only
time that new file handles might be opened. We just don't also need
it again simply because we're adding a file to the cache.
Setting the cmake option OIIO_DISABLE_BOOST_STACKTRACE to ON disables the use of
Boost stacktrace at compile time. This avoids dependencies on helper libraries
like dbgeng.dll.
* test: Add a corrupted test file
* fix(IC): upon tile read error, mark the file as broken
The idea is to keep fruitlessly reading from a corrupted file.
* fix: race condition in TIFF reader
TIFFInput::read_native_tiles had a very subtle race condition. There
is a section where it has a task_set of decompression tasks that are
sent to the thread queue. When the method ends, the task_set goes out
of scope, and its destructor will wait for all the remaining tasks to
finish. All fine, right? Except that other variables in the function
include unique_ptr's with allocated memory that the tasks need. So
what can happen is that if the compiler generates the code such that
the destructor of the unique_ptr happens before the destructor of the
task_set, the memory that the unique_ptr held may be deallocated
before some of the tasks still running in threads have finished.
In practice, we seem only to hit this race condition for particular
cases involving corrupted files, because the early `return` seems
quite likely to run those destructors before all the threads are done,
but it was possible to encounter it for valid files as well, if the
timing is just right. Shocking that we never had it reported.
The solution is just to have an explitit tasks.wait(), which will
block until all the tasks have finished before hitting the actual end
of the method and letting the other destructors to run. And for the
"early return" case, change it to a break so that we go through the
wait call at the end.
I believe that this fix addresses TALOS-2023-1709 / CVE-2023-24472.
There are two ways to fail the FITS magic number check: fail to read
the bytes, or the bytes don't match the magic number. It seemed
suspicious that we call close() for the latter but not the
former. Combine them into a single error response (they had the same
message, anyway).
Also, make sure FitsInput::init() fully clears some other data
structures. Oversight?
I don't know if any of these things are real bugs, but they look
sloppy and potentially problematic, so fixing them up.
* Make lookups of const unordered_maps thread-safe
* added back deleted end brace
* clang-format
---------
Co-authored-by: Larry Gritz <lg@larrygritz.com>
* Be more careful constructing strings from what we read in the header;
corrupted files may make them not be zero-terminated.
Fixes TALOS-2023-1707 / CVE-2023-24473
* Watch out for pixels with palette indices that exceed the size we
allocated for the palette.
Fixes TALOS-2023-1708 / CVE-2023-22845
* Check alpha_type for valid range.
Refactoring in #1641 incorrectly caused this to be set on the last layer
rather than the composite image as before. Now set it on the composite
image and every layer, as all of them use unassociated alpha.
* Better Typedesc testing of 'tostring' with fmt style formatting
commands, and fix a bug that uncovered.
* Refine 'make CODECOV=1' behavior
* Beef up python-imagecache test to test getpixels, more attribute,
invalidate.
* Better testing of python IB construction from numpy array.
* testtex: Use correct guard to avoid compiling unused code for
coverage analysis testtex: get rid of unused 3d warp code.
* Add texture-threadtimes test
* Eliminate redundant code in exroutput.c by implementing
write_scanline using write_scanlines.
* Add Exif write support to PNG output.
Stores any Exif data from the given ImageSpec into the PNG eXIf chunk.
* add test for writing Exif data in png
* add test reference output for libpng versions not supporting the eXIf chunk.
* Update testsuite/png/ref/out-libpng15.txt
Co-authored-by: Joris Nijs <joris.nijs@esko.com>
Sets color space metadata. While conceptually, the main job is to call
`attribute("oiio:ColorSpace",name)`, this is a central spot to add
logic that adjusts or removes other contradictory metadata.
The new 2023 one on the Intel site seems to require a glibc too new to
be compatible with the one on the ASWF CentOS-7 based containers, so
has been breaking our CI for a couple weeks.
We never had a test for `maketx --lightprobe`, so add it to
testsuite/maketx.
Immediately discovered that it only worked correctly for float images,
fails for half or other types. Fix.
* More extensive testing of texture3d: handle vs filename, subimages,
>4 channels, closest sample interpolation.
* Fixes for tex3d > 4 chan that turned out to be broken.
* Some rearrangement for the >4 channel case
* Refactor and simplify accum3d_sample_bilinear so multiple data types
all use the same templated code path.
* This all reduces the amount of untested code in OIIO by another 0.4%.
Would you believe we never had a testsuite test for environment
mapping? Well, it wasn't totally untested, in the sense that OSL
testsuite had one, and obviously it's used in production all the
time. But OIIO's testsuite didn't have any tests of
TextureSystem::environmemt(). We now remedy that.
* Add testsuite/texture-env, contains several environment tests,
including handles vs filenames, subimage selection, different interp
and mip modes, ask for >4 channels.
* New CMake build option OIIO_TEX_IMPLEMENT_VARYINGREF can be used for
testing to disable implementation of the old 1.x style of batch
texture calls. We don't use them, they are deprecated. This option
give us a way to exclude them from code analysis, because we really
don't care.
* Fix bug with >4 channels for environment batch shading!
* Avoid unnecessary spec lookup in TileID constructor.
* Avoid building testtex test_hash during code coverage -- it is an
extremely specialized test, not run in the testsuite, it shouldn't
really count againt good code coverage.
* testtex: better control over temp file creation
* Additional texture tests to improve coverage of udim generation,
and deadling with cache limits.
* Better ImageCache unit tests for perthread create/destroy and
individual tile retrieval. Fix a bug in get_tile for nonexistant
subimage or miplevel.
* Dead code removal: ImageCacheFile::udim_to_concrete() and
find_file_no_add(), which were no longer called anywhere.
* IC stats: fix messed up 'slowest' list ordering when there are broken files
* IC stats: when stats level is high, enable printing of some stats that
we previously skipped over when they were "uninteresting" (e.g., too
little time spent on something, no problematic files, etc).
* IC testing: test a variety of get_pixels error conditions
* IC testing: more tests of IC::imagespec() and get_imagespec(), incluing
error conditions.
* More get_texture_info query testing
* Minor bug fix for int-to-float data conversion in get_image_info
* Test IC query for resolution[3]
* Test IC::reset_stats()
* Fix ImageCache::reset_stats in cases where threads ended
* When checking code coverage, don't abort for failed tests
* Rejigger imagebufalgo_test a bit for better IBA test coverage
* Test zover
* Have testsuite/oiiotool-fixnan test with deep images
* Add to the extisting --fixnan tests a test for 'error' mode.
* [Excluded from backport to 2.4] Make testsuite/oiiotool-color test all colormap names
* Test 2D filters in filter_test
* Exclude python scripts in src/doc from coverage analysis
* Remove redundant attribute-seting code from the python internals
* Remove unused python_array_code function
* Exclude some imported code from coverage testing
* Re-enable and improve coverage testing from 'make CODECOV=1'
* Exclude unused/unexposed parts of xxhash.cpp from compilation
We didn't have thorough tests of pbm/pgm/ppm files, add some.
That immediately uncovered a problem with pbp files.
By spec, pbm (1 bit) files use 1 to mean black, 0 for white.
Confusing, because pgm (grey) and ppm (rgb) use 0 for black.
We had no testsuite exercise of --pdiff, add it.
Along the way, I saw tha we were getting the "you didn't have any
output" warning when using pdiff, so fix that.
And some other minor changes: formatted output update, some other
tightening up of things.
A little bit of jiggling the contents of XYZtoLAB_color to work around
what I think is a compiler error with icx -- it was generating NaNs
for reasons I was unable to understand or fix, but not if I rearranged
the code slightly. All the other compilers and versions had correct
behavior.
Seems like a reasonable assumption for BMP files. It's not strictly
correct -- there are header fields on V4 and V5 BMP files that
communicate color primaries and gamma, but honestly, I do not have any
idea if anybody uses them or whether they're more misleading than
helpful. For now, assuming that a BMP file is meant for output on
an sRGB display seems like it's likely to be true.
It should not have been set in platform.h (inside the cpuid function!)
in the first place. And apparently, though this was ok for Windows ARM,
on Mac ARM it's an error because that symbol is already defined.
So only redefine it if it's currently undefined. And move the whole
check and definition into simd.h where all the other ARM_NEON stuff
lives.
* Build: fmt external version flexibility
The current default behavior is to
1. Check if there's already a copy of fmt found.
2. If not (or if BUILD_FMT_FORCE is ON), download it locally.
3. Copy the relevant fmt .h files as part of the OIIO install in
OpenImageIO_ROOT/include/OpenImageIO/detail/fmt
This patch adds a new INTERNALIZE_FMT option, which defaults to ON,
whch is the old behavior. But if set to OFF at build time, the fmt
headers will NOT be copied into OIIO's install area (instead, it will
have dummy files that in turn include the real ones).
Set INTERNALIZE_FMT to OFF if you are really sure you want users of
OIIO to use the system install of fmt and not have OIIO install a new
copy.
When an externalized fmt is used, also make sure our exported cmake
config files cause the downstream user to find fmt and pass on the
dependency properly (necessary because strutil.h actually requires
properly including fmt headers).
Eliminate extraneous include_directories of fmt
Bump the auto-download version of fmtlib to 9.1
A few minor fixe to the term code, add a "term:filename" optional hint
to redirect output rather than go to stdout, and also change to the
new std::format notation for output.
Fixes#3721
Please read the comments in #3721. This is the "local" fix for the
build break due to the type mismatch. It is a band-aid. And it may be
the best solution for the 2.4 branch if we don't want to potentially
break ABIs by changing the definition of any public types.
Still pending is to examine the issue of whether it was a mistake to
define vbool4 storage for neon as uint32x4_t or if we should change it
to int32x4_t to better match the non-simd reference implementation.
After debating that (and identifying somebody with access to an
ARM-based machine to test the solution for us), we may return to
tackle this more fundamental change.
IOMemReader::pread catch out-of-range read positions.
Fixes#3711
Along the way, I noticed that the system pread & pwrite return a
ssize_t rather than a size_t as our IOProxy methods do, so I need to
catch the negative value returns that they use to indicate errors.
* There was lots of duplicated code between iinfo and `oiiotool -info`.
Refactor much of that into a new src/libOpenImageIO/printinfo.cpp.
These are internal functions, no change to public API.
* Test iinfo --hash, --stats, and with nonzero origin and full size
* Test iinfo and oiiotool --dumpdata of deep image
* Test iinfo for subimages, mips, -f, -s
* Some conversion to new print instead of printf
There is probably more such refactoring to do later, I did the easy
parts for now and that's a start.
Kai Pastor (dg0yt) made an important observation that the way I
recently made OpenImageIO_VERSION be a cache variable might lead to
unexpeted behavior by inadvertently retaining a version number upon
recompiles if the cache itself was not cleared, and this can happen
unexpectedly even when the user isn't trying to override the version.
This patch changs the behavior as follows:
You must use the special `-DOpenImageIO_VERSION_OVERRIDE=...` to
override the version.
* if you do so, you are playing with fire and get what you deserve
if you misuse it or change branches and don't remember to clear
the cache.
If you do a normal build (without _trying_ to override the version),
* you will get the real version number because it IS NOT cached
* if you switch branches, you'll always get the right version number
* ...but switching branches without clearing the cache can still, as
always, have other subtle behavior differences versus doing a
truly fresh build from scratch.
So in short, it still allows a way to override the version if you
really want to, but if you are not trying to do that, you can no
longer get yourself inadvertently into unexpected trouble.
When reading PSD file, if reading the thumbnail doesn't pass basic
sanity checks, we consider that a hard error reading the file. One
check that we do is to make sure that our computation of the byte
width of the uncompressed thumbnail (`bpp/8 * width`) matches what the
file says the `widthbytes` is.
BUT... it seems that it might round up to an even (or maybe 4-multiple?)
of columns, presumably for scanline alignment reasons. And so we were
considering whole PDF files unreadable if they fell into this case.
This PR primarily loosens the test to allow some padding up to a
multiple of 4.
Fixes#3658
* Maya IFF documentation says IFF file has max resolution of 8192, and
our implementation only allows writing RGB and RGBA, so check these
limits when an IFF file is opened for output.
* Make sure scratch space allocates enough for tile padding.
* Check for non-zero image origin coordinates (which are not allowed
in IFF files).
* Change some 16-bit loop variables to uint32_t to avoid possible
overflow.
Fixes TALOS-2022-1654, TALOS-2022-1655, TALOS-2022-1656
In open(), check for non-zero image offsets, which is not supported by
BMP, and issue an error if found. If the spec passed in has nonzero
spec.y, other code in the write functions would fail because they
assumed spec.y == 0 and could therefore access memory incorrectly.
More generally, also put checks in the write functions that issue
errors and take early outs if they find that they are called on a
BMPOutput that is not currently open, for example if open() fails but
the user doesn't check the return codes and proceeds to call the write
functions anyway.
This addresses TALOS-2022-1653 / CVE-2022-43594-CVE-2022-43595
There was a subtle error where we had a reference to an ImageSpec for
one of the subimages, confusingly called m_spec, which we altered, but
then didn't make the same change to the this->m_spec, and in a later
spot we referenced the `m_spec` meaning the one in `this`. This could
lead to a crash.
Fix, but not only that, also rename to avoid the confusion. `spec0` is the
alias we us for the spec for subimage 0, and `spec_s` is the alias we
use when referencing subimage S, and we really never use m_spec
anymore to avoid any confusion. This should make it easier to reason
about this code in the future and avoid these kinds of mistakes.
A second error resulted from the situation where a failure in open()
would return before one or both of m_buf or m_tilebuf were allocated.
If the return status of open() were ignored and the caller
subsequently attempted calls to write_scanline or write_tile, invalid
memory could be accessed. Fix this with a variety of guards not only
based on the allocation status of these buffers, but also issuing
errors (and exiting early) from write_scanline or write_tile if they
are called on an ImageInput that is not in fact open.
Addresses TALOS-2022-1651 / CVE-2022-43592 and
TALOS-2022-1652 / CVE-2022-43593
Fixes#3660
In investigating this issue, I also noticed that now that IOProxy
support can be relied on, the old code that in the psd reader that
directly fumbles with libjpeg is not needed and can be replaced by
just a couple of lines to make an ImageBuf read from an IOProxy.
And it fixes the bug, too. So I'm not even going to try to figure
out what was going wrong with the old code.
I uncovered a very subtle problem that can happen with the interaction
between ImageBuf, using an IOProxy, and the underlying ImageCache.
Remember that when using an IOProxy, it is critical that the lifetime
of the proxy outlasts any ImageInput, ImageBuf, or ImageCache that was
told about it.
Consider the following code:
IOProxy proxy(...); // temporary proxy in this local scope
ImageBuf A ("foo.exr", 0, 0, proxy);
We've set up an ImageBuf that will read from an IOProxy. Behind the
scenes, it made an ImageCache entry that remembers the IOProxy, as it
has to, since it needs to use that proxy for subsequent reads.
Now we read the image, and we use the `force=true` parameter to
force an immediate read of the file:
A.read (0, 0, true);
It reads from the proxy. But the IB and the underlying entry in the IC
still know about the proxy. But in the user's mind, they have fully
read the image, will never read from it again, so of course (they
think) it is safe to close or destroy the proxy, knowing that they
made it last past its use:
proxy.close();
But -- oops! -- the cache entry still has a pointer to the proxy, and
even though it won't need to read from it again, it still will access
it, close it, etc., but the user's proxy may by that point be out of
scope.
This is *technically* not a "bug" in the sense that it is the user who
violated the contract by closing or destroying the proxy before its
last use by anything they passed it to. But in context, it's very easy
to be confused by this -- the user thinks that the only observer of
the proxy is the ImageBuf, and they're done using the ImageBuf at that
point, but because they don't explicitly mention an ImageCache, they
don't realize that the ImageBuf passed the proxy reference along to
the underlying IC that could be backing any ImageBuf.
It's the user's fault in some sense, but let's just say that we have
given them an "attractive nuisance."
That's a long bit of background explanation, but I wanted to force
myself to explain it clearly and make sure it was spelled out in the
commit comments. Now here is the fix:
* When IB::read does an immediate forced read involving an IOProxy,
after the read is complete, tell the cache to invalidate the entry
for the file so that it no longer retains a pointer to the proxy.
* When an IB itself is destroyed or cleared, if it used an IOProxy,
also invalidate the IC entry for that file.
* Unrelated cleanup: avoid a redundant invalidate call in init_spec().
* Make it so that when we build against OCIO 2.2, it is possible to
specify the built-in configs. The logic had previously precluded
this, because at one point it checks to see if the passed config or
contents of env $OCIO was an existing file. Now you can set OCIO to
"ocio://default", for example, and it will be used by OIIO.
* Expose getAliases (I think that has been there for a while but we
didn't expose in in our ColorConfig wrapper).
* Expose isColorSpaceLinear (new to OCIO 2.2)
* New OIIO global getattribute query: "opencolorio_version" returns the
human-readable version string (such as "2.2.0").
* Some miscellaneous other cleanup in color_ocio.cpp
* Change in color methodology for test suite: Previously, for every
call to oiiotool in the testsuite, we passed a `--colorconfig`
argument to oiiotool pointing it to the sample colorconfig that we
ship in the testsuite. Here, I'm changing it so that the cmake setup
for tests sets the OCIO env variable for all tests, so no explicit
--colorconfig is necessary, and thus you won't see a path to the
config in any of the test output files if they need to echo the
command line arguments (and you'll see several test outputs altered
in this way). This is a prelude to later changes that will use the
built-in configs themselves for tests, when available.
More interesting changes coming in subsequent PRs.
When writing a zfile, anything that causes open() to fail will leave
m_tilebuffer at 0 size. Subsequent calls to write_tiles, and also
to close (which cleans up unwritten parts of the buffer at the end)
did not check for this condition and would access out of bounds memory.
While I was in there, I converted the zfile.cpp error messages to the
new formatting convention.
Fixes TALOS-2022-1657
Lately, it has been experiencing intermittent availability problems
that causes our CI to fail while setting up dependencies.
What happens if we don't use the problematic ubuntu-toolchain-r at all?
Nothing, it seems. So eliminate it.
I think that once upon a time, it was necessary to add this repo to
get certain new packages? But maybe they are mainstream enough now
that it's not necessary to add this repo? In any case, it doesn't seem
to hurt anything, it's one less moving part that can't fail if it's
not there.
We use this all the time in our internals, and I'm tired of having to
write Strutil::print. Too verbose. By putting it in the main OIIO
namespace, OIIO internals can reference as simply print(), and
external users now have a choice of saying Strutil::print() or
OIIO::print().
* Add casual tests of oiiotool with no args (print brief usage message)
and when an unknown command.
* Also changed the meaning of -q (quiet mode) to be more quiet -- in
particular, not to print the full help message for errors when quiet
mode is used, only print the error.
* oiiotool test sequence errors.
* Test (and fix) --selectmip and --subimage with out of range indices
Changes:
* Scorecards and analysis -- allow newly launched job to cancel
running old job from the same PR.
* Scorecard only runs on main repo, no need to run on forks.
* Nightly CI only runs on main repo, no need to run on forks.
* Test oiiotool control flow syntax mistakes
* Test oiiotool --set with array of strings
* oiiotool: better adjust_geometry unit testing
Helper utility additions:
* oiiotool --no-error-exit
`--no-error-exit` causes an error to not exit immediately, but
rather to try its best to continue execuing command line
operations. Use with caution! This is mainly for unit tests and
debugging.
`--unittest` also turns this on automatically, so that tests that
are probing failures don't cause an exit.
* unittest.h: If OIIO_UNIT_TEST_QUIET_SUCCESS is defined, don't print
anything extra for unit test success
Make extra sure to clear the oiio:ConstantColor, oiio:AverageColor,
and oio:SHA-1 metadata when writing textures, if those things are not
going to be set.
* Unit test adjust_geometry
* Tests of --set for various implicit and explicit types not
previously covered in the testsuite
* Comment out Oiiotool:get_position, which seems to be unused
* Exercise --iconfig for explicit types
* Add test for --origin and --originoffset
* Add test for --ociodisplay
* Add test for --unmip and --selectmip
It's similar to trim_whitespace, but returns the trimmed result, rather
than operating in-place on a `string_view&`.
Also add unit testing of trim_whitespace and trimmed_whitespace
Along the way, improve the runtest.py script by giving the
info_command() fuction the same silent, concat, and failureok
parameters that the oiiotool() function has. (This lets individual
tests assemble iinfo command lines with a little more flexibility.)
* In the case of DDS cubemap files where a cube face was not present,
we were filling m_buf with 0 bytes before allocating it.
* Add check for invalid bits per pixel.
* Add regression test for another bug that was perviously fixed in
2.4.3.
In one case, we actually had a check for this, but an assignment to an
int made the nonsensical offset appear negative, and we only tested
whether the necessary offset it was bigger than the buffer size.
Keeping it as (unsigned) size_t makes the test work as intended.
In another case, there were several places where we never checked that
we were staying within the exif block, and here we address this by
changing the utility decode_ifd so instead of passing it a pointer to
the ifd, it passes the offset (the pointer turned out to always be
inside the buffer) so it can check the extent for subsequent accesses.
Also some fixes related to squashing undefined behavior sanitizer
cases.
Also, change `operator<<` to output the string, not the hash itself.
These changes help push us more toward the original intent of
ustringhash being a drop-in replacement for ustring, any time it's
more desirable to be using around the hashes rather than the pointers.
* oiiotool tests for expressions var() and getattribute()
* Make sure we test oiiotool IMG[] expressions and several specific
special attributes to retrieve.
* Add coverage of oiiotool --keyword and --clear-keywords
When reading the image (which can have signals to jump around in the
array), we checked for out-of-bounds x coordinate, but not y, and some
corrupted files could overrun buffers as a result.
The existing syntax `TOP.name`, if the name wasn't found, produced
an error.
This PR adds another syntax: `TOP[name]` which is identical if the
named metadata is found. But if not found, this syntax just evaluates
to the empty string, with no error.
This test was taking almost 30 seconds to run, without a lot of images
being read or written. Turns out it was just a large number of separte
oiiotool invocations, and the overhead of all those shells and
programs launched adds up. Most of them were just firing up oiiotool
for the sake of doing an `--echo` to print things. A little bit of
restructuring to make fewer oiiotool invocations (each one doing a
bunch of echo commands, each testing a different thing) cuts the time
for the whole test by 50%.
I also made the runtest.py script allow certain test debugging by
letting a test set the `wrapper_cmd`, and if so, will prepend every
oiio utility launch with it. For example, a test could say
`wrapper_cmd='time'` to time every command.
Add a hidden --unittest command to oiiotool, only functional for Debug
builds, which enables additional testing of internal utility funcions
and other things that are difficult to fully exercise in the testsuite
from normal oiiotool commands (could be done, but would take tons of
additional tests to get into particular deep edge cases).
Prototype by adding unit tests for scan_box. More will be added later
now that the mechanism exists.
Simplify string_to_dataformat
Make sure to use --list-formats and --runstats during testing.
* We mis-named color_test, and that caused it to never get run!
* imagespeed_test is meant for benchmarking only, run by hand.
We don't make a testsuite test, so don't even build the source
(which would never be counted for code coverage because it doesn't
run) during CI.
* Add a test for cineon -- we never directly tested it before
* Don't compile code used only for writing. We don't supporting
writing cineon, so there is no point compiling the cineon module
used only for writing.
* And of course, once cineon was in the testsuite, we immediately
found a problem via the sanitizers! So fix buffer overflows when
reading FilmEdgeCode in Cineon. (And clean up the equally sloppy,
though not technically buggy, code in DPX as well.)
On any one platform, our testsuite was only testing the old C++ and
the new C core exr implementations.
Rig it to run the openexr-oriented tests twice -- once in each state
(when using OpenEXR new enough to support the core library).
Doing this cleanly required adding SUFFIX and ENVIRONMENT optional
parameters to the oiio_add_tests utility macro. Once that was added,
the texture batch testing could be implemented that way as well,
removing a bunch of hackier special purpose code.
* Protect against 0-sized allocation
* Exception safety in IffOutput destructor
* Using sonar.coverage.exclusions as a better way to exclude files
from coverage analysis
* Null pointer safety
I realized that the recently added ustringhash class is confusingly
similar to the (obscure, barely used, mostly forgotten) ustringHash
functor. But also ustringHash doesn't need to exist, because it's
equivalent to `std::hash<ustring>`, if we define it. So now we do define
it, and also `std::hash<ustringhash>`, and deprecate ustringHash, to
be removed entirely for 3.0.
Also, add a ustring ctr from ustringhash (explicit only for now),
and `operator<` for ustringhash.
And add from_hash(size_t) method to ustringhash, analogous to
ustring::from_hash().
Modernize a few places we still used `(void)` to just `()`.
Give ustring and ustringhash a `using rep_t =`
rep_t is the type of the underlying representation (const char* for
ustring, size_t for ustringhash).
Modernize some typedef into using.
Make `==` and `!=(ustringhash, const char*)` be constexpr for C++ >= 17
In particular, defining NOGDI before including <windows.h> achieves
several things amongh others:
- No longer causes ERROR symbol to be defined to 0. ERROR being defined
prevents using e.g. google/glog library, since it wants ERROR to mean
something else.
- No longer defines some possibly widely used names e.g. GetObject.
16 bit per pixel (e.g. 555, 565, 444) formats probably have never
worked. One part of the code was assuming 5 bits in the mask, another
was shifting by 4 bits, and the 16-bit pixel fetch code was reading just
one byte.
The code assumes "char" is "signed char" (as on x86), but on ARM "char" is
"unsigned char".
Co-authored-by: Joachim Reichel <joachim.reichel@posteo.de>
In some cases (at least on Windows, with static linking) the oiiotool
main returns zero, but the full process returns 3 error code due to
abort() being called. And that is called because a global shared
image cache is already destroyed, but the image references from another
global (Oiiotool itself) try to use it, get a call into a "pure virtual
function" stub that is put by MSVC C runtime into a destroyed object
vtable, and that ends up in an abort().
Work around that by making sure various member variables holding
ImageRecRef are cleared before main() finishes and global destructors
are called.
Most of the cost in HDR file reading was converting from RGBE to floats,
inside ldexpf(), which at least on Windows/VS2022 spends most of the
time inside _ctrlfp(). But we only have 256 possible exponents in total,
so just use a precomputed table.
On my PC (Windows 10, VS2022, Ryzen 5950X) this change gets file read
time for an 8x resolution .HDR image from 1.10s down to 0.27s
HDR code overhaul to support IOProxy (#3218) changed all reads into
a `pread()` with explicit file position tracking. Turns out, that is
quite a larger overhead compared to simple sequential reads; even more
so on Windows.
On my PC (Windows 10, VS2022, Ryzen 5950X) this change gets file read
time for an 8x resolution .HDR image (with RLE compression) from
8.28s back to 1.10s just like it was in OIIO 2.3.
On Windows the extra cost when using pread() is a bit more extra mutex
locks, but the real cost is in enventual ReadFile; looks like any seek
operations make it take some sort of "way slower" path with various
callbacks and whatnot inside the kernel.
---
Note repoduced from discussion #3588 from LG:
Switching from IOProxy::pread to using IOProxy::seek + IOProxy::read makes the IOProxy stateful and not thread-safe without an external lock (which this HdrInput has, so it's not going to break anything), precluding any future removal of the locks from read_native_scanline to make the whole ImageInput able to read concurrently. That's probably not important for this file format, but for others (especially crucial those usable as texture), it would be desirable to rely on those stateless pread calls and remove the lock from the ImageInput.
I think this patch is fine. We don't expect a lot of concurrent reads from hdr like we do for tiff and openexr that we use extensively for highly threaded on-demand reading of texture. It fixes the recent performance regression, and it doesn't really cost us concurrency because we already had the lock in HdrInput. But in general, we want to prefer eschewing read/seek and using pread instead, so we can expect maximum concurrency from multiple threads using an ImageInput (as we do in ImageCache/TextureSystem).
So I believe that a revised and more complete diagnosis is that the problem isn't the pread vs read per se, but that for RLE-compressed images we were calling pread separately for every 4 bytes! The calls to read are faster simply because they are buffered (in FILE) and require fewer OS system calls. But the real sin is that we call it way too many times.
I think that the rle case could go back to pread if we want, but do just one pread per scanline (rather than one per pixel value run), asking for the maximum amount that will be needed to read the scanline for the worst case rle representation (it's ok if it's the end of the file, pread will just read up to the end and return the true number of bytes read). In other words, we'd be doing the buffering ourselves.
But anyway, the real misdesign of this HDR reader is ancient, and consists of doing many many tiny reads. Even the buffered fread would be sped up a lot by not doing it separately for every 2-4 bytes of the file. I don't know if it's worth fixing for this format. Depends on whether anybody really needs to maximize its performance or expects multiple threads reading the same hdr file to be fully concurrent.
Add a new build-time cmake cache variable: FORTIFY_SOURCE
Default to empty, but can be set to 1, 2, or 3, corresponding to
setting the _FORTIFY_SOURCE macro available in recent versions of
clang and gcc. (I'm not sure exactly which minimum compiler version is
needed for each fortification level, except that for level 3, gcc must
be 12+.)
Fortification involves replacing several "unsafe" memory-related
functions such as memcpy, memset, strcpy, etc, with special versions
that do bounds checking, aided by some compiler smarts for
understanding the likely bounds of different buffers. If I understand
correctly, at level 2 it can figure out bounds of constant-sized
arrays, and at level 3 it can figure out certain dynamic cases as
well.
There are two use cases for this:
1. For our own CI, this is yet another bit of static and dynamic
anslysis to enable (currently, just in the gcc12 test) to possibly
catch bugs.
2. Users who are building OIIO to deploy it in security-sensitive
environment may wish to build with some fortification level enabled to
help prevent certain memory errors or security issues, understanding
that it may slightly impact performance.
* Refactor the clang-format job to be part of the aswf batch of jobs
rather than separate (reduces redundant logic in the workflow file).
* Give proper names to distinguish the two Mac jobs' artifacts for
broken builds and their caches.
* Revise scorecard workflow triggers: no need to run weekly, and run
on push only to the main repo, no reason to run on forks.
In addition to address and leak sanitizers, also turn on the undefined
behavior sanitizer in our CI run. We can do it in the same job as
asan/lsan.
Some things that come along for the ride, including fixes for things
that ubsan uncovered:
* src/build-scripts/ubsan-suppressions.txt can hold suppression list
(though we don't have any now, it just sets up the space for it for
future needs).
* Split the "targa" test so that the parts that directly need the OIIO
python module (which are all related to thumbnails) are now in
"targa-thumbnails" test. That has to be disabled for the sanitizers,
since we can't arrange for python to b linked to the sanitizer
libraries.
* Found vint4::store has a potentially unaligned integer store.
* Found Strutil::concat is not necessarily safe if either of the
strings are empty.
Also remove three raw test reference output files that we apparently
no longer use (corresponding to platform + libraw combinations that
aren't in our current set of CI configurations).
Something changed in libtiff master and it doesn't build shared libs
by default, breaking our "bleeding edge" CI test. So when we build it,
be sure to ask for it. It can be overridden in build_libtiff by
invoking it with env variable `LIBTIFF_BUILD_SHARED_LIBS=OFF`.
Comparing the time it takes to load DXT/BCn compressed DDS images via
OIIO, I noticed that it's several times slower than in another codebase
(Blender). So I started profiling, and fixing some low hanging fruit.
- After decoding a BCn block (always 4x4 pixels), it was written into
destination image using quite many branches in the innermost loop.
Now it still takes care of not writing outside of a destination image
that might not be multiple-of-4 in size, just with way fewer branches.
- BCn decompression is multi-threaded via parallel_for_chunked, using
at least 8 block rows per job (i.e. 32 pixel rows). This is
similar to internal multi-threading done by EXR or TIF readers.
- Avoid one case of std::vector resize (which clears memory) followed
by immediate overwriting of that buffer with bytes read from the file.
Switched to using unique_ptr for that array, similar to how other
input plugins do it.
Performance data: I'm timing how long it takes to do "iinfo --hash" on
24 DDS files, each 4096x4096 in size. Quite a lot of time there is taken
by just the SHA hash calculation which is not related to loading, so
also have timing info for just DDSInput::read_native_scanline where
all the actual DDS loading/decoding happens.
Timings on PC (Windows 10, VS2022 Release build), AMD Ryzen 5950X.
- BC1: total 3.68s -> 2.76s, read_native_scanline 1.54s -> 0.59s
- BC3: total 3.74s -> 2.84s, read_native_scanline 1.83s -> 0.66s
- BC4: total 1.26s -> 0.82s, read_native_scanline 0.74s -> 0.24s
- BC5: total 2.29s -> 1.57s, read_native_scanline 1.31s -> 0.46s
- BC6: total 7.50s -> 4.30s, read_native_scanline 4.67s -> 1.60s
- BC7: total 6.18s -> 3.09s, read_native_scanline 4.34s -> 0.84s
Possible future optimization: implement DDSInput::read_native_scanlines,
that could avoid one extra temporary memory buffer copy. This extra
buffer clear & eventual copy from it into final user destination pixels
is non-trivial cost from profiling. But it's also quite involved
to handle all the possible edge cases (e.g. user requests scanlines
not on 4-pixel row boundaries); the code would have to do similar
juggling as EXR reader does to handle "not on chunk boundaries" reads.
It could potentially also interplay with DDS cubemap/volume texture
reading ("tiled") code paths, for which there is no test coverage at all
right now. Maybe for some other day.
* DDS: fixes for A8, L8, A8L8 formats
While developing #3573 at one time I had them working properly, but then
while fixing the address sanitizer failure for 3-channel formats I
botched them again. Note to self: never again do a "yeah I'll add tests
later", since if I had all of them running all the time this would
not have happened. 🤦
* DDS: extend test coverage for more formats & channel size cases
With more test images recently added (https://github.com/OpenImageIO/oiio/pull/3459),
start using them for DDS tests. This covers now:
- Alpha, Luminance and Alpha+Luminance formats,
- RGB formats with rarer channel sizes (rgb10 a2, r3g3b2),
- Several "broken" DDS file cases
- Removed usage of sample-DXT1 since it is well covered by others.
Adapted from the similar document I wrote for OSL.
This serves both as a checklist for me to make sure I don't miss any
steps during a release, but also so that other people could take over
release procedures if necessary or desired.
I also rearranged the order of some lines in CMakeLists.txt for
clarity (but did not change the logic).
This patch allows ParamValue, ParamValueList, and ImageSpec to support
attribute/getattribute of metadata consisting of uint8[] arrays, which
is how we store certain binary blobs or raw byte arrays. We translate
this to Python as a numpy array of type uint8 (dtype='B').
This fixes a limitation where it was not possible to directly retrieve
or set the "ICCProfile" metadata (which is stored as a uint8 array)
from the Python APIs, as you could easily do from the C++ APIs.
In the process, I found similar bits of code in several different
places, in various states of incompleteness. I refactored to make a
single make_pyobject() utility and then used that call in place of
several other redundant pieces of functionality.
For a while, we have found embedded ICC profiles in TIFF, JPEG,
JPEG-2000, PNG, and PSD (PhotoShop) files and store them in the
metadata as a byte array with the name "ICCProfile".
With this patch, we further add the facility of parsing the binary ICC
profile data to extract a number of other fields that we add to the
ImageSpec metadata so that "oiiotool -info -v" or "iinfo 'v"
will print this info.
Important caveat: You can read this for informational purposes, but
changing it (e.g. with spec.attribute()) does not change the contents
of the ICCProfile byte array, which is still the canonical ground
truth of the profile. You have to replace the "ICCProfile" byte array
to actually change the profile.
* Add `Strutil::utf16_to_utf8(const std::u16string& utf16str)`.
This is subtly different from our existing utf16_to_utf8() that
takes a std::wstring, because wchar_t is not the same on all
platforms. If it really and truly is utf16-in-char16_t, you need
this and can't use the utf-in-wchar_t which may be 32 bit chars.
* Add `Strutil::utf8_to_utf16wstring()` (identical to the old
utr8_to_utf16) to make it more self-documenting as returning a
utf16-encoded wstring and NOT a u16string. Note that wchar_t is not
necessarily char16_t on all platforms (it is on Windows, but it's 32
bits on Linux).
* Deprecate the ambiguously named `utf8_to_utf16()` for OIIO >= 2.5 and
remove it for OIIO >= 3.0.
Rather than specify the version in the usual project() statement,
project (OpenImageIO VERSION 2.5.0.0 ...)
this patch decouples it
set (OpenImageIO_VERSION "2.5.0.0" CACHE STRING "Version")
project (OpenImageIO VERSION ${OpenImageIO_VERSION} ...)
thus allowing the possibility of overriding the version number at
build confuguration time (`cmake -DOpenImageIO_VERSION=3.1.4.1`).
This should be used with extreme caution. Probably never used by most
people. I don't need to describe all the ways that things can go
sideways if you make your build of x.y.z.w actually be arbitrarily
different from the x.y.z.w that everyone else uses and that is
documented.
That said, there are two scenarios I have in mind for which this
ability is very useful:
1. Sometimes, we add or deprecate API features "in advance" by having
an `#if` enable or disable code based on the version, essentially
putting in place changes that can't be enabled yet because they
would break compatibility rules, but we want to integrate the code
now and have it become active upon reaching a particular future
version in which that incompatibility will be allowed. The version
number override makes it easy to test this by quickly doing a build
"as if" we had already bumped the version number to see what will
happen when we actually do.
2. Every once in a while we need to do an internal build that, to
conform to version compatibility rules, needs to appear to still be
part of a past version family. This lets individual sites play some
games with version number overrides rather than have to create new
branches and tags for certain one-off internal releases.
--iccread reads a named file and directly adds its contents as the
"ICCProfile" metadata of the top image (as a byte array).
--iccwrite takes the "ICCProfile" metadata of the top image and
writes it to a named file.
Suggested by SideFX: explicitly shut down the OpenEXR thread pool
before exiting oiiotool, because sometimes on Windows it seems to hang
otherwise. A better solution is probably possible on the OpenEXR side,
but I don't know what it would be. (And in any case, we'd still need
to do this be be well behaved when using OpenEXR versions earlier than
where a fix might land.)
Solution tracked down by Edward Lam.
Oops, at some point, something about the way we set up the CI test
that runs the sanitizers, and ever since then (embarrassingly, perhaps
as much as a couple years ago), the "sanitizer" test has not in fact
been running the sanitizers. Here we fix that up, get sanitizers
running again.
* Minor png refactoring to fix some memory leaks for broken png files.
* Remove unused string variable from read_info() and simplify the
surrounding "if"
* Split png test into png and png-damaged, because I just can't seem
to squash some very minor memory leaks for broken png files, I think
it's not worth worrying about and I'll just exclude the broken case
from testing for the address sanitizer.
- If a DDS pixel format header contains "this is a normal map" flag
set, then handle that properly (#3525):
- For a DXT5, this means it's a "DXT5nm" format. Normal X & Y
components are in green and alpha channel respectively. Compute Z
component from those, and the result is a 3-channel tangent space
normal map.
- For ATI2/BC5 format that also has a "normal map" flag set, compute
the Z component automatically too, and result is a 3-channel image.
However a bunch of ATI2/BC5 normal maps out there in the wild do
not have that bit set, so also have a global "dds:bc5normal"
attribute to optionally turn this on.
- Fix support of R10G10B10A2 format - was producing garbage data due to
code assumption that truncation down to 8 bits never needs to happen.
- Fix bit expansion ("too dark") of very low-bit formats, e.g. R3G3B2.
Code was simply shifting the bits, instead of properly repeating
the high bits into new low bit positions (i.e. a 3-bit "111" needs
to become an 8-bit "11111111" and not a "11100000").
- Support "RXGB" (DXT5, with red & alpha swapped) as well as "BC4U"
and "BC5U" formats (the latter two are just BC4 and BC5, produced by
some older tools).
- Support alpha-only (e.g. A8) formats.
All this mostly prompted by doing DDS improvements in Blender, and
wanting to get Cycles renderer (which uses OIIO) to the same feature
parity. See https://developer.blender.org/T101405 and
https://developer.blender.org/D16087.
Reading uncompressed scanlines would copy 4 bytes from each pixel into
an int and then decode the bits, incrementing where it was reading
from by Bpp bytes each time. But when Bpp == 3, the last pixel of
every row would therefore read an extra byte past the end of the heap
storage. Allocation padding would in practice keep it from crashing,
and that meaningless extra byte was not used in the decode because it
knew it only needed the first 3 of the integer. But found by address
sanitizer!
This was an experiment that was added to master at one point but
removed before we branched for a release. But we forgot to remove a
few headers. Removing them now is not a compatibility break, since we
never shipped a supported release that contained the library that
implemented the functions declared in these headers.
This was part of a set of obsolete calls where we recently removed the
implementations, but for link compatibility, made matching shim calls
and wrap the modern versions. But I missed one case.
Changes since RC1:
- ImageCache/TextureSystem both have added a `getattributetype()`
method, which retrieves just the type of a named attribute. #3559
(2.4.4.0) NOTE: THIS IS AN ABI BREAKING CHANGE
- Python: Fix the ability to `getattribute()` of int64 and uint64
metadata or attributes. #3555 (2.4.4.0)
- Build: Improvements to the generated cmake config files when
building static libraries. #3552#3557 (2.4.4.0)
- Support for gcc 12.1. #3480 (2.4.2.1) #3551 (2.4.4.0)
- Support building with clang 15.0. #3563 (2.4.4.0)
- Fix cross-compiling on Android failing due to `-latomic` check. #3560
(2.4.4.0)
- Fix building on iOS. #3562 (2.4.4.0)
Note that we bumped the version to from 2.4.3.x 2.4.4.0 to reflect the
ABI break introduced by the last minute by the IC/TS addition of a new
virtual function. ABI will not change again for 2.4.
ImageSpec and ParamValueList have had (since 2.1) a getattributetype()
method, which simply returns the TypeDesc of the named attribute, or
TypeUnknown if no such attribute exits. This goes along well with the
related getattribute() and attribute() family of methods.
The getattribute() method was not present in ImageCache and
TextureSystem, despite their having very similar
getattribute()/attribute() interfaces.
A recent issue made me realize that in the Python bindings, the IC and
TS bindings for `getattribute(name)` did not work properly, always
returning None, and *required* you to use the full
`getattribute(name,type)` variety where you have to supply -- and
already know -- the type. This isn't especially Pythonic, where you
should just be able to ask by name and get a polymorphic result. And
when I went to look at how to repair this, it was going to be very
awkward without getattributetype(), but trivial with it.
So this patch sets up getattributetype() for both IC and TS, and
exposes them to both C++ and Python APIs, and also fully fixes the
getattribute() so that it works properly with the name only.
Unfortunately, because getattributename is a new virtual method, it
changes the vtable, breaking the ABI, and thus cannot be backported to
any supported release that makes ABI compatibility promises.
Currently when cross-compiling for Android build fails due to usage of
TRY_RUN(). In the particular case it tries to figure out whether atomics
can be used without the libatomic library.
Fortunately we don't need this check because on Android NDK atomics are
built-in and libatomic.so is not present. Only static dummy libatomic.a
exists for compatibility with -latomic flag.
Just a couple missing pieces were preventing getattribute() from
retrieving attributes that were stored as int64 or uint64. That
doesn't come up very often, but one place is retrieving certain
statistics that use 64 bit counters. (This comes up in ImageCache,
among other places.)
While we're at it, there was no existing test dedicated to the python
binding of ImageCache. So to add these getattribute calls, I made one.
There is an important caveat about ImageCache.getattribute(): In this
process, I discovered that the flavor of IC.getattribute(name) where
you only pass the name is BROKEN, and will always return None. You
MUST use the form where you pass the type, like
x = ic.getattribute("foo", "int")
This is unfortunate and not very Pythonic, but it is the workaround
for now. I will addess getting the name-only form working in a
separate PR.
Closes#3537
TBB::tbb is added as a library dependency via TARGET_NAME_IF_EXISTS.
Therefore if an executable links OpenImageIO static library without the
executable itself depending on TBB, then TBB will not be linked in and
result in link time errors.
This fixes builds in projects that compile OpenImageIO as static
libraries and don't themselves depend on the image libraries used by
OpenImageIO such as PNG, JPEG or TIFF.
When OpenImageIO is built as a static library its targets have e.g.
PNG::PNG among INTERFACE_LINK_LIBRARIES. If the including project does
not itself depend on PNG, the following error would be observed:
Target "<...>" links to target "PNG::PNG" but the target was not found.
Perhaps a find_package() call is missing for an IMPORTED target, or an
ALIAS target is missing?
When enforcing the max open files limit, there's a heuristic where we
only "try" to get the lock, and if we can't, rather than block, we
just defer the cleanup step until the next time around. So what if we
go slightly over the limit temporarily?
Interesting edge case: what if max_open_files is relatively low, but
the number of threads is very high and you have a high contention rate?
Then you can end up *significantly* over the limit, because almost all
the threads say "somebody else has the lock, I'll leave it to them to
clean up" and blithely open new handles.
So this patch modifies the logic so we do the defer trick only if
we're only a little over the limit. If we've exceeded the limit by
16 handles, grab the lock unconditionally and do the cleanup.
* ArgParse::add_version(str) tells ArgParse the version string, and
automatically adds an option --version that prints the version string
and exits.
* Add this to oiiotool, iinfo, igrep, iv, maketx.
* OIIO::getattribute("version") retrieves the version string.
JFIF/JPEG files only support 1 and 3 channels. For > 3 channels, we
had just been discarding the extra. For 2 channels, we had been
discarding the second, but maybe that's not always right. Try to
distinguish between files whose names appear to be YA -- in which case
output the Y as grayscale (JFIF has no way to represent alpha anyway),
and other cases which we now promote to RGB and just make B be black.
* For 2-channel DDS files, label them as Y,A if the flags indicate
luminance and/or alpha, otherwise label them as R,G.
* Do not set "oiio:BitsPerSample" for cases where the dds.fmt.bpp field
is not assumed to be valid. MS docs say it's valid only if the flags
field indicates RGB, LUMINANCE, or YUV types.
The implementation of boost::filesystem::path::extension() and the
equivalent for std::filesystem disagreed about the result of
extension(".foo"). C++17 correctly considers ".foo" to have no
extension.
This was envisioned to be useful in the very earliest days of OIIO,
but we never really finished it or used it in any helpful way.
In addition to being essentially dead and untested code, it ties us
to boost.asio, which we use nowhere else.
So just get rid of it this vestigial piece.
* Any use of `-i:ch=...` would result in a crash because of
dereferencing an empty shared pointer.
* In the case of `-i:ch=` specifying missing channels, improve the
warning message to make it clear which channels were actually present.
Extend ImageBufAlgo::compare() to take relative metrics, which if
nonzero check the symmetric mean relative error, in addition to the
current absolute error. To be formal, this means that a value
comparison will fail if
abs(A-B) > failthresh && abs(A-B)/((abs(A)+abs(B))/2) > failrelative
Setting the failthresh to 0 is equivalent to only checking relative
error, and setting failrelative to 0 is equivalent to the (previous
behavior) of checking only absolute error.
The idiff utility is extended to take new `-failrelative` and
`-warnrelative` command line arguments.
The hope is that there are various kinds of image comparisons within
tolerances -- like we frequenty use for unit tests, shader
verification, etc. -- where this metric is more helpful than absolute
error.
Set up SonarQube to run weekly (seems unnecessary to do on every PR,
it's a LONG run). Also required some build system fixes to enable code
coverage analysis.
This is handled by the "analysis.yml" workflow (into which I also
merged the scorecard analysis we were doing). This workflow runs once
weekly or on only those pushes/PRs with specially named branches.
There's no reason to run this expensive analysis on every last PR or
push.
A number of fixes come along related to things found by Sonar as I was
setting this up:
* Fix buffer warnings
* Address sonar warning: was reading wrong size into buffer
* warning reduction in dpxoutput, simplify string ops
* Clean up dubious uses of sprintf
* Add assertions
* Save a needless string construction in ParamValue::get_string_indexed
* Fix string safety warnings
* iptc: simplify encoding string data and improving safety
Stranely, this made slightly numerically different texture test
results on icc. Switching its math mode to precise brings it back in
line with the other compilers. The differences aren't visible, so this
isn't an issue in practice, but for tests it made it hard to compare
to reference images.
Hide nearly all of the implementation of the parallel_for family of
methods rather than expose them as inline. This gives us more freedom
to change the implementation in the future without breaking ABI.
Deprecate the parallel methods that take task functions whose first
parameter is the thread ID. The only reason they existed is because
our cobbled together thread_pool used that in its inner workings. But
there are good reasons why we should not expose that:
* We never used it anyway.
* It was not very useful, since sometimes it was a real thread ID, but
other times it was -1 in cases where the calling thread executed the
task directly.
* It is inconsistent with other thread pool implementations that we may
wish to try in the future.
So better just to not expose those thread IDs at all. Mark those
versions of the parallel loops as deprecated and schedule them for
removal in OIIO 3.0.
If TBB is detected at build time, support will be built in that allows
either the old OIIO built-in pool, or TBB, depending on the setting of
a new global OIIO attribute, "use_tbb" (default 0), if set to nonzero
will use the TBB thread pool.
In my experiments, the TBB thread pool seems slightly better -- I
think because there is less overhead, and maybe the clever
work-stealing it does elps to load balance. It's not used by default,
set the attribute if you want to use it (assuming the build
included. After we've had a chance to test more thoroughly, we may
change to use TBB by default. I'm interested to hear people's thoughts
on the matter.
One case where you almost certainly will want to use the TBB thread
pool is if you're using libOpenImageIO from within an application that
uses TBB for its own threding -- that way you're using one pool for
everything, rather than have two separate thread pools that don't know
about each other (and probably twice as many threads as you have
cores)..
Incorrect use of reserve(), then writing past the end of the vector.
It should have been resize(). But it can be cleaned up even further by
using assign().
If udimfile is null, it would get dereferenced anyway. Was missing
a `return` in that case.
While I was there, I realized that ImageCacheImpl::inventory_udim and
resolve_udim had no reason to be virtual, so I removed that
qualifier. (It was probably a cut/paste error, as the
identically-named methods in TextureSystem are virtual, on
purpose. But they don't need to be for this class.)
All read_native_* methods of ImageInput are required to be thread-safe
with any other read_native_* call or with the varieties of read_tiles()
that take an explicit subimage and miplevel parameter.
JpgInput::read_native_scanline() does a seek_subimage() call internally,
so it's obviously not thread safe without additional locking.
The current description of IOProxy::pread() and IOProxy::pwrite() is a
little ambiguous. This commit expands the description to explicitly tell
what the functions are and are not required to do.
Should not have passed an argument. Went undetected for a long time
because I guess we didn't previously call the erase() method. (Until
I did when working on something unrelated.)
Virtual destructors don't significantly differ from regular virtual
functions, so it is customary to add override specifier when virtual
destructor is overridden.
Current code assumes that CMAKE_INSTALL_INCLUDEDIR and
CMAKE_INSTALL_LIBDIR has exactly one filesystem component. This is not
the case on e.g. Debian, where standard paths are as follows:
CMAKE_INSTALL_INCLUDEDIR=include
CMAKE_INSTALL_LIBDIR=lib/x86_64-linux-gnu
Fortunately the Config.cmake file is installed into a path that is
itself relative to CMAKE_INSTALL_LIBDIR, so we can get
CMAKE_INSTALL_LIBDIR from CMAKE_CURRENT_LIST_DIR and then get
CMAKE_INSTALL_INCLUDEDIR by applying relative path from
CMAKE_INSTALL_LIBDIR to CMAKE_INSTALL_INCLUDEDIR on top of that.
Fixes: c8dfbd7066
The current limit has been already hit when AMD released
second-generation Epyc processors (codename Rome) back in 2019. Top
models have 64 cores per CPU, or 128 threads. This means 2 socket
configuration has 256 threads.
Fourth generation Epyc processors (codename Genoa) are rumored to have
96 cores per CPU, or 192 threads. This means 2 socket configuration has
384 threads. While the CPUs have not been announced yet, it makes sense
to prepare for them in advance.
Revealed by clang-tidy. Use length-safe versions.
Also get rid of some dead code by disabling some memory-clearing
code that we don't need (we're not using SHA-1 for cryptographic
or security reasons, just for comparing images).
Python 3.8 stopped loading DLLs from PATH on Windows, meaning anything that relied
on that environment for shared library resolution breaks. This PR works around that
behviour by manually adding PATH back to DLL resolution using os.add_dll_directory()
in a new __init__.py which just re-exports everything in the current .pyd.
The new behaviour is enabled by default, but is easily disabled at runtime if the
environment variable OIIO_LOAD_DLLS_FROM_PATH is equal to "1".
Found this coincidentally during some address sanitizer runs.
Some tiff files out in the wild that have RichTIFFIPTC tags
inexplicably mark them as if the data was LONG instead of BYTE or
NOTYPE. I guess early in our history, we came across one of those and
thought it was correct, because we were always treating it as if it
came in the form of longs. But that's not always true, and in
particular, the size in the tag I think is always expressed in
bytes. So this could make for buffer overruns (generally harmless, but
caught by the sanitizer!).
So this patch cleans this all up.
And also we update a few sample images with bad RichTIFFIPTC tags to
strip out the in the incorrect form of the tag.
Recently, we added the ability to do stochastic sampling in texture
lookups, by choosing the MIP level stochastically rather than reading
two MIP levels and mixing them.
This PR adds the ability to also stochastically sample the anisotropic
filter footprint -- instead of multiple anisotropic samples, choose
just one sample well distributed over the footprint axis. On real
scenes, this seems to speed up overall rendering (not just texture) by
up to 2-3%, for scenes where average anisotropy is around 3-4.
It's much less dramatic a speedup than the stochastic MIP selection,
which sped up texturing by around 40% and in practice sped up overall
rendering by around 15% for us. Why doesn't cutting the number of
texture samples by another factor of 3 only speed up by 3% instead of
another 20%? Basically, it's because the stochastic MIP mapping *by
definition* means that each texture call touches half the number of
tiles as before. In contrast, those multiple probes along the
anisotropic axis are almost always within the same tile, and
frequently touch a very nearby pixel within the tile, so not only are
we not needing to look up a new tile from the tile cache, but also the
very memory we need to read is highly likely to be in L1$. But still,
we'll take 2-3% rendering gain any time we can get it, that's
equivalent to a lot of money if you have a really large compute farm.
Some other changes that come along for the ride:
Add a --bluenoise option to testtex (makes for prettier 1spp results),
and add tests for the aniso sampling.
Some internals changes to precompute min w,h of mip levels so there is
less per-texture-call rifling through the ImageSpec of each MIP level.
Sadly, this did not produce any consistently detectable speedup in
overall rendering.
And I changed the interface for stochastic texturing a bit. The old
way involved asking for new MIP modes, StochasticAniso or
StochasticTrilinear, in addition to being sure to set TextureOpt::rnd
to the stratified random number you are using for sampling. Now we are
deprecating StochasticAniso and StochasticTrilinear. Just use the same
old mip levels you always did. So the new interface can be summarized
as follows:
* On each texture call, setting opt.rnd >= 0.0f is itself a request for
stochastic texturing on that call (opt.rnd will default to -1.0, and
any negative value does the old fashioned non-stochastic filtering).
* A ShaderSystem-wide option "stochastic" (int) indicates which things
should be sampled stochastically, for those lookups that have rnd >= 0.
It's a bit field: 0 means no stochastic sampling (even if rnd >= 0),
1 means to do stochastic MIP level selection 2 means to do stochastic
anisotropic sampling (so 3 means to do both, and higher bits are reserved
for future extensions). The default is 0, so renderers need to enable
it to get any stochastic texturing.
Finally, let me give a reminder that you ONLY want to do stochastic
texturing for shading that involves multiple stochastic samples per
pixel, like orinary surface shades. For displacement (shade once per
mesh vertex) or "imager" (shade once per pixel), you just have a
single shade that you want perfectly filtered, so you probably don't
want to set opt.rnd >= 0 in those cases.
Previous code was doing ioread one pixel at a time. Change it to
read one scanline at a time, similar to e.g. how BMP reader does it.
On a 8192x8192 rgba8 image (256MB), on Apple M1 Max "iinfo --hash"
time goes from 2.19s down to 0.65s, and is now similar to the time
it takes on an uncompressed TIF or BMP image.
When viewing this .md file in Github UI, everything seems to work. But when creating an actual PR, the links to CONTRIBUTING and CLAs point to nothing that would exist. Looks like they try to be relative links from the "create a new PR" page. Workaround seems to be to use absolute links instead.
Implements support for BC4, BC5, BC6H (unsigned and signed) and BC7
GPU texture compressed formats in DDS reader (#2338).
- BC4: loaded into 1 channel uint8,
- BC5: loaded into 2 channel uint8,
- BC6H: loaded into 3 channel half,
- BC7: loaded into 4 channel uint8.
This is implemented by replacing whole of libsquish (~4700 lines of
code) with bcdec.h (https://github.com/iOrange/bcdec, ~1300 lines of
code). bcdec.h has an advantage of being faster, smaller and supports
all BCn formats for decoding.
bcdec.h license is MIT or Unlicense (aka public domain), updated all
the relevant third party / license files that I could find.
* Add a SECURITY.md detailing our security policy and the private
email address to contact to report vulnerabilities.
* Create scorecards.yml, to run OSSF scorecard to check supply chain
security issues. It has some false positives, but it's only advisory.
* Make sure actions we use don't get write perms on our repo.
* Use exact SHA's for actions versions to guard against hijacked
actions repos that might try to substutite something else at the
version tag we use.
* Tried to use harden-runner action to ensure no actions on the runner
overwrite source code or have unauthorized outgoing net connections
during the GHA run, but currently leaving it disabled because of too
many false positives -- it flags every dependency download as a case
of altering source code. Which it technically is, but it doesn't
matter for any jobs that aren't producing any artifacts that will be
consumed or relied on by users.
Learned about this from a security talk.
On Linux, we'd been using gettimeofday, which gives results with usec
resolution. Switch to clock_gettime, which has nsec results.
In practice, the round trip time of a pair of "get time" system calls
is about 30-50ns in either case. But the new call at least nominally
returns ns resolution values. Another way of saying that is: the
answers we get should be more accurate with this call, but the call
itself is the same expense as before, so we should still be just as
cautious about not inserting timers around events that are so brief
that the timers themselves are going to be a significant addition to
the cost.
This only effects Linux (and maybe BSD?). On MacOS, we use
mach_absolute_time(), and on Windows we use QueryPeformanceCounter(),
so neither of those platforms is affected by this patch.
Oops, calling it "m_max_tile_channels" (the name of the member
variable) instead of "max_tile_channels" (the name of the attribute)
actually prevented properly setting or retrieving the attribute by
name.
ustring is a unique string (that lives in an internal hash table) whose
representation is the pointer to the characters that live in the table.
ustring hash is its alter ego -- it also refers to a ustring in the
internal hash table, but the representation is just the hash itself.
It presents an interface that is a subset of that of ustring, and in
many ways they can be used interchangeably. For ustringhash,
accessing the characters requires using the hash to find the string in
the table -- which is not hard, but more expensive than accessing the
charcters with a ustring, where the representation is the pointer
directly to the characters.
For some uses where you don't need access to the characters in any
performance-critical paths, this may be a more convenient
representation. In particular, it's well suited to a GPU that doesn't
have access to the character memory. Another interesting difference is
that from run to run, a ustring may have a different literal value,
since there's no reason to expect that the pointer to a string like
"foo" will refer to the same memory location every time the program
executes, but in contrast, the hash is guaranteed to be identical
from run to run.
In the process of adding unit tests for ustringhash, I ended up
overhauling all of ustring_test.cpp, as I saw that the ustring tests
were extremely cursory. Did a tiny bit of (non-breaking) touch-ups
on a few ustring methods as well.
If the BMP image we're reading declares a certain sized palette, but
then has pixels that contain values outside the range of that palette
size, we would read past the end of arrays.
With a 1x1, 2bit image of a Targa 1.0 file, the entire image was only
22 bytes, so the seek to 26 bytes before the end to look for the
Targa 2.0 data... obviously failed.
Don't look for the Targa 2.0 block if the file itself is not long
enough to hold both the header (18 bytes) and the block (26 bytes).
This tiny change is a quality-of-life fix for me, addressing the fact
that there are two different simd.h files in our code base. I never
would have done that on purpose, but one of them is part of the
imported "squish" source code that we use for dds encoding.
For years I've been frustrated that reflexively typing `CMD-P simd.h`
in the editor has a 50/50 chance of pulling in this file, when in fact
there is literally a 100% chance of my wanting
src/include/OpenImageIO/simd.h.
So finally put my editing brain out of its misery by renaming the file
that I never want to edit.
When doing thread "wedge" tests, most of the time spent is for the
1-thread case. Who cares about 1 thread anyway? So we make two
improvements with new command line options:
* --minthreads sets the minimum number of threads that will be
used. For example, `--minthreads 2` will skip the single-thread case
and start the wedge with the 2 thread case.
* --lowtrials is an optional maximum number of trials just for the 1
or 2-thread case (overriding --trials just for those thread counts,
if it is a lower value).
By default, the old behavior is still the default. But these allow a
bit of customization in the tradeoff of how much time to invest in the
very low thread counts in a wedge of thread timings.
You'd think there would be no way that basic failure like this could
occur after 14 years, right?
But the TIFF reader code was a little mixed up about the fact that the
internal m_next_scanline field is relative to the starting scanline of
the image, not an absolute value. But in the case of the code that has
to deal with handling the stray scanlines at the end of a strip-read
file neglected to set it properly and the end of the stripped part.
Obviously, for TIFF files with a first scanline of 0 (the vast, vast
majority), failing to adjust by this offset doesn't matter. And also
as long as the reading continued to be strictly sequential, it also
was asymptomatic. But in the presence of certain multithreaded random
access patterns, it makes a difference. Oops.
The udim_lookup_mutex_pool is not necessary. Just using an atomic
pointer for udiminfo.icfile is enough.
I'm seeing about a 5-8% overall speed improvement in testtex-based
intense udim benchmarks under heavy threading, a ~2% overall
improvement in a real rendered scene that uses a bunch of udim
textures (I think there are bigger bottlenecks elsewhere that I'll
move on to next, but every little bit helps).
Some testtex enhancements coming along for the ride:
* UDIM-oriented tests (automatically decuded if the test textures
contain `<UDIM>` in their name.
* --maketests will construct random test textures and use them (handy
if you need lots of textures for a big test, but don't want to store
them). The tests are based on a file pattern using std::format-like
replacement with an index. Also, if the pattern contains `<UDIM>` in
the name, it's understood to be a udim pattern and many individual
udim piece files will be created.
An example of these two items in action is:
```
testtex --maketests 200 "tmp/foo{:04}_<UDIM>.exr" --threadtimes 8 -iters 100000000 -runstats -v --handle -maxfiles 1000 -trials 5 -wedge
```
fmt master seems to have just deprecated the ability for
fmt::sprintf("%s") to default to printing anything that allows stream
output. This breaks use of Strutil::sprintf formatting ustrings.
This isn't yet in any tagged release of fmtlib, but I'm trying to get
ahead of it.
The quick fix is to just change those handful of places to use
std::format style calls, which we are slowly converting to anyway.
Also throw in a fmt::formater for ustrings for good measure.
It's a lot of data, you don't need it if you don't want to run the
full testsuite, and a lot of places and distros have policies against
live downloads happening in the build step. Also, downstream projects
that need to build OIIO for their CI or other reasons don't want to
accidentally download a lot of test data they won't need.
So change the default of OIIO_DOWNLOAD_MISSING_TESTDATA to OFF.
Build with -DOIIO_DOWNLOAD_MISSING_TESTDATA=ON to turn on auto
download of the data images, or else run
src/build-scripts/install_test_images.
Now, this does have the following danger: That people who build OIIO
and do want to run the tests -- and don't know to turn this option
back on -- will end up with very few tests able to run.
We largely address this by observing that quite a lot of our tests
that depended on oiio-images actually depended only on one file
(grid.tif, which is not especially large). So by making a copy of that
file in testsuite/common, quite a few additional tests can run even
without oiio-images.
Though, to be clear, there are still many tests that want oiio-images,
particularly tests that validate support for various image file
formats, which often depend on large collections of images that are
stored in oiio-images. So remember, if you are developing OIIO, the
best policy is to run the whole testsuite. But on the other hand, if
you submit a PR, they'll all be run in CI as well.
For failed CI builds, save the cmake caches in a safe place, don't zip
them up in situ, because then when I unpack them in my build area
locally, they overwrite my working cache.
The Intel OneAPI LLVM-based C++ compiler, as we download it from Intel
for our CI, bumped from 2022.0.0 to 2022.1.0.
New warning popped up -- newest Intel LLVM-based compiler warns about
overriding certain floating point math flags.
Also had to adjust thresholds on a couple texture tests that have a
few pixels change slightly due to numerical differences.
Another round to hide Imath externally to OIIO public headers as well
as speed up our own builds.
* Remove the external reference to Imath::M44f from color.h, and instead
use M44fParam.
* The `<OpenImageIO/Imath.h>` includes the most commonly used Imath
headers including half.h. But this is unnecessarily expensive (and
presumptuous about wanting other Imath types as well as fmt.h) if
you just need to know about `half`. So we create a new
`<OpenImageIO/half.h>` that only includes what you need for `half`
(as before, with logic to handle the changing location of the half.h
header depending on whether you're using Imath 3.x or OpenEXR 2.x).
* Several files that included Imath.h, but in reality only needed half
all along, change to including OpenImageIO/half.h.
* simd.h leans more on vecparam.h and other template trickery to
reduce exposure of methods that only work in the presence of Imath
headers, and make fewer use cases that would care whether the Imath
headers are included before or after simd.h (which is less
controllable for unity builds).
* Vec3Param gains a `cast<V>()` templated method that allows easy
casting to foreign vector types even if Imath headers are not
included prior to vecparam.h being included (which is less
controllable for unity builds).
We have a template OIIO::bit_cast which has served us for a while, but
it turns out that C++20 has std::bit_cast which operates ALMOST the
same -- unfortunately, the tempalte arguments are reversed from the
way we did it!
So here we:
* Set up a new `bitcast<>` that works exactly like C++20 std::bit_cast.
* Implement our old bit_cast in terms of the new one, for now.
* Set up `#if` trick so that the confusing OIIO::bit_cast will be
marked as deprecated beginning with OIIO 2.5, and will be removed
entirely from OIIO 3.0. (Farther out than that, we may choose to
re-introduce bit_cast as truly being identical semantics to
std::bit_cast. But before that is possible, we need to completely
ween downstream users off the old one.)
* Took the opportunity to expand the set of compilers for which
certain bitcast overloads use Intel intrinsics. Previously, we only
did this for the Intel compiler, but it seems to work for modern
versions of clang and gcc as well. Do a little bitcast testing in
fmath_test.cpp to verify that the use of these intrinsics doesn't
break for any of the compilers we use for CI.
Over the span of several individual PRs, we've made the code base safe
for "unity" (a.k.a. "jumbo") builds -- a feature of CMake that
automatically combines multiple .cpp files into a single compilation
unit (basically making temporary .cpp that `#include`s each original
cpp) in order to, allegedly, speed up compilation by not re-parsing
headers multiple times, and other sources of per-module overhead.
We discovered early on that on a modern computer where modules are
built in parallel (another automatic feature of CMake), jumbo builds
are simply not an overall performance improvement for OIIO. Basically,
the efficiency gain from removing redundant parsing is mostly offset
by the loss of parallel efficiency -- fewer and larger modules means
there are periods where all cores are not able to find work to
do. (Reductive example to illustrate the basic problem: If you have 8
independent modules to compile, and each takes 1 second, 8 cores can
compile the entire collection in parallel in 1 second. But a unity
build that batches those files in groups of 4 and achieves a 2x
speedup, thus compiling each set of 4 modules in 2 seconds instead of
4, only can do two of these compilations in parallel, so it takes 2
seconds. We've cut the total work in half, and yet it takes twice as
long in real time to complete, and 6/8 of your cores are idle.
OK, but that's when you have several cores. If you had only one core
(or maybe two) so you weren't using much parallelism in the first
place, then the unity build would look like a win, since you'd really
see the improvement from the amortized overhead of the combined
modules.
Most of us don't have single core machines on our desks, or laps, or
even in our pockets (or on our wrists?).
But we do have single core machines (or close to it) in our CI runners!
We get 1.5-2 cores, roughly, in the slice of VMs we get to run CI on
GitHub Actions. So maybe this is the one place where a unity build would
be of benefit.
That's the theory anyway. Does it pan out? I tested it, and the answer
is basically yes. I can speed up the "build" step of our CI by
somewhere in the vacinity of 30% (varies a LOT from run to run). I
tried a whole bunch of different approaches, summarized in the table
below. I picked the best basic strategies revealed by those tests and
set up the CI to use them. (I didn't preserve all the other
permutations.)
| Name | try 1 build | try 2 | try 3 | try 4 |
|:----------------------- | ----------- | -------------------- | -------------- | -------- |
| **linux gcc9 BASELINE** | 14:19 | 11:46 | 11:56 | 13:32 |
| linux gcc9 group | 9:39 | 8:06 | 9:58 | **7:58** |
| linux gcc9 batch 8 | 8:06 | **7:17** | **8:02** | **7:59** |
| linux gcc9 batch 16 | 8:29 | 7:32 | **7:56** | 9:35 |
| linux gcc9 batch 4 | 8:39 | 8:38 | 8:52 | 10:31 |
| linux gcc9 batch 2 | 13:34 | 12:43 | 11:22 | 9:32 |
| linux gcc9 batch mix 4 | 7:22 | 9:14 | 9:01 | 9:39 |
| linux gcc9 batch mix 2 | 9:11 | 7:45 | 10:55 | 9:36 |
| **linux icx BASELINE** | | 14:39 | 15:07 | 15:06 |
| linux icx group | | 11:59 | **10:36** | 10:26 |
| linux icx batch 8 | | **9:40** | 11:31 | **9:14** |
| linux icx batch mix 4 | | 10:06 | 11:58 | 11:16 |
| **macos11 BASELINE** | 25:41 | 27:53 | 26:30 | 28:32 |
| macos11 group | 10:56 | 20:24 | 19:28 | 19:24 |
| macos11 batch 8 | 16:43 | 17:38 | **18:34** | **17:42** |
| macos11 batch mix 4 | | **9:22** | 19:12 | 18:26 |
| **windows BASELINE** | 20:17 | 23:27 | 28:43 | 19:13 |
| windows group | 23:05 | 15:57 (tests broken) | 21:57 (broken) | 15:33 |
| windows batch 8 | 21:16 | 15:57 | **17:38** | **15:08** |
| windows batch mix 4 | | **15:30** | 23:19 | 15:16 |
Notes:
* Timings vary wildly from run to run. You never know which VMs you'll
get or how overloaded they are. But over 3 trials, the trends were
clear.
* I rigged it to clear the ccache, so these are FULL uncached builds.
* This is only "build" time. The CI run also includes time to set up the
VM, download docker images, build or install dependencies, and of course
run the testsuite. So the overall speedup of the entire CI run will not
be as improved overall as these numbers indicate for the build step.
But still, saving several minutes per job means the CI tests turn
around faster, and that's what we want.
For C++17 and beyond (and maybe earlier in some compilers), we already
used std::gcd. Now for C++14 compilers without std::gcd, we roll our
own simple gcd implementation instead of using boost::gcd. Maybe it's
slightly slower, but this is nowhere near a performance-critical
path. It's worth it to take one more step at eliminating boost.
Reminder: When opening a file by name, first it tries to open assuming
that the file's extension indicates the format. If that doesn't work,
it tries all known format readers to see if one will work.
This patch includes several improvements to this process and surrounding
code in imageioplugin.cpp:
* When trying all formats because the extension was wrong, instead of
traversing the input_formats map in alphabetical order of the list
of all format names + extensions, try the three most common formats
we encounter (openexr, tiff, jpeg) FIRST, then try the other formats
in alphabetical order. We do this by traversing the
format_list_vector, which contains all format names in the order
they were declared, and ensure that those common formats are
declared first. Basically, this means that if the mislabeled file is
one of those most common formats, the correct reader is found after
only a couple tries, instead of having to try a dozen or more bad
guesses of rarely encountered formats.
* Debug mode messages that make it easier to understand which readers
are being tried for which files.
* Comments clarify that input_formats and output_formats hold not just
the map from format names to creators, but also map extensions to
creators. (That confused me while I was debugging this, so I figure
the clarification is needed.)
* Establish a new extension_to_format_map that lets you get the format
name if you know the extension.
* Get rid of input_extensions and output_extensions maps, which weren't
ever used.
* Some other minor refactoring.
If you know the hash of a ustring, this tells you the ustring.
(Caveat: in the very unlikely chance of two ustrings having the same
hash, this will return the first one it finds in the table.)
I realized that --attrib (and several related actions) only set the
attribute as directed for the first subimage if it's a multi-image
file (such as a multi-part exr), or all of them if the -a flag is
used. But there was no ability to set it for a particular (non-first)
subimage, like many other oiiotool actions have the ability to do.
So in this patch, I refactor the following commands to use the
OiiotoolOp and related macros, which automatically make it responsive
to the `:subimages=` optional modifiers:
--attrib
--sattrib
--eraseattrib
--caption
--orientation
--clear-keywords
--iscolorspace
As a bonus aside, one thing this lets you do is set a different
compression method per-subimage using, e.g.,
`--attrib compression:subimages=3 dwaa:100`.
But if you set compression with `--attrib compression`, do not also
use `--compression`, which will override it and also apply to all
subimages.
When this symbol is defined, this marks as deprecated the ErrorHandler
class methods that use sprintf formatting notation. It's not enabled
by default, but a downstream project can define the symbol in order to
help root out any uses of the old style formatting functions, to aid
identifying and changing those spots to the new style, long before
they are truly deprecated or removed from OIIO.
This change also sets up an OIIO_INTERNAL symbol that will be present
when OIIO itself is compiling, but not a downstream project using the
OIIO headers.
This is the first tiny step to fixing all the little things that break
in a unity build. This one is pretty small fry -- the oiiotool set of
modules only needed some minor rearrangement of header file include
order.
* Initial implementation of `ImageBufAlgo::st_warp`
* Simplify default filter logic, and share it with `ImageBufAlgo::warp`
* Add st_warp to oiiotool
* Add tests for bespoke `st_warp` validations that are done before IBAprep
* Add st_warp test to oiiotool-xform testsuite
* Address review notes
- Remove ST buffer datatype guard
- Simplify additional ROI prep
- Remove unnecessary buffer reallocation
* Use sample accumulator type for source/dest iterator USERT
* Add `--st_warp` section to `oiiotool.rst`
Cut/paste error inadvertently deleted a critical cmake line that I
wasn't intending to alter or remove, that resulted in the command line
utilities not being tagged for part of the install step. Oops!
Oh boy, never leave anything unbenchmarked.
Turns out the boost::algorithm functions we were relying on underneath
many Strutil "case-insensitive" comparisons were ridiculously slow.
We thought they were good enough because they allowed specification of
locale, so we could just pass the static classic locale, and so they
would be inexpensive because they didn't have to query the current
locale. But this is wrong, they were still ghastly.
So here I rewrite Strutil::iequals, iless, starts_with, istarts_with,
ends_with, iends_with in terms of a new (internal) Strutil::strcasecmp
and strncasecmp (which underneath handle differences in platform, and
use the locale-independent versions). The net result is that most of
those case-independent comparisons speed up by a factor of
50-100x. Wow.
I still need to tackle the family of 'ifind' related functions. They
are a bit trickier. But I'll leave them for another time, because I
need to roll this present fix out right away to fix a real production
bottleneck.
(Worth noting: iequals is instrumental when you're searching a
ParamValueList for a particular name, which is what happens when you
look up attributes from an ImageSpec, which is what happens when you
call get_texture_info(), which is what underlies OSL gettextureinfo()
calls in the cases that they cannot be constant-folded during runtime
optimization. So this came to my attention when analyzing a slow scene
whose shaders had a pathological explosion of gettextureinfo calls that
couldn't be optimized away.)
Reminder: isConstantColor() finds out if an ImageBuf is constant
color. It uses the thread pool to check several areas in
parallel. Once any task finds that its subregion is not constant, it
sets a flag so that as other subregions are pulled off the task queue,
they will take an early out if one of their siblings already knows the
image is not constant.
Turns out this isn't much of a big savings in practice, because lots
of regions may be in flight by the time one region realizes it's
non-constant, and once a region task starts, it doesn't check the flag
again.
This patch speeds up isConstantColor by a pretty big factor for
non-constant images (it can't really help for images that ARE
constant) in the following ways:
1. Before doing anything, it checks to see if the FIRST TWO pixels of
the image are unequal. If that's the case -- and it's a fairly
common case for non-constant images -- then it immediately knows
the whole image is not a constant and doesn't need to look at
anything else or launch any of the threads at all. (If it is a
constant image, or if the first two pixels are just coincidentally
identical, this quick check of only two pixels didn't really cost
us any masurable time.)
2. In addition to checking the result flag at the start of each
parallel job, now we also check if the flag has been changed as we
iterate to each pixel. So pretty much as soon as one thread finds
unequal pixels, the other threads can spot this and stop their
tasks early, no need to finish looking at their entire region. (To
make the more frequent checks of the flag inexpensive, I changed it
from an atomic to just a regular bool. It doesn't need to be
atomic, because the strict ordering is not important at all here
and it doesn't change the result if the threads' access to the flag
is unsynchronized.)
When outputting OpenEXR files, in a case where the "compression"
request was for "dwaa" (no compression specified in the main name,
like "dwaa:100") we accidentally forced a default quality level of 4,
which is the default we wanted for zip compression. But that's not a
good default for dwaa!
As I learned recently, a "unity" (aka "jumbo") build is where multiple
.cpp files are combined into one translation unit -- imagine a unity.cpp
that simply has
#include "file1.cpp"
#include "file2.cpp"
...
#include "fileN.cpp"
and so you compile unity.cpp instead of the separate file?.cpp files
individually.
Turns out that CMake understands this concept and can do it for you
automatically!
The benefit of a unity build is that file1...fileN probably include
most of the same headers, expand the same templates, etc., so a bunch
of the per-file work of the compiler can be done once rather than
redundantly for each file.
There are two potential downsides, however:
1. It may not be safe to concatenate your cpp files! For example, if
both file1.cpp and file2.cpp each contain a `static int foo;`, that
may have been safe when compiled separately, but is not allowed to
happen twice in one compilation unit. Similarly, if you have headers
that don't have proper guards against multiple includes, etc. So one
should expect a whole lot of little fix-ups are needed for this to
work properly. (We'll come back to that topic.)
2. Combining source file into these "jumbo" modules can make heavily
parallelized builds on many-core machine not be able to load balance
or keep all the cores busy. (Simplified examples: if you have 16 .cpp
files on a 16 core machine, each core can compile one cpp file in
parallel with the others. But if you mash the modules into just one
huge cpp file, give that to one core, and your other 15 cores are
idle, so the full build probably takes much longer.)
I tried this out, including the many fixups implied by (1) above, and
at first the unity builds were slower on both my laptop (8 cores) and
workstation at work (32 cores), because of downside (2) explained
earlier -- harder to take advantage of parallel builds when there are
fewer, bigger, compilation units. Some tweaking of strategy got me to
the point where I could always get the unity builds to go faster, but
not by a whole lot when many core were available. Slighty faster, but
not worth the trouble. A bit disappointing, nearly abandoned the whole
idea.
HOWEVER, in situation where you are limited to very few cores (like in our
CI, which allocates 2 cores, and it sure seems more like 1-1.5 for the
Windows and Mac CI runners), the unity builds are substantially faster --
there's already not much parallelism to exploit, so you come out ahead
with the savings of that redundant per-file work we talked about.
So my current thinking is to go ahead and make the changes that allow
unity builds. I don't particularly recommend them when highly parallel
builds are available to you, but it might help to cut our CI
turnaround time down on the GitHub runners. And maybe it will help in
other situations for other people.
Ok, then. The present patch introduces these concepts and makes the
CMake and other build system changes to allow unity builds. (N.B. It
won't WORK yet, so don't try it!) After we get that out of the way, in
subsequent PRs I'll show all the changs to the code that were
necessary to fix all the little things that went wrong when source
files got combined.
* Fix warning from duplicate define
* Fix warnings about bool/int expressions
* Fix discarded result warning in PSD input
* Fix codepage warning from unicode symbol in string constant
* Ensure the term object is initialized with the stream we intend to output to
* Fix pointer casts in paramlist test
Signed-off-by: Chris Kulla <ckulla@gmail.com>
Co-authored-by: Larry Gritz <lg@larrygritz.com>
In FFMpegInput::close(), some of the tear-down calls would happen
regardless of if their pointers were already null. Seemed safe, but
maybe not always? So check the pointers.
Fixes#3371
When trying to break filenames into the base name and any "RESTful"
arguments (such as "/foo/bar.exr?name=value&dog=cat"), we were
just looking for presence of the "?" to determine if it was a RESTful
thingie and using that as the split position.
But this is thrown off by an odd Windows notation where names that
start with (get this) "\\?\" somehow designate long filepaths. See
See: https://docs.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation
So the solution is to change Strutil::get_rest_arguments to explicitly
check for this pattern at the start of the string.
In OpenEXROutput::write_deep_scanlines and write_deep_tiles, don't
just pass the DeepData along -- handle the case of where the DD (the
buffer the caller is presenting) does not match the data types that
you said the file is going to hold. In this case, we need to make a
local temporary DD that holds a copy of the passed-in data, but with
the formats transcoded.
Some changes to DeepData were done to support this.
GHA is retiring the windows-2016 image, we won't be able to use it soon.
Also bump openexr versions in a couple places.
There's some in-progress work on a VS2022 test, but ignore that for now,
it's commnted out.
I'm a total convert to fmt (and/or std::format, std::print) and slowly
converting all prior use of sprintf and iostream to it.
If I print an Imath::V3f or a simd::vfloat3,
fmt::print ("v = {} {} {}\n", myvec[0], myvec[1], myvec[2]);
That's ok, I guess, but it's clunky and a big pain to put the format
specification for every element separately if it were a vfloat16,
say. I would like to be able to do this:
fmt::print ("v = {}\n", myvec);
and in fact, it does print all elements (thanks to some nice behavior
by fmt)! But this does not work as expected:
fmt::print ("v = {:.3f}\n", myvec);
fmt's tricks will print all elements, but only using the default format
spec, and the spec above applied so the "whole thing", not each element,
if you see the distinction.
So this patch sets up the custom formatter infrastructure to be able
to print array-like things (and in particular, the Imath and
OIIO::simd vector and matrix classes) by specifying the format spec
used for the elements.
The important elements to note:
* detail/fmt.h adds custom formatter helpers for (a) any class with
operator[] and size() ("index_formatter"); (b) any class that lacks
these but nonetheless has an array-like memory layout and a known
length ("array_formatter"). I also added a nonstandard extension to
the format spec for these, which optionally lets you use a comma
separator rather than just the space between elements.
* These building blocks are used to specify custom formatters for
the commonly-used Imath types (in our Imath.h) and our internal
SIMD vector types (in simd.h).
* simd.h also adds a few other goodies, including an `is_simd<>`
template, and `size()` constexpr methods to each vector type. These
make certain template metaprogramming tasks easier for these types.
* A new "type_traits.h" header for collecting a bunch of little utility
templates, such as `has_subscript<>` and `has_size_method<>`.
JPEG expresses pixel aspect ratio implicitly as header fields
"xdensity" and "ydensity" (which we call "XResolution" and
"YResolution" in our metadata after the TIFF names). We also have
"PixelAspectRatio", so the three of them, if they are all used, form
an over-constrained system. Plus, they may come both in the JPEG
header and also (not necessarily having the same values) in any Exif
block.
And on top of that, the way Nuke and PhotoShop infer the aspect ratio
is BACKWARDS! (If pixelaspect = width/height of a displayed pixel,
then it's also = ydensity/xdensity. But Nuke is the other way
around. Only for JPEG, it gets it right for TIFF and OpenEXR.)
Anyhoo, there were some edge cases in which we were still getting
confused (specifically, that we were making some files that Nuke was
not properly recognizing as anamorphic). This patch tightens up
these cases.
Create a small shim class template called
Vec3Param (and its specialization, V3fParam) that does nothing but
implicitly convert from and to things that looks like 3-vectors
(thanks to some template magic I developed a while back for the Imath
project). And similar for template MatrixParam (and specializations
M33fParam and M44fParam) to pass a square matrix.
Basically the idea is that we can replace the public API function
parameters where we take an Imath::V3f and instead use a
V3fParam. User code that passes an Imath::V3f still works because
V3fParam accepts and converts it. Implementation code can implicitly
convert the V3fParam to an Imath::V3f, so it doesn't need to be
changed, either. Only the function declaration itself. (Same basic
situation for MatrixParam and Imath::M33f/M44f.)
So in the process, now we have laundered our public API so that it
doesn't reference the Imath types and headers any longer. This confers
two big advantages:
1. It allows OIIO itself and any downstream client code calling OIIO
to use *different* versions of Imath if they want.
2. It allows client code to not need Imath at all, for example if they
have their own proprietary vector type that they prefer to use
internally.
I'm not necessarily 100% sold on this approach, but I think it's got
promise and am seeking feedback. It seems to work, and I was able
to compile OSL against this altered OIIO with *no changes*.
The use of ends_with instead of iends_with made it so that
capitizalized files (especially on case-independent filesystems like
Windows or some Mac filesystems) were rejected.
(And yes, I did grep the whole code base for ends_with to be sure that
this was the only place we made this mistake.)
Fixes#3362
icx is the *new* Intel LLVM-based compiler, and it behaves a lot like
clang (versus icc, which is the "classic" Intel compiler).
* An additional CI test in ci.yml for icx.
* Tweaks in compiler.cmake and platform.h to detect and set options for
this compiler.
* A variety of warning fixes encountered for this compiler.
* Add a new cmake build-time option `ENABLE_FAST_MATH` (default: OFF)
for enabling fast-math optimizations that trade IEEE compliance for
speed. The Intel compiler enables it by default (!), so this build
option makes icx behave like the other compilers (not fast math
unless explicitly asked for), which makes the CI tests and other
behavior more consisent across compilers and platforms.
* Note: The bug that prevents some versions of fmtlib from working
with icc does not appear to be a problem for icx.
I finally learned how the "strategy" stuff works and refactored our
CI workflow based on it.
In short, we previously treated each CI test as its own universe, with
a huge amount of repeated build logic. These changes cut down on
redundancy by, for each set of related tests, collapsing them into
just one recitation of the build script logic and a handful of
variables that create the different test variations.
Currently, the categories are (a) all tests based on ASWF containers and
corresponding to VFX Platform specs, (b) "off-year" combinations based
on the native Ubuntu installations, (c) Mac tests, (d) Windows tests,
(e) the clang-format checker.
This refactor cuts the size of the ci.yml file by more than 50%, and
makes it much easier to add or change individual test configurations
in the future.
Strutil::print wraps fmt::format to render the formatted output into a
std::string, then sends the string to Strutil::sync_output(), which
ensures that every string printed (or sent to a FILE or stream) is
done so atomically, using an internal mutex, and with a stream flush
after each string. But this makes it a lot more expensive than it
needs to be, when you don't need such care -- either because buffered
output is ok, or because you don't need the thread safety.
This patch:
* Adds an optional `flush` parameter to sync_output(), defaulting to
true, which controls whether sync_output flushes or not.
* Introduces Strutil::sync::print, which is the same fully
synchronous, thread-safe, non-buffering behavior that Strutil::print
has traditionally had. Use this when you need those features.
* Redefines Strutil::print to be an alias for fmt::print, which means
- There is one fewer layer of template expansion and generated code,
and possibly fewer internal string copies, so that should make for
both faster compilation and faster runtime performance.
- Results are buffered, no sync after every output, so much faster
when you don't need that.
- No internal mutex.
On the platforms we use -- and with the current implementation of
fmtlib -- fmtlib's underlying use of stream and FILE output appears to
be adequately thread-safe. But we can't 100% guarantee this on all
platforms or all future versions of fmtlib, so users who need the
stronger guarantees (and are willing to sacrifice performance) can
simply use sync::print.
* New CI test downloads the Intel icc compiler and builds with it.
* A variety of minor fixes to address warnings from icc that were
not apparent from the other compilers.
* Some minor changes to just a few tests and reference output where
the icc-built compiler produces slightly different (usually LSB)
results.
These are cases where we declared a template, then later gave its full
definition, and we marked the declaration of the OIIO_FORCEINLINE but
omitted it from the later definition.
This addresses some compiler warnings from icc.
Slight reworking of -Werror logic to let the default behavior be
changed by setting a cmake variable. In particular, this allows us to
make a CI build disable -Werror when we need to.
Previously, having the "OpenmageIO_CI" env variable set trumped all
else and forced -Werror. Now it just contributes to the default behavior,
but setting `-DSTOP_ON_WARNING=OFF` and cmake config time can still
disable it.
Seemed like a good idea when we wrote it, but subsequently,
std::string_view was accepted into C++17 without a str() method. So
for the sake of future full interchangeability with std::string_view,
stop using this method. We will eventually deprecate it.
Special case some methods of ParamValue, ParamList, and
ImageSpec::attribute for ustring and string_view data, to reduce some
redundant copies and/or ustring conversions.
The internal method init_noclear() is augmented with a flag that allows
the caller to specify that any string data it might point to is already
ustrings, and thus does avoid the costly default behavior of converting
strings in the data block to ustrings. (This is not publicly exposed, so
users can't screw it up.)
So the new ParamValue constructor directly from a ustring can use this
flag to avoid passing the already-ustring through the string-to-ustring
conversion a needless second time. But also, the copy constructor, move
constructor, and assignment can use the flag and avoid the extra
conversion, since by definition they are copying from the contents of an
existing ParamValue which is guaranteed to be already ustring-ified.
While I was at it, I turned the optional arguments of ParamValue
constructors into "strongly typed" flags, deprecating the ones that
take a raw bool, which might be confusing.
c_str() was cute, but it's a little dicey and it's not present in
the C++20 std::string_view. We didn't use it in a ton of places,
so replace with other idioms so that in the future we are better
able to interchange with std::string_view or replace our own entirely
once we have flying cars and C++20 minimum.
* Allow ustring and OIIO::string_view to auto-cast to fmt::string_view
(helps some fmt formattting of types without custom formatters).
* When std::string_view (C++17) and/or std::experimental::string_view
(some compilers pre-C++17) are available, add OIIO::string_view
auto-construction from and auto-casting to those types.
...and stop using them internally as well.
For some of the IBA functions, we were unable to fully mark them as
OIIO_DEPRECATED (so instead that hint is commented out) because the
function signatures of the deprecated and new ones can be
ambiguous. We're just going to have to delete those when we are at a
compatibility break point, without going through a period where they
are present but marked as deprecated.
Just like we had done for the other libraries recently. And need
to make sure that the imath libs are added explicitly to the python
module's needs, since for hidden static libraries, they will not get
picked up merely by linking to libOpenImageIO.
Add an OIIO build-time option for forcing our exported
OpenImageIOConfig.cmake to skip the find_dependency of Imath and
OpenEXR.
This is helpful if you're at a site that is trying to build OIIO
against statically compiled Imath/OpenEXR, and want to be super sure
the downstream projects are definitely not going to encounter that
find_dependency(Imath) when the read the OIIO config files.
(They could do their own `-DOPENIMAGEIO_CONFIG_DO_NOT_FIND_IMATH`,
but I want even that to be unnecessary.)
There was that one pesky place where we were still including Imath
headers, from simd.h in the case where OIIO_SIMD==0 and we wanted to
fall back on some Imath::Matrix methods.
Ugh, but this backfired because when including this file from Cuda,
OIIO_SIMD==0 and it included the header. But what happens if a
downstream app (a) includes this file from Cuda, and (b) uses a
DIFFERENT Imath (so names clash) or doesn't want to use Imath at all?
So one more change:
* Remove the include of Imath.h from simd.h entirely.
* Most places where we fell back on the Imath::Matrix methods, just
replace it with scalar code, it's not complicated. (For some technical
reasons, this required some minor surgery on the matrix44 class, but
not anything that changes anything from a caller's point of view.)
* The one complicated case is matrix44::inverse(). Punt! Just let that
only be implemented if either (a) there is SIMD available, or (b)
there is Imath available (i.e., included by the downstream code
prior to this header). That's it. If you want to invert a 4x4 matrix
without SIMD, you need to arrange inclusion of the correct
ImathMatrix.h yourself before including this header.
For vector displacement maps or normal maps that are in tangent space,
it is heplful to be able to record whether the data are intended to use
a local left-handed or right-handed coordinate system, and for that to
be querably from the shader accessing the map.
The `maketx --handed` option, or `oiiotool -attrib -otex:handed=...`,
or adding "handed" metadata to the configuration ImageSpec being
passed to `IBA::make_texture()` does this. It's easy for texture
formats like openexr that allow arbitrary metadata. For TIFF textures,
we embed it in the ImageDescription tag (as we do for several other
texture-specific hints).
Skip 3 bytes between "software" and "background (key) colour" for the "software
version" (was simply missing). Avoids that m_alpha_type is overwritten with
wrong values.
The very latest fmt master changes a behavior: fmt functions no longer
automatically know how to format any type that can output to an
ostream, and instead need custom formtter objects. This breaks A LOT.
Fix by defining the preprocessor symbol `FMT_DEPRECATED_OSTREAM`,
which restores the old behavior (for now, at least).
While I was at it, I moved the fmt inclusion logic out of strutil.h
and into a new separate header, detail/fmt.h. This is in turn included
from strutil.h, so nothing in user code needs to change.
It turns out that this can crop up all the time, sometimes from
malformed metadata inside image files. We weren't noticing it because
in many cases, things ended up being turned into ustrings twice, and
the sequence string-with-nuls -> ustring() -> c_str() -> ustring()
will launder it. As I was tracking down and eliminating some of the
double ustring constructions, the metadata strings that were malformed
all along started becoming symptomatic.
Bad paradoxes follow from a ustring contain embedded NUL characters,
including ustring.c_str() and ustring.string() being different
character sequences, as well as ustring.hash() not being equal to
strhash(ustring.c_str()). One should definitely be able to count on
ustring(cstring) == ustring(ustring(cstring).c_str()).
So this patch checks for embedded NUL characters at the appropriate
time of ustring construction, and truncates the character sequence at
the point of any internal NUL character.
* `OPENIMAGEIO_IMATH_DEPENDENCY_VISIBILITY` is a new CMake cache
variable at OIIO build time that controls whether the Imath library
dependencies will be declared as PUBLIC (default) or PRIVATE target
dependencies of libOpenImageIO and libOpenImageIO_Util.
* For *downstream projects* that consume OIIO's exported cmake config
files, setting CMake variable `OPENIMAGEIO_CONFIG_DO_NOT_FIND_IMATH`
to true will skip the find_depencency() calls for Imath and OpenEXR.
To clarify, this is not a variable that has any effect when building
OIIO, it's something set in the downstream project itself.
* Add a `OIIO_IMATH_H_INCLUDED` header guard in OpenImageIO/Imath.h,
which allows something downstream to pre-define it to prevent OIIO
headers it needs to read from using the contents of that header.
Most builders of OIIO won't care and shouldn't change the defaults or
use any of these. But here's the specific case it addresses: If you
are building OIIO against static libraries of Imath & OpenEXR, but the
libOpenImageIO will be consumed by a downstream project that wants to
use a different Imath version, this can help you avoid some build-time
conflicts in the downstream project.
PPM by specification is Rec709! Make it so.
PNG files were getting marked as sRGB or Gamma if certain header
attributes were found in the file, but if they were absent, we didn't
designate a color space at all. Change so that the default assumes
sRGB, in the absence of any metadata telling us for sure.
Fixes#3084
One new method that should have been inline and non-virtual (so that
this whole set of changes does not change ABI by shifting the vtable
around) was accidentally marked virtual -- cut and paste error. This
patch fixes it to restore ABI compatibility.
IPTC data blocks seems buggy, and I don't know why. Our code?
Something in libtiff? Since nobody is depending on it and none of our
testsuite appears to fail when disabled, only write the IPTC block if
the user explicitly asks for it. Also, when reading them, don't
overwrite other metadata of the same name.
We wrote our string_view before C++17 std::string_view was finalized,
and the std one doesn't have a c_str() method. For conformance with the
standard, I want to remove this over time from ours.
Part 1: There were some places we did not need to call c_str() at all,
and some places where it was much more appropriate to just use a
std::string.
As Doug Rogers points out in Issue 3304, it is customary for Windows
applications to use wstring extensively for Unicode support. But it's
especially critical for unicode filenames, because unlike Unix-like
OS's where fopen() and similar functions support UTF-8, on Windows
those functions are strictly ASCII, and Windows users are used to
calling _wfopen() and other nonstandard functions that expect
filenames to be std::wstring if they want to support Unicode.
This patch:
* More clearly documents through all of our classes when parameters that
represent filenames or searchpaths are assumed to support UTF-8.
* Augments ImageInput, ImageOutput, and ImageBuf so that their methods
that take file names or path lists have additional flavors that take
wstring (always implemented as an inline shim that translates the
UTF-16 wstring to a std::string with UTF-8 encoding to pass on to the
usual method).
* Augments ImageCache and TextureSystem in just a couple places, the
functions where a filename is turned into a handle. All the many
other functions that take ustring for function names were left alone,
because they are called very frequently and are performance-sensitive,
so we don't want any translation from wstring to string happening on
every call.
We do not attempt to add wstring-based versions of functions that take
strings for other purposes -- that is, that are not intended to
represent filenames.
C isspace() takes an int, but on Windows when building in debug mode,
the implementation of isspace() can hit an assert if the value is < 0,
which of course it can be if you're looking at a character in a string
whose high bit is set (because char is signed).
Easiest solution is to make our own Strutil::isspace that does exactly
what we want and is safe.
Fixes#3300
This makes sure that two new preprocessor symbols are defined in
oiioversion.h: `OIIO_USING_IMATH_VERSION_MAJOR` and
`OIIO_USING_IMATH_VERSION_MINOR`, revealing the version of Imath being
used by the build of OIIO (which may not be the same as the version
found or being used by the app that's calling OIIO). They also get
defined as CMake variables in our exported cmake configs, allowng
downstream cmake build systems to be able to easily test whether the
Imath they found is compatible with the version that we use in our
APIs.
This patch sets out to minimize the extent to which Imath is a
necessary source dependency for using OIIO C++ header files and
APIs. The major changes are:
* Remove the unconditional `#include <OpenImageIO/Imath.h>` from
fmath.h and simd.h. Those forced Imath headers on nearly all all
code that uses OIIO's public APIs, whether or not they needed the
very few calls where Imath types are used as parameters.
* Make sure all public OIIO non-virtual methods and freestanding
functions that have Imath types as parameters are enclosed in header
guards like `#ifdef INCLUDE_IMATHVECTOR_H`. Downstream client code
needs to include Imath headers before including OIIO headers, if
they want to call those few particular functions.
This means that client applications who don't need those particular
functions, but call other parts of the OIIO APIs, don't need to
`#include` any Imath headers (or pay the extra compilation time for
doing so), and don't need any knowledge of which specific version of
Imath is used by the OIIO they are using.
The small number of public API calls that will no longer be visible
downstream unless the client code also include Imath headers (before
the OIIO headers) are:
- color.h: ColorConfig::createMatrixTransform (needs ImathColor.h)
- fmath.h: type conversions involving `half` (needs half.h).
- imagebufalgo.h: warp(), colormatrixtransform() (needs ImathMatrix.h)
- simd.h: conversions between our SIMD types and Imath types (Vec3,
Matrix33, Matrix44) and vtransform() calls involving those types
(requires ImathVec.h and ImathMatrix.h), and SIMD load/store
involving `half` data (requires half.h).
These changes necessitated adding including of Imath.h in a number of
internal .cpp files, the ones where those types are actually used.
The following header files do still unconditionally reference Imath
types or headers:
- texture.h: Some virtual methods of TextureSystem use Imath types
as parameters.
- imagebufalgo_util.h: For `half` support. We use this a lot
internally, but presume most client code never uses this header.
- simd.h, ONLY IF USE_SIMD=0 (which is generally only used for
development debugging and if porting to obscure platforms that are
neither x86 nor ARM).
The bottom line is that the vast majority of downstream callers of OIIO
APIs who need ImageInput/ImageOutput, ImageBuf, ImageBufAlgo, and
generally everything in the OpenImageIO_Util library, do not need to
use Imath types, include Imath headers, or even be aware of which Imath
release OIIO is using.
Why would we do all this?
Because this paves the way for optionally building OIIO to link against
static libraries for OpenEXR and Imath, essentially embedding and
completely hiding an OpenEXR/Imath inside libOpenImageIO.
That allows libOpenImageIO to use the very latest Imath (3.1, say,
with the new faster core API), even though that libOpenImageIO is
being consumed by an application that for whatever reason is forced to
use an entirely different (older) OpenEXR or Imath library. In other
words, it lets OIIO use OpenEXR/Imath fully embedded without
transitively passing that OpenEXR version as a downstream dependency
(except for the few use cases that need to call the tiny number of
OIIO API methods that require Imath types.
There is a small chance that client code that use Imath types and
SHOULD have been includeing Imath headers themselves actually omitted
those includes and dependd on the fact that certain OIIO headers
included them. Now that they don't, it may require the client-side
code to add those includes. I'm going to claim that this is a "bug" in
the client code (they didn't include a header that defined a type they
used), and not a breaking incompatibility in OIIO's public calls
themselves. Thus, I feel comfortable with these changes being in OIIO
2.4 (today's master) and not necessitating a 3.0 release (in which
arbitrary loss of backward compatibility would be allowed). So watch
out for the occasional need for OIIO-calling code to need to add an
`#include` for Imath types the code was always using, but neglected
to explicitly include.
Our Strutil utilities for utf8 <-> utf16 conversion were based on
Windows specific calls that we had added even before C++11 was our
minimum. Replace the Windows-specific code with std calls, and
expose them in strutil.h for all platforms (why not?).
We were passing along, in the exported cmake config files, a demand
for downstream clients to use a C++ standard version equal to that
which we used to build OpenImageIO itself. But that's not
necessary. We may choose to build with C++17 (for example), but all of
our public APIs are safe to use for C++14.
Split the build-time options to CMAKE_CXX_STANDARD for the standard we
build this project with, and DOWNSTREAM_CXX_STANDARD for the standard
we are declaring is the minimum to *use* the library.
The main event:
* Assume that for a TGA 1.0 file (too old to have the header field
that says if alpha is associated, unassociated, or meaningless), if
the alpha is zero everywhere, it means that it's meaningless, not
unassociated, so in that case we should skip the
auto-premultiplication. (See issue 3277)
Public-ish stuff:
* Be more consistent with Targa-specific attributes all being called
"targa:foo" (most were, but one was "tga:alpha_type").
* Document all the targa-specific attributes (a few were not).
* Add "targa:version" to reveal whether the file was a TGA 1.0 or
2.0 version of the format.
Internal stuff, opportunistic:
* Some more minor conversions of string fomatting to fmt style.
* Convert some std::cerr debugging code (commented out) to the DBG macro
we use anywhere.
* The `std::unique_ptr<uint8_t> m_buf` should always have been
`unique_ptr<uint8_t[]>` because it's not a pointer to a single value.
I realized recently that if you said something like
oiiotool foo.exr --ch R,G,B,A -o out.exr
and there was no "A" channel... it would just silently fill the channel
with 0 in the output.
This patch adds a warning if --ch is told to copy a channel that doesn't
exist in the input.
Ugh, there's a situation where PhotoShop is misinterpreting an alpha
channel for certain OIIO-written TIFF files, but it's fine with some that
came from elsewhere, and the only thing that appears to be different is
that the non-OIIO one entirely lacks the EXTRASAMPLES TIFF tag.
So I'm adding an option (use with extreme caution) allowing you to
tell the TIFF output to not output this tag, even though clearly it
should be.
Some rogue app out there is writing 4-channel TIFF files without the
ExtraSamples tag in the header that explains that the data is a valid
alpha channel.
We were using TIFFGetFieldDefaulted to retrieve this tag, and guess what?
The defaulted behavior changed between libtiff 4.0 and 4.3!
Changing to TIFFGetField actually turns out to fall back to the right
logic on our side, which boils down to assuming a 4th channel is alpha
if there's no other information. That seams reasonable -- RGB + mystery
channel is almost certainly RGBA.
* Bugs fixed in build_opencolorio.bash, it botched some of the directory
choices when customizing where to place and build it. Also, switch to
the more standard BUILD_SHARED_LIBS cmake variable to control whether
to build dynamic or static libraries.
* Correctly handle the cmake variable VISIBILITY_MAP_FILE being set to
"" on the command line as a way to disable using the visibility map file
entirely.
* FindOpenColorIO.cmake changes:
- Prefer and use an exported cmake config for OCIO if found (is should
be present for OCIO 2.1+), unless caller set OPENCOLORIO_NO_CONFIG=ON.
- New hint OPENCOLORIO_INTERFACE_LINK_LIBRARIES lets caller force a
list of interface link libraries. This is helpful when purposely using
a static OCIO library, which itself has static dependencies, and you
need to name those specific library filepaths.
* Simplify some of the logic for setting extra link flag arguments for the
two main libraries.
* On Linux, also set link flags `--exclude-libs ALL` to hide symbols from
any embedded static library dependencies, to prevent clashes with other
versions of those libraries that may be compiled into a DCC app.
I realized that a call to make_texture() with the "maketx:ignore_unassoc"
attribute set did not properly set the configuration attribute up when
opening a new ImageBuf.
```include/OpenImageIO/typedesc.h:620:10: warning: 'const' type qualifier on return type has no effect [-Wignored-qualifiers]
auto parse(format_parse_context &ctx) -> decltype(ctx.begin()) const // c++11
```
Co-authored-by: Thiago Ize <thiago@solidangle.com>
The docs said that passing a tuple for min or max param of the clamp()
function gave per-channel values, but passing a single float gave one
value for all channels.
In fact, the implementation used the float to clamp only the first
channel. This patch fixes the behavior.
Fixes#3264
In addition to C++ code examples for each IBA function, work toward
also having equivalent Python code. Also, where applicable, show the
equivalent operation using oiiotool.
We had forgotten to mark --colorcount and --rangecheck as commands
that intentionally outputs to the console. This omission meant that an
oiiotool command consisting only of a those would warn about no
output. But we didn't really want it to.
How do you craft an oiiotool command that will examine a file and add
an alpha=1 channel if it's a 3-channel file, but leave the existing
alpha alone if it's already a 4 channel file?
This patch adds control flow, so you could express the above like this:
oiiotool in.exr --if "{TOP.nchannels < 4}" --ch R,G,B,A=1 --endif -o out.exr
The following slate of new fetures is added:
* `--set name value` sets a "user variable".
* Expression evaluation will expand identifiers that are the names of
user variables. For example, `{i}` is replaced by the value of user
variable `i`, and `{i+3}` returns 3 more than the current numerical
value of `i`.
* Expression evaluation has been extended to support the following
numerical comparison operators: `<` `>` `<=` `>=` `==` `!=` and
`<=>` (the last one being the 3-way comparison, like in C++20).
Also logical operators `&&` and `||`. All return 1 for true, 0 for
false. Note that the comprison and logical operators have the same
precedence as `+` and `-`, so use parentheses liberally to
disambiguate complex mixed expressions.
* New functions recognized within expressions:
- the value held by a named user variable: `var(name)`
- logical negation: `!`, `not()`
- string comparison: `eq()`, `neq()`
* `--if condition commands... [--else commands...] --endif`
* `--while condition commands... --endwhile`
* `--for variable range commands... --endfor` where the range may be,
like Python, any of `end` (implied begin=0, step=1), `begin,end`
(implied step=1), or `begin,end,step`.
Also needed to modify ArgParse so that it has the notion of whether
it's "running" -- actually calling the actions for each argument, or
if it's merely parsing but taking no action -- and to be able to set
and retrieve which command line argument index an action is responding
to.
This little change makes this work:
ImageBufAlgo::add (buf, { 0.2f, 0.2f, 0.2f });
instead of the more awkward
ImageBufAlgo::add (buf, cspan<float>({ 0.2f, 0.2f, 0.2f }));
and for anyplace tht has Image_or_Const parameter types.
New OpenColorIO and fmt versions are out.
* Bump "latest releases" test to fmt 8.1.0, and OCIO 2.1.1.
* Re-adjust CI test fmt versions across many tests to give a better
spread over the supported releases.
* Bump the default version to download and use if none is found to 8.0.0.
We had --line (wrapping IBA::render_line) and --box (wrapping
IBA::render_box).
This adds the obvious --point, wrapping the existing IBA::render_point.
The Python bindings for ParamValueList and ImageSpec allow a "dict"
style (associative array) interface:
imagespec["key"] = value
val = imagespec["key"]
This patch makes some minor improvements:
* Clarify in the docs that like a regular Python dict, the associative
retrieval will raise a KeyError if the key is not found.
* Add `__contains__` python method as a synonym for contains(), thus
implementing `key in dict`.
* Add `__delitem__` python method as a synonym for erase(), thus
implementing `del dict[key]`.
* Add an `ImageSpec.get()` method that is a synonym for getattribute
(which retrieves the item or returns None or a default if not
found), mirroring the behavior of Python's `dict.get()`.
Somehow I never implemented a copy() method for ROI, which of course is
super useful if you want to make an actual copy of an ROI and modify
the copy.
I also realized that ImageSpec.copy() unnecessarily/incorrectly set
return_value_policy to reference_internal, but actually it should be a
real copy, which it has to be anyway because the implementation
creates a separate copy.
* Docs: we previously only documented the "width=" optional modifier.
Add height, color1, color2.
* Change behavior ever so slightly so that if the "height=" modifier is
not supplied, the height defaults to the "width".
* Use the new pybind11 v2.9.0 for 2022 and "latest" tests.
* Bump the default version built by build_pybind11.bash to 2.8.1.
* Adjust the pybind11 version used in many of the other CI tests, to
give an even spread across versions, and to make each of the
"vfx platform" CI tests use the pybind11 version that was available
near the start of its repsective year.
For each ImageSpec ctr that takes a TypeDesc, add another one
that's the same but takes a string_view (which is just a simple
pass-through of the TypeDesc-from-string_view).
Also tagged Strutil and ustring::fmtformat as 'nodiscard', to produce
warnings if people call it and ignore the results. (As was the case in
my own bug that I tracked down in the course of this patch.)
When doing operations with filters that have negative lobes, you can
sometimes get visible ringing artifacts in very high contrast regions
of HDR images (you tend not to see these artifacts in images that have
range [0,1] because the ringing is very low amplitude unless the input
is very large).
You can do "highlight compensation" to reduce the artifacts when they
occur by doing --rangecompress, which does a log transform on the
input, then the filtered op, then --rangeexpand to transform back to a
linear space. But that makes for kind of verbose and clumsy command
lines, and makes it easy to err by doing the transforms in the wrong
sequence or omitting one.
The -otex command takes a highlightcomp=1 modifier, which does this
bracketing automatically, foolproof, with a compact notation.
This patch adds support for optional highlightcomp=1 modifier for all
of the oiiotool commands that let you specify a filter= option:
--rotate, --warp, --resize, --fit, and --pixelaspect.
* Clean up intra-document section references.
* New explanations about input and output configuration hints.
* More code examples in both C++ and Python.
* Fix some formatting flubs.
This was an obsolete experiment in adding a C binding. We abandoned
this effort (a replacement will happen eventually), but this one file
was left behind when we intended to remove it all.
Errors encountered while reading tiles were getting lost as we used
ImageBuf::Iterator to iterate over an image. Have the iterator keep a
flag for whether it encountered an error, set to true if the
ImageBuf::retile indicates an error occurred during the read.
Also, some ImageCache errors might get lost on the way to
TextureSystem calls, fix that as well.
TextureSystem is designed primarily for tiled images (typically 64x64
pixels each), but to handle untiled images, it just treats the whole
image as one big tile. That means for very large untiled images, you
might end up with one VERY large tile, and if the image is large
enough, the size and/or offset computations therein may overflow a
32 bit int.
This patch attempts to make all the offset computations in the texture
system be 64-bit safe.
After setting up the new IOProxy helper methods for ImageInput and
ImageOutput, here we go back to the readers/writers that had used
proxies for a long time, and switch them to using the new helpers.
Add two new global OIIO attributes: "limits:channels" and
"limits:imagesize_MB" which are imposed limits for validating a
reasonable number of channels and total uncompressed image size,
respectively. We presume that an image file header describing an image
that purports to exceed these is most likely corrupted or even
maliciously crafted. The limits, therefore, are designed to avoid
crashes, absurd memory allocations, or other cascading errors that
would result from taking such a claim at face value (for example,
a file that appears to have tens of thousands of color channels).
The limits default currently to a maximum of 1024 color channels, and
a maximum uncompressed data size of a single 2D image to 32GB. These
seem reasonable today, but if such large sizes become likely to be
seen in the wild in legit files in the future, we may change the
defaults. They may be adjusted now simply by setting the attributes,
of course, as should be done by applications that legitimately expect
to see input that exceeds them. (Applications also may make the
validation limits even smaller if they wish.)
I've modified the TIFF reader to check these limits as a first step.
We'll expand to doing such validation checks on other readers over time,
as we have opportunities to visit them.
This string utility computes the "edit distance" between two strings.
Currently it only uses the Levenshtein metric, but it has room for
future expansion by taking a parameter that can specify which metric
to use. Levenshtein is an algorithm for computing how many single
character insertions or deletions are necessary to turn one string
into the other.
I have a few ideas for where this can be useful, but as one
demonstration, I also modify the ArgParse class internals so that if a
command is not found, it will print a suggestion naming the valid
command option that has the closest edit distance (but only if one
choice is <= 2 edis away, and only if it's not a single-character
option, in order to reduce the number of useless suggestions). Here's
its behavior in action:
$ iinfo --stat foo.exr
iinfo error: Invalid option "--stat" (did you mean --stats?)
...
* The main event here is that we're now telling OpenJPEG to use
threading when compressing/decompressing JPEG-2000 files. This
improves the speed of JPEG-2000 reading on my laptop by 2-3x! The
threading feature was added to OpenJPEG 2.2 for decode, and 2.4 for
encode.
* CI: Build OpenJPEG 2.4 for the "latest releases" test, and build
their master for our "bleeding edge" test.
* Fixes to deal with the in-progress 2.5 deprecating a struct field.
* Generally clean up our detection of what OpenJPEG version we are
using.
* Some clean-up of the FindOpenJPEG.cmake file, use PATH_SUFFIXES.
Also fix longstanding bugs in PNM input:
* We weren't correctly emulating random scanline access, as is expected
(but in practice rarely used) of all ImageInput implementations.
* We weren't marking the color space in any way, now we do mark as
"Gamma2.2", which is the correct/preferred interpretation according
to the PNM spec.
* Add full IOProxy support to both input and output.
* This necessitated a major refactor, including sucking the I/O specific
code from rgbe.{cpp,h} into hdr{input,output}.cpp and getting rid of
the old files.
* Add a testsuite entry for hdr (shockingly, there was none before)
As well,
* Add a DDS test, apparently we never had one.
* Remove DDS setting of "texturetype", we don't ever use that, it's only
"textureformat" that we use. Not sure how that crept in there years ago.
Helper functions ioproxy_opened() and iotell() should have been const
all along, since they just return back information and should not
change state of the proxy, and also ioproxy() should have both const
and non-const versions.
* On Mac CI, use openvdb from Homebrew now that it's OpenVDB 9, which
properly supports OpenEXR/Imath 3.1.
* Update our FindOpenVDB.cmake target names to match the names used in
OpenVDB's own FindOpenVDB.cmake (even though we don't use their module,
may as well stick to their convention, especially if they eventually have
a properly exported config file).
For some reason, in the early days I got in the habit of having OIIO
input readers call their color space "GammaCorrected2.2" (for example)
when they appeared to be a straight-up gamma. (Generally only used for
certain fairly old formats that date from before sRGB existed.)
But there appears to be a widely used convention of naming OpenColorIO
spaces "GammaX.Y", so I'm switching OIIO to conform that
convention. It's more straightforward and increases the chances that
it will match the names of color spaces that already exist in people's
OCIO configs.
As we move toward fully equipping all image format implementations
with full IOProxy support, it's clear that a lot of repeated code is
error-prone. I first tried to collect this in a mixin class, but then
re-implemented after realizing that we can use the existing PIMPL
ImageInput::Impl and ImageInput::Impl classes, without changing any
ABI or making new headers.
This patch add a couple data members to the Impl data -- an
`IOProxy*`, (the current proxy, used for both passed-in and created
proxies) and also a `unique_ptr<IOProxy>` used when we have to create
our own and want to make sure it's destroyed with the ImageInput or
ImageOutput.
Then we add a number of protected member functions to ImageInput and
ImageOutput, not meant to be ever called by "users" of the OIIO APIs,
but used internally by II's and IO's to encapsulate several common
tasks related to managing IOProxy's. This coalesces redundant code
that would need to appear separately in most of the plugins that
support IOProxy, helps to reduce cut-and-paste errors from having to
make these changes separately in many places, and centralizes the
logic so that as our ideas of how to implement IOProxy evolve, we can
easily change it in just one place and have it apply to all plugins.
GIF revise with new IOProxy utility methods
JPEG simplify with new IOProxy utilities.
To try to combat long template instantiation times, did some refactoring
on the ImageBuf iterators:
* Pushed more functionality from the templated Iterator and
ConstIterator to the non-templated base class IteratorBase. This
should reduce the amount of code that is separately instantiated for
every data type, in every module that uses iterators.
* Pushed many implementations from inline defined in imagebuf.h to
separately compiled in imagebuf.cpp. The most important methods that
are called on every pixel are still inline, but we are switching to
separate compilation for all the methods that are only called upon
construction (once per image being iterated over) and most of the
code that is called only when we wrap to the next scanline.
* Some other miscellaneous refactor to remove redundant computation in
those classes.
Maybe somebody else is building this as a subproject or has a varible
called "CI"?
* For our CI runs, set an env variable "OpenImageIO_CI" (not just
generic "CI").
* Make sure STOP_ON_WARNING is true for our CI runs, even for release
branches -- but now check OpenImageIO_CI, not CI, in case somebody
else has set that.
This was deleted a few months back, but I think I did not realize
it still served a purpose. We don't use JasPer directly, but LibRaw
does, so if we are using static LibRaw, we still need it here.
Fixes#3209
When the make_texture option monochrome_detect is used to collapse 3
identical channels to 1, it was leaving the first channel name in
place (for example, "R"). That was confusing, because at this point,
we're not trying to represent a red image, we're trying to represent a
grayscale image. A strategic call to `default_channel_names()` clears
it up, imposing the standard "Y" we use for a single channel luminance
image.
Also amended the documentation of IBA::channels(). It turned out not
to be the way I called channels() that was the problem here, but by
the time I figured that out, I had already seen that its docs could be
improved.
Fixes#3204
* --onlyformat or env IMAGEINOUT_TEST_ONLYFORMAT to test only one format.
* --no-delete to preserve the temporary files for inspection
* other debugging checks
* Add a DDS test, apparently we never had one.
* Remove DDS setting of "texturetype", we don't ever use that, it's only
"textureformat" that we use. Not sure how that crept in there years ago.
The previously existing max and min take 2 images and compute the
pixel-by-pixel, channel-by-channel max or min of the two images.
The new maxchan/minchan take one multi-channel image and produce a
single-channel image where each pixel contains the max or min of the
*channels* of the corresponding pixel of the original image.
It is equivalent to this (for an RGB image):
oiiotool in.exr -ch R in.exr -ch G in.exr -ch B -max -max -o max.exr
Except, of course, that --maxchan is a single command, is much more
efficient, and doesn't require you to know the number or names of the
channels of the input.
This was a specific user request.
Note: in the course of making the tests, I noticed that we did not
have documentation or testsuite coverage of the python bindings for
IBA.min() an max(), although they were implemented. So I added that as
well.
* Make PROJECT_VERSION_RELEASE_TYPE be a cache string, so individual
build configs or sites can override it.
* Change OpenImageIO_IS_SUBPROJECT to PROJECT_IS_TOP_LEVEL (and with
the opposite sense), to match CMake 21+ behavior. And no need to set
it ourselves if using cmake 21+.
ImageBufAlgo has both premult and repremult (as well as unpremult).
But oiiotool only exposed unpremult and premult, but not repremult.
This little patch rectifies that oversight.
Just to remind everybody of the difference:
`premult` just multipies the color channels by alpha, like it says.
Note that for a pixel that has Alpha = 0 but nonzero color, this will
crush the color to black.
That's fine if you have an *original* image that's unpremultiplied and
you want to do a one-time conversion to premultiplied.
But there's another use case, which is when you start with a
premultiplied image, you need to temporarily unpremultiply it (like
for a color space conversion), and then re-premultiply it. In that
case, you may have those glowing pixels in the original image, the
unpremult will preserve them, and you DO NOT want to crush them to
black with the subsequent premult. So there is a "repremult"
operation that's like premult, but preserves the color values when
alpha == 0. This is the preferred operation when you are doing the
sequence of "unpremult, do something, re-premult" so that the glowing
pixels make it through the round trip.
When I implemented `oiiotool --invert`, I knew that the default should
be to only invert the first three channels (presumably R,G,B) and not
anything beyond, because we wouldn't want to invert the Alpha channel.
But I botched it, and instead what happened is that if given an RGBA
image, it would just leave the A black in the result image.
So this patch has two improvements:
1. If there are more channels in the source image than we are
inverting, make sure to copy the other channels unmodified, don't
just leave them blank.
2. Allow optional modifiers chbegin= and chend=, so users can change
the default if they want a different channel set to be inverted.
This new IBA/oiiotool command adjusts the saturation of a color image,
with a scale factor. Scale = 0 makes the image gray scale of
equivalent perceptual luminance, scale = 1 leaves color unchanged,
scale > 1 increases color saturation.
This adds a new cmake variable in our build system, TIME_COMMANDS,
that when set ON will adjust the compile and link rule launch to
include a time command. This is probably only useful when also setting
`CMAKE_BUILD_PARALLEL_LEVEL=1` so that build are happening serially.
The use case is to investigate how long each module takes to compile,
for optimizing the build.
* Add a test for MacOS 11
* Updated supported versions of dependencies
* Move linux clang test to clang13 on vfx2022 container
* Switch clang-format test to use clang12
In the early days of OIIO, it was 100MB. Ten years ago, we bumped it
to 256MB. I've heard a rumor that computer memories have grown in the
last 10 years, and so have the size of images some people deal
with. So it seems reasonable to raise it to 1GB.
`sscanf()` still has locale-dependent behavior that we wish to avoid
because it can lead to subtle bugs for people whose machines are
configured for certain locales.
This patch removes from the OIIO codebase almost every remaining
sscanf use (and definitely every one that could harbor locale bugs).
Instead, we use the rich set of parsing functions we've added to
Strutil over the years, which are locale-independent, and in most
cases are also faster than the standard C/C++ equivalent.
The official names we use are "box2" and "box3", but users of Imath
know these as box2f/box3f, so accept these as synonyms when
constructing a TypeDesc from a string.
* Typo in webp section -- said "gif:LoopCount" when we meant
"webp:LoopCount".
* In the code, did not actually set the "oiio:LoopCount" metadata
that the docs said would be set.
* Opportunistic fix: TIFF docs said we only suported IOProxy for input,
but we have supported it for output for a while now.
* `parse_values()` and `scan_values()` both can parse a series of
values in a string, with optional prefix, suffix, and separator. The
scan versions just figure out the values, the parse versions
optionally can update a string_view& like the other parsing
functions in Strutil.
* `scan_datetime` parses the int components of a string that looks like
"YYYY:MM:DD hh:mm:ss" (and a few minor variants).
* Fix stoui -- it was previously casting stoi results, which is not
correct.
* Add `from_string<>` specialties for `double`, `int64_t`, `uint64_t`.
When outputting to formats that don't support negative pixel origins
(such as JPEG), we crop away the negative areas. But we did not
consider the case where the entire data window was in the negative
region!
TIFF has an absolute limit of 4GB file size due to use of uint32
offsets within the file. If you try to write a big enough TIFF image,
it will fail. This is not a software problem, it's a limitation of the
original TIFF file specification itself.
The underlying libtiff supports a nonstandard "bigtiff" extension that
uses 64 bit offsets. This patch switches to this automatically for
images whose uncompressed size would get too close to (or exceed) the
4GB limit. Any such files can be read by OIIO-based software, as well
as anything else using a relatively modern version of libtiff, but may
not be readable by independent TIFF reading implementations.
This is a mixed bag, though. On one hand, this at least lets us write
files that would have been an error before. But note that some files
that would have worked as classic TIFF before -- because their
*compressed* size was still under the 4GB limit, even if their
uncompressed pixel size was not -- will now be written as bigtiff
because we have to commit at the time we open the file, before we've
seen the pixels and know the final compressed size. We just can't
know for sure, for an image with > 4GB of pixels, if the file will get
lucky and be below the limit by the time the scanlines or tiles are
compressed. So for those files, not all apps will be able to read
them (though anything using libtiff >= 4.0 or OIIO will be fine).
Fixes 3156
When ImageInput::create() or open() is called, it first uses the
extension of the filename to guess which reader plugin to try (for
example, "foo.jpg" will try the jpeg reader). If that fails, however,
the default behavior is to then try EVERY reader to see if any will
open the file, in case it guessed wrong, or the extension is used by
more than one format, or the file was simply given the wrong
name. Thus, ultimately, OIIO will never fail to open a valid image
file in a known format, even if the file is wildly misnamed. However,
it's a lot faster to open a correctly named file, because it will try
that one first, before trying a long succession of readers in the hope
that eventually one will work.
There are three cases where this is not the desired behavior:
1. You are confident that files are named with the correct extensions,
and you want a corrupt or invalid file to fail QUICKLY, without trying
every format reader.
2. If the file is corrupt, rather than see a generic "Could not find
any format reader that will open the file" because they all failed,
you want to see the more specific error message for why the (presumed)
correct reader could not succeed.
3. When doing regression tests to make sure that sample corrupted
files are correctly reporting the right errors, we really do want to
see error messages from that one reader we are testing.
This PR exposes a global OIIO attribute, "try_all_readers", which
defaults to 1, but when set to 0, changes the behavior so that ONLY
the one format implied by the file extension is tried, and OIIO does
not subsequently try every known reader.
This new oiiotool command lets you set global OIIO attributes, i.e.,
making calls to OIIO::attribute(). It allows oiiotool command lines to
set some additional OIIO-wide options for which no specific oiiotool
command flags could otherwise access.
When building against OpenEXR >= 3.1.3, exr output now allows zip/zips
compression to take a quality metric (such as compression
"zip:4"). The range is from 1-9, 4 is the default, and it's a control
over speed of compression vs compression ratio (higher number is more
expensive but may result in smaller files). This only affects
compressed output -- it's lossless and will be read to the same pixel
values upon input.
This is also incentive to upgrade to OpenEXR >= 3.1.3, because its new
default (and ours) is 4, versus the old (implied, not directly
controlled) 6. This change was made because compared to 6, it's
significantly faster to write exr files with zip compression 4 (tens
of % faster), while only increasing file size by 1-2%, so that seemed
like the right tradeoff. Lower numbers make much larger files for
little speed gain, and larger numbers than 6 make for much slower
writes of exr files for very tiny compression improvements. But YMMV
and you may want to experiment to find the right trade-off for your
use cases.
OIIO 2.3 marked it as deprecated. This new change is for master (2.4+)
only -- removing all support for Field3D.
F3d is effectively unmaintained, is known to be incompatible with
the latest versions of OpenEXR/Imath, and we haven't found anyone
prepared to speak up as a user of it via OpenImageIO support.
One more boost extraction: tokenizer.
The only place we used it is Filesystem::searchpath_split, which can
easily be rewritten in terms of the Strutil parsing functions.
While I'm at it, add a new version that returns the vector instead
of needing to pass in a ref.
Suggested by Luke Emrose, the idea here is that we introduce two new
MIP interpolation modes: StochasticTrilinear and StochasticAniso.
These work similarly to Trilinear and Aniso, respectively, except that
instead of sampling both of the bracketing MIP levels and blending
between them, in stochastic mode we use a random variate to select ONE
of the two MIP levels with probability proportional to the computed
blending weight.
If these mip modes are used, the caller needs to pass a
well-stratified value in the new "rnd" field of TextureOpt. Also note
that the method only really makes sense in the context of a renderer
that is using many samples per pixel. As a single texture lookup, it
looks horrible and noisy, as you can see by the testshade-based tests
we do of this, which are 1spp and looks pretty rough.
My benchmarks (in a hard stress test of the texture system using
`testtex --threadtimes 8`) show that this approach has speedups of
35-40% when using between 1-4 threads, dropping to more like 25-30%
speedup as threads are in the 12-32 range (that makes sense, it's
doing the same amount of total I/O).
In this first implementation, in order to preserve ABI compatibility
if we decide to backport this feature to the release branch,
TextureOpt::rnd is added as a union co-member with the unused "bias"
field. For TextureOptBatch, we had to add it outright, but it's
guarded by an `#if` and will not be backported to 2.3. (We may change
our minds and not backport it at all, I just wanted to leave the
option open with this initial patch. Also, there will probably be
additional further ABI-breaking changes coming later that will
definitely not be backported.)
In this first phase, we only use the stochastic component to turn the
2-MIP-level lookup into one level lookup. It has been further
suggested that we could apply stochastic techniques to anisotropic
filtering (instead of many samples spaced across the major axis of the
filter footprint, take one sample distributed across the axis), and
even for the ordinary bilinear or bicubic within each level (instead
of 4 or 16 weighted texels, just retrieve one texel chosen with the
weights as probabilities). But these deserve to be benchmarked and
have their quality evaluated separtely and deliberately, and so will
be relegated to a later investigation and implementation.
We will have a C API eventually, but it will be implemented
differently, and this cruft will probably get in the way. If the new
code needs any of this (particularly the test cases), it's easy enough
to revive it from the project history).
When requesting that raw image be returned as "ProPhoto" color space,
we were using the wrong gamma. Also, at user request, add
"ProPhoto-linear", which is the same but with linear response. (This
is analogous to the existing "sRGB" and "sRGB-linear".)
Fixes#3152
Thiago Ize points out that TextureSystemImpl::fill_gray_channels
assumes that the `float*` addresses it is passed are big enough for
a vfloat4, and also properly aligned as such. Not true.
This is an edge case anyway, only called in particular cases where
we are doing 3/4 channel lookups on 1/2 channel files with the option
turned on to expand single channel gray to RGB gray, and also doesn't
do enough SIMD to have a perf impact. So just recode as scalar and
have no more alignment mismatches to worry about.
Also make sure at least one of the Windows CI tests is explicitly
turning SSE on.
Fixes#3128
Field3D is essentially unmaintained, and is currently incompatible with
OpenEXR/Imath >= 3.
In this first stage (which will apply to OIIO 2.3):
* By default, Field3D support is disabled. To enable it, you must
invoke CMake with `-DENABLE_FIELD3D=1`. No warning or other messages
are printed by the build configure step when not enabled.
* If requested, but an incompatible OpenEXR is found, it will be disabled
with a clear error message.
* If requested and enabled, a warning will be printed indicating that
Field3D support will be removed from future versions (2.4 and beyond).
* Add a new test, texture-texture3d, that provides the same testing that
texture-field3d did, but with openvdb.
We have options that allow adding random dither to images before
output, which when converting float images to 8 bit or less, can
greatly reduce the appearance of banding artifacts. This PR makes a
number of improvements:
* When the option is used, it should apply dither any time you are
reducing an image with > 8 bits/channel to an output with <= 8
bits/channel. Not just from float/half, but also from uint16, for
example. (When outputting > 8 bits per channel, there are enough
gradations that you tend not to see banding artifacts, so dithering
is not necessary for 16 bit output, even if coming from a float data
source.)
* Change the actual dither we use to blue noise. The old dither just
used a hash to add uniformly distributed random numbers to each
pixel value before quantization, completely uncorrelated from pixel
to pixel. Blue noise is specially constructed to still have a
uniform distribution of the output range, but it ensures that
spatially, similar values are distributed as far away from each
other as possible, rather than having similar values sometimes
clumped together. The result is a MUCH NICER appearance.
* Of course, this means the we needed to store a blue noise table in
the library, and some access routines for it. We incorporate a
public domain blue noise texture as a table.
* ImageBufAlgo::noise() and `oiiotool --noise` now take "blue" as a
noise name to return blue noise. Also, make "white" the (preferred)
synonym for "uniform" (to disambiguate the fact that both white and
blue noise are uniformly distributed; the difference is that white
is independently random for each pixel, whereas blue shapes the
spectrum to favor high frequencies, making it better suited for
dithering and sampling purposes.
* ImageBufAlgo::bluenoise_image() returns a reference to a stored
periodic blue noise image.
* oiiotool -d would let you say "uint10" or "uint12" to indicate 10 or
12 bit output (for formats that support it), but had no way to ask
for less than 8 bit output (for example, TIFF supports that). So we
add "uint6", "uint4", "uint2", "uint1" as recognized names for the
`-d` option in those rare cases when you specifically want to
produce a low bit depth file. And furthermore, if you ask for
`--dither` when using lower bit depths, the dither amplitude is
adjusted to be right sized for the bit depth you are using.
* Many format output modules that support 8 bit output honor the
option to dither the output, but in the "built-in plugins" chapter
of the docs, most did not list this option in the section that
describes which special output options it knows. So now we've added
this to the docs for all the output plugins that did support it.
Turns out that heif files that contain alpha can have either
associated or unassociated alpha. We had been assuming
associated. With LibHeif 1.12 and beyond, you can ask, so now we
do. Thanks Frédéric Devernay for pointing this out.
Similar to other formats that optionally have unassociated alpha:
* The default behavior is to associate/premultiply automatically when
reading, but set metadata "heif:UnassociatedAlpha" to indicate that
it was unassociated in the file.
* Open configuration hint "oiio:UnassociatedAlpha", when nonzero,
means to please keep the data unassociated and not premultiply. In
that case, metadata "oiio:UnassociatedAlpha" will be set in the
ImageSpec to indicate that the returned image (not just the file)
consists of unassociated values.
I did a couple other minor docs touch-ups while I was at it.
Fixes#3129
New ":C=name" modifier for --dumpdata will cause the dumped image data
to be formatted with the syntax of a C array, that can be dropped right
into source code.
$ oiiotool --dumpdata:C=foo image.exr
// image.exr : 256 x 256, 4 channel, float openexr
float foo[256][256][4] =
{
{ /* (0, 0): */ { 0.517036676, 0.261921108, 0.017822538, 0.912108004 },
/* (1, 0): */ { 0.653315008, 0.527794302, 0.359594107, 0.277836263 },
...
};
Without this modifier, --dumpdata will keep the same formatting it
always had.
This works pretty well for ordinary boring flat, single subimage,
non-MIPmap images. The option should probably not be used (or at
least is very unpolished) for multi-subimage, MIP-mapped, deep, or
other oddball cases (but you probably don't want to shove those into
a single C array anyway).
We had long ago established that the one in fmath was a slightly
faster formulation, as well as having the property of squashing
NaNs. There were a few remaining places where we used a different
clamp so fix them.
The DSLR raw format reader has the ability to pass hints to the
underlying LibRaw requesting that the pixel data be returned to us in
particular color spaces.
* Add new supported color spaces "DCI-P3" and "Rec2020" when using a
LibRaw new enough to understand them -- which means the upcoming 0.21.
* A new name to request the data be returned using sRGB color
primaries but a linear response curve: "sRGB-linear".
* The old name we used for sRGB-linear, "linear", is now considered
deprecated since it's ambiguous -- since its first use, new color
spaces "ACES", "DCI-P3", and "Rec2020" have been added, all of which
are also linear response.
When reading tga files, make sure that every seek and every read
checks results and issues error messages. Also double check that
large allocations succeed.
Also improve error propagation in several spot of iinfo, oiiotool
Owing to our C++11 focus in previous years, we had a number of places
were constants were declared as `static constexpr`. When done in
headers, this potentially causes separate copies of those constants to
be instantiated in each module that includes the header. It's not
really harmful the way we're using it for small constand objects, but
it's wasteful to potentially have dozens of copies of each those
constants by having it repeated in each module.
C++17 has a new idiom, `inline constexpr` that is more like we want --
makes it a true compiler constant that doesn't get instantiated in the
memory of each module.
We still have to cater to C++14, and in platform.h we already had a
C++-standard-dependent definition of OIIO_INLINE_CONSTEXPR that was
almost correct. Fix it (it wasn't previously used, but I must have
seen this coming at some point) so that it expands to
`inline constexpr` for C++17 and beyond, but stays as
`static constexpr` for C++14, and then use it for these constants.
Fixes#3117
If think that our inventory of UDIM tile files won't find names with
wildly out-of-range tile indices, you haven't been working in a
production studio! When those indices imply tile numbers < 0, we were
running off the beginning of the vector when filling in the udim
table, to hilarious results.
Oiiotool::extract_options was kind of fishy and brittle before --
splitting the string at ':' characters, then split each piece (which
should form a `name=value` pattern) at the first '=' character. There
are probably many malformed option strings that would have had weird
results with the prior naive parsing method. And it has been noted
that it can't handle a value that is intended to be a Windows filename
(like "C:/foo"), nor a searchpath (like "/usr/bin:/usr/local/bin").
Rewrite to parse it more carefully as name=value pairs delimited by
':'. In the process, we now have the ability to have the value be
enclosed by a single or double quoted string, in which case the
contents of the quote may themselves contain a colon, for example, the
following command and modifier is now no longer a problem:
`--foo:filename='C:/blah':count=3:`
We were already passing this on to libtiff when it handles the
compression. But for the case where we were calling zlib ourselves
(to facilitate multi-thread compression, which libtiff doesn't do),
we were just using the default compression.
This is inspired by a great [blog post](https://aras-p.info/blog/2021/08/05/EXR-Zip-compression-levels/)
by Aras Pranckevičius in which he made a similar change to OpenEXR
and benchmarked the results. He reported approximately a slightly more
than 2x write speed increase, while getting files that were only around
2% larger.
I tried replicating his experiment for TIFF, and while I don't quite
get results as good as he did, it still is really interesting. I
tested with 100 images randomly selected from the [Animal Logic Lab
USD scene](https://animallogic.com/usd-alab/) (they are almost all
4k^2, a mix of 1- and 3-channel images), converted from half exr to
uint16 TIFF images. I ran the tests for both scanline and tiled
images.
Compared to the default zip level 6, using level 4 resulted in 3%
bigger files, and around 40% faster write times. That seems like it
is a potentially very helpful tradeoff.
I tested the same set of images also with zip levels 8 and 2. For
level 8, the files were only another 2% smaller than the default level
6, but the write speed was 45-50% slower, so there is little value to
asking it to work harder on compression -- it just burns up time. For
level 2, it went somewhat faster (maybe another 10% faster than level
4) but the files started getting bigger, level 2 was 12% bigger than
level 6, that's definitely getting into territory where people might
mind the larger files.
So I want to add the control now, though I'm deferring a decision on
changing the default value until I can do some more extensive
tests. In particular, I would like to assess how it behaves with 8 bit
files (I was testing uint16, specifically), how well it works with
rendered images and with natural/photographic images (I was testing a
set of textures, which may not have the same characteristics), and
perhaps also see if results are better or worse for lower resolution
images.
The old libOpenEXR/libIlmImf (default) implementation handled
multithreading on the exr side of things, inside the readPixels method.
The new (experimental, enabled only by option) OpenEXRCore
implementation leaves the threading to the caller, and we weren't
doing it, so the performance was unsurprisingly awful.
This patch turns the implementations of the exrcore-based
read_native_scanlines and read_native_tiles (and also the deep
versions thereof) to use parallel-for loops, restoring parallel
processing to the exr reader when using OpenEXRCore.
Reading of whole scanline and tiled images now is on par or slightly
faster (a few %) than when using the main OpenEXR library. Reading of whole
scanline and tiled deep images is substantially faster (2-3x!) than
before.
I did some extra refactoring while I was there, simplifying and
eliminating some redundant error handling and using a new
DecoderDestroyer utility class.
STILL TO DO:
* See the long comment at the end of read_native_scanlines -- the error
message strategy implemented here is temporary, I'd like to improve it,
will probably involve both some work here as well as on the openexr
side.
If set to ON, will include the testtex program in the install.
The default is OFF, which builds it for tests, but doesn't install it
with the real command line utilities.
In forward looking code (current master means OIIO 2.4, which won't
be released until mid-2022), get rid of some stray logic catering to
OpenEXR releases 2.2 and older.
Background: Old style CMake was to use FindFoo.cmake modules, which
set FOO_INCLUDES and FOO_LIBRARIES variables. Best new cmake practices
are for the upstream packages themselves to export a FooConfig.cmake
and friends, which set up cmake targets. Of course, many packages don't
provide exported configs, so we use the old Find modules for many of
our dependencies.
I've slowly been migrating our Find modules to set up targets in addition
to the variables, and favoring use of the targets over the variables.
But I was wrong about this. It's more than just a choice of style.
When targets are used, use of transitive linkage (especially for static
library versions of dependencies) make the dependency targets show up in
our own exported OpenImageIOConfig.cmake. But to a downstream package,
now our config talks about a target like Libsquish::Libsquish, but it
doesn't know what that is, and it doesn't have the FindLibsquish.cmake
module that we used (internally) when we found that dependency.
This was causing particular problems for Libsquish on Windows with
vcpkg, but I see now that this could be a problem elsewhere as well:
field3d, libheif, openvdb, and webp.
So this patch:
* Reverts from the ill-advised targets back to the old style
variables, for the dependencies where we use *custom* Find modules.
(We presume that a dependency's exported configs, as well as any
Find modules that are distributed as part of CMake itself and define
targets, are done correctly, and will be available to downstream
packages, and so we continue to use targets in those cases.)
* Beefs up our FindLibsquish.cmake module to correctly find debug
versions of the library, to use in debug builds.
* Alters build_webp.bash to allow WEBP_BUILD_TYPE to override the build
type, which had previously been hard-coded to Release.
* Removes search for HDF5, which as far as I can tell was unused
directly by OIIO.
Instead of a build-time option for enabling (and always using) the
experimental OpenEXR "core" library...
* Build support for it by default any time we're using a new enough
OpenEXR version (>= 3.1). You can still hard-disable it being compiled
in at all with `-DOIIO_USE_EXR_C_API=OFF`.
* Select whether it's actually used with the global OIIO attribute
"openexr:core" (default to 0, disabled, for now).
This easily lets an app select support at runtime, and also may be
selected by user with environment variable
`OPENIMAGEIO_OPTIONS="openexr:core=1".
This all is intended to make it easier to benchmark use of this new
feature, switching back and forth without a rebuild.
* Honor env $OPENIMAGEIO_FONTS as an additinal font searchpath.
* Global attribute "font_searchpath" as an additional font searchpath.
* Refactor and simplify the code that assembles the list of directories
to search.
When $OCIO was not set at all, the call to OCIO::GetCurrentConfig()
was checking it again and writing the info message that none was found
to stdout, which we really don't want. And as Patrick pointed out
recently, asking for the "current" config is suspect, since who knows
exatly what the app has done.
So we're making the following executive decision: At least from OIIO's
viewpoint, if you want an OCIO color config, you either have to name it
expliticly, or have it determined by the `$OCIO` env variable. That makes
cleaner logic, and also rids us of the info messages.
I also now have the exported cmake config set
`OpenImageIO_HAS_OpenColorIO`. This makes it easier for downstream
consumers of our exported OpenImageIOConfig.cmake to know if the OIIO
build they have found has build-in support of OpenColorIO.
When it comes time to read_image() into the ImageBuf's internal
memory, we should have said to convert to `m_spec.format`, which
describes the buffer we allocated. Instead, we asked for `convert`,
which is the user request, which at that time can still be UNKNOWN,
which indicates to preserve the file's data format, which in fact does
not necessarily match the internal buffer.
This becomes especially troublesome for images with mixed per-channel
data types, which will obviously never match the internal buffer since
ImageBuf only internally stores one data format for all channels.
This is a very low priority change, but every little bit helps when it
comes to limiting boost's footprint as thing we used to require it for
become available in the standard library.
Add tests to be sure we have reasonable error reporting for at least
two important cases: a specified OCIO config file is not found at all,
and an OCIO config is present but cannot be read (in particular, if
the config is newer than the library). Because the exact error spewage
is different for each OCIO version, and apparently also depending on
the exact version of the yaml library that OCIO used, we have a few
different possible output references.
* oiiotool --nostderr (new hidden command)
This hidden flag causes oiiotool errors to go to stdout instead of the
default stderr. It only does this for oiiotool's own error and warning
calls; if there are other places within library functions that
directly output to stderr, they will not be redirected by this
command.
The main purpose of this is for testsuite and debugging. I don't
expect ordinary users to have any use for this, and so it is currently
hidden from the oiiotool --help explanation. If it seems useful for
something other than testsuite, I'll expose it in oiiotool --help.
It seems that for the case of OCIOv1 reading a v2 config, OCIO itself spews a bunch of warnings to stderr. And apparently the order is nondeterministic, because I just could not get any set of reference error output to match stably from run to run in CI. But the problem is, if I don't capture stderr, I also don't get the oiiotool error messages, which are the very thing I'm trying to test!
So what I did is add a new flag to oiiotool, --nostderr, which causes oiiotool's error and warnings to get sent to stdout rather than stderr. This lets me capture the usual output and the error messages from oiiotool, for this test, while allowing OCIO's stray stderr output to go into the ether where we don't care about its order.
I'm not sure if there's any non-CI, non-debugging application for oiiotool --nostderr, but maybe it'll turn out to be generally useful.
When OIIO that was built against OCIO 1.x, but encounters an OCIO 2.x
config file, OCIO could throw an exception that was not caught by
OIIO. The problem was the call to OCIO::GetCurrentConfig(), it should
have been inside the try/catch block.
Calling close() in open will forget too much state. All we want to do
here is make sure the TIFF file itself is closed at that point (so it
can be re-opened in append mode to add a subsequent subimage).
* CI test for VP2022, fix revealed python binding bugs
* CI test against ASWF's latest VFXPlatform-2022 docker container,
including Python 3.9.
* Pesky crash resulted in testsuite/python-imagebufalgo, laboriously
traced to IBA_color_range_check releasing the GIL too liberally. The
scoped release cannot contain the tuple construction!
* Audited the rest of py_imagebufalgo.cpp and found the same mistake in
only one other place: IBA_histogram. Was asymptomatic so far, but fixed
now.
* Presumably we just got lucky and these bugs were never symptomatic
until we started building against Python 3.9.
* Noticed that the recent change in build tree layout where we now
copy the test images into the build/testsuite area, resulted in HUGE
GHA build artifacts for failed tests. Modify ci.yml use of
upload-artifact action to exclude those directories.
* more tweaks
oiiotool --runstats tries to report how much time was taken by each
operation. But it has always had a major flaw: Because we try to read
input files lazily and use ImageCache underneath, only in certain
circumstances do the true cost of reading the files show up in the
entry for "-i"; often the cost gets wrongly credited to the other
command (like --add or -o) whose operations triggered the I/O.
This patch addresses the problem, and I think correctly credits I/O
time where it belongs.
The workhorse here is a new helper class within oiiotool,
OTScopedTimer, which does some tricks to try to account for both
direct read time and ImageCache I/O time, subtracting those from the
op it was instructed to time and instead adding it to the counter for
"-i". Because the destructor apportions time to the right commands,
this eliminates a whole bunch of spots where we did that by hand.
Also fixed a few places where ops directly called ImageRec::read()
instead of Oiiotool::read(), which would also result in bypassing some
timers we relied on. I made the ImageRec one be private, so harder to
make that mistake in the future.
Use a cmake idiom so that you need an explicit `-DCMAKE_INSTALL_PREFIX=`
in order to write over /usr/local or any potentially sensitive area.
In the absence of this being set explicitly, it will configure the
cmake build so that an "install" ends up in ./dist (which is what the
Makefile wrapper did all along).
Before doing all the work of copying the image, first see if the
`--ch` is actually specifying any change versus the image as it
stands. If not, skip all the copying and just stick to the image we
have.
The main benefit here is if you have a --ch primarily for "conformance",
e.g. you want to reduce to a known subset of channels and remove any
extra that may be unanticipated in the input, but it's possible that
the input will come to you already with only the desired channels. As
an example,
oiiotool in.exr --ch R,G,B ...more ops... -o out.exr
Previously, if in.exr aleady had channels R,G,B (and no other), there
would be some unneceessary expense of creating (in memory) a new image
for the output of --ch copying data to fill it in. With this patch,
the --ch would very nearly turn into a no-op. In both cases, if
in.exr's channels were something other than R,G,B, channel copies
would need to be done identically in both old and new code.
I realized after this was merged that I often share and copy a lot of
these little dependency build scripts across multiple projects, so I
wanted to remove the OIIO-specific naming.
To help make testsuite pass even with nonstandard relative path to the
build area, slightly change how we reference the OCIO config that is
in the testsuite.
Fixes python error about OIIO_TESTSUITE_ROOT not defined in the case
where the testsuite dir wasn't found in the standard places. This can
happen if you put the build directory in an unusual location that's
not right in the main project area.
* Ensure that "CMAKE" built "OpenCV" libraries can be found.
* Fix incorrect url to "openexr-images" repository.
* Fix typo in "FindTBB.cmake" module.
Co-authored-by: Thomas Mansencal <tmansencal@wetafx.co.nz>
* New SETVARIABLES option sets named cmake variabes to true if the
package is found.
* DEBUG should print variables even if not found. Duh, failing to find
the package is the most important debugging situation.
For historical reasons, our wrapper makefiles assume that we build in
`build/ARCH[.variant]`, so maybe build/linux64, build/linux64.debug, etc.
I think this is a throwback to literally decades ago when I would
build multiple architectures from different machines that viewed a
common shared directory of code, this allowing them to build
simultaneously and/or erase one without needing to rebuild the others,
etc.
But now I find that:
* It's contrary to modern cmake practice, and there are many benefits
to having our build directory structure identical to the vast majority
of other open source projects.
* I never want or need those simultaneous builds in practice, haven't
for a long, long time.
* ccache means that there is no longer an onerous cost to, for example,
having to blow away a release build and do a debug build and then
blow that away and do release again. It's just a few seconds to do a
ccache-based full rebuild of a recently-built tree.
So this patch switches all our assumptions to having a single level
hierarchy to the build and dist areas.
Also, I'm changing the assumption that the oiio-images (and other data
sets used by the testsuite) live "up a level", but instead added code
that either copies it from there (or another area settable with the
cmake variable OIIO_LOCAL_TESTDATA_ROOT) or downloads it, into the
build/testsuite directory itself. This simplifies a great number of things
in the test scripts as well as the test reference output (a lot of which
changes slightly to accommodate the new paths to test files).
* raw: Fixed output when raw:Demosaic == "none"
* raw: Include metadata regarding filter pattern
* raw: Added missing call to set unpacked status
* raw: Implement "balance_clamped" mode to help HDRI processing
* raw: Implement raw:apply_scene_linear_scale feature
* Fix up formatting with clang-format
* Update testsuite reference output
* raw: Fixed error message format handling
* raw: Updated documentation to include new raw features
* Ensure that all the `build_*.bash` scripts don't download anything if the packages appear to already be there.
* Env variable `OIIO_DEP_DOWNLOAD_ONLY` can be used to do the download only (if not present) and skip the actual build.
This combination allow the scripts to be run twice: once to do all the network I/O to download the packages we'll need, and a second time to trigger the builds.
The existing --info and --stats seem a little confusing and
inflexible. Unlike the majority of oiiotool arguments which are
evaluated in order, these merely set flags which apply to the entire
oiiotool command (specifically, they cause all image inputs to print
the metadata or image statistics at the time that the input are read
from disk). There isn't a way to make it apply to a single image, nor
to print the metadata or stats of any *computed* image, leading some
people to awkwardly save a computed image and re-read it merely to
print information about it.
So in this patch, we add new commands
* `--printinfo` which is like `-info -v` but is done immediately for
the "current" image on the top of the stack (even if it's a computed
image and does not correspond to any disk file).
* `--printstats` which is like `--stats` but is done immediately for
the current image on top of the stack.
Furthermore, `--printstats` takes an optional modifier `:window=`
which can be given a rectangular region in either of the notations
we use ("WxH+X+Y" or "xmin,xmax,ymin,ymax"), for example:
oiiotool blah.exr --printstats:window=30x30+1000+500
prints the statistics of the 30x30 region whose upper left corner
is pixel (1000,500).
The code to implement this is actually pretty short and straightforward,
but this patch turns out to touch a lot more. I needed to plumb
some of the info-printing internals for this support and did some
refactoring along the way.
Also, upon adding these to the docs, I realized that it would be
better to reorganize a bit to separate descriptions of the "flags"
from the "actions", and so many of the oiiotool arguments got shuffled
between different subsections, and in a few cases the descriptions
themselves got rewritten for clarity.
To help applications that need to deal with UDIM textures, the
TextureSystem now exposes the following public API calls:
* is_udim() : is a filename (or a `TextureHandle*`) referring to a
UDIM pattern / virtual file?
* resolve_udim() : given a filename (or handle) and s,t coordinates,
return the handle for the concrete file corresponding to those
coordinates.
* inventory_udim(): retrieve the full list of concrete filenames, as a
vector indexed by `(utile + vtile * nvtiles)`, and the nutiles,
nvtiles counts. Empty filenames in the vector indicate UDIM tiles
that are not populated.
Some extra functionality falls out naturally from these and existing
ingredients:
* inventory_udim + get_texture_handle together can get you the list of
handles corresponding to each found udim tile.
* resolve_udim + get_texture_info together can retrieve the attributes
of the udim tile corresponding to a particular texture coordinate.
Additionally, I have done some refactoring and performance
improvements of the internals:
* The internal UdimInfo that's stored inside the ImageCacheFile record
is now a vector, not a flat_map, so conversion of texture
coordinates to a handle is now O(1) rather than O(log n).
* Per Luke Emrose's suggestion, use `rcontains()` when searching for
UDIM pattern markers, since they will almost always occur close to the
end of the full pathname.
* Added TextureSystem and ImageCache methods that lets you find out
the filename that corresponds to a handle. (That's bound to be handy
for somebody.)
* Beefed up the udim tests to be sure to test every udim notation that
we support.
* At the suggestion of Luke Emrose, add rcontains() and ircontains(),
which are like contains/icontains, but start the search at the end
of the string rather than the beginning. This is measurably faster
than contains() for the case when you are reasonably certain that if
the search pattern is found at all, it is highly likely to be close
to the end of the string. (They're approximately the same speed when
the pattern is not found.)
* To verify this, I added a bunch of benchmarking to strutil_test.
* The benchmarks also revealed that contains was about 2x slower than
calling find! It should be the same. I don't know what the underlying
boost::algorithm::contains is doing differently than find, but whatever.
Changing our implementation of Strutil::contains to simply call find
doubles its speed.
* The new implementation of contains has an extremely minor behavior
change for the degenerate case of `contains("", "")`, which used to
return true and now returns false. Since the right behavior here is
debatable at best (is it false because the first empty string
contains nothing? or is it true because containts(a,b) should return
true if a==b?), I'm going to accept this behavior change as well
worth the 2x speed improvement (besides, "contains(a,b) should have
the same result as find(a,b)!=npos" is also a reasonable
expectation).
* The benchmarks also reveal that the case-insensitive functions (like
ifind) are not just a little bit slower than their case-sensitive
siblings (like find), they are 20x or more slower! I don't think
there is anything to do about that, but I did note it in the
documentation of those functions so that we are disuaded from using
them except when a case-insensitive test is really necessary.
The primary issue being addressed here was that get_texture_info() on a
udim pattern filename (like "foo_<UDIM>.exr") was unable to retrieve
info, except for an extremely narrow range of queries, because of course,
different UDIM "tiles" might have different attributes or be missing.
So in consultation with Lee Kerley, we decided that get_texture_info should
work as follows:
* If the attribute exists and has the same value for all of the
matching UDIM file files, then succeed and return the value.
* If any of the tile files does not contain the attribute, or if the
attribute doesn't have the same value in all found files, then it's
a failure and the value cannot be retrieved.
This necessitated a more rebust accounting of all the tile files
comprising a UDIM pattern, by doing a full inventory the first time
the UDIM pattern is encountered, rather than (ick) iterating through
all possible tile files every time we retrieve an attribute, as we
were apparently doing before. And so the following improvements were
also made:
* get_texture_info now works for more queries -- in fact, any query
that would be possible to ask about on an individual file, without
restriction. (Though, as I said, the query will fail if the value
isn't identical across all the found tile files.)
* The info queries (even the few that were previously supported) will
be MUCH FASTER than before, because we are doing the tile file
inventory once per udim pattern, rather than on every query.
* That inventory will also pre-build the association between u,v tiles
and the concrete filename (and texture handle!), so that each
individual texture call does a simple map lookup to get all the way
to the texture handle, without any string manipulation to first
construct a concrete filename, then look that new filename up in the
cache to get a handle, etc. I'm expecting this to reduce some of the
overhead of using UDIM files with the TextureSystem.
Note: I added an ImageCacheImpl::find_file_no_add at some point in
this work, eventually not needing it by the time I got to the final
version of my implementation of this PR, but I'm going to keep it
because I can forsee its usefulness and I don't want to delete it only
to write it again some day.
We now think that this C API should be removed, and replaced by a
different one coming soon.
Merely commented out the option and other lines in the top-level
CMakeLists.txt, since almost the exact thing will be needed eventually.
Also kept libOpenImageIO-C/CMakeLists.txt since we'll eventually need
one much like it. But deleted the .cpp and .h files, since those will
be totally replaced.
For the upcoming OCIO 2.1, the library names will incorporate the
major/minor version numbers, so our Find module needs to take that into
consideration.
Work towards eventually using OCIO exported configs (though still not
100% working now, may need fixes on their end).
Change CI to use the OCIO's that come in the aswf containers instead
of building from scratch in many cases.
Sometimes we use a filename, or pieces of it, as a regex.
But filenames can have characters that are interpreted differently when
part of a regex pattern (like `.`, which in a filename is literal, but
in a regex means to match any character!).
So this handy utility inserts the right backslashes so the filename will
end up as a literal match when part of a regex.
Furthermore, there's an optional parameter `simple_glob`, which when
true, will also substitute `?` with `.?` and `*` with `.*`.
We then use this to fix a latent bug in scan_for_matching_filenames():
it could have incorrectly matched files during the search.
I noticed when working on an ImageCache thing that even the smallest
change to imagecache.h was causing a substantial amount of the
codebase to needlessly recompile even though very few places really
need to use the IC APIs directly.
It took a little bit of juggling of some declarations, but this patch
allows imagebuf.h to be able to treat ImageCache and ImageCacheTile as
opaque, forward-declared types, without the need for every file that
needs imagebuf.h to also have to parse imagecache.h and everything it
might pull in.
This speeds up my development cycle substantially in many cases, and
it also speeds up compilation of user code (by a small amount) by
avoiding parsing of imagecache.h in circumstances where the user code
only needed to know about ImageBuf but not ImageCache.
There's a downside to this -- it's possible that user code that truly
needs both imagebuf.h and imagecache.h only includes the former
(inadvertently relying on the fact that the latter was transitively
included). These apps may need `#include <OpenImageIO/imagecache.h>`
to be added after upgrading to OIIO 2.3. Is that really really bad?
Added this handy function that figures out if the first string contains
any character present in the second string.
Also rewrite the comments describing parse_until and parse_while for
better clarity and correctness.
Some of the OpenEXR sample files had an attribute "version".
All along, I assumed this was telling me the version of the OpenEXR file
itself, so I was passing it along and also renaming "openexr:version".
Turns out I was wrong! This is just arbitrary metadata. Who knows why
it's in some of these files? Maybe nobody. But it's not meant to indicate
the file format version, and indeed it does not correctly reflect it.
In fact, there is no simple API call in the OpenEXR API that tells you
the file format version, you have to infer it from the features
present (multi-part or deep or long attribute name indicate changes
that came along with OpenEXR 2.x, but there is no version number in
the file per se).
So I'm just going to stop doing this rename, let "version" be "version",
it's just some user's metadata, I guess.
A few image formats (but not most) support "thumbnails" -- low-res preview
images.
Traditionally, we have shoved them in the ImageSpec metadata as a byte
array (and ints for the width, height, nchannels). This is less than
ideal: It makes the metadata very bloated, taking more memory and
being more expensive to copy metadata, and potentially wastes time
doing I/O on this data that most applications don't want or
need. (Remember that any alterations to the image will make the
thumbnail "wrong", also. So it's rarely useful except for applications
that specifically want a preview image.)
Discussion in Issue 2902 led us to think that thumbnails should be
treated explicitly by the API, rather than shoved into the metadata in
this way. This patch is the attempt to overhaul thumbnail handling:
* ImageInput has a new `get_thumbnail()` method that can retrieve the
thumbnail (if it exists) into an ImageBuf. The base class / default
implementation does nothing; the image formats that support
thumbnails should have their readers override this method. They are
encouraged to read the thumbnails "lazily" if possible -- only
if/when this method is called, and avoid the cost, memory, and I/O
if not requested.
* ImageOutput likewise has a new `set_thumbnail` method.
* These are implemented for Targa and PSD, the only two formats for
which we previously had implemented thumbnail functionality. (There
are probably other formats that support thumbnails, but for which we
never added OIIO support. Leave that for another day.)
* ImageBuf has new `has_thumbnail`, `get_thumbnail`, `set_thumbnail`,
and `clear_thumbnail` methods.
* ImageCache has a new `get_thumbnail` method.
* oiiotool and iconvert attempt to preserve a thumbnail from input to
output (but of course, only if both file types support it).
* ImageBufAlgo::IBAPrep clears the thumbnail in the destination image,
on the theory that pretty much everything IBA would ever do will
likely invalidate any preview image that it started with. This gets
us to the point with oiiotool where a straight copy (like a format
conversion) `oiiotool in.tga -o out.tga` will preserve the thumbnail,
but just about anything else `oiiotool in.tga --stuff -o out.tga`
will not transmit the original thumbnail to the output (which would
almost certainly be wrong).
There is more work to do in the future. Some file formats that support
thumbnails don't yet have OIIO support for it. I'm not sure we ever
really want (except in the case of a truly straight copy) for oiiotool
to keep any input thumbnails; maybe if a thumbnail is wanted for
output, it should always be created anew (by downscaling) so that it
definitely matches the main image? Maybe also oiiotool needs a command
to explicitly extract and save the thumbnail (is that needed for any
legit use? or only as a debugging tool?).
Imagecache repeatedly has to determine whether a filename is a type
of file that's "procedural" and does it expensively -- every time, it
creates the type of ImageInput in question just to ask it if it
supports("procedural").
We can ask this of each plugin type once, when we first register it.
Then just look it up when we need it.
This is strictly internal, has nothing to do with public APIs.
OpenCue-launched apps are encouraged to honor the setting of
CUE_THREADS to limit the number of threads they use. This change makes
oiiotool (and other OIIO-based tools and scripts) better citizens when run
as part of an OpenCue job.
CUE_THREADS has lower precedence than the more OIIO-specific
`OPENIMAGEIO_THREADS`.
For ordinary command line usage, this should not change behavior, since
you wouldn't expect CUE_THREADS to be set.
(Also fix docs which incorrectly named `OPENIMAGEIO_TEXTURE_OPTIONS`.)
And make sure to switch all of our internal uses of those to the
preferred new equivalents!
Also, hide some deprecated things from the documentation system so that
new people won't be tempted to use them.
These are all things that have been deprecated starting with OIIO 2.0
(or in some cases, even earlier). Multiple releases (2.0, 2.1, 2.2)
have gone by with them commented or documented as deprecated, but
without our adding the warnings. It's about time. We're still not
taking them away entirely, I usually don't do that until we've had at
least one release where it's been marked as OIIO_DEPRECATED, and also
only do true removals of API at major releases. But we're laying the
groundwork for being able to totally remove these when we eventually
have a 3.0 release.
This patch modifies the build system to make it possible to control
whether OIIO should link Boost in the form of static or dynamic
libraries, independently from the LINKSTATIC setting, without
having to use the fully custom route (BOOST_CUSTOM).
This is done by preserving the value of the Boost_USE_STATIC_LIBS
variable, if it has been defined by the user. If the value
of Boost_USE_STATIC_LIBS has NOT been explicitly defined, it is
set to match the value of the LINKSTATIC variable, which should
result in Boost being linked in the same way as other dependencies.
Slightly amends PR 3012
* Add docs for this recently-added feature.
* Change the make_texture configuration hint to "maketx:uvslopes_scale"
for symmetry with all the other texture conversion configuration names.
Everything works as before, including setting "uvslopes_scale"
metadata in the file. (But still accept the name without the
prefix as a synonym.)
* Change it to a float for a bit more flexibility.
Ordinarily, --mosaic takes the N images you have, arranges them into a
rectangular array of "cells" where the cell size is simply the biggest
image of the collection.
This adds the optional modifier `:fit=WxH` that lets you specify a
cell size expliticly. Images will be sized to fit the dimensions in
the same way as the `--fit` works (maximally filling the dimensions
while preserving aspect ratio of the original image).
This is particularly helpful for creating a contact sheet from full-size
frames, without having to do any resizing as a separate step.
The existing semantics of `oiiotool -info`, `-info -v` and `-stats`
are that these options set flags so that each time a file is opened
for the first time, it will print the metadata and/or stats. But these
do not allow you to create new images and print their metadata or stats
(at least, not without writing to disk and reading those files).
This patch extends the expression substition syntax to recognize the
following new metadata notations:
* `METABRIEF` : a string containing the brief one-line description that
would be printed with `oiiotool -info`.
* `META` : a multi-line string containing the full metadata that would
be printed with `oiiotool -info -v`.
* `STATS` : a multi-line string containing the image statistics that would
be printed with `oiiotool -stats`.
Some interesting use cases are illustrated with these examples:
# Roughly the same behavior as `oiiotool -info img.exr`:
oiiotool img.exr -echo "{TOP.filename} : {TOP.METABRIEF}"
# Perform computations on an image and print statistics about
# the computed image (not the original):
oiiotool img.exr -blur 3x3 -echo "blurred stats: {TOP.STATS}"
# Print statistics on intermediate results... and keep going with
# more operations:
oiiotool fg.exr bg.exr -over -echo "{TOP.STATS}" -o comp.exr
# Write metdata as "burn-in" on an image:
oiiotool in.exr -text:x=10:y=20 "{TOP.META}" -o burned.exr
The implementation in the business logic of oiiotool.cpp was very
simple. The more extensive change behind the scenes to make this
possible was a massive refactoring of printinfo.cpp, which previously
just wrote everything to stdout in a totally undisciplined fashion,
and transforming it to very carefully write to output streams instead,
which could be passed a stringstream to render the metadata listing or
stats output as a string. Also needed some refactoring to allow it to
take either the name of a file to read anew (as it did before), or an
existing ImageBuf/ImageRec (to support the new feature).
Internally, ImageBuf has for some time now supported strided or
non-contiguous buffers (for IBs that reside in memory, rather than in
an ImageCache). It was originally anticipated to accommodate cases
where we would want pixels or scanlines to have particular alignment
or spacing, such as to be even multiples of the SIMD width.
But we never allowed the kind of IB that "wraps" an application's
memory buffer to specify the strides, in case the app buffer was in
some layout that was not contiguous. This patch adds that ability, by
amending that constructor with stride values (which are optional and
default to AutoStride, which indicates that contiguity should be
assumed). This is 100% back compatible for source code.
This lets you do some cool things with in-memory buffers:
* More flexible about the kinds and layouts of buffers that you can wrap
to look like an IB.
* Can easily make an IB that wraps an interior rectangular subset of
another buffer (including another ImageBuf).
* To a limited extent, makes it possible to have an IB that is a "view"
of another IB (if the second IB is in memory), including strided pixels
or scanlines -- and even if the strides are negative (this lets you make
an IB view of a buffer where copying the buffer can reverse left/right
or top/bottom, however you can't use it to change the channel order or
layout within each pixel).
* I think this also lets you make a 2D IB that wraps a 2D slice along any
axis of a 3D volumetric IB.
Some other changes that came along for the ride:
* Added `ImageBuf::contiguous()`, which tells you if the memory layout is
contiguous (will return false if it's not an in-memory buffer of some
kind, so ImageCache-backed IBs are NOT considered contiguous).
* Renamed some internal types and fields, in particular, changes like
`size_t ImageBufImpl::m_pixel_bytes` to `stride_t m_xstride` to make it
clear that the values are strides and can be negative.
There are several alpha choices. We were assuming that color values
were unassociated (need to be premultiplied) unless it was marked as
premultiplied alpha. But it seems that, actually, we should only
premultiply if it's "useful alpha". [Indeed, their terminology is
the worst.]
Rename the m_alpha member to m_alpha_type to clarifywhat it holds.
Also add the alpha type to the metadata as "tga:alpha_type" to make it
easier to inspect the value in tga files.
I don't think issues have been reported, so maybe this wasn't a
problem in practice for reasons I've long forgotten about, but doing
some other work in ImageBuf, I noticed what I think was a thread
safety problem.
ImageBuf reads are done lazily, a lot of operations call
validate_spec() or validate_pixels() which can trigger a call to
init_spec() or read() if they have not yet been done, and those had a
spin mutex protecting them from being called at the same time. But the
underlying read() and init_spec() themselves did not have any lock,
which made me worried about what would happen if one user thread was
explicitly calling read() while another was calling it implicitly
through validate.
Also, spin_lock is probably inappropriate for something that could
trigger an entire image read from disk -- it's meant specifically for
situtions where the lock only will be held for a few instructions.
So, this patch:
* Renames ImageBufImpl::m_valid_mutex to m_mutex, reflecting that it's
not merely guarding access to `m_spec_valid` and `m_pixels_valid`,
but to potentially anything internal to the ImageBufImpl that needs
thread safety.
* Changes that mutex from a spin_mutex to a std::recursive_mutex, to
reflect that it *may* block longer than a few cycles, so blocked
threads should yield rather than spin. And recursive to simplify the
hand-off logic between validate_spec, validate_pixels, init_spec,
and read, the former two may call the latter two.
* Add locks to read() and init_spec(), which previously lacked them (I
think in error), in addition to the existing ones in the validate
methods.
* For some locking ImageBufImpl methods, add a param that lets the
caller (which will always be internal code) specify that the lock is
already held.
* Add a timing test of recursive_mutex to spinlock_test, just to verify
that I'm not doing anything to totally botch performance. On my Intel
Mac laptop, an uncontested lock acquisition/release is 6.4ns for a
spin_mutex, 16ns for a std::mutex, and 27ns for std::recursive_mutex.
So this is definitely a more expensive lock, but 27ns is still very
reasonable, and *most* operations don't need to acquire the lock at
all (first it checks the m_spec_valid or m_pixels_valid, once they are
set there is no more locking).
* Improve ImageBuf docs, adding more explanation including some guideines
about which methods are considered thread-safe.
* strutil.h already had a symbol called OIIO_FORMAT_IS_FMT defined to
0 or 1 that reveals (to clients of OIIO) whether Strutil::format()
behaves like printf (the old and current way) or like std::format
(the future way). Now we adjust so that the caller may pre-set the
value to force one implementation or the other, at least for the
"inline" functions of the headers.
* As an aid to downstream packages trying to ensure that they aren't
using the function slated for eventual deprecation or behavior
change, you can now define OIIO_HIDE_FORMAT to cause a deprecation
warning now on those functions that will eventually change or be
deprecated. This can help to identify any places where you call the
future-deprecated functions and change them to a "safe" variety.
ImageInput and ImageOutput have ways for the caller to say that there
should be no "fan-out" in the internals of what the reader or writer
needs to do. But TIFF wasn't honoring this.
This patch provides an additional but optional parameter for --bumpslopes option to mormalize the derivatives in UV space and scale them by a user defined constant. The scale factor is used to prevent half floating point overflow at time of the UV normalization. A value of 256 is recommended to prevent floating point overflow with texture resolutions up to 32k. The constant value is stored as an extra attribute in the output file for reverting the scaling when using the texture. Note that this options is only meaningful with height maps.
Co-authored-by: Pascal Lecocq <plecocq@imageworks.com>
* Add TypeDesc support for metadata that is a 2D or 3D bounding box,
represented as an array of 2 VEC2 aggregates (for 2d) or VEC3
aggregates (for 3d), having the BOX semantic. The basetype
determines whether it's an integer-based (generally for pixel
ranges) or float-based (generally for 3D spatial) bounds. The new
text representations are "box2" and "box3" (for float-based), and
"box2i" and "box3i" (for int-based).
* Somehow we never had a C++ test thoroughly exercising all the corners
of TypeDesc. Now we do: typedesc_test.cpp
* The test revealed a number of edge cases we don't handle well, now
shored up.
* New fmt 8.0.0 is out -- make sure we test against both that and their
current master.
* Starting with fmt 8.0, and in accordance with C++20 std::format's
final incarnation, the format string needs to be constant, for the
sake of compile time checks. To use a nonconstant format string, as
we do for these wrapper functions, we need to lean on vformat.
* For Strutil::sprintf, just `using ::fmt::sprintf` and eliminate the
wrapper entirely. (Though we express the wrapper inside
`#ifdef OIIO_OXYGEN_ONLY` so that it fully appears in any docs we
generate.)
Add custom UDIM support and Houdini UDIM convention support
For our (Animal Logic) in-house renderer, we need support for an additional UDIM format of `/path/to/file/textureName_u##v##.ext`
Added second line to ImageCacheImpl::resolve_udim to make this work.
Additionally add support for Houdini UDIM specification
* New Ptex (since 2.3.1) has exported cmake configs. Use those if
available and seem to work.
* Rename FindPTex.cmake -> FindPtex.cmake to match the capitalization
in the Ptex project itself, thus avoiding warnings from CMake.
* Fix our FindPtex.cmake to set up Ptex::Ptex_dynamic target. Use
that instead of the old style variables. That makes a seamless future
transition between the old Find module and future exported configs.
* Our FindPtex.cmake got the version number all wrong. Fix.
* gh-installdeps.bash tried to install ptex-base. That's not Ptex! So
stop doing that.
* New build_Ptex.bash does a simple clone and build of Ptex. Do so for
a few of the CI test cases so that we are more thoroughly testing Ptex
in CI.
LibRaw 0.19 and earlier use auto_ptr in public headers, which was
deprecated in C++11 and removed in C++17.
When this combo is detected, issue a warning message from CMake.
When using filesystem::directory_iterator, I was using operator++,
which can throw exceptions for OS errors. I should have used
increment(ec) instead. Throw in a surrounding try/catch just in case,
since even increment is not marked as noexcept.
* Use g++11 on the bleeding edge CI test
* Remove OIIO_CONSTEXPR20 decls from string_view -- gcc 11, even in
C++20 mode, doesn't have std::string fully constexpr plumbed, it's
not ready for this.
* Suppress gcc11 warnings about arithmetic with enums that occur in the
OpenCV headers.
* Suppress gcc11 warning (correct) in the spot where we were trying to
force a crash for testing purposes.
We were double-printing error messages. If maketx/oiiotool is itself
printing errors returned by IBA::make_texture, it is not necessary to
pass std::cout to make_texture so that it can echo the message to the
console itself.
It's meaningless to try to use make_texture on input images that are
"deep". We never even thought to check for this, which led to a
cascade of problems resulting in a segfault.
There's something wrong with the behind the scenes tricks we're using
to doctor the internal string representation. Is it because of big
endian? The "alternate" string rep? Something else different on
Apple's libc++ on M1?
Fix for now by just disabling that clause. The only harm is that
ustring internals will take more memory on that platform. That's
better than crashing!
When I have more hands-on access to an M1 (or when GitHub Actions
supports running CI tests on M1), I'll return to this for a proper
fix.
A few varieties of encoded videos with bit depth 12 and having alpha
were not marked as such, and falling to the default case that reported
them as 8 bit RGB.
The docs were ambiguous about what would happen for many-channel
image, and the behavor in fact was to copy and color transform only up
to the first 4 channels, leaving additional channels in the
destination undefined (though in practice they turned out black).
Clarify now that for colorconvert, colormatrixtransform, ociolook,
ociodisplay, and ociofiletransform, the first 4 channels are assumed
to be RGBA to undergo the transformation, and any additional channels
will be copied unaltered from source to destination (not set to black).
In ImageInput::read_scanlines and read_image, if somehow the image has
a spec with 0 size, it will lead to a divide by zero error. Check this
and fail safely.
For BMP in particular, do some sanity checks to catch corrupted headers.
(This is in response to Issue 2982)
For TBB 2021, headers moved around, and our FindTBB.cmake was failing.
But actually, starting with TBB 2021, TBB exports a proper cmake config!
So take advantage of that by preferring to load TBB from a config file.
Fixes 2963
Both global and per-package controls have been added to our
checked_find_package cmake function, to cause it to first check for
an exported config for the dependency, and only if not found, revert
to potentially using a FindPackage.cmake module.
This reflects our general leaning toward assuming that a package that
exports a cmake config probably does a better job of it than we would
do writing a Find module for it.
Rename FindOpenJpeg -> FindOpenJPEG to match the conventions used by
that project, in particular for its exported CMake config files.
(We don't yet rely on their exported configs, I need to do some
homework to see if they exist and are reliable as far back as the
oldest OpenJPEG release that we support.)
Add helper script to build OpenJPEG
We still had a few stray localtime() calls (which is not reentrant,
because it depends on a global buffer), even though all along we had a
"safe" wrapper in Sysutil::get_local_time, which internally calls safe
versions localtime_r or localtime_s (depending on the OS).
* Writing: Write single channel BMP as 8 bit palette images (not 24/32
bit images). This fixes a problem I discovered related to the recent
change that reads 8 bit palette images as if they were single channel.
The lack of symmetry there made problems for round-tripping an image
and having it end with the same apparent number of channels as it
started with.
* Reading: respect a configuration hint "bmp:monochrome_detect" that if
set to 0, will disable the detection of monochrome palettes and spoofing
to look like a single-channel image. The default is 1.
* Read RLE-compressed images correcty now! Both 4 and 8 bpp varieties.
* Beef up the testsuite/bmp tests, now test against a wider set of sample
BMP images from bmpsuite (including rle compressed images).
OpenImageIO exported config has a dependency on Imath/OpenEXR. For
Imath 3, I accidentally used OpenEXR_VERSION when I should have said
Imath_VERSION. Makes no differece if they're in sync. But should
somebody downstream have a separately compiled Imath and OpenEXR that
just might be different patch versions, it could break builds by
rejecting if the Imath version appears to be behind the OpenEXR
version. There's no guarantee that Imath and OpenEXR will always be
released exactly in sync.
Found some files that encountered the problem described here:
https://prepression.blogspot.com/2017/06/metadata-bloat-photoshopdocumentancestors.htmlhttps://feedback.photoshop.com/conversations/photoshop/photoshop-corrupt-ancestors-tag-in-xmp-causing-giant-file-sizes/5f5f45f74b561a3d426ba97f
The pathologically long lists of "photoshop:DocumentAncestors" tags (a
4MB PNG of which 3MB consisting of XMP) was tripping us up with an
O(n^2) handling of appending to lists for those attributes that could take
minutes per file.
It seems like this was a widely acknowledged problem that has been fixed
in Photoshop 20.0.2 (January 2019) and later.
Solution:
* Just skip the problem tag; I can discern no real use for preserving
it in any OIIO context.
* More generally, be on the lookout for pathological or corrupt XMP by
keeping track of the length of lists and just cutting them off from
further appending when they seem senselessly long. I can imagine XMP
with lots of attributes, but it's hard to imagine that any single
attribute having thousands of list entries is intentional and useful.
* Fix some places where we were needlessly generating and then tearing
down std::string's.
This makes several minor formatting changes, mostly related to how
clang-format 11 has improved ability to track the alignment of = in
successive statements, even when one of those statements is split
across two lines.
This started out as just trying to fix our CI on the "bleeding edge"
building against OCIO master and complaining about a recently committed
OCIO deprecation of Config::parseColorSpaceFromString.
But it turned out that the new OCIO v2 method, getColorSpaceFromFilepath,
didn't work quite the same way and broke our tests. So I ended up having
to implement a bit more of a complicated set of functionality:
* Config::parseColorSpaceFromString now wraps the OCIO version if it's
available, but implements the equivalent logic if OCIO is new enough
that the function is deprecated.
* As a bonus, our pCSFS now works (didn't before) in cases where lack of
OCIO support or missing config meant the OCIO functionality wasn't
available. That is, it works on our meager set of default color spaces
that are present even without OCIO.
* New Config::getColorSpaceFromFilepath wraps the OCIO version if OCIO
is new enough, but if it isn't, or if there's no OCIO, or if OCIO ends
up not figuring it out and returning the default rule, we fall back
to Config::parseColorSpaceFromString.
* Add OCIO to VFX Platform CI builds -- because it's not in the aswf-osl
docker images, we weren't testing OCIO in as many versions as we had
previously thought.
* Bump default OCIO version built by build-scripts/build_opencolorio.bash
to 2.0 (from 1.1.1).
the temporary image buffer is created with only 3 channels, so the get_pixels call at the end of the
read_unmipped method must account for that when creating the ROI passed to it.
This also bumps compiler minimums to gcc 6.1, clang 3.4, MSVS 2017,
and icc 17 (all because those are the first versions to fully support
C++14).
This has been coming and has been socialized for a long time, with
nobody voicing serious objection. So here it is.
This WILL NOT be backported to release branches. The new minimums only
affect OIIO 2.3 and later.
When last discussed on the oiio-dev mail list, I said I would aim to
introduce this change on May 1. I will not merge this PR earlier than
that, and I will tag a developer preview release before that happens.
The new `--missingfile` argument (taking a value of either `error`,
`black`, or `checker`) specifies the policy for what oiiotool does
for input files that are entirely missing (not present but broken).
In all cases, an error messages is issued, but for the cases of
black and checker, an image is substituted and computation continues,
rather than immediately ending the attempt to execute the command
sequence.
On a personal project a while back, I wasted a full day chasing down
inexplicable behavior that turned out to stem from TWO implementations
of ustring being in the same application, a result of inadvertently
linking against both libOpenImageIO and libOpenImageIO_Util. This led
to two ustrings that contain the same character sequence, but with
different pointers because they resided in different ustring tables.
There is no way that anybody without very deep understanding of ustring
and a lot of patience could have tracked down the underlying issue.
This patch eliminates the possibility of duplicated modules in
libOpenImageIO and libOpenImageIO_Util. Now libOpenImageIO depends on
and links to the latter. This should all be automatic for users -- as
before, link against libOpenImageIO if you want everything, link
against libOpenImageIO_Util if you just want the util helper classes
but don't need any of the image I/O functionality. But now, if you
accidentally specify both... well, that's just the same thing as
asking for libOpenImageIO.
Fully separating libOpenImageIO from libOpenImageIO_Util requires new
OIIO_UTIL_API
Make whether namespace includes the patch number be dependent on
whether it's a dev or release branch. We generally want the patch
in the namespace for master, but not release.
This teaches oiiotool's command line expression substitution (`{...}`)
syntax to understand that
getattribute(name)
should be replaced by the value that OIIO::getattribute(name,...) would
retrieve. The `name` may optionally be single or double quoted, but doesn't
have to be if it's a simple identifier.
Some examples:
# Which SIMD features does this OIIO use?
oiiotool --echo "{getattribute('oiio:simd')}"
# Which dependencies are used for each format?
oiiotool --echo "{getattribute(library_list)}"
# Which file extensions are associated with each format?
oiiotool --echo "{getattribute(extension_list)}"
The cv::Mat::create method does the same thing as the constructor but "Allocates new array data if needed." And when not needed, which is my use case when all the images I process are the same size, create does nothing. This is more efficient than the implementation where a new mat is constructed and then moved to the destination. This is a minor optimization, but it does avoid allocating an image size block of pixels when it is not necessary.
Sorry this has taken so much tinkering to get right for downstream
projects. It's tricky business.
First, we start by adding testsuite/cmake-consumer -- a test that
consumes OIIO and its exported cmake modules, and is independent of
the rest of OIIO's build system, more accurately simulating whether
things will work for a downstream client.
Adjust our exported module, it needs to break out the OpenEXR 3 vs 2
cases separately, as well as whether it found an OpenEXR/Imath config
file or not. I think I have all the cases covered now. I also needed
to specify find_dependency of IlmBase and Threads, for the sake of
openexr 2.4 with config, and add the INCLUDES directory lists in the
case of no config files found.
Some adjustment of our library target_link_libraries -- better PRIVATE
hiding of things not needed in the interface by downstream builds.
In particular, the OpenEXR components and OpenCV should be private, not
public.
Add m_allgray field and color_table_is_all_gray method to detect
that the color table entries have the same r, g, b values and indicate
a single channel gray scale image rather than a 3 channel rgb image.
Temporary allocations were made and freed for each scanline read or
written. Do it once per image and reuse. When reading, eliminate one
temp buffer by writing data into its final location, also eliminating
a needless extra copy of each scanline.
We broke things subtly for older OpenEXR when we added the 3.x
support. Without this fix, our export CMake config files weren't
properly exporting the OpenEXR include path as the interface includes
for OIIO libraries. This doesn't affect the OIIO build itself, but it
can affect downstream libraries that use OIIO as a dependency and
rely on our exported cmake config files.
We also remove the need for OpenEXR/OpenEXRConfig.h entirely through
header configuration.
Ever since adding OpenEXR 3.0 support, we had some cases where
installations that built the 2.x IlmBase and OpenEXR portions
separately and installing to separate areas.
Also make some minor changes to ci.yml to make sure we are documenting
the right openexr versions used for each test.
It seems that for CMake, --config should only be used in conjunction
with --build. Who knew? Things are breaking right and left now, especially
on GitHub Actions CI. But also on my laptop.
Hypothesis: maybe this was silently ignored in older cmake versions, but
now with CMake 3.20 is a hard error?
This does not (YET) change the minimum to C++14. For now, you can still
build for C++11 -- and with compilers that only support C++11 -- by using
CMAKE_CXX_STANDARD=11.
By changing the default, we hope to more thoroughly test everybody's
ability to compile C++14, ahead of the actual cut-off when we make that
the true miimum.
Also, the latest OpenVDB 8.0 does has C++14 as a minimum, so compiling
in C++11 mode will automatically disable OpenVDB support if OpenVDB >=
8 is what is found. We expect an increasing number of our dependencies
to drop support for C++11, so we want our default compilation mode to
be one that doesn't silently drop format support without
users/builders being aware of it.
In many cases, this informative debugging code was left over from when
parts of this code was strictly a command line utility. It's not
appropriate for a library call, especially on Windows.
Make sure every last output to std::cerr is guarded by some
preprocessor symbol that would only be enabled when one of the
developers is trying to do some "printf debugging". And similarly,
std::cout should only be used for standalone command line
utilities that are by design supposed to output to the console.
We were treating some recoverable warnings from jpeglib as fatal errors.
Removed testsuite/jpeg-corrupt-header since now we can read it just
fine despite the very minor corruption! Also, turns out the image was
inappropriate, sorry, I had never actually seen it before because
OIIO wouldn't read and display it and I assumed it was just garbage.
Apparently all the recent farmhash changes are fine with C++14 mode,
and fine with C++11 mode on gcc 4.8, but gets some errors when
usig in newer compilers (gcc6+) using C++11 mode.
Add a CI test with gcc10 but with C++11 compatibility to verify this fix.
We will remove this test once we retire C++11 compatibility.
For the case of applying a color transformation to just one color, we
were sloppy and set the strides to 0 (just one color means we are not
iterating from point to point using the strides).
OpenColorIO 1.x didn't care, but OpenColorIO 2.0 throws an exception if
the strides don't look valid.
Thanks to Thomas Metais for the fix suggestion.
Change imageio.cpp convert_from_float to use the convert_type function
from fmath.h, rather than defining redundant functionality in that
module.
This has the added benefit that the ones in fmath.h have some interesting
specializations to use SIMD ops and other optimizations that the one-offs
in imageio.cpp lacked.
Ordinarily, oiiotool will exit upon encountering an error (such as an
input file not being found). When operating on a frame range or with
frame number wildcards, that means that it will exit without attempting
the rest of the frame range.
The new `--skip-bad-frames` command line argument means that an error
will skip the rest of that iteration, but not exit, instead starting
fresh on the next frame in the sequence. (But if any frame in the
sequence encounters an error, the exit code of the oiiotool command
will still be EXIT_FAILURE.)
GitHub Actions in its wisdom has removed pre-installed Boost from the Windows
images, causing our CI to fail.
Instead, get boost from vcpkg, just the pieces we need.
Also needed some JPEG adjustments for Windows CI build.
Added utility function OIIO::get_extension_map() which returns a
`std::map<std::string, std::vector<std::string>>` listing all
image file formats and their presumed file extensions.
`oiiotool --list-formats` helps to test this and is useful in its own
right.
Fix reading tiled tiff files when vertical read size is less than tile size, just for the code path where we use TIFFReadRGBATile (used for certain rarely encountered types of TIFF files). TIFFReadRGBATile returns the image data in bottom-to-top order, and we had been correctly reversing it all along, but neglected to account for the issue of a partially-filled tile ending up with the padding at the beginning of the buffer rather than the end.
All of our prior tga tests had valid thumbnails. The code skipped the
thumbnail if the "offset to the thumbnail" was 0. But if the offset was
nonzero but the thumbnail record at that spot had a 0x0 size, we just
ran merrily along, and on Windows in debug mode even it an assert.
This patch makes it understand that a 0x0 thumbnail size means no
thumbnail.
I added an image from Doug Rogers, the reporter of the problem, to the
oiio-images repo in the targa directory, and updated the OIIO testsuite
to also check that image. (So testsuite will fail for people who don't
update their oiio-images repo.)
Also changed some stray refs to &vector[0] to vector.data().
Fixes#2900
The very old OS2_V1 variety of BMP files -- which dates from the OS/2
and Windows 2.0 circa 1987 -- could have undefined results by relying
on the cpalete header field that didn't exist in that early
version. Our failure to initialize it led to expecting color palettes
of undefined size (which could either cause a read failure or even a
crash if it vastly over-allocated).
Some other minor touch-ups to the bmp code while we're at it:
* Zero-initialize all the other fields that don't exist in the OS2_V1
version of the header.
* Don't set the Resolution metadata if the values in the file aren't
positive (including for the V1 header that lacks it entirely).
* Fix reading V1 header resolution on big endian architectures (I just
happened to spot that potential bug).
* Fix longstanding error (that I only just noticed) in reading 4 bpp
images with odd horizontal resolutions, wherein the very last column
was not filled in correctly (that's the change at lines 188 and
193).
* Beef up one of the error messages invoked for failures while reading
the color palette.
* Rig the BMP reader to know the "dib" is another extension often
associated with BMP files.
* Add a regression test case for this format variant.
* Make sure CI tests bmp -- the GHA CI did not download the bmpsuite and
so was not actually running the testsuite/bmp tests.
* Add "bmp:bitsperpixel" (when not 24 or 32) and "bmp:version" metadata.
This is primarily for debugging and understanding which encodings the
BMP file is using.
* Recognize nonsensical values for the header's field for number of entries
in the palette. Add a test for that.
This patch ensures that two ustrings will never have the same hash value.
The basic strategy is that the usual 64 bit hash of the characters provides
an initial guess. If no other string has already used that hash (as is
almost always the case), we use it. However, if a different string already
took that hash value, we rehash, and repeat until we find an unused hash.
We maintain a "reverse hash map" that relates hashes to ustrings to
help figure this out efficiently. (Note: This adds at least another 16
bytes per unique ustring to maintain the reverse map. Do we care?
Probably few applications are in the business of creating more than a
few million distinct strings.)
Because we are recording the collisions as they happen, this also
greatly simplifies (and reduces to trivially low cost) the
recently-added ustring::hash_collisions() function, because there is
no complicated search when it is called.
When making a texture out of float values, but writing ultimately to a
`half` file, it's not too hard to end up with valid but large float
values that are outside the range representable by a `half` (fp16)
value and therefore will get converted to an `Inf`, which nobody wants
in their texture.
This patch prevents Inf from showing up in the output of make_texture
by clamping output buffers to the range of `half` when the source data
are float and the destination file is half.
Fix minor warnings and deprecations of C++20.
Bump to more modern ubuntu images for recent-year tests and make some
related adjustments. Also make sure that the "latest" and "bleeding edge"
tests are using python3, they were inadvertently still using python2.
A recent commit in the OpenEXR master changed which headers are
transitively included by other OpenEXR headers, and that broke our
build until this patch adds the missing headers explicitly.
The change may or may not be reverted on the OpenEXR side, but this
patch to OIIO at least unbreaks our CI in the mean time, and has
no down-side.
Update old calls to Strutil::format to new fmt conventions, convert
some sprintf-like calls to std::format like calls.
This is not meant to be comprehensive; there is more work to do.
It's just an opportunistic bite at getting some of it done.
The oiiotool --colorconvert operator (and the other color conversion ops)
have an optional modifier ":unpremult=1" that will bracket the color
transformation with an unpremultiply/premltiply. But the implied
colorconvert that happens upon input and output with --autocc has no such
option, which is an oversight. This patch adds the ability to control
with autocc transformations happen in an unpremultiplied state (which I
believe they SHOULD, generally speaking).
Also beef up the docs and help messages in some areas where I
coincidentally realized that -i and -o explanations omitted some of
the possible options.
There were some flaws in the logic of IB::set_write_format that could
cause a crash when used in conjunction with a file format that doesn't
support per-channel data types (even if the set_write_format call
specified a single type rather than per-channel types!).
There are basically two problems I fixed here.
First, in ImageBuf::write, where we are dealing with the fact that
set_write_format previously registered a request, be extra careful
that the "default" data format is the "biggest" type out of the
per-channel choices, and also make sure that if all the channels are
identical, we just use the default type and clear the channelformats
vector.
Secondly -- and more importantly to the crash itself -- in
ImageOutput::to_native_rectangle, if the output file format does not
suport per-channel data types, just use the main format. Do NOT
consider this case a hard error, which previously took an early out by
returning NULL for the location where the native data was copied. It
was that NULL that, when subsequently copied from (not noticing that
it was NULL indicating an error), actually produced the segfault that
caused the crash. In other words, don't worry, be happy -- just do
your best by treating the unsatisfiable request as the next best
thing, as if it were the same type for all channels.
Homebrew just upgraded to Qt6, and that breaks us. Peg to Qt5 as a
temporary measure to let our CI pass, until somebody figures out what
needs to be fixed for full Qt6 compatibility.
I have submitted a PR to Imath that adds "interopability" constructors
and assignment, that use some templates to allow seamless
construction, assignment, and passing of "foreign" vector types to
Imath vectors. This makes it easier library A that uses Imath vector
types in its API to interact with app or library B that has a custom
internal vector type, without a ton of ugly casts and copies.
(See https://github.com/AcademySoftwareFoundation/Imath/pull/91)
These created a few minor conflicts with OIIO's internal SIMD classes
and their construction and casting from Imath types. This patch cleans
up the ambiguities:
* Make Imath::V4f -> simd::vfloat4 and Imath::M44f -> simd::matrix44
constructor `explicit`.
* Make the simd types `simd()` method (returning the underlying raw SIMD
type) also have a variety that returns a ref to a non-const vector,
to ensure proper working of the _MM_TRANSPOSE4_PS macro.
Now I am going to commit a sin. This MUST get backported to OIIO 2.2
in order for it to not break against the upcoming Imath 3.0. This
doesn't break ABI compatibility, nor change OIIO's primary advertised
public APIs, but it does change the technically/informally public APIs
of simd.h in ways that introduce the possibility that an app using
those simd vector classes *might* need minor source code editing if
they relied on those implicit constructors. Such software might need
certain `simd::vector4 foo = imath_bar` to become `foo = vector4(bar)`
and the same for certain matrix44 assignments. It is usually a big
no-no to backport any change that could require editing of source
code. But since the alternative is not being able to use the new
Imath, I think this is the lesser evil, and maybe nobody out there is
doing the thing that would need to be changed?
Need some include fixes to more gracefully handle the case where
Imath_ROOT points to a custom build (say, of Imath 3.x), but there is
a system installation of a different version (say, OpenEXR 2.x in
/usr/include, which could easily end up earlier in the path list).
Also update some reference output due to recent bug fixes in OpenEXR,
where error messages changed.
* Better handling of lists/sequences -- append into attribute arrays
instead of allowing each value to overwrite the previous one.
* Better inference of unknown attributes that look like int, float, or
rational, rather than cramming into strings.
* Fixed what I now think were bugs in handling rational values.
Turns out it doesn't -- we're still seeing timeouts on the Windows CI
tests, even with the OpenEXR that we thought fixed the threadpool
problem.
But let's keep this change anyway.
Background: In our GitHub Actions CI, we were doing environment setup,
dependency setup, OIIO build, and testsuite -- all in one action
"step" because each stage is its own shell process, so I couldn't
figure out how to communicate between them (for example, a dependency
setup might build and install a dep package, but then the later build
stage needs the CMake Pkg_ROOT variable set to that location).
Doing it all as one "step" was clumsy, not best GHA practice, and it
made it hard to understand which stages were taking different amounts
of time and how to optimize them.
The trick, it turns out, is that $GITHUB_ENV is the name of a file
that will be executed at the start of each step. So you put commands
into that location that restore whatever env variable you need to
restore between steps, or any other setup you need.
So in this patch, we:
* Elevate env setting to be for all steps.
* Split build-and-test.bash into separate build and test scripts.
* Separate setup, dependency installs, build, and test into separate
GHA steps.
* Each step's script, at its very end, calls new script save-env.bash,
which puts commands to restore the environment into the file named
by $GITHUB_ENV, which will be read at the beginning of the subsequent
step.
* Merge gh-installdeps and gh-centos-installdeps scripts -- there was a
lot of redundancy between them anyway.
The net result of this is that when we examine the test results, it's
much easier to see which stage a failed test had trouble on, and
especially how much time each stage took. That's really handy with the
recent changes to add caching, and helps us verify that the caching is
working as intended, since it is expected to take a big bite out of
the dependency and build stages, but not the test stage.
This speeds up CI builds by 1/2 - 2/3 of total time, for PRs that are
minor changes by reusing the compiled results from previous runs.
Most of the remaining time cannot be squeezed down -- it's the
instantiation of docker containers and running the testsuite
itself. But the build (including of dependencies we build from
scratch) is dramatically faster.
Currently, this depends on ccache, so is for Linux/Mac only. I'm also
working on doing something similar for Windows, but it's not ready for
review yet.
Do some adjustment/simplification of the macros that set up the right
decorators on the new inline farmhash.
Some additional constexpr's, including Strutil::strhash().
Add to hash_test verification of the actual results of each of the
hashes, so we know that we don't break them over time, that they give
the same result on all platforms, and most immediately, to 100% verify
that the new constexpr inline version of the hash precisely matches
the old procedure call one.
Augment the existing benchmarking of the hashes to include
farmhash::inline. It tries a variety of string sizes. I was worried
that the constexpr/inline version would be a performance hit because
it can't use any of the SSE intrinsics (which, ugh, aren't declared
constexpr in the headers). But it turns out that the inline (but
SSE-free0 version is a little faster on some tests, a little slower on
other tests, overall a wash and never differing by more than
10%. Maybe that means that the inlining/constexpr makes up for the
lack of SSE? Maybe it means that the compiler optimization is adding
SSE back in well enough that losing the intrinsics isn't as bad as we
would have thought? In any case, the upshot is that this doesn't
represent a performance it, so I may consider a future change to just
go strictly with the inline constexpr versions and eliminating the
separately compiled version altogether. Will mull that over.
* cmake: Make LibRaw Jasper and LCMS2 detection optional, add LIBRAW_NODLL
* cmake: Fix OpenColorIO defines on Windows to use correct cmake mechanism
* cmake: Add missing LibRaw defines (needed when linking statically)
Some fields have disappeared from certain structures, so we need to make
adjustments to the various conditional compilation guards.
This patch just makes OIIO build against the new libraw, and is intended
to be backported to the release branch.
This patch does not incorporate any new fields added to libraw
structs, representing newly retrieved metadata. A separate PR patch
will address the newly added capabilities (TBD whether or not that
ends up safe to backport and when).
Bump CI "current releases" test to OCIO v2, it's released!
Oops, we used the wrong name for the env variable, and that CI test
wasn't getting the OCIOv2 RC all along!
Add OIIO::ColorConfig::OpenColorIO_version_hex().
Have oiiotool --help print the OCIO version.
Also Bump the pybind11 version for the "latest" test, and update
INSTALL.md a bit to document the latest versions known to work of
several dependencies.
When we wrote our `span<>`, we based it off the draft for std::span,
which at that time specified span::size() as ptrdiff_t, which is
signed. By the time std::span got finalized in C++20, it was changed
to size_t, which is unsigned. Ick. Switching to size_t will be an API
break, so we can't do that until we're at a version where it's ok to
make breaking changes.
But we can stage a few changes that will make that future transition
easier.
* Add std::size() and std::ssize() wrappers for both span and
string_view. This gives downstream apps the opportunity to avoid
span::size(), instead using std::size(the_span), which is guaranteed
to be unsigned, and std::ssize(the_span), which is guaranteed to be
signed, and both of these will work now as well as after we make the
other breaking changes.
* Doing this properly required adding a couple 'constexpr' in places
where they should have been all along.
* Define macros OIIO_SPAN_HAS_STD_SIZE and OIIO_STRING_VIEW_HAS_STD_SIZE
making it easy to know if std::size and std::ssize are defined for
span and string_view
* OIIO_SPAN_SIZE_IS_UNSIGNED will be defined when OIIO::span switches
to having an unsigned size like std::span (making it easy for client
software to know if span::size is signed or unsigned).
Turn this on automatically for OIIO >= 3.0 so we don't forget.
If client code predefines OIIO_SPAN_SIZE_IS_UNSIGNED before
including span.h, that forces span::size to unsigned now (good for
experimentation and compatibility testing, but I don't recommend
anybody ship software in that state unless they really know what
they are doing and don't mind potential incompatibility with the
out-of-the-box OIIO).
These SHOULD NOT break any existing code or ABIs that uses our
existing span and string_view classes or their size() methods. It only
adds options that will help bridge compatibility.
A variety of small problems were fixed:
* Symbols in libOpenImageIO-C that were not marked as OIIOC_API.
* MSVS not being happy with array length determined by a variable
* Issues with Windows paths
This is a weird thing we're doing in this very artificial test,
removing a file when the IB reading it is still alive. That's safe on
Unix, where a file with an open file handle still be safe to read even
if it's removed. But it's not kosher on Windows. Fix by simply making
sure that in this test, we aren't doing the unadvised thing of having
the IB still active on the file we're deleting. This should fix the
unit_imagebuf test that is failing our Windows CI.
* Try to address the OpenEXR thread pool hanging when we call exit().
* Strive to remove all exit() calls after any file reads happen,
in oiiotool, iconvert, and idiff.
* This DOES NOT stop all hanging. It's still a bit of a mystery whether
there is anything we can do to fully prevent it without a fix on the
OpenEXR side.
* Temp workaround: for the Windows CI tests, use ctest commands to
allow each test to timeout 3 times before giving up. To be clear,
this does not solve the problem, it only masks it. But it helps
us "pass" those tests, exposing any other problems occurring with
them, and we will return later to find the source of any residual
timeouts.
* Update ref output for several tests where the differ slightly under Windows.
* Make runtest.py convert text output line endings cr-lf -> lf before
comparing to reference output. (This eliminates unnecessary diffs.)
* Update many tests' run.py file to use double quotes instead of single
quotes for arguments containing some characters -- looks like the Windows
shell invoked by Python system() does not handle the single quotes quite
the same way as Unix shells.
* oiiotool: when printing the command line after errors, genericize
identifiable filenames so they render the same on Windows and POSIX,
and strip directory name from the oiiotool filepath. This makes
error output comparison more equal between Windows and POSIX, making
several more tests pass.
In total, this seems to get us down to failing only 7 tests (not counting
spurious timeouts that cause failures).
Similar to C++17 std::filesystem::path::generic_string(), but for a
filepath sent as string_view.
While I was there, I noticed that a few Filesystem functions that I
declared as `noexcept` actually call things that can throw, so I put a
few try/catch in stratgic spots. Nobody had reported a problem with
this, and maybe it would be fine, but it's the right thing to do.
* Change to finding a consistent Python version (3.7)
* Install numpy
* On Windows, convert crlf to lf before diffing reference output, so we
don't get mismatches just because of LF vs CRLF.
* Update ref output on testsuite/python-imageinput
* Fix windows detection in runtest.py oiio_relpath
* Fix some semicolon problems that were making bash command lines conused
Along for the ride:
* Have ci-build-and-test.bash send OIIO_CTEST_FLAGS as extra args to ctest
* Remove some needless verbosity from the dependency building scripts.
With some recent heic related packages, we lost the ability to build
against some older libheif versions that we purport to support. We may
eventually want to raise the minimum supported libheif, but in the
mean time, this restores the prior build abilities.
Use pybind11 functions to set up python module, it's got superior logic
to what we were doing.
Bump pybind11 version default in build_pybind11.bash
Suppress clang warnings about python register use
* Re-enable vcpkg installation of freetype on windows
* Add TESTSUITE-README.md file
We'll use this file to document how the testsuite works and how users
should update it when necessary.
* Re-enable windows tests
Fix a few issues parsing rational values with large denominator in OpenEXR header.
Incorrect array index can cause divide by zero crash (observed consistently on Windows with repro image).
Conditional comparison was assigning 1 or 0 to int f declaration instead of gcd value.
Changed gcd type parameter to long long to avoid unsigned int->long int overflow on windows (long int is 4-bytes on windows).
Co-authored-by: Chris Crosetto <ccrosetto@gmail.com>
* fixes to make vs2019 CI run. Not passing yet as some tests fail
* added explicit ctest timeout (180sec) + misc fixes to run CI local
* disable Windows VS2019 tests again as they are not passing yet
Co-authored-by: Dieter De Baets <dieter.baets@unity3d.com>
OSL has some concerns about how ustring hash collisions could impact
its strategy for strings on GPU.
This patch adds some instrumentation to ustring to detect hash
collisions. Added static ustring methods total_ustrings() and
hash_collisions(). The latter tells you the number of collision pairs,
including optionally enumerating them into a vector of ustrings. Also,
`getstats(verbose=true)`, in Debug builds only, will print all of the
colliding pairs. These are both somewhat expensive if there are lots
of strings (my laptop takes about 4 seconds to find and print all
collisions for 10M ustrings). So if you are concerned about
performance in debug builds, call getstats(false) instead if you have
a truly enormous ustring table.
This addresses slight ringing at image edges when using a combination of
black wrap mode and a filter with negative lobes. We need to be sure that
we really clamp the filter at the source image edges.
But we only want to do the clamping selectively. Works great for --fit,
not so much for more general warp and rotate. So add it as an internal
parameter to the underlying warp we're doing.
Needed to update a bunch of the fit-related output reference.
There are several ways to make an image fit a new size while preserving
aspect ratio. Originally, fit() would do what we now call "letterbox"
mode, which tries to maximally fill the new size, retain the original
aspect ratio of the visible image, exactly fill one of the dimensions,
but not lose any visible image data -- this means that a dimension that
cannot be filled will be padded with black pixels. This is still the
default behavior.
But additional modes are not possible: "width" means to exactly fill
the horizontal dimension (the vertical may be letterboxed, or may be
cropped, depending on if it's too big or too small), "height" means to
exactly fill the vertical dimension (the horizontal may be letterboxed
or cropped).
* Add `fillmode` parameter to IBA::fit() in both C++ and Python.
* Old fit() without this parameter are still present for
back-compatibility, but are deprecated and will eventually (when
we're at an ABI-breaking release in the future) be removed.
* oiiotool: Add optional `:fillmode=` modifier to the `--fit` command.
* More comprehensive tests in testsuite/oiiotool-xform, testing every
combination of portrait, landscape, shrinking, growing, each fill mode.
* Update docs
The WebPData::size is a size_t, on 32 bit systems it will balk at our
putting a uint64_t in it. Downcast explicitly so the warning doesn't
flag us.
Also some changes to ensure adequate testing of webp in CI. (Even though
we only CI test 64 bit systems, so this was not related to the specific
build break being fixed... but an excuse to make sure we are hitting it
a reasonable amount in CI.)
Filesystem::IOFile constructor now upon an error opening the file will
set the error message appropriately.
Move IOProxy::error() implementation from inline to be in
filesystem.cpp to hide its details, use a static mutex to make error
set and retrieval be thread-safe (we presume that the rare chance that
multiple threads will be setting or getting IOProxy errors
simultaneously are rare enough that it's not a thread performance
issue, and if the user is getting file I/O errors, a bit of minor
locking is the least of their worries).
Better checking of these errors in OpenEXR input and output open
operations. Hopefully this means that exr file I/O errors will
percolate up more helpful error messages.
TL;DR: This could crash during openexr output, turned out to be a subtle
race condition.
Long explanation:
Metadata gets prefixed with `<formatname>:` in cases where it's only
meant to be understood by the reader or writer of one format. Formats
that support arbitrary metadata filter out any items that are prefixed
by the name of a different format (so you don't, for example, when
converting file formats, incorrectly copy a TIFF-specific metadata
hint into an OpenEXR file as literal metadata).
The specific code we used in the OpenEXR writer tried to do this but
got the threading locks wrong, and this could lead to very rare, subtle
crashes. It could have been fixed by a 3-line very local fix, but...
I also noticed that the same code was replicated in Field3D output (and
quite incorrectly -- in the cut and paste from the openexr code, it was
still looking for "openexr:" prefix instead of the correcd field3d one!).
Since this idiom was happening (wrong) in two different places, and
seemed likely to crop up again some time, instead of fixing locally I
added a new API call, `is_imageio_format_name()`, to do this centrally
and correctly. So most of this patch is implementing that and then
using it instead of the incorrect code elsewhere.
Also add python bindings and docs for the new API function
Upon reading cmake docs mre carefully, it's clear that when you pass a
version to find_package (which does much of the heavy lifting inside
checked_find_package), it DOESN'T MEAN you want that version or
later. It means you want to find a version that is "compatible" with
the named version. The meaning of "compatible" is up to the package.
Some packages define it as being >= the requested version. Others
define it as being the same major release, or same major AND minor
release, or exact.
But generally speaking, we only care about the minimum version (e.g.,
"I want FooBar 8.0 or later"). And it's not guaranteed that is what
the package's compatibility will do. In fact, FooBar may be using
compatibility mode "SameMajorVersion", which will make our find of
FooBar succeed for 8.0, 8.1, 8.2, but if a user encounters FooBar 9.0,
our build will reject it, even though it's probably fine.
So I augmented our checked_find_package wrapper to take optional
VERSION_MIN to give the real minimum version, when that's the semantic
we really want (in which case we would omit the ordinary version
number used for the compatibility check and let VERSION_MIN do all the
work). There's also an optional (and usually omitted) VERSION_MAX that
you can use if you are sure that we must be less than a certain
version (the max is exclusive, you must be <, not <=, whereas the min
is inclusive >=).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Some permissions on GH changed due to security concerns, and that broke
the setup-nuget action we were using... which turned out to be obsolete,
anyway, and switching to the new recommended replacement fixes everything.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Inspired by OpenColorIO's beautiful docs, I'm adding the "sphinx_tabs"
extension to our docs, and testing them out in one little spot in
ImageBufAlgo to see if we like the effect.
Please look at https://openimageio.readthedocs.io/en/rtd/imagebufalgo.html
look at the first couple screenfuls to see the example of how the
tabs look. Opinions are welcome. Should we do this kind of thing more
extensively?
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Detect if OIIO is being built as a subproject, and if so, set
`OpenImageIO_IS_SUBPROJECT` (a.k.a. `${PROJECT_NAME}_IS_SUBPROJECT`).
* Move all the testing logic out of the main CMakeLists.txt (and a few
testing-related macros from oiio_macros.cmake) and into a new
src/cmake/testing.cmake.
* Move all of the packaging logic out of the main CMakeLists.txt and
into a new src/cmake/packaging.cmake.
* Make all the testing and packaging inactive if this is being built
as a subproject. Also disable the clang-format target for subprojects.
* Move fancy_add_executable and add_oiio_plugin from oiio_macros.cmake
into their own files (leaving oiio_macros.cmake empty and
eliminating it).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
As OpenEXR works toward their 3.0 release, which includes splitting
Imath (the vector, matrix, math, and 'half' stuff) out into a separate
repo, their top of tree master finally breaks our "bleeding edge"
build.
This is a little tricky, because now our support of this dependency is
attempting to span 3 eras: 3.0 with the full split of openexr from
imath, recent openexr+ilmbase 2.x that supplies reliable exported
cmake configs, and older 2.x that predates cmake configs. Eventually
we'll phase out older versions and clean up.
Fixes in this patch generally include:
* Add a new CI test of VFX2021 + openexr/imath master. (This will go away
when 3.0 is released and that's what we use for the main 2021 case.)
* For new enough versions of OpenEXR, use the exported config file and
targets, rather than the old style OPENEXR_LIBRARIES style variables.
We have kind of a mixed idiom use because we're still supporting older
versions that predate use of the config files.
* Move imath includes to new OpenImageIO/Imath.h Deal with the fact
that the Imath headers now live inside `<Imath/ImathBlah.h>` rather
than `<OpenEXR/ImathBlah.h>`. But of course, you need to start
including things just to find out the version. It's tricky. The
logic is to look at `<OpenEXR/OpenEXRConfig.h>` to determine the
version we're dealing with, then include Imath stuff from one or the
other depending on what we found. I rearranged quite a few includes
to ensure that all this logic only happens in a couple spots and the
rest gets included transitively. As it turns out, quite a few of our
`#include`s of Imath headers were redundant.
* OpenImageIO-C depended on OpenImageIO publicly, but actually it
should depend on OpenImageIO privately, but export its include
needs.
* OIIO's exported config needs to look for OpenEXR.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Use __builtin_bswap when available. These seem to be 8-15x faster than
the std::swap-based implementation when multiple data values are
being swapped. (It's not dramatically better when swapping a single
value, either way it's pretty fast.)
Also add correctness unit test and benchmark for swap_endian.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The core ImageInput and ImageOutput (and auxiliary classes they use, such as ROI, ImageSpec, DeepData) now have C99 interfaces that wrap the native C++ classes. In addition to being good for using OIIO with C, this is an important step along the way to simplifying the implementation of Rust bindings (and other languages that have a simple ffi interface to C but are tricky to wrap C++).
Still to do: ImageBuf, ImageBufAlgo, etc.
* Bump breathe release to the latest (we'e been holding back because of
errors that we fix in the other elements of this patch, below)
* Changing declaration formatting, particularly in the ImageBuf and
ImageBufAlgo chapters, eliminates function ambiguity warnings and even
python exceptions we were getting in Breathe before.
* Document missing texture option fields and improve formatting.
* Instruct Doxygen to not catalog deprecated ImageBufAlgo functions.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We were running some CI tests only for PRs or for certain branches.
This may save some time, but I found that I was, more often than not,
naming my branches to trigger all tests to run. That's silly. Just run
all the tests every time, it's not that expensive.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
OCIO master recently renamed the symbol in its headers that we used to
detect its version, breaking OIIO's build against the current OCIO
top of tree master.
This change makes our FindOpenColorIO.cmake work with both old and new
OCIO.
Also, cut out other useless OCIO building (for our purposes) by having
our build_opencolorio.cmake disable buildig of GPU tests (which,
recently, became controlled by a different option than is used to
disable building all the other tests).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
ImageBuf methods subimage(), nsubimages(), miplevel(), nmiplevels(),
and file_format_name(), which are concerned with the file to which the
ImageBuf refers, only make sense for an ImageBuf that directly refers
to a file -- either by being constructed/reset from a filename, or in
some cases, being a direct unmodified copy of such an imageBuf. All
other cases, including pixel-copies of other ImageBufs (even those
that came from files), it is merely an in-memory image and doesn't
really refer to a file.
Clarify the docs (and fix the implementation) to ensure that for
any ImageBuf that is not a simple reference to a file, these methods
return a suitale default value.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Accommodate changes for 0.20.2 that Homebrew now has.
* Mistake in variable naming meant we weren't testing libraw master as
we previously thought.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
There is a big bug lurking in how we read TIFF files with "separate"
planarconfig (RRRGGGBBB, versus "contig" RGBRGBRGB). I knew that
libtiff when reading scanline files doesn't support true "random
access" to scanline for most compression methods -- because you can't
seek to the start of a scanline in the middle of a compressed "strip,"
you have to read whole strips, or read individual scanlines
consecutively. But what somehow I've misunderstood for years (if not
decades) is that for "separate" planarconfig, this means that you
can't jump from scanline i in plane 0 to scanline i in plane 1! You
have to read all the scanlines (at least within one strip) in a single
plane before moving to the next plane.
This means that we get all sorts of read failures when reading TIFF
files meeting all of these conditions:
* "separate" planarconfig (these are rare in general)
* Scanline-oriented (tiles are fine, because they're read whole)
* More than one channel (with one channel, separate and contig are
basically the same)
* Certain compression codecs (turns out the separate+scanline combo
tests in the libtiff example suite are uncompressed, which is how
our testsuite failed to catch this).
* rowsperstrip > 1 (if rps == 1, every scanline is a new strip, and
that worked)
* written by an app that doesn't use OpenImageIO (because OIIO TIFF
output forced rps=1 for separate planarconfig)
Things were very broken. There is a comment in tiffoutput.cpp that says
// I can only seem to make separate planarconfig work when
// rowsperstrip is 1.
and forces rps=1 for separate planarconfig output. So at some point
many years ago I realized that something odd was happening here, but I
apparently I thought it was a bug in libtiff when writing with rps > 1.
Probably we were writing correctly all along, and the real bug was in
the reading. This also explains why we had mostly not seen this
problem before, which is because all OIIO-written files avoided
producing tiff files that fell into the failures case.
This patch mostly fixes it and also makes a number of performance
enhancements:
* Fix misunderstanding in read_scanlines where it was loose with thread
safety in retrieving the rows per strip.
* In read_scanlines when deciding the "chunk size" to limit temp
buffer size, and in read_image when deciding how many scanlines to
read at a time via read_scanlines, these computations weren't the
same, leaving slivers to be read alone instead of staying aligned
with strips. Make those both compute the same way, so from read_image
on down, we end up reading chunks that are always aligned with
strips.
* TIFF read_scanlines used to try to read whole strips, except for
reading individual scanlines that didn't comprise a whole strip
length. Modify it to recognize that the one exception is the last
strip in the file, in which the strip is shorter than usual if the
total number of scanlines is not an even multiple of rps. This
ensures that the last strip is read as a strip, not as individual
scanlines.
* Imagecache autotile behavior is modified to force the pseudo-tile
size to be a multiple of the strip length. Again, that keeps the
resulting reads to always be in units of strips, which is much more
efficient and is the safe way to seek and read separate
planarconfig. Also, always turn on autotile for any tiff file with
rowsperstrip > 1, because it's already organized that way in the
file. This is not only more efficient generally, but also makes it
so that ImageCache will only do reads aligned safely to strip
boundaries.
This is not a complete solution, in the sense that an app making
individual read_scanline calls for solo scanlines, or read_scanlines()
that is not aligned to strips, can still have read failures for TIFF
files that are just the right combination of separate planarconfig +
certain compressions + scanline order.
However, with this patch, the following common cases will ALWAYS work
correctly now: (a) read_image or read_scanlines of the whole image
size; (b) any reads mediated by ImageCache or TextureSystem; (c) all
images read by ImageBuf (which always use either (a) or (b)); (d) all
operations of oiiotool (which use ImageBuf).
After this patch is merged, solving the common cases, I'll return to
the topic to tackle the remaining issue of broken reads for these
files when doing individual ImageInput::read_scanline reads or
read_scanlines with ranges not aligned to strips. To make those work,
I'll need some kind of ugly internal strip buffering.
I'm going to leave the "force rps = 1 when writing planar separate tiff"
as it is, because even after this set of fixes, software using old
versions of OIIO will not be able to correctly read rps>1 planar files,
so we wouldn't want to be inadvertently writing them, at least not
until all the pre-fix OIIO releases are unlikely to be used by
anyone. Not writing planar files with rps>1 is not much of a
sacrifice, considering how rarely planar files are used at all.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Now that we use Linux and the ASWF docker containers for the
"clang-format" test, there's no reason to have logic in this setup
script for Mac to choose a more restrictive set of package for that
case.
Same for clang-tidy -- we don't have a test like that anyway, and
if we add one, we will surely do it with one of the Linux containers.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
There was no formal error setting! Errors were echoed to the optional
output stream, if supplied. But that's not good enough.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Although we were careful to make sure that the new versions of
ImageInput `read_scanlines`/`read_tiles`/`read_image` methods were
thread-safe when called concurrently, one thing that slipped through
the cracks was error reporting. A failed read (or other) call would
return an error flag and use error() to set the error message, which
could be retrieved via geterror()/has_error().
Except the flaw was that the error itself was a single std::string in
the class. It was guarded by a mutex, so at least the threads wouldn't
clobber each other over access to the string. But it means that the
following call sequence would do the wrong thing:
thread 1: read_tile A --> fails
thread 2: read_tile B --> fails
thread 1: geterror() --> gets both errors appended, oops!
thread 2: geterror() --> gets an empty error message, oops!
Why hadn't we noticed until now? Because read failures of individual
scanlines or tiles are rare -- usually the failure or discovery of a
missing or corrupt file is in open(). So I guess it's not often that
two concurrent reads would both have just the right timing to have the
error retrieval problem described above.
There is no easy fix, so a bit of a bigger, ABI-breaking, change was
necessary to make this work properly once and for all:
* ImageInput and ImageOutput error messages are now thread-specific. That
is, geterror() retrieves errors generated by method calls made by the
same thread.
* ImageInput and ImgageOutput now use a PIMPL idiom to store some data
behind an opaque pointer. This is good -- it means that the details
are hidden from the public API, and can use classes we don't wish to
expose (which we will do here with boost::thread_specific_pointer) as
well as allow us to change what's behind the pointer without breaking
the ABI.
* The error message uses a thread_specific_pointer.
* We therefore removed the m_errormessage member that was unfortunately
in the public class. Also removed the mutex and the m_threads, which
can now be safely tucked behind the PIMPL.
* With the mutex hidden from view, we now re-jigger references to it
to use the lock/unlock methods we had already. In fact, instead of
using lock_guard<mutex>, we can use lock_guard<ImageInput> because
the presence of lock/unlock methods make ImageInput (and ImageOutput)
themselves compliant with the BasicLockable concept.
Note that this changes the data layout of ImageInput and ImageOutput,
and therefore is an ABI break. So this is a change meant for master
only (future OIIO 2.3+), and will not be backported to the 2.2 release
branch.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* WebP input overhaul (including animated webp and metadata)
Huge overhaul of webp reading. Since webp support was first added in
2011 (with no major changes since, other than a couple minor bug
fixes), we have been using the basic/simplified webp API, which had
many limitations. This patch rewrites it to use the more complete
API. This results in the following changes and new capabilities:
* RGB images are kept as 3-channel instead of forced to be RGBA upon
input.
* Skip alpha premultiplication for RGB files. And for RGBA, skip the
dubious color space transformation when doing premultiplication of
RGBA files. I think this has been wrong all along.
* We can now read animated webp images (akin to animated GIFs, they look
like multi-subimage files).
* Much more efficient to read just the header, the pixels are read and
decoded lazily.
* Switch to the WebPDecode*Into API calls, which lets us preallocate
and manage the decoded image buffer (as a unique_ptr). Also switch
the encoded buffer from a std::vector to std::unique_ptr, which
avoids an unnecessary zeroing out of the buffer.
* Close the file handle immediately after we're done reading, rather
than leaving the handle open until the client is done with the
ImageInput.
* Read Exif metadata properly from webp files.
* To use the more sophisticated API, we need to find and use the
libwebpdemux library.
* Since there is a "LoopCount" metadata here, similarly to with GIF, I
decided to make an OIIO metadata "oiio:LoopCount" that will work
with both. GIF saves with this, too, and "gif:LoopCount" is now
considered a deprecated synonym.
Fixes#2721
N.B. This only covers webp input, not output. A similar overhaul should be
done eventually on the output side.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Cuda 10.1 seems to have trouble because of the `__attribute__` used in
`OPENIMAGEIO_PRINTF_ARGS`. Declan Russell suggests modifying the `#if`
to avoid using it for `__CUDACC__`.
LG further observes that `OPENIMAGEIO_PRINTF_ARGS` is only used for
the declaration of two functions, it seems like a nice decluttering to
just directly use the short attribute exactly where it is needed
instead of having a lengthy explanation and possibly defining
`__attribute__` everywhere as a side effect.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Co-authored-by: Declan Russell <declan@solidangle.com>
For IC::add_tile() and IC::get_tile(), the documentation advertised that
passing chend < chbegin meant "all channels". But the implementation was
setting chend=nchannels but not setting chbegin to 0.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Different classes and modules had minor inconsistencies on how errors
were reported and retrieved. Let's fix these up.
* geterror() should always take a `clear` parameter that allows the
caller to control whether or not retrieving the error clears the
internal error message (default is true, clear it after retrieving).
* Any class with a geterror() should also have a has_error() that merely
reports whether there is an error message pending retrieval.
* geterror() will always *append* to an existing error message that
has not yet been retrieved, with a newline separating error
messages. Previously, some of our geterror implementations appended
and others replaced.
* geterror() will trim a single trailing newline from the message it
is passed, thus storing error messages that are not
newline-terminated and making it so that callers of geterror() don't
need to wonder whether or not they should terminate their error
messages with a newline or what happens if you make the wrong
choice.
* Usage guidance is therefore that individual errorf/errorfmt() calls
not include a trailing newline (though if one is present, it will
be harmlessly stripped), and that whoever retrieves with geterror()
should, after outputting it, add a newline (which they seem to
invariably do already).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
One of our CI matrix entries -- building with clang -- has
mysteriously broken with compile errors complaining about reallocarray
being inconsistently declared in stdlib.h and gif_lib.h. It just broke
a few days ago, unrelated to any changes on our part. I have no idea
if it was the compiler or gif library that got updated and then broke.
Whatever. Insulate ourselves from random changes on GHCI by converting
this test to use one of the ASWF containers that we can count on to
have stable dependencies installed.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Bring back the idea of a Findpugixml.cmake file because it turns out
that (for pugixml <= 1.10) the exported config files are not reliable.
* Fix CI scripts to build PugiXML (with a few different versions). We
previously were relying on the embedded version for all CI tests, which
masked problems associated with using an external PugiXML.
* Fix deprecation warnings about xml_document.load(), the new call is
xml_document::load_string().
Signed-off-by: Larry Gritz <lg@larrygritz.com>
It was setting it, but later, too late for the particular case where
constant_color_detect was used and found a constant color and also the
config spec came in without any tile preference (still 0's). So move
the default initialization up to the top.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This is good for optimization, allowing "de-virtualization" of function
calls in cases when the compiler can be sure that there can be no further
subclassing.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This used to work, was inadvertently broken in #2398, in Nov 2019, and
no release of OIIO 2.1 ever had it working! (I guess people don't
come across 16 bit iff files very often?)
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Boost date_time, though never explicitly called in OIIO code, can be transitively pulled in by boost threads. This seems to be a problem on Windows when you build a partial boost containing only the libraries you think you need.
The solution seems to be to set BOOST_ALL_NO_LIB on MSVC, even for the dynamic linking case.
What data format does oiiotool write for its outputs? The ones
specified by -d, if supplied, but otherwise it will make a guess based
on the types of inputs it has seen.
This patch makes several improvements:
* Multi-subimage files were getting very confused because different
subimages may want different data formats, and it was at best
remembering data formats overall and by channel name, but not taking
subimages into consideration. Make this much more specific by
remembering both raw channel names (for all subimages), as well as
specific channels of named subimages.
* The way the input data formats were remembered had repeated (and
non-matching) code depending on whether the input was read
immediately or lazily. That is now cleared up, with the logic
refactored and expressed only once.
* The -d directive can specify a type (for all channels), a type for a
particular channel name (`channel=type`), and -- this part is new --
a type for all channels of a named subimage (specified as
`subimage.*=type`) or just one channel of a specific subimage
(`subimage.channel=type`).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This can cause cache re-reads unnecessarily because it thinks you're
asking for potentially different configuration hints every time you
set the ImageBuf to a different subimage.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
A bug (issue 2713) revealed that an alloca in IBA::colorconvert could
cause a stack overflow on an image that had an unusually wide number of
columns per scanline.
This patch takes a number of measures to help root out these problems
in the future:
* The OIIO_ALLOCA macro is amended to assert (in DEBUG builds only)
upon any attempt to alloca more than 1MB, which is surely a sign that
we have used alloca in an inappropriate place.
* A new OIIO_ALLOCATE_STACK_OR_HEAP macro is used to encapsulate an
idiom I have used before, to alloca small things but new[] big things,
and ensure in both cases that the memory is freed when the function
exists scope. I choose arbitrarily for "big thing" to mean <= 64 KB,
which is big enough for one 64x64 tile or one scanline of a 4k image
if it has 4 float channels, that means most of the time we'll safely
allocate on the stack, but for anything larger, it will use heap.
* Use OIIO_ALLOCATE_STACK_OR_HEAP in those problematic spots of
colorconvert.
* Use OIIO_ALLOCATE_STACK_OR_HEAP in a few other places that seem
prudent.
* Grepped through the code for all instances of OIIO_ALLOCA and I think
that all the potentiallly "big" ones are now using the safe idiom.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
For "mip-mapped" TIFF files where we use the multiple images in the file
to emulate MIP-maps rather than what we ordinarily consider subimages,
a call to `ImageInput::spec(subimage,miplevel)` was not properly
recognizing invalid subimage values and returning an empty ImageSpec
as advertised. Basically, we assumed that they would only ask for a
valid subimage and miplevel and didn't properly handle out of range
values.
Same problem for spec_dimensions().
Signed-off-by: Larry Gritz <lg@larrygritz.com>
For both, allow an optional modifier ":n=" to specify the number of
images from the stack to be combined by having their channels or
subimages appended (versus the default of combining just the top 2
images).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Fix cause of a subtle bug: splitting an empty string returned one
string piece (also empty) instead of 0 pieces.
Beef up the Strutil unit tests in this area.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This was subtle. Empty subimage include/exclude specifications would
confuse compute_subimages(), which in turn would MUCH later do a
mis-allocation and lead to a situation in --resize where it would
iterate over subimages that had not need allocated.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Switch from old cmake macros that were marked obsolete in cmake 3.12 to
the new FindPython.cmake module.
Instead of defaulting to python 2.7, default to whatever is found.
But user can still set PYTHON_VERSION variable to request a specific
one in case there are multiple.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
ImageBuf uses ImageCache underneath. But this can lead to the following
unexpected behavior if you don't understand this:
1. Make an ImageBuf for an image, read it (even just header info).
2. Done with it, destroy the ImageBuf.
3. Some other part of the program, or even a different process, changes
that file on disk.
4. Make another ImageBuf with the same filename, thinking you're getting
the updated image, but in fact it's relying on the cached representation
which doesn't contain the changes made in step 3.
So these tiny changes try to make this much harder to shoot yourself in the
foot:
* New ImageBufs automatically call cache->invalidate on the filename
before they start reading, to ensure that if the file has changed on
disk, anything in the cache is invalidated.
* When you destroy or clear an ImageBuf, it calls cache->close() on the
file, to release the handle (especially helpful on Windows, where an
open file handle can lock the file to other processes).
There is a chance that some image operations will have degraded
performance, if you are mixing direct ImageCache use (including
TextureSystem) and ImageBuf reads of the same files -- since creating
and destroying the IB's will flush tiles from the cache and close cached
file handles. But I think that this should not be a problem in practice,
since most apps that are heavy users of ImageBuf (like oiiotool) aren't
heavey users of IC/TS, and vice versa. And eliminating this confusing
behavior is probably more important than occasional over-eager
invalidation of the cache.
Also clarify in the docs (and imagebuf_test) that if you use the
variety of ImageBuf ctr where you give it a custom ImageCache, it's
important that the ImageBuf outlives the ImageCache, as it is a
non-owning reference.
Fixes 1987
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This new command takes two images as arguments, and appends all the
metadata (only) from the first image onto the second image's pixels
and metadata, producing a combined image.
This can be useful for users who have some (either non-OIIO-based or
just badly implemented) application that operates on image files but
fails to propagate the metadata, by having a simple way to transfer
metadata from one file to another:
bad-app in.exr -o out.exr
# oops, out.exr no longer has the crucial metadata from in.exr
oiiotool in.exr out.exr --pastemeta -o out2.exr
# Now out2.exr has the pixels of out.exr, but with all metadata
# from in.exr added to or overwriting that found in out.exr.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We have many classes that have methods
.attribute(name, intval)
.attribute(name, floatval)
.attribute(name, stringval)
.attribute(name, TypeDesc, anyval)
For ImageSpec.attribute() and OIIO.attribute(), the anyval was
restricted to only accept tuples. Fix these to bring them in line with
many other classes' attribute calls in which the 'anyval' really can
be just about anything -- tuples, lists, numpy arrays or other buffer
objects, or scalar values.
Fixes#2674
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Seems that cmake's FindGIF.cmake isn't good at discerning minor
releases (prior to GIF 5.x including more info in its headers), so
saying "find_package(GIF 4.1)" doesn't work the way we'd like, and
seems to fail to find a legit 4.1 installation. Go back to
find_package(GIF 4).
We seem to have broken this in PR 2681, and I'm worried that people
with gif_lib 4.1 will find themselves mysteriously lacking GIF support
in their OIIO builds. I may do an emergency tweak release of 2.2 just
to address this.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
When transferring blocks of pixels (for example, by ImageInput.read_image
or ImageOutput.write_image), "half" pixels ended up mangled into uint16,
the only type of the right bit width.
But NumPy has a numpy.float16 type! A little bit of extra hinting to
pybind11 makes this work as users probably expect.
Fixes#2689
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Don't copy from the data pointer if it's null, that indicates an error.
Related to issue 2688, but doesn't fully solve it.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
For a few rare edge cases of photometric interpretation, we fall back
on libtiff's ability to auto-translate to an RGBA image for us. It
only does this for the whole image at once, so we do it for the first
scanline we read (storing the whole image in a buffer, skipping the
conversion for subsequent scanlines -- and it can tell it's the
"first" scanline because the buffer isn't yet allocated).
The bug was that for these cases (such as a photometric YCbCr), if the
file was *multi-subimage*, we were not clearing the rgba buffer, so
upon reading the second subimage, it thought its buffer was already
filled, so it wasn't actually doing the read and convert, just using
whatever was already living in the buffer. Oops.
Also, I noticed that we should clear m_use_rgba_interface for each
subimage -- it was previously just setting it when it found those edge
cases. We never saw a symptom from this, but I realized it would be
problematic for a file where the first subimage was YCbCr and the
second subimage was regular RGB, it wouldn't know to fall back to our
usual code path.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
It's been endless trouble, because the nature of the test is malformed
images that fail in inconsistent ways. I can never seem to make a
stable set of reference output for it, especially for OpenEXR
development master, which is always adding new fixes or tinkering with
exceptions and error messages.
So just don't run that test in the automated CI.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Plain old bug in all cases:
--croptofull botched the subimage count by reserving space for results
based on the subimages in the input image, but then only producing
results for the first one if -a (or :allsubimages=1) was not used.
Output specific bugs:
There are some operations that happen ephemerally upon output, mostly
to account for output file formats that don't support some feature of
the top image (like action_croptofull if the output format doesn't
support differing display and data windows, or action_channels if
there are more channels than are supported by the format, or
action_colorconvert when --autocc is used).
In these cases, we were forgetting that those operations may tend to
drop all but the first subimages, if no -a or :allsubimages=1 is not
used. But that's NOT the behavior we want in this case, because for
output specifically, we never want to drop subimages. This is because
we want a simple format conversion like
oiiotool in.xyz -o out.pdq
to faithfully copy, not "narrow" to one subimage, even if some other
operations on their own will tend to drop all but the first subimage if
not instructed to preserve them all.
So the solution is that we need to add ":allsubimages=1" modifier to
these ephemerally called adjustments.
Periodic compare of analogous build scripts in OIIO & OSL and synchronize
to ensure that my current most favored idioms are in both.
The biggest change is that I split the checked_find_package into a separate
cmake file, for easier copying between projects.
Also fix some minor typos.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
HEIF is a lossy format, and it seems libheif 1.8 changed internally
in such a way that the bit-for-bit comparsions of the write-read
cycle we do in imageinout_test started failing. (Bummer on our Mac
CI, where the homebrew version of libheif updated automatically and
then started failing.)
The easy solution is that just for this test, write zeros to the image,
not 1.0 values (which were coming out 254, not the expected 255).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
In the case of a dependency not being found, we look for
src/build-scripts/build_{package}.bash and if it exists, suggest using
it. But the scripts were all lower case, and the packages were not, so
it was often not giving the hint! Also, in some cases, the package
might be named 'TIFF' but the script was 'build_libtiff.bash'.
So now, look for a build script with the package name straight, and
also all-upper and all-lower-case versions, and "lib<lower>".
Rename build_ocio.bash to build_opencolorio.bash, because it's the one
case where the script didn't match any of those obvious
transformations.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This new file contains the hashes of commits whose only function was
to reformat large amounts of code.
Ordinarily, this would make for confusing 'git blame' output, by
making old code written by author A appear to have a new modification
date and authorship by B, when in fact we know that B just did some
reformatting. This can make it especially hard to discern code history
after after commits that establish the use of "clang-format".
A new git feature (requiring git >= 2.23) allows pointing to a file
containing hashes of commits to be ignored by git blame:
git blame --ignore-revs-file .git-blame-ignore-revs
To avoid having to remember this all the time, consider adding this to
your git config:
git config blame.ignoreRevsFile .git-blame-ignore-revs
Note that git supports this feature but does not dictate the name of
the file. I'm just following the convention I'm seeing pop up in other
projects of calling it ".git-blame-ignore-revs".
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Sometimes the true minimum version of a dependency that we are able to
build against is lower than the minimum *recommended* version (perhaps
because some important feature was added in a subsequent release that
we would like to expose if available).
This patch allows checked_find_package to take two new optional
arguments: RECOMMEND_MIN is the minimum *recommended* version (but not
the true rejection minimum) and RECOMMEND_MIN_REASON is some text to
add to the error message if there is a succinct explanation for why we
recommend a higher version than the true minimum.
As an example, we will (currently) build against any OpenEXR 2.x. But
if it's not at least 2.2, we can't support DWAA and DWAB compression.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
These were so out of date and unhelpful, it has to be better to
have no instructions at all. Tear it down and make room for somebody
to do it right.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Oops, ParamValue.type ought to be a TypeDesc, just like in C++. In
the Python binding overhaul of 2.0, it seems that I made a mistake and
unintentionally changed it to appear to Python as a string (e.g.,
'int' rather than TypeDesc('int')).
Fixing it to be the way it was originally intended would involve a
breaking change and not appropriate until such things are allowed by a
2.x -> 3.0 transition, so we'll just have to live with it.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The thing about IOProxy is, once closed, it can't be reopened. So it's
important that if an IOProxy-based file is in the IC, its ImageInput
should not be reclaimed by closing it (and that might not help the
file handle count anyway, since the main use of an IOProxy under those
circumstances is probably to read an image from memory).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
It's important not to mess with a passed-in IOProxy* in Input::close(),
because that proxy may not still exist.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Add support for reading DPX files from any IOProxy
* Replace memcpy_s function with memcpy
* Fix IOProxy bug (not just for DPX) seeking from end
Signed-off-by: ra-mt <radu.arjocu@masstech.com>
I just like this better than the prior habit of putting it one PAST
the column width, which made the backslashes go in column 81, ick.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Last week, Vcpkg apparently had some flaw that recently caused it to
fail when building/installing libpng (among other things). It's been
fixed in the vcpkg main branch, but that hasn't propagated to the
vcpkg that's installed on the GitHub Actions CI runners, and that's
causing our Windows CI tests to fail.
Rather than wait out the fix, I'm just taking it into my own hands by
building all our dependencies from source on Windows. This means adding
bash scripts to download and build zlib, libpng, and libjpeg-turbo (and
maybe those will be helpful to others as well).
**Late breaking news** They beat me to it, vcpkg has been updated and
the GitHub CI Windows tests are passing again.
But I had already done this work! So I still would like to switch to
this methodology because:
* Sometimes it's faster to build ourselves rather than have vcpkg
build, since vcpkg always builds both release and debug, but we only
need the release builds of the dependencies.
* This makes it easier to customize the specific version of a
dependency, or particular build options, which are hard or
impossible to change for a vcpkg build.
* These same dependency-building scripts could be useful for people who
are not on Windows or not interested in using vcpkg.
Also some minor changes to the build_libtiff and build_openexr scripts
to synchronize some of the idioms I use in these scripts.
Rename workflow.yml to ci.yml to adhere to convention and also make it
so we can have multiple workflows in the future.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Fixes a compilation error, that appears on alpine linux.
/builds/lmarz/aports/testing/openimageio/src/oiio-Release-2.1.18.0/src/libOpenImageIO/exif.cpp: In function 'void OpenImageIO_v2_1::read_exif_tag(OpenImageIO_v2_1::ImageSpec&, const TIFFDirEntry*, OpenImageIO_v2_1::cspan<unsigned char>, bool, int, std::set<unsigned int>&, const OpenImageIO_v2_1::pvt::TagMap&)':
/builds/lmarz/aports/testing/openimageio/src/oiio-Release-2.1.18.0/src/libOpenImageIO/exif.cpp:789:20: error: comparison of integer expressions of different signedness: 'unsigned int' and 'OpenImageIO_v2_1::span<const unsigned char, -1>::index_type' {aka 'int'} [-Werror=sign-compare]
789 | if (offset >= buf.size()) {
|
When the texture cache reads scanlines in random order, the same pixels could
be read many times over. Now the file offset for scanlines is cached for quick
random access.
* OpenImageIO/imagebufalgo.h: remove OIIO_API from enums
* OpenImageIO/strutil.h: remove OIIO_API from inlined lower and upper functions
* libutil/plugin.cpp: cast return value of GetProcAddress to void*
* libutil/sysutil.cpp: define _POSIX_C_SOURCE for localtime_r declaration
* libutil/thread.cpp: cast t->native_handle() to HANDLE type
* ptex.imageio/ptexinput.cpp: use perr.c_str() for %s string format
* Fix some incorrect error calls in ImageBuf internals.
* Simplify because we know that ImageBuf::error() and warning(), with
just one string argument, doesn't need a format command.
* Make sure that if no message bubbles up, we say "unknown error".
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We weren't setting up the proper handler callbacks for libpng internal
errors to bubble up to the caller.
Tell libpng to turn off sRGB color profile check. There are many
false positives where embedded ICC profiles that don't match its
expectations (but that other people and apps consider fine) trigger
errors in libpng.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Another step toward switching to std::format notation.
The basic strategy is to make sure that all the classes that have
errorf (etc) methods using printf notation will also have errorfmt
(etc) methods using std::format notation. These will coexist for a
while to let people switch, before the printf ones will be marked
deprecated and then eventually removed.
In some cases where the methods aren't really part of the public
API, but only are used internally, we are already skipping to the
deprecation marking.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Policy change:
There are some headers that we just have to make part of the OIIO
distribution, but they aren't officially part of the supported OIIO
public API. We are moving them into an include/OpenImageIO/detail
subdirectory.
Anything in there is, therefore, allowed to be changed arbitrarily for
any minor version numbered release, including source back-incompatible
changes that would ordinarily require a change in major release
version if they were part of the public API. They nonetheless cannot
change in patch releases in ways that would break ABI or linkage.
Initially, this includes the imported headers from fmt and pugixml,
but in possible subsequent patches, I may move other headers there, or
split some headers to have declarations be public but certain inline
implementation details split into separate files in detail.
We were also able to move SHA1.h out of the public include area and
locally into libutil, since it should only be needed by SHA1.cpp.
It was also used by imagebufalgo_compiler.cpp, but didn't need to be,
so we changed the direct use into use of the hash.h wrapper.
Also add a build_pugixml.bash to make it easier for people to install
that dependency. We will eventually migrate to a fully external
pugixml, but right now old versions of OSL depend on it being part of
OIIO. We'll change on the OIIO side after another release.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We relied on it for years and are very thankful for its existence.
It has been optional for some time now, as we have come to use the fmt
library, which is more similar to the upcoming C++20 std::format.
Now it's time to excise tinyformat completely.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Bump Linux "latest deps" to use the newly release LibRaw 0.20.
* Document that we can successfully build with CMake 3.18.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Resizing an enormous image by a large scale factor could end up having
this allocation, which is xres*ntaps*sizeof(float), end up big enough
to overflow the stack. It's only done once per thread worker, so there
is no perf penalty to making it a heap allocation just in case it's
too big.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This is the key to writing a multi-subimage file from ImageBufs.
We previously could do that from C++, but not from Python.
Add a test to testsuite/python-imagebuf. Also rearrange the python in
that test a bit to cleave off a couple pieces into separate functions.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The `__getitem__` method taking an integer index correctly set the
return value policy so it returns in internal reference rather than
a copy, but the `__getitem__` method taking a string did not.
Also fix indentation.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
fmt is supposed to (if you include fmt/ostream.h) automatically
support any user type that defines an `operator<<()`. And so it did
through version 6.1.2.
Starting with 6.2.x, the presence of `operator bool()` throws it off,
particularly for our TypeDesc, which started rendering as "true" rather
than "uint8" or whatever. It seems that the fact that it can convert
to bool takes precedence, it formats as a bool, rather than notice that
there's a stream ouput option.
In this patch, we add a custom formatter for TypeDesc, which solves
the problem for `format("{}", typedesc)`, but unfortunately not for
`sprintf("%s", typedesc)`. I've filed an issue, but in any case, the
workaround to avoid incorrect formatting when using sprintf is to be
sure to use typedesc.c_str().
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Trying to force 3.7 when Homebrew has moved on is swimming upstream.
VFX Platform is also talking about moving to 3.8 next year anyway.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This is a little kooky maybe, but it was something I was playing a couple
years ago, finally inspired by Chris to clean it up and PR it.
This patch adds a "term" format which, instead of writing files,
outputs a preview image to a true color text terminal window.
Is any of this useful? I have no idea. But it's kinda fun.
The quick demo of this is:
oiiotool myfile.exr -o out.term
It will only do something useful if on a color terminal emulator with
24 bit color capability and Unicode. (I've tested it on the
gnome-terminal on Linux, and iTerm2 on Mac.)
The best quality results are on Mac if you are using iTerm2, it can
additionally do an actual pixel output (iTerm2 has as a nonstandard
feature the ability to send inline images to the terminal). This is
the "iterm2" method, but it's only expected to work on Mac with
iTerm2. It will automatically use this method as the default if you
are running iTerm2.
Next best is (thanks, Chris) is to display two pixels (vertically) in
each character cell, by outputting the Unicode "upper half block"
glyph (\u2508) with the foreground color set to the top pixel's color
and the background color set to the bottom pixel's color. This is the
default if you're not on a Mac running iTerm2. To demo (if you're on
have iTerm2 but want to see this method):
oiiotool myfile.exr -attrib term:method 24bit -o out.term
But some terminals and fonts don't support unicode, or don't display
this character, or display it with an ugly border. So you can also use
an alternate method:
oiiotool myfile.exr -attrib term:method 24bit-space -o out.term
where doesn't use the unicode trick but instead outputs a space (' ')
for each 2 vertical pixels, simply setting the background color to the
average of those two pixels.
There's also a "256color" method that just uses the 6x6x6 color space
in the 256 color palette -- which looks horrible -- and an
experimental "dither" which does a half-assed Floyd-Steinberg
dithering, horizontally only, and frankly is not an improvement unless
you squint really hard.
In all cases, the image will automatically be resized to fit in the
terminal, as well as converted to sRGB so it looks kinda ok.
Along the way, I added a new ImageOutput::supports("procedural")
query. If the existing input supports("procedural") query says
whether the input image is being generated procedurally and doesn't
correspond to an existing file on disk being ead, then the output
supports("procedural") is the opposite -- is the output doing
something procedural that does not correspond to a file being written.
(The specific use is to use this to avoid oiiotool attempting to
rename the file to move it into its final position.)
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This is a step toward making sure we are habitually testing on ARM64.
It is incomplete at the moment, several individual tests fail and need
to be tracked down, so the test is marked as "allow_failures" so it
will never appear to fail.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
By using the explicit `cast<TypeDesc>()` instead of allowing the
template system to figure it out, we were losing the `const` detection
and making it fail to match templates. It used to work, but a recently
added change in pybind11's master branch caused this to finally fail.
Also change the "latest releases" to use pybind11 2.5. (Only semi-related,
2.5 was fine, it was post-2.5 where we broke.)
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Either -a or optional modifier :allsubimages=1 ought to have erased
an attribute from all subimages, but it was erased from only the first.
Add a test to verify that --attrib and --eraseattrib both apply
correctly to all subimages when requested.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We previously disabled tests for a format when its library was not
found, but didn't consider if the library was found but the
`ENABLE_FOO` variable was set to intentionally disable it.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
SIMD improvements for emulating 8-wide on 4-wide HW: for many methods
on 8-wide SIMD types when running on 4-wide hardware, it's better (as
revealed by benchmarks) to implement as two 4-wide ops rather than
falling back to a purely scalar path.
Add operator*(vfloat{4,6,16}, float). On SSE, it hardly matters, no
better than letting the float be promoted to vfloat. But NEON has
vec*float.
Add NEON implementation of blend, min, max, reduce_add.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
A few weeks back, I rearranged a bit of ci-build-and-test.bash to invoke
cmake directly instead of the makefile wrapper. But in the process, I
accidentally caused it to always build `--target install` instead of
using the optional target `$BUILDTARGET` which is how clang-format runs
were supposed to have been triggered.
The result was the 'clang-format' test did a full *build* instead of
running clang-format. This was both much more expensive, and also
pointless: we weren't testing clang-format during CI at all. Oops.
Thus, some recent checkouts weren't properly formatted. We also amend
the minor fixes that bring us into format compliance.
I also noticed that I can use apt-get to install OCIO on the Ubuntu
instances used by GH CI, so modify to only build OCIO from source if a
specific version is requested. This speeds up some of our CI test
cases.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
To help track down build issues with MSVS 2015, re-enable appveyor.
Also split GHA CI to test windows for both MSVS 2019 and 2017.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Doesn't trigger an error on other compilers, or on newer MSVS. But this
minor rearrangement makes VS2015 happy, so let's just go with it and
not look under that rug.
Partially addresses #2611
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Was browsing some NEON docs and took the opportunity to fill in
implementations of a few more simd methods.
I don't have a way to directly test this myself, but hopefully
simd_test.cpp has enough unit tests to uncover if I've used the wrong
intrinsics for any of this.
I'm hoping that soon GitHub will add a way to run CI jobs on ARM CPUs
for Mac, if nothing else, so we'll have a way to verify this
architecture.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
They already existed for float{4,8,16}, but were never implemented for
float3.
Add to simd_test.cpp both unit sanity checks and benchmarks for these
functions (including for the other classes where it already existed,
they were nonetheless not unit tested).
Also beef up simd_test by testing and benchmarking load/construcution
from scalar. Rename some test functions for clarity.
Along the way I realized a difference between scalar std::round and
our vector simd::round() in how they handle exact middle values
i+0.5. Because our round is more efficient (one instruction, whereas
std::round cannot be done in one op on x64, because "round midpoint
away from zero" is not a native hw rounding mode), put a big fat
comment warning about the difference.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This seems wrong. EXR standard metadata "worldToNDC" is exactly what it
says, not screen space. I'm not sure how we ended up with the historical
accident of reading "worldToNDC" from exr headers and auto-translating
it to "worldtoscreen".
The hope is that nearly no one is affected by any of this, since the
primary use case for these matrices was shadow maps, which were never
fully implemented in OIIO's TextureSystem anyway (they were last on
the list, and the VFX world switched to ray tracing before anybody got
angry enough at my procrastination to force me to implement it).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Added a line for vcpkg users to discover additional opencolorio features they can build with that tool. This will help avoid people trying to build manually to enable features available via that relatively obscure package manager. Looking in the vcpkg repo isn't as helpful for people unfamiliar with that package manager or how it is configured.
Test more images that have been part of the openexr-images suite for
a while. Three cases:
* New images that weren't in the suite when I first set up these tests.
* Images that were "broken" because of (originally) missing features
in our exr support, that over the years are now supported.
* Images I was unaware of or somehow fell through the cracks.
In the process, improve and uniformize a number of error messages in
oiiotool, iinfo, openexr reading.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This will make it much easier to investigate failures or update ref
output when failures only happen on CI -- any test that fail tests
will produce a downloadable "artifact" consisting of the outputs
of all the failed tests.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
In at least OpenEXR 2.4 and 2.5, is it quite broken!
But OPENEXR_VERSION_STRING seems ok, so use that.
Issue filed against OpenEXR, but I don't have time to fix it myself tonight.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
One-line utility function to open file, write a string, close file,
return if it succeeded. Analogous to the existing read_text_file().
Signed-off-by: Larry Gritz <lg@larrygritz.com>
On Fedora, Python must be run using version specific binaries, python2 or python3. Since we go to the trouble of finding python for the main library, we might as well use the discovered binary name and path.
Brief background: OpenEXR sorts channel names alphabetically. Most
formats dont allow channel naming at all, let alone forcing a
particular order contrary to the order the user tries to write it to
the file. OIIO promises that if there is RGB, they will be first and
in that order, next will be alpha if present, then depth, then
everything else.
So our exr reader will reshuffle channels into this canonical order
before presenting to the calling app. In addition to literal R, G, and
B being first, also obvious synonyms are accepted and understood to be
first ("Red", "Green", "Blue", and also "Y" in case it's just a
1-channel intensity and then maybe alpha).
A user noticed an interesting edge case: what if it's a normal map or
positional info, so the channel names are x, y, z? You want it in that
order (coincidentally alphabetical), duh. But the current heuristic
will reorder it as Y (mistaking for intensity), Z (mistaking for
depth), then X (a non-special name, so goes last). Ick!
The solution is that we need a more sophisticaed heuristic. This patch
changes the logic to first just sort by layer, then within each layer,
look to see if there are channels "x", and either "y" or "z", and if
so, then we can conclude that it means something spatial and that Y
isn't really intensity and Z isn't really depth for that particular
layer, so an alternate ordering is used. In all other cases, we
continue to use the old heuristic.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
OIIO::clamp had the property that if `val <= low`, it sets `low`.
But actually, when `val == low`, we want to use `val`, not `low`.
This may seem like the same thing, and it is for ordinary types. But
OSL uses this clamp template for "dual" types (implementing automatic
differentiation), and it causes a case where `val == low` but `val`
carries derivatives that we want preserved, not written over by the
0-derivatives of the fixed `low` value.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
SSE2 lacks an instruction for 32 bit SIMD int multiply. SSE4 has
it. We simulate it with a trick using unsigned multiply on SSE2. Chris
wondered if this alternate code path was adequately tested for
negative numbers, so I added a test to be sure.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Make sure the opname is always canonicalized
* Refactor to allow setup() to call skip_impl() to skip the impl and
copy instead.
* --ummip : make OiiotoolOp subclass (but note, it doesn't honor the
`:subimages=` modifier).
* --chnames : now OiiotoolOp and honors the `:subimages=` modifier.
* --chsum : now OiiotoolOp and honors the `:subimages=` modifier.
* --chappend : now OiiotoolOp and honors the `:subimages=` modifier.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Turns out no code changes were needed for gcc10!
* Also updated some dependency ranges in INSTALL.md.
* Make a new CI test "latest" -- using the latest supported release of
the toolchain and dependencies. That includes gcc10 in this case.
(This is different than the "bleeding edge" test, which tests against
the development master of many dependencies.)
* Make several test matrix entries (including bleeding edge, static
libs, and sanitizers) no longer test on every push. That makes
things take too long. They all still run nightly, and if you're
pushing a branch with the name "RB", "release", or "master", so they
will catch problems before releases. Also, all tests run any time
the branch name is "gh", so if you are preparing a patch
specifically to make changes to CI, make sure to do that. We will
probably continue to fine-tune exatly when we run all tests, versus
the common configurations.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
In the changes of PR #2584, set_and_check was the wrong macro to use
when the path might be empty -- because it is an error if the path
doesn't exist. The "check" part bites us. So just use a regular "set".
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* The sanitizer test was using wrong setup script -- we switched to
using a container, but didn't change the dependency install script
accordingly.
* Mac seems slower and less reliable on GH, so switch the clang-format
test to run on Linux.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Build "bleeding edge" against fmt master
* Tests that build against OpenEXR 2.4.0, bump to 2.4.1.
* The "sanitizer" test, use ane aswf/ci-osl:2020 container. This is a
slow test, so may as well speed it up by using the container with
pre-installed dependencies.
* Modify ci-build-and-test.bash to use straight cmake build and install,
not the makefile wrapper.
* No need for atomic lib check on Apple
Signed-off-by: Larry Gritz <lg@larrygritz.com>
oiiotool dates from pre-C++11, and has grown by accretion. As a
result, the individual actions are implemented using multiple
different idioms, and there is a lot of complex boilerplate.
A recent PR added subimage selection widely to all commands that use
the idiom of an implementation that inherits from the OiiotoolOp
class. The convinced me that we want to migrate as many actions as
possible to this idiom. I also realized that the current use of
OiiotoolOp could be greatly simplified by using lambdas in most
cases, which weren't available in the pre-C++11 ancient times when
oiiotool was first being developed.
This patch:
* Has the OiiotoolOp base class contain (and accept in its ctr) a
std::function for the meat of the impl() and setup() methods.
* If init() and setup() are the only bits that need customization for
a particular action, they can just be passed as lambdas rather than
needing to subclass OiiotoolOp.
* This eliminates a huge amount of per-action boilerplate, many helper
subclasses and macros.
* In front of every action implemetation (however it is done), I
inserted a comment with the command line argument (e.g.,
`--colormap`) which is an aid for me to quickly find the
implementation in this huge file.
This patch DOES NOT change any non-OiiotoolOp action idioms to
OiiotoolOp. That will come later, first I wanted to make these
improvements.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
For a large number of oiiotool commands (anything whose implementation
inherits from the OiiotoolOp class), allow the optional modifier
`:subimages=...` to control how the operation applies to different
subimages (when dealing with multi-subimage images).
Of course, previously, the default is that most commands operate only
on the first subimage, discarding the others. The global `-a` (and for
many commands the optional `:allsubimages=1` modifier) cause all
subimages to be preserved and the operation to be performed on all of
them.
This new `subimages=...` allows more fine-grained control by
specifying which subimages will be active, as a comma-separated list
of either numeric indices (starting at 0) or subimage names. The
special string "all" means all subimages (thus `:subimages=all` is
equivalent to `:allsubimages=1`). Any subimage number or name preceded
by a "-" excludes that subimage from the active list. The operation
is performed on only on the active subimages, with excluded subimages
merely being copied from the first input image. The presence of any
`subimages` modifier implies that all subimages should be preserved.
For example, if a.exr is a "multi-part" OpenEXR image (what OIIO calls
multi-image):
# color convert just the first subimage, discard the rest
oiiotool a.exr --colorconvert linear sRGB -o out.exr
# color convert and output all subimages: -a
oiiotool -a a.exr --colorconvert linear sRGB -o out.exr
# color convert just the subimage named "beauty", but output them all
oiiotool a.exr --colorconvert:subimages=beauty linear sRGB -o out.exr
# color convert all EXCEPT the subimage named "normal"
oiiotool a.exr --colorconvert:subimages=-normal linear sRGB -o out.exr
# color convert subimages 1, 2, and 5, leave the others unmodified
oiiotool a.exr --colorconvert:subimages=1,3,5 linear sRGB -o out.exr
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Seems that the default version of the 'breathe' sphinx plugin is broken on
RTD.
* Change minimum breathe version to the specific one that is the most
recent release that I verified runs correctly on our docs.
* Remove unused and rusting sphinx extensions.
* Remove obsolete Doxyfile config settings
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Have our CI "bleeding edge" test libraw master
* Update Fuji reference image and other reference output related to changes
in libraw.
* Update install instructions to indicate LibRaw 0.20 support
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The existing `oiiotool --subimage <n>` replaces the top multi-subimage
image with a single specified subimage.
This patch adds an optional modifier, so that `--subimage:delete=1`
performs the "inverse" operation: removing just the one specified
subimage from the top multi-subimage image.
One way this could be useful is if you want to perform some operation
on just one subimage, but sill leave the others intact. For example,
oiiotool many.exr -subimage 0 -colorconvert foo bar many.exr -subimage:delete=1 0 -siappend -o new.exr
This command loads a multi-image file, extracts just the first image,
applies a color transformation, then loads the file again and deletes
the first image, then appends the two pieces, yielding a new
multi-image file whose first image is the color corrected first
subimage, then unaltered copies of all the other subimages.
Upon adding a test case to testsuite/oiiotool, I noticed that the
existing tests for subimage operations did not have reference output
images, and thus were not being tested after all! Fixed that now.
(Those commands were all working just fine anyway, just didn't have
their tests quite enabled.)
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Codecs which set the CODEC_CAP_DELAY capability led to infinite
loops during reading (see #2571). Once AVERROR_EOF is returned,
no more packets are going to be received, regardless of internal
buffering. break out of the loop in that case.
I've verified that this resolves the livelock following decoding
of the first packet in the Notcurses AVI test case which
illustrated this problem. I've also verified that said AVI
advertises the CODEC_CAP_DELAY attribute, matching the proposed
fix. In addition, I verified that OIIO continues to decode all
my other test cases without trouble. I hand-inspected the media,
and verified that the frame counts were precisely correct.
It is legal to pass NULL/0 into the decoder regardless of
CODEC_CAP_DELAY, and thus this whole conditional can really
be eliminated (I have no mention of it in my FFmpeg wrappers).
I haven't bothered to do that here, but it's a thought for the
future. Closes#2571.
Nick Black points out that in many places we have used "writeable", which
is (depending on which dictionary you check) is either a less commonly
used variant of "writable", or just an outright misspelling.
Even charitably interpreting it as an alternate spelling, let's standardize
on the more common variant.
There's only one API function where this comes into play, so to preserve
link compatibility, we'll have both ImageBuf::make_writable() and
make_writeable() as a deprecated synonym that will eventually disappear.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We were getting warnings from newer cmake versions in cases where we
had modules named FindFoo.cmake, but the "found" variable was FOO_FOUND
(instead of matching the module/package name with Foo_FOUND).
Clean that up, as well as imposing a few more namespaced imports to
change to a more modern notation.
* Use the aswf docker images to make sure we have one test that is an
EXACT VFXPlatform 2019 test (all versions, and on CentOS image).
* The aswf docker also lets us test against g++ 4.8 on GHA, which I
had never quite figured out before. That lets us disable the old
Travis CI tests for now, since the only important use for them was
to run the "oldest supported version" tests.
* In the future, we may use aswf containers for more test matrix entries,
as they get fleshed out to support 2020 and future 2021. But for now,
it's just 2019.
* Set up nightly CI test.
* Make CI tests skip for any pushes or PRs that consist of changes only
to .md and .rst files (i.e., if only docs change, don't bother with
an expensive build and test).
I was investigating thread behavior while fixing a totally unrelated
bug, and I spotted a way that redundant I/O happens when the same tile
is needed by multiple threads at once. (And it really did happen, I
could see the extra reads in the stats.)
A small additional method to `unordered_map_current` allows us to
simplify the logic in `add_tile_to_cache` fully avoid the redundant
read.
Testing against our production benchmarks, this makes no measurable
difference to performance, even under heavy threading. I think that's
because we make sure that all our texture files are prepared with
'maketx', and thus are tiled, so even the presence of a few redundant
tile reads is essentially undetectable in a full render. However, for
somebody who is using untiled textures (such as jpegs), which will be
read as whole images, this might be a bigger performance gain because
those redundant reads are redundant reads of the entire file.
Both the JPEG and PNG readers, because the can only read scanlines in
strictly ascending order, have a section where if they need to backtrack,
they do so by closing the file and re-opening.
But in this close/open, any 'config' settings they were passed when they
opened the first time would get lost. And this could include the IOProxy
they were supposed to be using.
The config info needs to be remembered and applied again if the file
is closed and reopened internal to the reader's operations.
* libOpenImageIO: Added OCIO Role accessors to ColorConfig
* FindOpenColorIO.cmake: Added OpenColorIO_STATIC definition (required for Windows static builds)
* py_colorconfig: Updated ColorConfig python bindings and unit tests
The idea of this "bleeding edge" entry in the CI test matrix is to
really use the very latest toolchain, in order to spot problems long
before users will encounter them.
To that end, let's make sure we are building and testing the current
master of libtiff and OpenEXR.
Ugh, homebrew just updated some packages (including numpy!) to be
python 3.8-based, but the 'python' homebrew package is still 3.7.
Need to jump through some hoops to make it work.
Some IBA functions specified the wrong version of the function, thus
leaving out the explanation from the header that was associated with
the other form of the function.
Some groups had bad identifiers, apparently, and were dropping
pieces of text.
Typo in the intro, oops!
After a call to TIFFReadEXIFDirecory, we needed to call TIFFSetDirectory.
We were doing it if the call succeeded, but not if the call failed.
Also, just TIFFSetDirectory is enough, we never needed to close and reopen
the file again.
This had been asymptomatic all along, but some changes in the current
libtiff master made the problems apparent.
When compiling OIIO against a static version of boost where only a subset of boost libs have been built:
(eg, boost configured with --link=static --with-filesystem --with-system --with-thread --with-headers)
We want to disable the automatic MSVC linking, which will otherwise also try to link against unused and non-existant boost libraries (such as date_time).
I've been thinking about overhauling ArgParse for a couple years, and
did most of the work in December when I was home from work. Then sat
on it except for slow burn weekend tinkering, but finally got a chance
to put the final touches on and write the docs this week.
The bottom line is that this is a major rewrite of the APIs for
ArgParse to make them much richer, more flexible, and more closely
resembling those of python's argparse module.
Rather than trying to explain the syntax here, I simply will refer you
to the extensive inline documentation in argparse.h, as well as the
examples in the code base of where it's used.
It sill respects the old syntax, so this change is hoped to not alter
the behavior of any programs using the deprecated API.
Start by reading argparse.h, it's design review of the API that is the
most important for those inclined to weigh in.
* New `find_pv()` method wraps `find()` but returns a pointer rather than
an iterator, and nullptr if the attribute is not found.
* Fix `add_or_replace()` -- it was previously failing to "replace" if the
new attribute had a different type than the existing one.
It is possible for the constructor for filesystem::directory_iterator
to throw an exception. Switch those to the ones that take an error code
reference and do not throw exceptions.
Also fix what must be a bug (how did it ever compile?) at line 208,
where we tried to find out if the status() (!!!) was a diretory, rather
than the path. Huh?
Closes#2517
Sometimes people try to make a texture from a source image that has
"overscan" (that is, when the pixel data window goes outside the
bounds of the "display" window that is assumed to be the 0-1 texture
space). Thiago Ize in particular noticed that this works fine when
outputting the texture as exr, but not as the default TIFF-based
mipmapped textures.
Alas, the only way this can work is if the texture file format has a
way to represent fully general display windows, and TIFF does not. So
it should never have been allowed, but there was no error message, just
strange runtime behavior.
This patch:
1. Tightens up some of the tests for whether the user is requesting an
overscan texture, so that it is an error to use overscan and the
output file type doesn't support display windows (previously this was
only checked if the sequence of operations also required an initial
resize, which was not stringent enough).
2. Fixes the TIFF output which I now believe should never have had
`supports("displaywindow")` return true; it should always have been
false.
`set -ex` exists the whole script if any individual command fails. I
think this will help ensure that people know when these dependency
building scripts have errors, instead of (hypothesized) seeming to
execute fully but actually the installation of that dependency was not
complete.
Note that some of the warnings related to the dpx/cineon readers highlight suspicious code, but I've opted to retain the existing behavior. The offending variables to not appear to actually do anything, they are just carried around as metadata.
* Fix resolution unit not properly set in JPEG export
Bug description
-----------------
To give an indication of the physical dimensions (pixels per inch, pixels per mm,...) of the image we can set the "ResolutionUnit", "XResolution" and "YResolution" attributes in the image spec.
For instance for PNG and TIFF this works perfectly, for JPEG however the resolution unit is not properly written.
Root cause in the code
------------------------------
In jpegoutput.cpp in the open() function the "ResolutionUnit" attribute was read and set in the m_cinfo.density_unit entry.
However this was overwritten by the jpeg_set_defaults(&m_cinfo) call.
Fix explanation
-----------------
The resmeta_to_density() already contained the "XResolution" and "YResolution" code, but not yet the "ResolutionUnit" code.
So in this fix I moved that code into resmeta_to_density().
Also, for CI, some changes to the scripts that download and build some
dependencies, since CMake will give an error if you ask it to generate
configs tat have paths to within the source tree, as we were doing. Move
it out of the source tree.
The place where the TGAUTILS samples were stored seems to have disappeared
from the web. So put them in oiio-images. Adjusted the targa-tgautils test
for the new location.
Also took the opportunity to clean up the pnm test and simplify how it's
declared.
Seems that Homebrew has removed the field3d package, as its scons
build system needs discontinued python2.7 and other parts have fallen
into disrepair and are having trouble on modern compilers and cmake.
New cmake variables `REQUIRED_DEPS` and `OPTIONAL_DEPS`.
`REQUIRED_DEPS` is a list of individual package dependencies that are
ordinarily optional (meaning, if not found, the build continues,
possibly disabling some functionality or format support), and forcing
them to be required (build failure if not found). Note that "ALL"
means everying.
`OPTIONAL_DEPS` is a list of individual package dependencies that
makes them optional even if ordinarily (or via `REQUIRED_DEPS`) would
be treated as optional. **Extreme caution necessary!** If you take
something required and force it to be optional, there's no telling
what else in the build might break as a result.
A typical use case might be if you have automated builds and want to be
sure that if for some reason they can't find all the dependencies (even the
optional ones), you will definitely fail the build. But maybe you have some
exceptions. For example,
cmake ... -DREQUIRED_DEPS=ALL -DOPTIONAL_DEPS="DCMTK;R3DSDK"
That means that you want any missing package dependencies to cause the build
to fail, *except* for missing DCMTK and R3DSDK, which you don't care about.
Also did some cleanup to turn PNG into an optional dependency, including
graceful disabling of ICO if libpng is not found.
* Updated INSTALL instructions.
- Include some of the various package manager that can be used to manage dependencies libraries.
- Removed some missing links in the instructions.
- Added vcpkg method for Windows build.
Signed-off-by: Mei Chu <meimchu@gmail.com>
* -INSTALL: Updated documentation per request. Now each package manager got a small section of its own that can be elaborated on in the future if necessary.
Signed-off-by: Mei Chu <meimchu@gmail.com>
The way we downloaded header-only library was a little janky inasmuch as it
did not actually "build" pybind11, which although there is nothing to compile
per se for this header-only project, nonetheless that's what would generate
the cmake config files defining the targets to import. And the presence of our
FindPybind11.cmake module inhibited cmake finding the config files even when
present elsewhere.
This patch:
* No longer automatically downloads pybind11 as part of our cmake config step.
* Eliminates FindPybind11.cmake module in favor of always expecting a
pybind11Config.cmake as part of a true pybind11 install.
* Correctly uses the imported pybind11::pybind11 target instead of
futzing with ${PYBIND11_INCLUDE_DIR}.
* Supplies a new src/build-scripts/build_pybind11.bash script that will do the
download and "build" of pybind11 (into the ext/ area by default) for those
users who need it and don't have it on their systems.
Previously, --autotrim when applied to a multi-subimage file would trim
all subimages to the nonzero region of just the first subimage.
This fixes, now examining all subimages and trimming them all to the
union of their nonzero regions (as --trim always did).
Technically, a multi-part OpenEXR can trim each subimage ("part")
separately, each having a different data window. But in practice, many
reading applications (including RV and Nuke!) seem to have trouble
reading files like that. So we go with the union approch.
A variety of minor rewrites of certain math functions to ensure they
generate better machine code and auto-vectorize more cleanly.
* Lots of inline -> OIIO_FORCEINLINE
* Improve clamp used in fast_sin and cos
* fast_safe_pow improve code gen with OIIO_UNLIKELY
* Introduce `OIIO_FMATH_SIMD_FRIENDLY` to allow application switching
between implementations that give the best scalar performance versus
sacrificing scalar perf to have the best SIMD vectorization of loops
containing the fmath function. This is anticipated to be very rare,
and of course we strive to be simultaneously fastest on scalar & simd,
but we have one or two cases where such a tradeoff exists.
* Lots of additional fmath benchmarks to help us judge how good they are
compared to std functions.
* New safe_fmod which not only prevents division by zero but even in other
cases is much faster than std::fmod.
* New fast_neg is faster than `-float`, in cases where you are ok with
-(0.0f) being 0.0f instead of actual floating point -0.0f. If you have
no idea what I'm talking about or why it matters, you definitely will
like this function!
* Most of these improvements were backported from OSL, made by Alex Wells,
Intel.
I was previously trying to use std::fma, but upon further investigation
it was turning into a single vfmadd instruction a lot less than I thought,
in many cases ending up as a function call (which itself was extra slow
when emulating fma on hardware that lacked it).
The better strategy seems to be just saying `a*b+c`, which on gcc &
icc compilers automatically turns into vfmadd when available on the
hardware, and adding a clang-specific pragma ensures this behavior for
clang as well.
Thanks to Alex Wells of Intel for pointing this out to me (in OSL land).
Versioning embedded in the shared library .so name should reflect ABI
compatibility level. For supported release branches, we guarantee
back-compatibility of the ABI, and therefore 2-digit naming
(major.minor) is appropriate, as we have always done. Changes to the
patch (and tweak) are guaranteed to not break ABI.
On the other hand, development in the master branch (as well as alpha
and probably to be safe betas also) may have ABI changes at any time,
so it is not sufficient for apps to link against libOpenImageIO.2.2.so.
They really should end up with major.minor.patch in order to not cause
trouble for people working from tagged developer previews in the
master branch. In master, "patch" level changes in master may change
ABI, but "tweak" level changes will not.
I was previously testing for whether the openexr version was >= 2.2,
which is when the ImfFloatVectorAttribute.h header was introduced.
BUT... OpenEXR has two build systems, and the CMake one (which I use
to build OpenEXR for OIIO's CI tests) worked fine, but the
autotools-based one had a bug that inadvertently omitted installation
of this and some other headers. This problem with openexr's buid system
was fixed in 2.4.
So my preprocessor tests for >= 2.2 was not good enough. This patch
refines the tests so that it's looking for openexr >= 2.4 (when
everything was fixed), or openexr >= 2.2 AND the header is present
(meaning it was built with cmake rather than autotools).
There are API changes in libraw soon-to-be 0.20.x.
Also, write a simple build_libraw.bash that can be used to build a
custom libraw from scratch (including from a specific commit, to test).
The reason I ended up down this road is because we needed to be able
to read .cr3 files, which are not in the released libraw 0.19.5, but
has initial support in the master branch.
Ref image updates for new libraw.
Pass an optional IOProxy* to ImageBuf ctr for reading, supply via new
call set_write_ioproxy() to set one up for subsequent write().
This is much easier than fumbling around with a PTR attribute in the
configuration hints.
Add tests to imageinout_test to check that proxy reads and writes via
ImageBuf works for all the same file types as they did for ImageInput
and ImageOutput.
In the process, found some subtle bugs in the PNG proxy support, needs
to ensure it starts at the beginning when we open, or else multiple
open/close cycles (as will happen in some cases for ImageBuf) will
get all confused.
`concat(s,t)` is semantically equivalent to `sprintf("%s%s",s,t)`, but
concat carefully avoids copying any of the characters more than once,
and it performs at most one temporary allocation (typically it does
not allocate temp memory from the heap at all, if the sum of the
strings' lengths are not too long).
Added both a std::string-returning and a ustring-returning version.
Also reworked the implementation of `Strutil::repeat()` to similarly
perform at most one one temporary allocation (typically none) and
not perform any unnecessary copies.
Inspired by work Chris Kulla did in OSL.
More actual (unreported) buglets revealed by -Wextra pointing out function
parameters that were unused.
* Failed to properly swap endianness when decoding Canon camera info.
* ColorProcessor::apply methods were not using chanstride param.
* texture3d fixes: some deriv results dropped on the floor
* Get rid of unused parameters in some internal functions. Not a bug
per se, but could be misleading about how the functions operate.
Regroup embedded or derived-from projects by license type, with each
license text reproduced only once. I think this makes this file much
more readable, shorter, and easier to see at a glance exactly which
parts are covered by which compatible licenses.
I don't know why, but in the past couple days, all the Linux tests on
GitHub CI have been failing because they can't build OCIO. Travis can do
so. Something changed on the GH end, I can tell because if I re-submit
CI for commits that passed just a few days ago, now they fail, too.
It seems to only be a problem when using Ninja. Falling back to makefiles
for just the OCIO build step appear to work fine.
More exploring what -Wextra turns up.
Found several buglets!
* The string version of getsym didn't pass along report_error.
* ImageBuf::getchannel did not honor its wrap parameter.
* ImageSpec::erase_attribute did not honor its 'searchtype' parameter.
* IBA::reorient did not pass its nthreads params to its children.
* IBA::computePixelHashSHA1 did not honor its nthreads parameter.
* Get rid of useless parameter in internal helper function parse_elements.
Fool around with -Wextra (optional, off by default for now), see what
turns up.
No code changes in this batch -- just suppress warnings by commenting
out param names for functions where they are not used (often because
they are base class stubs and the like). For functions that are part
of the public API where we use Doxygen to extract the declarations, we
can't comment out the param names, so instead mark them with
OIIO_MAYBE_UNUSED (and tell Doxygen to make that word go away in the
docs).
We have used this for a long time to write animated gifs. Just updating
from its git repo.
Also noticed that we neglected to list it in the docs where we
acknowledge 3rd party embedded software and licenses. Sorry.
To make the non-Conan build match the Conan build (and I rather prefer
it this way, as it more closely matches the eventual layout of the
install area), instruct to put the executables in build/bin and
libraries in build/lib.
Not all the dependencies set up right. In particular, the Conan central
repository doesn't have all the dep packages, or in some cases doesn't
have new enough versions. Still contemplating how to address that. But
this is a step in the right direction and maybe somebody with more
Conan knowledge can take it across the finish line.
* Add GH CI test for Linux + clang
* Needed to port build_llvm.bash from OSL to get the right clang.
* Some cleanup of build_ocio.bash, including renaming varibles more
clearly and instructing the OCIO build to only make static libs if
requested.
* Also touched up the travis -- we have recently pared the travis tests
to just a few, so there's no perf-related reason to restrict some tests
to just PRs, may as well do them on every push.
* GH CI: install libsquish & qt5 to get a more thorough build a tests.
Like we do for robin-map or pybind11, auto-download the fmt headers
into ext. Remove embedded copies.
Disadvantages:
* One more build step, requires a download. Maybe additional things can
go wrong? (Mitigating: we already do this for pybind11 and robin-map,
with no complaints that I'm aware of.)
* This is sometimes a pain for people building behind firewalled networks
that don't have internet access. (Mitigating: but they had to get the
OIIO code from the net in the first place, so they could get fmt, too.)
Advantages:
* It's just bad form to fully embed other projects if you can help it.
Some distros have rules against it, and ASWF (if we ever go that route)
strongly discourages it.
* Makes it WAY easier for me to bump which version of fmt we prefer
to use -- change one line with the tag we want, rather than updating
several header files in their entirety.
* Allows users to use a system install of fmtlib, or to use a
particular/alternate version than what we had embedded.
* Easier for us to benchmark new releases of fmtlib.
* Tried to expand and clarify explanations of the constructors and
reset methods in particular, after an issue was filed with some
confusion.
* Added checks to ensure that for the ctrs and reset methods that take
an ImageSpec and allocate (or wrap) a buffer, if the spec doesn't
have enough information to do the allocation or know the size of the
wrapped memory (spec lacking nonzero width, height, depth,
nchannels, and a concrete data format), the buffer stays marked as
UNINITIALIZED.
* Added `ImageBuf::reset()` as a synonym for `clear()` and a reset
variety corresponding to the "wrapped app buffer" constructor.
This makes it so that every ctr has a corresponding reset method, and
vice versa. As I was clarifying the docs, I realized that this would
be better if fully symmetric. Because these are not virtual functions,
these changes should not break ABI compatibility and should be safe to
backport.
* In a few spots nearby I renamed NULL to nullptr, couldn't help myself.
For apps using atomic.h, on architectures where certain atomic operations
need libatomic, they'll need to know about it. The `${GCC_ATOMIC_LIBRARIES}`
should have been a "PUBLIC" designaton for target_link_libraries, not a
private one (since atomic.h, which calls them, are public).
This patch checks for libsquish already on the system, and if found,
uses it. If not found on the system, it falls back on the embedded
version in src/dds.imageio/squish. CMake option USE_EMBEDDED_LIBSQUISH
can be used to force use of the embedded version even if found on the
system.
Background: Our DDS support (by Leszek Godlewski in 2010, as a GSoC
student) relies on an embedded copy of libsquish (MIT license) for DXT
compression. Embedding was a reasonable choice at the time, but in
2020 it's easily installed an Ubuntu, Homebrew, Vcpkg. Furthermore,
it's also a Debian package and runs up against their rule that we
should never embed code that is available from other Debian packages
that can be used as dependencies.
This is all I want to do in this patch -- comply with Debian rules,
and also reduce OIIO build time by relying on a prebuilt libsquish if
found on the system. This won't break anybody's builds at this time if
they don't have libsquish already installed, and should be safe to
backport to 2.1 for Debian compliance.
But in the future, we might make it a purely external dependency --
expect it on the system, and if not found, simply disable DDS support
(like we do with many other optional dependency such as libwebp or
libheif).
Recent changes (2347) seem to rely on some pybind declarations that
are newer than the minimum version we checked for. Bump the pybind11
required version.
If no pybind11 was found on the system, it would correctly auto-download.
I botched the test where, if a pybind11 was found but was an insufficient
version, it would not auto-download as intended. This was just an incorrect
test of PYBIND11_INCLUDES when it should have tested PYBIND11_FOUND.
I also augmented the logic to print a more helpful message pybind11
is found but is insufficient version, and furthermore if it's found
in the ext/ area (the auto-downloaded one) but still too old, it will
remind you that it needs to be blown away and refreshed. You'll see
something like this in the cmake configure output:
-- Pybind11 was found in /Users/lg/code/oiio/oiio.lg/ext/pybind11/include
-- ... but was version 2.4.3 (minimum is 2.4.4)
-- Try removing ext/pybind11 and let me download a newer version.
Lots of changes, a combination of:
* DASSERT -> OIIO_DASSERT (better name with no namespace pollution,
same behavior as before)
* ASSERT -> OIIO_ASSERT (better name and better behavior -- no abort
when in release mode, but error still printed; does both error and
abort when in debug mode)
* Many cases of ASSERT -> OIIO_DASSERT, downgrading to checks that only
need to happen in debug mode.
* Removal of a few assertions that are just not necessary nor helpful
(e.g, return if x is null, then immediately pointlessly assert that
x is not null).
* Change a few assertions to just plain old error handling, no assertion
needed.
Make it a first class thing the API knows about, not the very weird back
door of passing via configuration hint.
For the sake of backward compatibility, the "old" method of passing
config attribute "oiio:ioproxy" with a PTR data will continue to be
supported for at least a couple major releases. But moving forward
for 2.2+ (which, remember, won't be a supported release for many
months), the preferred and documented way will be through these new
API calls.
It is still the case that only some format readers/writers support
IOProxies -- it is not only lots of work to extend support to more
formats (volunteers?) but since we rely on 3rd party libraries for
some of our important formats, if they don't have support for some way
to supply custom low level I/O function overrides, it would be very
hard to have a way for OIIO client applications to use proxies for
those formats. So callers should always check supports("ioproxy") on
the format before relying on it.
Background: unordered_concurrent_map<> is a thread-safe map that is
parameterized by the type of the underlying "shard" maps. The default
underlying map type is std::unordered map. Though for a while, as used
by ImageCache, we have been using robin_map as the underlying map
type, having found it has superior performance.
PR 2314 improved performance even more by taking advantage of the fact
that robin_map has a find(key,hash) that lets you supply the
precomputed hash value (in situations where you already had it), thus
avoiding redundant hash computations.
BUT... we failed to notice that unordered_map doesn't have this kind
of find() variety, and so even though the u_c_m is templated on the
map type, this meant that it would no longer work with
std::unordered_map. That's especially painful since the underlying
type defaults to this.
The solution, in this patch, is to make a helper function
`find_with_hash(map, key, hash)` that uses the hash if the map type
has a find(key,hash) method, and otherwise falls back to the standard
find(key).
There is some template metaprogramming involved, and we add a couple
of handy idioms to platform.h (it sort of belongs there because the way
to express the idioms there depends on whether it's C++ 11 or 14+, and
platform.h is where we sort that kind of thing out).
Troy Sobotka pointed out to me that the implementation of colorconvert
where unpremult=1 (i.e., unpremult, then transform, then premult) was
wrong: the unpremult operation was careful to preserve the color value
of pixels where alpha==0, but the premult we did at the end was just a
straight multiply of the color by the alpha. This would have the
effect of crushing those "glow" (A==0, RGB > 0) pixels to black in
their round trip, whereas we want the original glow to be preserved.
So really, the kind of premult we want for a one-time conversion of
unpremultiplied to premultiplied is subtly different from the "re-premult"
that we want as part of a round-trip operation.
The fix in this patch is to add IBA::repremult() to do this modified
operation (multiply color by alpha, unless alpha is 0), and also that is
the correct operation to use in the unpremult-transform-repremult round
trip that we do as `colorconvert(unpremult=true)`.
Also noticed that the recent CMake overhaul inadvertently left
testsuite/oiiotool-color un-tested, so re-enable that (which also lets
us test this fix).
* Relax version compatibility specification to just matching major version.
* Specify targets cmake name correctly.
* Make sure install include dir is listed in the public include dirs of the
libraries.
* Some other cleanup and genericization.
I believe this corrects some deficiencies in the generated
OpenImageIOConfig.cmake that projects can use that exclusively (with
imported targets) and have no need for a copy of
FindOpenImageIO.cmake.
We used the assertion macros, but didn't include the header. I guess
coincidentally, all other files that included refcnt happened to have
included dassert first? But 3rd party code that wants to use refcnt
for some reason shouldn't have this pop up as an error
Something changed about homebrew versons, I guess, and the combination
of python + cmake on both GitHub CI and TravisCI broke OSX. Modifying
install_homebrew_deps.bash to be sure it upgraded to the latest
version of both packages seems to solve the problem.
Never reported, not on CI tests, but I've seen it happen. Only on MacOS, only
for builds with EMBEDPLUGINS=0, only for tests involving Field3D files.
I seem to be able to fix it (or at least make it asymptomatic when I
run tests) by forcing all the open texture file handles to close
before the DSOs are unloaded, for a few of the utilities.
To reduce thread contention on the IC/TS caches, change from unique
locks to reader-writer locks.
* unordered_map_concurrent: Change the shards from spin_lock to
spin_rw_lock. Most operations (insert, erase, holding an iterator,
and find, which returns an iterator) are unique (write) locks. But
the underappreciated retrieve() method is stateless and so can be
a reader lock!
* In imagecache.cpp, change a bunch of the find() calls and fiddling
with iterators into simpler calls to retrieve(), which should be
able to happen concurrently.
On a highly threaded machine (52 cores) this resulted in a 6% gain
(geometric mean) across a set of production render frames that we like
to test with, but that disguises what's really happening. Most of the
individual tests in that suite don't change at all (or vary within the
+/- ~1% we expect from timing noise and run-to-run variation. But a
handful of the tests sped up by 20-40%! So I think you are unlikely
to see any change for a few threads, or even many threads for most
situations. But for highly threaded renderings where the access pattern
really results in the cache locks as a major rendering bottleneck,
this should help a lot.
There is still one use left, in ImageCache. I'm not sure how to get rid
of that yet, because the C++11 std version only works for global or static
variables, but we are using it in a class. We'll come back to that later.
In the mean time, this gets rid of all other uses.
One thing this does is fix a link error I get on MacOS when building
with EMBEDPLUGINS=0.
The ffmpeg we were using was from a repo whose owner just took it offline.
This was making our TravisCI Linux builds fail. But it's well tested in
the GitHub CI matrix, so just remove it here.
Also... remove another Linux test from Travis that seems redundant, as
it is nearly an identical configuration to one in the GH test matrix.
And for that matter, we don't need to do the clang-format check on
both testing platforms, so remove it from Travis, too. Now we only use
Travis for the gcc 4.8 tests and Mac with Python 2.7 (I've had trouble
getting both of those working on GH CI).
The dataptr() function returns nullptr if the offset is out of proper
range. But the array_to_spec function calling dataptr did not check for
a nullptr return, ended up segfaulting.
This can happen with JPEG files that have certain malformed (we think)
sections of their camera maker notes.
Oh boy, I discovered that something in the new fmt make Cuda unhappy,
and this in turn bites OSL. So I made it so that to really get out of
such a jam, one could `#define OIIO_USE_FMT 0` before including
strutil.h, and then the fmt headers will not be included nor called,
falling back on tinyformat where possible (for sprintf, to_string) and
for std::format style simply fail to format. This is sufficient for
now, we don't actually need the formatting to work from within Cuda at
the moment.
Also fixed some out of date comments while I was there.
We previously exposed the PV constructors from just a plain int, float,
or string, but there wasn't an easy way to construct from arbitrary data
like there is in C++.
This updated version of fmt has its floating point conversions locale-independent as we wanted.
Switch completely to relying on fmtlib instead of tinyformat by default, even for the printf-style formatting and to_string() implementation. (It's still controlled by a preprocessor symbol so we can switch back if there's an emergency bug.)
Be more careful about %d vs %u string formatting: tinyformat ignored the distinction, fmt does not.
Summary of performance results:
* float->string (printf style and to_string) is almost 10x faster than before, and now is ~3x faster than C++ std::sprintf to buffer.
* int->string is as fast or faster than std::sprintf to a buffer (despite also doing allocation!).
* A 6-item conversion to string of mixed int/float/string values is 5.6x faster than before when using printf-style, 1.5x faster for fmt-style, and those two now approximately match and are each 2.7x faster than std::sprintf to pre-allocated buffer.
The default is 0, old behavior. If you set to nonzero, you are
promising the ImageCache that it will always be able to identify a
texture file's format based on the filename extension (e.g. ".exr"
always means it's an openexr file, ".tif" or "tx" for TIFF files, and
you will NEVER call a file ".tx" if it's really openexr, or if you do,
you are ok with it being unable to open it). When you enable it with
that promise, it can eliminate a few redundant file opens and header
reads, by not needing to verify file format types at certain stages of
processing.
I'm not sure I would recommend enabling this in general. But if you
are operating in a closed environment where you are sure that file
extensions will match their types (such as a studio with rigid naming
conventions, where all possible textures have been converted with
maketx), this can eliminate some unnecessary network traffic and file
server workload.
One test is to write an image file to disk, then write the same image
as a "file" to an in-memory buffer, and make sure resulting bit patterns
match exactly.
Every once in a while, it would spuriously fail. I'm pretty sure it's
because of metadata not matching exactly if the two writes happen on
opposite sides of a minute boundary!
This patch attempts to preempt that by setting "DateTime" metadata to
a fixed value, so that individual format writers don't, in its absence,
supply the "now" value, which could vary from run to run.
* Detect failed allocations when ImageBuf tries to allocate local memory
but there is not enough memory -- catch the exception, leave the IB
uninitialized, set an error in the IB. This at least allows oiiotool
to issue an understandable error message if --create or --pattern fail
because too high a resolution was specified.
* oiiotool --create and --pattern take a new optional parameter:
`:type=name` that overrides the default behavior of allocating all
internal buffers as float. An important application of this might be
if you are creating a huge image that would not usually fit in memory
as floats, might as a half or uint8. For example,
oiiotool -pattern checker 64000x32000 3 -o big.exr
writes a really big checkerboard, but will fail on a machine that can't
muster an allocation of 23 GB RAM. However,
oiiotool -pattern:type=uint8 checker 64000x32000 3 -o big.exr
will create and write the same checkerboard using only 6 GB RAM.
* Noticed that oiiotool -o is the only command that used `:datatype=`
as an optional type override. All others use `type=`, so change to
that. (The old one is silently accepted as well.)
Difference between padded scanline width and number of pixel values was
not properly considered or zeroed out. This was (nondeterministically)
botching occasional CI tests.
The new preferred assertion macro, OIIO_ASSERT, tests a condition, and
if not true, prints an error message and aborts for debug builds. For
release builds, only tests and reports the error, but does not abort.
OIIO_DASSERT is identical for debug buils, but does nothing at all for
release builds (not even test the condition).
ASSERT and DASSERT are the old macros, but because of namespace
pollution and less desirable behavior, we consider them deprecated
want to phase out their use. (In a separate review we'll convert calls
to these.)
Moving forward, then, the attitude about assertions should be:
* OIIO_ASSERT for when you want an error message in all cases, and also
abort for debug builds (but not release builds).
* OIIO_DASSERT when you want error/abort for debug builds, and nothing
for release builds.
* ASSERT/DASSERT - old names and behavior, phase out over time.
* assert() - roughly equivalent to OIIO_DASSERT, ok to use as well,
especially if you are desiring to have a particular header or module
free of dependency on dassert.h.
* Overall: remember that you should only use assertions to test for
"impossible" conditions that indicate a programming error. Never use
them in place of error handling, or to detect bad input.
We already had strip, which trims chars from both sides. Sometimes you
just want to trip the left or right.
Also add some missing unit tests. And saw a > vs >= improvement in
in string_view::substr(). (I don't think it was wrong before, just
failed to take an early out for an empty substring.)
We relied on vsnprintf underneath, which is locale dependent. This is
important for OSL, which generates code that uses this (it will botch
the formatting of floating point values in OSL printf, format, error,
and other similar calls, if the user has their system set to a locale
with commas as decimal separators).
OSX, FreeBSD, and Windows have vsnprintf_l which allows passing a locale
(for which we use the "C" locale). But Linux does not.
Instead, we use the public domain stb_sprintf, which is locale independent,
for its vsnprintf implementation.
Also found a spot in rla output that could botch certain metadata when
in the wrong locale.
Each texture query may perform many samples, and each sample may fetch a
tile (or more). EVERY time it was making sure that the 'used' atomic int
was set to 1. That's potentially a lot of contention.
Idea: Only set it once for all those samples. We really only need to mark
it 'used' per texture call, not per sample.
So I augmented find_tile() with an optional parameter mark_same_tile_used
(defaulting to true), and the effect is that if it happens to be the
most recently used tile in the microcache, it will only call use() if
mark_same_tile_used is true. So then, when looping over sampling, it
passes sample==1 for that param, having the effect that for the first
sample only (or of course if any other tile is needed than the last one)
will it mark used and diddle the atomic variable.
In our production scene benchmarks at SPI, when run in a highly
threaded configuration with 52 cores, I'm seeing a 2% average speedup
in overall rendering time (geometric mean of many tests; on a few
tests I'm seeing up to 20% gain, which is crazy, but many other scenes
merely break even).
* Since it uses an ostrstream internally, make sure it is set up to be
locale-independent (floats will always render with '.' decimal).
* Add a version with a parameter that allows you to set the number of
items joined to something other than the length of the sequence,
truncating or padding with default values as needed.
And I illustrate its use in make_texture.
* Fix places that could overflow int32.
* Speed up read/write by about 40%, mostly by avoiding the pointless
zeroing out of allocated memory in cerain std::vector temporary buffers.
* Replace fgetpos/fsetpos with Filesystem::ftell/fseek
A while back we added errorf() in various places, which henceforth
will be the place to do printf-style formatting, trying to migrate to
a point where the "non-f" error() will eventually follow std::format
(python-like) formatting codes. But for now, error() and errorf() do
the same thing, we will only change the meaning of error() down the
road when it's safe.
This patch just rounds up a lot more places in the internals where
we use error() and changed to errorf().
Ultimately this was to simplify the header reading and make sure that an
error message is returned to make it easier to understand where things
failed.
We were pegged to a fairly old commit. Also, modern versions have
changed their directory layout a bit (tsl/... -> include/tsl/..., so
compensate for that.
These utility functions are natively 64 bits, so you don't need to
worry about what nonstandard functions to call. Prefer Filesystem::fseek
and ftell over std to guarantee safety for enormous files.
Appveyor seems to have changed something on their end that is breaking
our vcpkg installs of dependencies. Now that I have GitHub CI working,
I don't feel like trying to figure out how to fix Appveyor. Let's just
retire it, since GH CI is testing the build on Windows adequately.
A number of calls in the Filesystem API were using std::string instead of
the more efficient string_view. This was leading to extra conversions in a few places in the code.
Also mask the usage of boost::system::error_code behind an alias so we can (one day) move to std::filesystem more easily.
Fix some internal roundtrip between string and path representations in the Filesystem module.
Two subtle bugs in Strutil::parse_string:
1. Toggling whether we are in an escape code based on the first
character in the original string, rather than the next character to
scan. Big oops!
2. Not realizing that a second backslash in a row should *clear* the
flag (i.e. `\\` does not leave you in an escaped state).
I beefed up the unit tests, looking at the example I added should
explain the edge case where this crops up -- a double-quoted string
with embedded \\\" within it, if you catch my drift. Just look at the
unit test, it explains better than I can do here. It's very hard to
talk and write (and reason correctly) about escaped strings with
embedded quotes!
* Simplify implementation of Filesystem wrappers
Use the version of boost::filesystem calls that return an error_code instead of catching
exceptions to make the OIIO noexcept.
Also remove extraneous calls to exists that were doubling the number of stat syscalls
* Reduce number of syscalls when opening files
Defer checking for filename existence when scanning for udim tokens in the filename
Avoid double-checking for file existence when measuring the file size, since the Filesystem wrapper already returns 0 on error
It's a wash for small thread counts, but for highly threaded renders,
I'm seeing 5% speedups to overall rendering across a large suite of
production scenes on a 52 core render.
The IBA::compare for deep images A and B would loop over pixels in the
union of A's and B's ROIs (so far, so good), then for each pixel would
loop from 0 to A's number of samples in that pixel. Oops! That means
that if A had fewer samples than B (or if A had no samples but B had
samples) in that pixel, it would not notice that the images were
different.
We've run into bugs with DWAA/DWAB compression of OpenEXR images in
the specific case of 1-channel images where tile size is either < 16
or not a power of 2. There was already special logic in the exr
output module that notices this problem case and switches to zip
compression.
However, in the case of outputting MIPmaps, the top level may satisfy
the conditions under which DWA compression works fine (power of 2,
tile size >= 16). But in the course of outputting the MIP levels, we
will eventually reach a tile size < 16, and the OpenEXR format doesn't
allow us to select a different tile size per MIP level (it's just
stored once in the header for the whole file).
So add some logic to our MIPmap generation (maketx, etc), so that
single channel exr texture maps preemptively switch from dwaa/dwab to
something safe. This still only affects 1-channel images, multiple
channels don't appear to tickle the bug.
I couldn't help myself, along the way I did a little refactoring,
coalescing several "if the texture is an exr..." clauses together
under one conditional, and also modernizing an initialization loop
and fixing a typo.
On OSX, Plugin::plugin_extension() returned "dylib". But actually, OSX
distinguishes between true shared libraries ("dylib") and what they
call "bundles" -- the latter is what we think of as a runtime-loadable
plugin and is traditionally given the extension "so". This choice
also corresponds to what is chosen by new CMake conventions where we
use "MODULE" as the argument to add_library() when making plugins.
I apologize for not understanding this distinction previously.
Allow overrides of the build_dir and dist_dir.
For SPI, build debug & release libs in same place for better staging.
Also some other local build updates that won't affect anyone else.
* Split python utilities into its own file
* Move shrinking contents of install.cmake into compiler.cmake
* cmake refactoring: mostly moving options close to where they are used
* Comment the heck out of compiler.cmake
* Tweak build_openexr.bash - fix some 2.2 probs
* The Python bindings of both ociotransform and ociolook mixed up the
order of inverse and unpremult params. Oh no!
* Python bindings of both ociotransform and ociolook sometimes called
"inverse" "invert", which would be bad for named parameter binding.
* Underlying C++ IBA::ociofiletransform issued an error message when
no color config was found. This is unnecessary, since the config is
only used to make a processor, no color spaces need to be known
since the LUT comes from the file.
The details are below, but the summary of what this enables is that if
you have rendered "panels" yielding images A, B, ... Z that all have
nonoverlapping data windows correctly stored in each file, then you
can stitch them together into the correct composite image with:
oiiotool A.exr B.exr ... Z.exr -paste:all=1:mergeroi=1 +0+0 -o out.exr
This works for deep files, too!
The individual changes:
* New `oiiotool --paste` optional modifiers:
- `:all=1` Paste the entire stack of images. Normally, only the top
two image on the stack are involved (`fg bg --paste`), but this
allows `fg1 fg2 ... bg --paste:all=1`, which means "paste all of
the fg images on top of bg").
- `:mergeroi=1` Normally, the foreground image will "clip" against the
extent of the background image's data window. This option makes the
result of paste be sized to the *union* of the data windows fg and
bg images.
* Extend IBA::paste() and `oiiotool --paste` to work with deep images.
* New `oiiotool --originoffset` offsets the data window origin
relative to its previous value (versus the existing `--origin` which
sets it to a specific value regardless of what it previously was).
* Change semantics of IBA::paste() and oiiotool --paste.
When pasting image F on top of B at offset (x,y,z), we now clarify
that pixel (0,0,0) of F will replace (x,y,z) in B. As before, only
pixels within F's data window will be copied.
Before, we were a little unclear and in fact the implementation was
that the data origin of S would be the pixel that replaced (x,y,z) in
B. These two interpretations are identical for ordinary images whose
data origin is (0,0). But for "crops" or "shrink-wrapped" images, the
old behavior was confusing since especially in the shrink-wrap case it
depended on the exact content of S, whereas intuitively we want the
paste position to mean the same relative offset regardless of the
pixel contents of F.
* Enable parallelizing of IBA::paste, making it much faster.
* oiiotool internals: give adjust_geometry an allow_size parameter,
have it handle the warnings when sizes are specified but shouldn't
be.
* Move the cut and paste tests from testsuite/oiiotool to
testsuite/oiiotool-copy.
* CMake revision: Allow disabling of individual formats and utils
`ENABLE_<name>=0`, as either a CMake variable or an environment
variable (or its antonym `DISABLE_<name>=1`), will turn off support
for the named format or command line utility. This is even the case if
the file-reading library is found, or if no external library is
required.
For example,
make ENABLE_DPX=0 ENABLE_cineon=0 ENABLE_maketx=0
A compelling use case is for a package that has OIIO as a required
dependency, but only needs a subset of its features. Building a fresh
copy of OIIO is one step in a CI test and we would like to minimize
the time spent on this step, knowing that the CI tests themselves only
use a fraction of OIIOs's functionality.
Be careful, obviously if you disable enough some of the core formats
or components, you will have an OIIO that is nearly useless. The selective
disabling of components is really helpful for ephemeral builds such as
Along the way, we have some other refactoring:
* Rename macro oiio_install_targets to more generic install_targets.
* Move check_is_enabled to its own file, add ENABLE_foo synonyms and
DISABLE_foo antonyms, in addition to USE_FOO. I'm starting to lean
toward ENABLE_foo as the general switch name, the others may go away
eventually.
* Move colors definitions to their own file.
* Overhaul build_openexr.bash for more options and standard nomenclature.
* Revise static compile defines
* Simplify Find cmake modules -- I now realize that many of the
PATH_SUFFIXES I was using were redundant, specifying suffixes aleady
appended by default by the find_*() commands.
* Make some warning suppression more targeted (push some global warning
suppression to local to modules).
* Simplify and deprecate OIIO_BUILD_CPP macros.
* Switch to cmake idioms for symbol visibility. No more HIDE_SYMBOLS
directive, hide by default. Expose standard CXX_VISIBILITY_PRESET
(default to "hidden"), VISIBILITY_INLINES_HIDDEN (default to ON).
* Experimental option USE_GENERATED_EXPORT_HEADER (defaulting to OFF
for now) uses CMake's ability to generate the export header rather
than using our hand-coded one. I haven't decided whether to switch
to using this or not. Will stew on it.
* Rename oiio_find_package -> checked_find_package for easier porting
between projects
* Fix syntax error in FindOpenImageIO.cmake.
* Purge old OPENIMAGEIO_ROOT_DIR for standard OpenImageIO_ROOT
* Clean up build_openexr.bash
`command --help` is a well-formed command that should have a "success"
error code of 0. Most of the command line utilities returned a failure
code. (Only oiiotool got it right.)
After a fresh build, sometimes a parallel test would fail because
certain tests tried to get ../oiiotool/src/... but that link is not
set up until oiiotool test runs. It is listed first, but sometimes
when doing parallel test runs (via CTEST_PARALLEL_LEVEL=...) the
timing can be just right to fail.
* More CMake refactoring and touch-ups
* Eschew directory-level include_directories and add_definitions in favor
of explicitly tying them to targets and using PRIVATE/PUBLIC.
* Use standard idiom CMAKE_CXX_STANDARD instead of custom USE_CPP.
* Remove last remnants of CSTD manual settings
* Use CMAKE_DEBUG_POSTFIX to add an optional suffix to debug libs
* For SPI, have the "spi" target build both debug and release libs, with
a "_debug" suffix for debug libs, and dist it all.
* CMake: change obsolete COMPILE_FLAGS to modern COMPILE_OPTIONS
* Use HOMEPAGE_URL, print CMake system name
* CMake cleanup with shared vs static libs
- Replace BUILDSTATIC with use of the standard BUILD_SHARED_LIBS.
- Eschew add_library(SHARED|STATIC) and prefer relying on the automatic
behavior of BUILD_SHARED_LIBS.
- Get rid of my fPIC option, rely on CMake smarts to do it automatically
for shared libs and modules.
- Add static libs CI test to GitHub actions
* Fix: Robust to freetype not being found
* Eliminate DEBUGMODE symbol -- unneeded
* Mark library targets' target_link_libraries and target_include_directories
with the correct PUBLIC and PRIVATE markers.
* Rely on cmake's transitive library dependencies to eliminate a lot of
explicit library link directives.
* Eliminate almost all CMAKE_BUILD_TYPE references -- it's not
considered safe for generators (such as MSVS and Xcode) that
configure/generate once and then can build multiple configurations
selectable at build time. It's still not perfect, but it's close.
* Eliminate platform.cmake -- unnecessary. Most of it worked around
cmake limitations from long ago, or defined symbols that we didn't
seem to actually need (and in some cases would have been
ill-advised). The one compiler option that might be important, and a
few helpful status messages, are moved to compiler.cmake and the
rest is thrown off the side of the boat.
* Add a sanitizer run to GH test matrix, remove from Travis.
* Properly hint search for pybind11 and robinmap
In a C++11 world, we can define these using int64_t and uint64_t:
simple, standard, identical size on all platforms.
The prior convoluted definitions were from an earlier time when we
couldn't count on these being present.
This seems to have gone unnoticed by the compilers except for a very
recent update of MSVS 2019...
But the opaque class ImageBufImpl we deine publicly in imagebuf.h
in order to use PIMPL idiom has a problem where the std::unique_ptr
doesn't know what deleter to use for an incompletely declared class.
Do some magic to make a public well-defined base class with a deleter,
keeping the full ImageBufImpl private to imagebuf.cpp.
Must take steps to install a newer cmake for the one Travis matrix case
where the old "trusty" image didn't contain a new enough one. In all
the other CI cases, a new enough cmake was already present.
Get rid of legacy cmake policy directives.
With newer cmake, change many add_definitions to more proper
add_compile_options. Get rid of unnecessary cmake definitions.
Refactor Makefile wrapper with better/standard target names: install,
build, config, etc. And don't replicate code for make vs ninja --
instead, use cmake --build.
Now that CMake 3.12 is the minimum, we can count on any FindFoo.cmake
having its searches automatically include hints for having defined
`-DFoo_ROOT=...` or a defined environment variable `Foo_ROOT`, without
our needing to separately define or handle them. This leads to a huge
refactor ond simplification, as well as a normalization of this as the
one true naming convention for dependency location hints (versus a
hodgepodge of FOO_ROOT_DIR, FOOHOME, FOO_PATH, FOO_HOME, and more).
A lot of the repetitive logic of finding dependencies has been
coalesced in new oiio_find_package macro, which wraps the build-in
find_package() with some nicities, including color coded clear status
messages about which deps were found, which optional ones were not,
and which the user disabled.
Along the way, we bump the Travis macosx setup to xcode11.
The original code was always asking ffmpeg to give it RGB uint8 frames.
This patch modifies it so that it asks for RGBA when alpha is present,
1-channel image when it's a greyscale movie, and retrieve as uint16
when >8 bits per channel are stored in the file.
If you are, to pick a hypothetical example at random, trying to
convert a 10 bit rgba quicktime of an fx element into individual
frames to save out as TIFF files, this will fully preserve the 10 bits
and alpha channel, whereas before you would just get back a crappy 8
bit rgb.
Also updated where the SPI build gets its ffmpeg, to pick up a newer
version.
It's not entirely working yet -- I can get a more or less full build,
but am having trouble with tests. But I'd like to merge the
refactoring and progress I've made so far.
The newly released pybind11 v2.4.0 and v2.4.1 are actually broken for
C++11 (accidentally used constexpr in a place where it's only valid
for C++14). Communication with the pybind11 team quickly resulted in a
v2.4.2 release that fixes the problem.
This patch adds logic to our build scripts so that if one of the two
broken versions are found on the system and we're building for C++11,
it falls back to our "download our own copy" logic. The new default
for downloading our own is updated to 2.3.0 (including updating some
ref output).
Several small, subtle, interacting issues were making it particularly
difficult for oiiotool to process frames within movie files (and,
less noticeably, other multi-subimage files). The fixes:
* The ffmpeg reader had no valid_file() method, which meant it fell
back to the default of doing a full open() and looking for success,
which is expensive in this case. Replace it with the cheap and naive
test of just ensuring that the file exists and its name has one of
the proper file extensions.
* ImageCache::read_untiled(), when reading an untiled image as one
single tile, closed the file immediately thereafter, which seems
like a prudent thing to do because once you have read the whole
image into the cache, you may as well free the resources associated
with the open file, since you don't need to read any more from
it. Except... if it's a multi-subimage file, you may need to read
other subimages. Especially for a movie file with many frames, you
don't want to do a full open/close cycle for every frame
individually just because you are using an ImageCache.
* In oiiotool, ImageRec::read was applying a heuristic to directly and
fully read "small" images, but fall back on the ImageCache to manage
"large" images. But it was just looking at the each subimage
individually. For a movie file, that means that it will force read
all the frames up front, which is silly. So adjust the heuristic to
multiply the frame size by number of frames as a better estimate for
the memory cost of the up-front read. This results in most movie
files of moderate size falling back to the cache, and thus deferring
the read until the frames are actually needed (especially helpful if
not all frames will eventually need to be read).
* Oh boy, oiiotool was always keeping handy a config spec to pass to
the image open, whether it was needed or not. But also some logic in
the ImageCache was causing certain kinds of "find file" operations
(which when asked to find an already in-cache file should just
return it) to invalidate that previously read file and close+reopen
when a config spec was supplied (in case the configuration hints
were meant to change the read behavior or interpretation of the
file from how it was opened before). These combine badly. So we
change oiiotool to only pass the config spec when opening via the
cache if there was a user option to actually set a configuration
hint, and therefore when that doesn't happen, many needless cache
invalidations were avoided.
It's the number of frames for movie files, number of subimages for
other files.
Set and positive only if it can be known inexpensively from the file
header. If absent or zero, it means there was no easy way to determine
the number of frames or subimage from just reading the header (in that
case, you'll have to discover the subimages the old fashioned way,
repeatedly seeking to subimages until you fail).
I realized that HINTS is higher precedence than PATHS, and that
allowed some simplification of the exr finding logic.
Clean up install_openexr script, more control over the generator type
and build configuration. Use `cmake --build` instead of `make` so it
will work on all platforms and not only make.
Current prototype works for 2 Linux tests (default and debug), an OSX
test, and a clang-format check. I want to commit this as a basis for
moving forward, after which I'll flesh out the rest of the test matrix
(including Widows and static analysis). I think this may be able to
eventually completely replace Travis-CI and Appveyor with an
all-GitHub solution.
Some things that came along for the ride:
* Changes to allow parallel running of tests. I didn't previously
realize that CTest could do this, but it saves a lot of wall clock
time in the testing phase. However, I needed to chase down some
aspects of how I set up tests that precluded parallel execution or
had inadvertently established order dependency to the test cases.
* For Travis as well, use Ninja to build, which is faster.
* Establish a minimum version of libgif of 4.1, and also take extra
care on OSX not to pick up an errant version from certain Framework
directories (such as for Mono).
* Updated ref output for yet more versions of libraw.
Two problems (sorry for not testing better):
1. I botched the template syntax for the Cuda specialization.
2. The Cuda specialization uses intrinsic __funnelshift_lc which is
not constexpr, so let's just un-constexpr this whole thing for now.
Quite some time ago (early in OIIO 2.0 development, after it was
separated from 1.8), we changed the ImageInput::read_native functions
to be thread-safe and stateless. This meant that a seek_subimage would
be called inside the read (which now had a lock) whereas it was
previously assumed that the user had already called it.
All good, as long as your seek_subimage checks whether it's seeking to the
subimage it's already on, in which case no work need be done.
DPX failed to do this, so every scanline read involved a costly header
examination. Somehow nobody noticed this until now, but it's fairly
icky. It can slow down DPX reads by 2x, and this is greatly exacerbated
in a heavily multithreaded environment.
I found and fixed a similar error for psd files.
While I was at it, I also added a read_native_scanlines so it can read
multiple scanlines (the building block of read_image working efficiently)
with less work, though in my timings it's only barely worth it.
* Fix potential deadlock in TIFF I/O: minor flaw with threadpool method
This has *very* occasionally popped up for a while. I've only seen it
on OSX for some reason, too hard to reproduce but once every several
days. But a totally unrelated set of changes caused me to see it
happen quite regularly and it's icky.
Long story short, our thread pool seemed to have two different ways of
finding out whether the current thread is a member of the pool, one of
them only used from the TIFF reader/writer, and they didn't match!
Specifically, the correct one tried to know if the calling thread was
doing work for the pool, either being a member of the pool itself or a
nonmember that was stealing work because the pool was too full to add
to. The other one merely said whether it was a fully fledged pool
worker thread. Turns out this difference could get you into trouble,
because you don't want to spawn new tasks from a work stealer, either.
It took me hours of debugging to isolate the trouble, it's too hard to
summarize the specific set of coincidences necessary for this
circumstance to lead to a deadlock.
* Back compatibility fix
Some synchronization with OSL build scripts, including making some
places that said "OpenImageIO_blah" into "${PROJECT_NAME}_blah" and
the like, for the sake of easy sharing between the projects.
build.py turns out to no longer be needed, but a new CMakeLists.txt for
the Rez distro makes downstream Rez packages happy. From Jon Ware.
This mostly concerns builds at SPI, but a tiny bit touches the generic
build scripts. Most of you will want to tune out now, unless you're
especially curious.
Changes for everybody:
* Switching to a 4-part formal version number (e.g. 2.1.3.0), so there
is an additional (rarely used) CMake variable
OPENIMAGEIO_VERSION_TWEAK. This gives me a little more flexibility
with certain tags and releases in the master branch but there are
very few contexts in which anybody will care about or refer to the
last tweak number, and in practice we may rarely or never use it.
* New CMake option `OIIO_NAMESPACE_INCLUDE_PATCH`, if turned on, will
make the "inner namespace" include the patch number (such as
namespace OpenImageIO_v2_1_3), whereas the default is the usual
major/minor (OpenImageIO_v2_1). In general, for external/supported
releases, we want just major/minor, since within a major/minor
family we guarantee link compatibility and specifically DON'T want
to force a recompile for patch updates only. But for studios like
ours who build for production directly from master, link
compatibility is not guaranteed at any time, so we make sure even
down to the patch level is carefully namespaced.
Changes that only have an effect if you're building at SPI:
* The *default* build (regular old "make") now uses the generic
default namespace. If you want to make the special namespaced
version we use for Arnold, build with `make OIIO_ARNOLD=1`.
* Change SpComp2 versioning from consecutive low integer SpComp2
versions (like v47, v48) to canonical multidigit numbers
corresponding to external releases, such as v20101 for OIIO 2.1.1.
Also install the bins in the same place (even though it's
nonstandard SpComp2 practice).
* A new build mode enabled by `make OIIO_SPIREZ=1` does the kind of
build we want for an experimental Rez packaging. Some notable details
and options:
- `REZ_PACKAGE_ROOT=path` overrides the usual root area where we
store the packages.
- `OIIO_REZ_NAME=name` overrides the default package name for the
Rez distribution (default is "OpenImageIO").
- An appropriate package.py is automatically generated.
- The Rez variants fold in the SPI compiler compatibility platform,
Python major/minor version, and Boost version.
- The build automatically sets the Rpath for the Rez install area
so that nobody needs to mess with their LD_LIBRARY_PATH to get
the libraries to be found (especially if you're just importing the
Python module).
* Make the appcfg xml for releasing the binaries to the facility.
* Instructions for building for the facility are now in
site/spi/README-SPI.md
* Tweak ustring test to avoid benchmarking sprintf and allow auto detection of thread count
* Tune the default size and shard count for ustring table to perform better on machines with many CPUs
Comment out the num_lookups variable by default (since it is not used)
Make sure ustring table mutexes are cache aligned
This is back-compatible, using argument lists like "--foo %s %s" works
as it always did.
But now the argument type specifier may be followed by an optional
colon (:) and then a human-readable parameter that will be printed in
the help message. For example,
"--foo %d:WIDTH %d:HEIGHT", ...
indicates that `--foo` takes two integer arguments, and the help message
will be rendered as:
--foo WIDTH HEIGHT Set the foo size
I think that makes some of the command line help messages even more clear
and self-documenting.
Most oiiotool commands copy metadata to their result from their first
input. Using `--metamerge` causes >2 arity image ops to try to merge
metadata from all of its input images (though earlier arguments have
precedence in the case of name clashes).
Also make IBAprep utility have a bitfield flag for the same purpose,
though it's not the mechanism we use for the oiiotool calls.
Along the way, added `ParamValueList::merge()` merges two PVLs together.
Also fixed a couple typos and updated the oiiotool --help message to point
to the new online docs.
The recent AttrDelegate and other notational improvements lead to a clearer and
more compact way to express and extract modifiers as ParamValueList instead
of parsed maps of strings.
* ParamValueList::contains() should be const
* ustring: make an explicit conversion to std::string
* AttrDelegate: make `get<string_view>()` work correctly without
improperly referencing a temp string whose lifetime is shorter than
the string_view.
* AttrDelegate: allow default `get()` template to return a string_view. You need
`get<T>()` to retrieve/convert some other type.
* ParamValueList: have getattribute of string do auto conversion.
If you getattribute() of any non-string type (like an int or float),
it will try hard to convert whatever it finds to that type, even if it
was stored as a different type internally. But if you asked for a
string, it would fail unless it was stored internally as a
string. Make these symmetric by always converting if possible.
In the process, move some bit depth serializing functions into fmath.h so
we can reuse it. (It's not float math exactly, but it is bit depth conversion,
which for historical reasons we also have here.)
The bones of it were there, just needed some enabling and realizing
that the ConvertToNative/ConvertToRGB (that were unaware of the
1-channel cases and returned errors) were irrelevant for
single-channel images and should simply be bypassed.
Fixes#2291
Aside: Our output of DPX doesn't currently support compression
properly. Fix that someday? (Our input of compressed DPX is fine.)
But it probably doesn't matter much, since the only compression method
DPX allows is run length encoding, which you would expect performs
poorly on real-world photographic or rendered images with infrequent
runs of identical pixel values. I had not previously appreciated this
fact about DPX. Why, why does the industry use this terrible format?
The FindBoost.cmake interface is broken if it uses boost's installed cmake
output (e.g. boost 1.70.0, cmake <= 3.14) at least in the case that boost
was installed to a non-standard prefix. Specifically it fails to set
variables as indicated in FindBoost.cmake documentation. So until that's
fixed force FindBoost.cmake to use the original path which is consistent
with its docs.
When scanlines are accessed out of order, to emulate true random access,
sometimes we close the file and re-open. But in this one spot, instead
of calling close_tif() we called close(), and the difference is that close()
also calls init() which resets the options that might have been set by
configuration hints -- such as "oiio:RawColor" and "oiio:UnassociatedAlpha".
In this case, we may want to close/open the TIFF file itself, but we don't
want to "forget" about any configuration hints from the original open call.
The circumstances that led to this being symptomatic are obscure and amusing:
- Scanline TIFF file with CMYK color space, and opened with a
configuration hint "oiio:RawColor" which causes the CMYK values (4
chans) to be preserved rather than it all get auto-converted to RGB
and appear to the app as if it were a 3 channel file.
- The image was held in an ImageCache-backed ImageBuf. And the IC has
the "autotile" feature turned on (so it will try to break it up into
tiles in the cache, instead of storing the whole scanline image as a
single tile).
- An IBA function was running parallelized over multiple threads, and
it happened that a thread needing a lower part of the image (high
scanline number) made the request for its tile before another thread
that needed a tile with lower scanline numbers. This caused the actual
reads of the scanlines to not be in proper order, therefore necessitating
a close and re-open of the file.
- Upon the re-open, it had forgotten that "oiio:RawColor" was set, so
even though the app had asked for the pixels to be written into a 2k
x 4 chan CMYK buffer, the confused TIFF reader thought it was in the
mode where it was emulating a 3 channel RGB file. So it would not
only convert to RGB when it should have left it as cmyk, but since the
RGB is only 3 channels, the 2048 pixels only filled in the left-most
1536 (that is, 3/4!) of the value slots in each scanline of the app's
buffer.
- When running valgrind on the app that was reading the image (for
unrelated reasons), it was complaining about reading uninitialized
memory when it would get to pixel 1536, mystifying everybody.
... And I'm so glad today I work on monster movies and cartoons, and not
nuclear power plant control software or life support systems.
Additional fixes included:
* And add a test case
* Python ImageBuf: add constructor that takes a configuration hint spec
* ImageBuf: when adding file to cache, careful with configuration.
If you open an ImageCache-backed ImageBuf, close, then have another
ImageBuf opened later with a different set of "config" hints... you
may not get expected behaviors. Instead, be sure to invalidate it and
add it to the cache in "replace" mode.
By not passing the nthreads param to the parallel_for_chunked loop,
it was always running fully threaded instead of being controlled by
the nthreads passed to it.
The logic was wrong in cases where the string_view was longer than the
ustring, but they has the same character sequence up to the length of
the ustring.
Add some unit tests to ferret out this case.
To do: fully flesh out unit tests for every last bit of functionality
of ustring and string_view.
Previously, any 32 bit channel would force the "single best type" to be float.
But what if it was a 32 bit int?
Instead, use our standard type_merge logic, where we've carefully considered
all the edge cases.
Needed to check in the figures -- we don't want to do an entire build to
generate a working oiiotool just to make docs on runthedocs.
Add sphinxtr extensions for some handy things I want moving forward
It actually boiled down to trouble with setting unsigned int array attribute
from Python int tuples.
Make sure to test this in testsuite/python-imagespec
GitHub lets you create issue templates, in fact multiple kinds, so
when somebody files an issue they see a menu of the categories. Each
category can have a different template.
I'm making four categories: (1) bug report, (2) feature request, (3)
build problem, and (4) question.
I'm also making it very clear for (3) and (4) that they should not
file an issue at all, but rather use the oiio-dev mail list.
Hopefully this will direct people toward filing more helpful bug reports and
feature requests, and redirect the usual build troubles or usage questions
to the mail list where they belong.
Replace wordy boilerplate with SPDX license identifier and link to full license.
This is a common practice nowadays, and is a much sleeker appearance IMHO.
* Use pretty function printing to show what func it failed in.
* Changing wording to more closely match built in assert().
* Make DASSERT simply define to built in assert(), which has the
advantage compiling without warnings and running correctly with Cuda!
The ASSERT (works in debug mode as well) and [D]ASSERT_MSG macros
still are not Cuda friendly because they specifically mention stderr.
* Slight reformat to make it more automatically discoverable as exactly
the 3-clause BSD license.
* Change copyright name from "me, et al" to "Contributors to the
OpenImageIO project." This is more accurate; it's always been the
case that copyright belongs to each contributor for their own
additions, we never have requested copyright assignment. The "et al"
was supposed to indicate that, but I think taking my name out makes
it even more clear.
We were not using certain libraw parameters correctly.
New default behavior is to decode to sRGB (both primaries and
response). But configuration hints "raw:ColorSpace" can request
various other things, like "ACES" (linear ACES), "Adobe" (Adobe gamut
with 1.8 gamma), and so on. Asking for "linear" gives you sRGB
primaries with linear response.
* Switch where I get gcc9 from in the Travis setup, the previous one has
broken (it was a "gcc-latest", which is now an in-progress gcc10).
* Fix a gcc9 warning
Background: ColorConfig is our wrapper around some OpenColorIO
functionality, so that apps whose only need for OCIO is for the sake
of OIIO color conversion functions, they can go though this thin
interface instead of directly needing to navigate the more complicated
APIs of OCIO. It also allows us to abstract whether OCIO is actually
available, if it's not it understands just a couple hard-coded color
transformations.
* Add getColoSpaceFamilyByName, which was some OCIO functionality we
did not expose before.
* Add methods that return a vector of strings containing the list of
all color spaces, looks, displays, or views for a display.
* Expose all this to Python. And add a simple unit test.
* Make sure one Travis test disables OCIO support, so we know what
breaks under those circumstances.
A few virtual functions in ImageOutput have had their order changed
for the sake of clearer documentation, and that introduces link
incompatibility in the vtbl. This is only in master, so we're allowed
to have breaking changes. But just mentioning it so it's noted in the
history.
Years ago we made Doxygen docs. I hated it, I think they look ugly and
are a poor user experience trying to make sense of them. So at some
point we stopped the charade and let it rust away. Though it lives on
a bit by the fact that I'd already established the habit of using
`///` for documentary comments.
This leaves us with the problem of how to make the header files
self-documenting, while also maintaining the beautiful LaTeX PDF book
that I consider the primary documentation of OIIO. I've been keeping
up as best as I can, but this leads to redundancy of the writing (at
best), and divergence (at worst, which is to say, always).
I'm not a huge fan of "restructured text" (rst), but the documentation
that Sphinx makes is fairly pleasing, it can generate html web pages
and also a PDF (via LaTeX) from the same source material, and I'm
really pleased with other projects that I see use sphinx-generated
documentation hosted on readthedocs.org.
Turns out that there is a Sphinx extension called "Breathe" that will
take Doxygen-generated XML and weave it into a Sphinx doc pretty
nicely. In short, this allows me to write the book I want (in rst,
though, rather than LaTeX) and simply include by reference the
documentation for specific API calls (anything from a single function
to an entire class), which are taken directly from the header
declarations via Doxygen+Breathe. And in addition to the PDF book, it
simultaneously makes nice looking HTML version of the book, which can
be hosted on readthedocs.
For this PR, I'm not replacing any of the old stuff yet, but I set up
the skeleton of this scheme and convert just a single chapter of the
book. I'm liking what I see, so I want to commit what I have so
far. Over time, I will convert the other chapters and eventually will
switch permanently to this scheme, assuming I don't discover a
horrible unintended consequence of this.
In the process of getting this first chapter done -- which happens to
be the one about helper classes like ImageSpec, TypeDesc, ustring,
etc. -- I did some very heavy editing of the header comments. No code
changes, but a LOT of beefed up comments, brining them into sync with
what was in the full latex documentation.
It worked with the global -a, but not the customary `:allsubimages=1`
modifier to that individual command. Either is supposed to make it
apply the channel shuffling to all subimages of a multi-subimage file
(for example, a multipart exr file).
`ImageBuf::set_write_format` took just a single TypeDesc to set the
pixel data type that would be used when the file writes. But there was
not a simple way to specify per-channel data types from the ImageBuf
API (it could be easily done with raw ImageOutput).
This patch extends `set_write_format` to take a `cspan<TypeDesc>`
which could be a single type, as before, or a span of type that gives
the per-channel request.
The GCD functionality moved several versions ago, from boost::math to
boost::integer. It's been "replicated" in both places for a while, but
at least on 1.69, it issues warnings. At some point it will be
permanently moved. So let's get out ahead of that potential breakage.
* Straighten out construction of ImageBuf from spec -- zero or not?
* Docs, header, and implementation were not all consistent about
whether the ImageBuf construct-from-ImageSpec (or the `reset()` call
of the same flavor) zeroes out the newly allocated buffer, or if the
pixel values are left uninitialized.
* Change the call signature to take an optional `zero` parameter -- which
defaults to true, meaning that the pixels are zeroed out when allocated
(this is the foolproof approach, at the cost of writing all the memory
of the buffer). If zero==false, the newly allocated pixels will remain
uninitialized (this is for power users who want to maximize performance
when they know that they will soon overwrite all the vaues in the buffer
and want to avoid any unnecessary writes).
* Touch up the docs and header comments to be consistent and explain the
behavior correctly.
* Revision: use an enum instead of a bool
I realized that there is a convert_type function in typedesc.h and a
convert_types function in imageio.h that are subtly and confusinginly
different in their meaning and arguments.
* Expand typedesc.h `convert_type()` to take an optional number of
values parameter (defaulting to 1, old behavior of converting one
value).
* Rename imageio.h `convert_types()` to `convert_pixel_values()` and
clarify comments to emphasize the important differece -- that this
one is dealing with pixel-like normalized values (e.g., the max
value of an int-like type corresponds to float 1.0).
So you should use convert_type to do a straightforward "cast" of the
value from one data type to the same value in another data type, but
convert_pixel_values to convert a pixel-like intensity to the
equivalent normalized intensity in a different pixel data type.
Sometimes for very large images, either the number of pixels (w * h * d)
or the total data size (npixels * pixelbytes) can overflow int32 math.
Find several places where this may be a problem and use casts to ensure
64 bit multiplication.
I don't claim this is a comprehensive fix, but I did a cursory check of
some obvious places and found these to squash.
* Switch to new TravisCI default 'xenial' ('trusty' is end of life)
* Install Travis linux dependencies per-matrix-entry. This should also
save some build time because we are only installing gcc versions used
for each test, as opposed to all tests needing to install all versions.
* Force 'trusty' for the test matrix entries that tries to approximate
VFX Platform 2017, which uses gcc48. That's because xenial's libraries
for the most part all use gcc 6 and the new string ABI, so it's a huge
pain to use gcc 4.8. It can probably be made to work, but it's a lot
of trouble and I gave up. We'll retire trusty when we retire support
for gcc 4.8 and C++11. Note that VFXPlatformm 2018 (and 2019 and the
draft for 2020) specify gcc 6.3 and C++14.
* Some revisions to the FindOpenEXR.cmake to get it working properly
on Trusty. The most important issue is that pkgconfig should only
be used if there is no manual override on `ILMBASE_ROOT_DIR` or
`OPENEXR_ROOT_DIR`.
We've been plagued by recently-introduced bugs with PNG and JPEG
reading that stemmed from the first IOProxy-related patch. The
problem was our use of the Windows ReadFile call, which seems to not
preserve the file pointer as we'd assumed (the MSDN docs are ambiguous
about this). I also began to doubt its thread safety, so in the end I
just replace the code with a simpler approach with a mutex and calls
to tell/seek/read.
Same changes to pwrite implementation.
Set up signal handlers in all the command line utils so that if they
crash, they will print stack traces that will help track down CI or
user-side failure cases.
This relies on Boost >= 1.65 for its stacktrace library. When built
against older versions of boost, stacktrace() will return an empty
string and the stacktrace-printing signal handlers will have no
effect. C'est la vie.
Default was printing different number of digits of precision on
different python versions. So be specific about the format for floats
to get uniform test output on all platforms.
Lots of important classes have "attributes", which are used as option
settings and the like, basically any kind of named/typed data that
objects carry around. The prototypical way we set and retrieve them
is:
obj.attribute (name, typedesc, &data);
found_ok = obj.getattribute (name, typedesc, &data);
This is very general, but it's kind of a wonky syntax. So this patch
introduces an additional synax that is super handy and aesthetically
pleasing:
obj[name] = data; // set, works for nearly any data!
data = obj[name]->get<type>(); // any type
mystring = obj[name]; // turns any data into a string
This magic is accomplished with a helper template called an
AttrDelegate, which we define in attrdelegate.h. This is pretty handy
and we anticipate using it more widely over time, as we add this syntax
to additional classes.
The first class we rig up in this way is ParamValueList, which
necessitates adding attribute() and getattribute() methods to begin
with. It's a natural fit, and you can get a feel for how it looks when
used by examining the new unit tests in paramlist_test.cpp.
Also added to ImageSpec, for which I think this will be a handy new
way to set and retrieve metadata.
There's a bunch more cleanup that should be fairly straightforward to
understand.
Also, we expose two new functions (pieces of which were used internally
before, though we have now made them public and refactored them):
std::string tostring (typedesc, void* data,
const tostring_foratting& fmt={})
Turns any data (described by the typedesc) into a string,
with optional fine-grained control over the formatting.
bool convert_type (TypeDesc srctype, const void* src,
TypeDesc dsttype, void* dst)
Copies with data conversion.
* Correct use of ZLIB_INCLUDE_DIRS (not ZLIB_INCLUDE_DIR!).
* Only add include dirs for modules that need it.
* Also, change from obsolete PNG_INCLUDE_DIR to correct PNG_INCLUDE_DIRS.
Oops, a few merges ago I changed the .travis.yml to "source" the
install_homebrew_deps.bash script (instead of making a subshell)
and this caused the "exit 0" early out in it to not just terminate
that script as intended, but to terminate the entire Travis test,
making it look like it passed even though it never got around to
running clang-format at all. Oh dear.
This fixes it. It also uncovers a few minor formatting changes
we should have made over the last few days, but missed.
The innards of IBA::resize() has to accumulate the weighted
contributions of source pixels under the filter window, and that
accumulator used 'float'. This is fine except when the input image is
'double', in which case you might want just a touch more precision.
So we define an Accum_t typedef template that is double when double is
needed, and float for all other cases. Adds the precision that double
image resize needs, but retains the speed and precision of the prior
float behavior in all other cases.
For anybody who doesn't know, CII is the Core Infrastructure
Initiative, which is a post-Heartbleed initiative to make sure that
critical (and really, any) open source projects are following best
development and security practices. The badge says the percentage of
their guidelines that we are currently following (you can click on it
to see exactly which ones), as we work toward full compliance. More
info can be found here:
https://bestpractices.coreinfrastructure.org/en
[ci skip]
I don't claim to understand exactly why is is problematic, but certain
files written by an older versin of "nconvert" have something strange
about their tags that was leading to a crash. Solved with use of
our existing safe_tiffgetfield() versus TIFFGetField when retrieving
the PREDICTOR tag.
Also refactored safe_tiffgetfield a bit so that for the calls that
"fail" (results are ignored), it doesn't overwrite the results. Do this
by retrieving into a temp buffer and then copying to the caller's
memory upon success.
TODO: How many other TIFFGetField calls ought to be replaced by
safe_tiffgetfield? All? Some? What are the circumstances that lead to
this inexplicable problem? Is there a better tag-reading idiom we
should be using? Investigating this on all on the side, but it need
not hold up this immediate fix to the one known problem case.
Background:
By default, most oiiotool operations ignore multiple subimages. For
commands that modify metadata in place, that means it will only modify
the first subimage (leaving the rest untouched). For commands that
remove inputs from the stack and produce outputs to be put back on the
stack, generally their output will only be based on the first
subimage, and subsequent subimages will be dropped.
`-a` says that any ops ought to work on all subimages if at all
possible; that means that in-place metadata modifiers will apply to
all subimages, and stack ops should produce outputs with the same
number of subimages as their inputs, doing the right operation to each
subimage in turn.
Most commands (but not 100%) also respect an optional
`:allsubimages=VAL` that override `-a` on that one command, `=1` turns it
on even if you didn't use `-a`, and `=0` turns it off even if you did use
`-a`.
My goal is to make all commands respect the `-a` and `:allsubimages=VAL`
options, in all cases except where there is no sensible interpretation
to multiple subimages. There are a few now where there is a sensible
behavior, but it's just not implemented.
This patch fixes the following:
* These commands previously honored `-a` but did not respect a
":allsubimages=" modifier, but now they do:
--sattrib --attrib --caption --clear-keywords
--iscolorspace --orientation --clamp --fixnan
* These were totally unaware of subimages, neither responding to -a
nor to ":allsubimages=", now they respect both:
--crop --fullsize --zover --fill
* These previously always operated on all subimages, but now work on
only one subimage by default (like the rest) and respect the -a and
":allsubimages=" modifier:
--origin --fullpixels --croptofull --trim
Internal to ImageBufAlgo functions, we are faced with the fact that we
have 1, 2, or 3 input images in addition to an output image, and they
each could be any of the 9 pixel data types we support (float, uint8,
uint16, half, int8, int16, uint, int, double). This means that the
template expansion of all possible combination would tend to be 9^2=81
specializations for unary functions, 9^3=729 for binary functions, and
9^4=6561 for ternary functions (I think mad() is the only ternary
function we have).
This is unworkable, leads to infeasible compilation times, so for a
long time we have made the following shortcut: handle the full
cross-product of just the 4 commonly used types (uint8, uint16, float,
half), and in all other cases convert to and from float. So that means
unary needs 16 template specializations, binary 64, and ternary 256.
But... other cases do crop up, so with this patch, I'm expanding the
range of cases that OIIO_DISPATCH_COMMON handles without
conversion. If all of the types (result, and inputs) are identical, we
handle it with the native type. This is only slightly more expensive:
unary functions need 9+12=21 template specializations (why 12 instead
of 16? because 4 combinations are already handled by the "all same type"
case, and therefore don't need to be instantiated again), binary
functions need 9+60=69, and ternary 9+252=261.
We think that, outside the "common" cases, the most common uncommon
ones are the "all types the same" cases. So accommodating this seems
worth the slight increase in compile time, which is nowhere near the
combinatorial explosion of the full cross-product of all types,
and in the process we make it even more rare that we fall into the
"plan B" case of converting to and from float to handle the rare
oddball cases.
This is in keeping with the nomenclature of the C++20 <bit> header which
contains the equivalent functions.
Also give them the constexpr and noexcept treatment.
Strutil::printf does its underlying output atomically, so if these
destructors are triggered from multiple threads, they won't get
"interleaved" on the terminal, making them unreadable, as they would be
with ordinary stream output.
A bit of material has been refactored out of .travis.yml into separate
scripts so it can be shared with circleci.
CircleCI is an alternative CI system like TravisCI or Appveyor.
Mostly I'm doing this as a prototype and to learn about it, because
it's under discussion as a potential favored CI system for ASWF.
But as it turns out, the VMs provisioned by CircleCI are more capable
CPUs than the ones with Travis. In particular, it looks like they
support AVX-512. Right now, the Travis VMs top out at AVX 1.0, so
although we have Travis test matrix entries that "build" for AVX2 and
AVX512 just to make sure we haven't broken the build, we aren't able
to run the tests when compiled for that architecture. But we can on
CircleCI. So I'm thinking of throwing CircleCI into the mix just for
certain build configurations that aren't possible with Travis, to get
better test coverage.
ArgParse had no unit test! Add one.
Also, add a new feature: %1 is like %* (catch-all for all
miscalleneous arguments that aren't dash-led commands and their
following data). But if %1 is present, its list will receive any
arguments that come BEFORE any other dash-led args, whereas the %*
will receive all the misc args that happen after the first dash-led
arg. If there is no %1, then %* will capture all the misc args
regardless of whether they are before or after the first dash-led arg.
Why is this useful? So you can have programs that require certain
arguments before any options. Like "git add ..." "git rm ...", you
would achieve something like that via:
std::string mode;
std::string files;
static int
callback(int argc, const char* argv[])
{
for (int i = 0; i < argc; i++)
files.emplace_back(argv[i]);
return 0;
}
static int
mode_callback(int argc, const char* argv[])
{
if (mode == "" && argc == 2)
mode = std::string(argv[1]);
else
callback(argc, argv);
return 0;
}
ArgParse ap;
ap.options(
"My git replacement",
"%1", initial_callback, "",
"%*", callback, "",
...
nullptr);
std::cout << "Mode was " << mode << "\n";
std::cout << "Files were: " << Strutil::join(files, ", ") << "\n";
Especially for CI or users systems that don't have the oddball locale
we used for some testing, make sure the tests complete rather than
crash with local exceptions.
Augment imageinout_test to try to write each format to a path that
includes a nonexistant directory (to test generally what happens when
asked to write to a place that doesn't exist or doesn't have write
permissions).
Also a bunch of other refactoring of imageinout_test.
This turned up failures in several format writers:
* DPX crashed (solved with PR #2186)
* HEIF silently failed, not returning any errors for this condition
(solved with PR #2188)
* PNG silently failed (solved with PR #2187)
* JPG silently failed and sometimes crashed (solved with PR #2187)
One consequence is that format readers or writers that relied on
IOFile (such as PNG and JPEG) would not properly notice certain I/O
errors.
This failure of JPEG masked a minor bug in the JPEG reader where we
could destroy the compressor that had never been created (crashing
inside jpeg library).
Turns out that libheif's `write_to_file()` does not throw exceptions
upon write errors, as it is advertised to do. Solve by using the more
general `write()` that takes a custom Writer class.
It's been reported that V-Ray writes images with this metadata field
present but empty. If you happen to take those images (and their
metadata) and turn them into a multi-part exr, the names all being the
same is problematic. The expedient fix is just to ignore empty names
on input. Suggested by Chris Allen.
ImageInput::create(name) and ImageOutput::create(name) are supposed to
create a new II or IO for given name, which can be a filename (such as
"foo.exr", the extension determines the format), or just an extension
(such as "exr"), or the name of the format itself (such as "openexr").
Except that the latter didn't work, due to a botched test of whether
the format name was already in the table. This has been a bug for
many years, so it seems!
* Add read-from-memory for JPEG.
* Add ioproxy read to PNG (only write was supported before).
* Had forgotten before to add correct supports("ioproxy") response to exr
and PNG.
Added a new unit test, imageinout_test, which for every supported
format writes an image file, reads it back and makes sure it matches
the original. For formats supporting "ioproxy", it also writes to
memory and makes sure it matches the disk file, and reads from memory
and makes sure it gets the original pixels back. I removed the
hard-coded tests for ioproxy on png and exr, this is more
comprehensive.
Over time, I'll add more to this test. It's a good central place to
test behaviors that should work for all formats.
Some new unit tests show crashes inside libopenjpeg, but only for
libjpeg<2.0 and only when we build with EMBEDPLUGINS=0 (apparently
it's fine when =1, which is the default, I don't know why).
I have no particular desire to try to track this down, especially
since it requires the combination of an obsolete version of libjpeg,
and a build mode that probably very few people use. So I'm taking the
easy way out: just disable jpeg2000 suppor for this unusual
combination. If you need EMBEDPLUGINS=0 and care about openjpeg,
upgrade to 2.0.
Non-SPI people can ignore this.
The motivation was to let us switch between several Boost versions,
including a new layout for our local boost install. But I took the
time to do a bunch of cleanup on the spi Makefile-bits.
Upon reading FITS specs, I realized that 16 and 32 bit int pixels in
FITS are signed, not usigned. Also beefed up our support for what
appear to be multi-channel FITS files and other oddities in the
image dimensions.
This patch adds a "max_mip_res" attribute for the TextureSystem that
limits filtered texture access to MIP levels that are no higher than
this resolution in any dimension. The default is 1<<30, which means
there is no effective limit. As an example, setting this attribute to
256 means that all texture access will be limited to the MIP levels
that are 256x256 pixels or lower. This can be useful for preview
renders where you want to cut down on the amount of disk or network
access for the sake of speed.
In a subsequent patch, I intend to also provide a way to set this
on a per-texture basis, but as a first step we'll start with the
global setting.
Restore its ability to run by hand from inside the
build/PLATFORM/testsuite/TEST directory, which had been lost recently
with some overhaul to the testing scripts.
Certain regex_replace tasks seem to crash on Windows with MSVS's
implementation of std::regex.
After considering a more sophisticated test for when we should use
boost.regex instead, I instead settled on just replacing the few uses
of regex_replace with our own string processing. None were very
sophisticated or truly needed regex functionality.
`IBA::colormatrixtransform()` uses a 4x4 matrix to apply a color
transform to all pixels in the image region. The matrix is, following
Imath, OpenGL, and RenderMan-ish conventions, laid out in the order
that assumes row-vector colors and does math as C' = C*M. (Another way
to put that is that the "translation" part of the matrix is the bottom
row, not the right column.)
`oiiotool --ccmatrix` does the same thing from the command line, and
is followed by an argument containing either 9 or 16 comma-separated
values to form a 3x3 or 4x4 matrix. (If 3x3, it will be extended in
the obvious way to 4x4 before passing to the underlying
colormatrixtransform() call. Additionally, the `--ccmatrix` command
takes optional arguments that will first invert the matrix or use its
transpose (the latter in case you are given matrix values in an order
that implies column-vector colors).
The unassociated alpha handling setting of the ImageCache was
overriding the setting specific to the ImageBuf itself, for the path
where for "forced" reads bypass the cache. Good intentions to still
honor the IC setting, but IB-specific request should take precedent.
When we recently made invalidate(file) have a 'force' parameter, we
botched it by not so forcing a particular call in invalidate_all().
In the line being changed here, we're dealing with a list of files
that definitely need invalidation due to special tests, so once
we've established that, we want to force it.
Pointed out by @ThiagoIze in a post-merge comment in #2133
Build for SPI Arnold:
make
Build for SpComp2:
make OIIO_SPCOMP2=1 spcomp2
or
make OIIO_SPCOMP2=1 spcomp2_install_local
Build completely generic, avoiding any special Arnold build logic:
make OIIO_ARNOLD=0
I know that the diffs are difficult to follow in this format. But
basically I was starting from a point where site/spi/Makefile-bits just
looked at the build targets, and if they looked like spcomp2 stuff, it
would include Makefile-bits-spcomp2, and otherwise would include
Makefile-bits-arnold. Those two files were already pretty similar,
lined up well in the diffs. So I just merged them, hiding the few
differences between them in conditionals, and then moved the result to
just plain old Makefile-bits and got rid of the separate customized
files. In reality, if you were to diff the new one to either one of the
old ones, it would be pretty close, with the obvious changes to merge
in the missing parts of the other.
It shouldn't change anything about how you use these. Just marking a bunch
of things that already didn't/couldn't throw exceptions as 'noexcept'.
There will be more of this moving forward, I'm just getting some basic
classes now.
HEIF is a container for HEIC-compressed images. HEIC is the still image
sibling to HEVC video compression. Roughly speaking, it produces lossy
compressed images about half the file size of JPEGs but with higher
visual quality.
At the moment, it's hard to find software that will reliably read
heic/heif, except for recent cuts of software within the Apple ecosystem
(the only ones I've encountered so far are iPhone camera images). But
that will surely change, since it's a superior format to JPEG in many
ways.
This is the first stab at supporting heif/heic images using the open
source libheif. I'm fairly confident in the reading, including exif
metadata, of RGB and RGBA images, including files with multiple images.
I've not yet figured out depth images.
This can also write images (but only single images per file, RGB or
RGBA), and although the exif metadata I write can be read again by
OIIO's heif reader, Apple's Preview.app does not recognize the metadata
for some reason. I don't know why, maybe I'll return to that later.
I also want to fix later the limitation of not being able to output
multi-image heif files.
Along the way, I converted decode_xmp() utility to be based on
string_view, and also changed encode_exif() to take an explicit
'endianreq' parameter so it's possible to request a specific
endianness. (Apple's Preview.app, which is the main way I was able to
test whether I'm writing them correctly, *requires* that the exif be
written as big endian, or it won't recognize the file as proper heic at
all. So that required some extensive changes in my exif blob code.)
The latest release of libheif, 1.3, doesn't fully support color
profiles. The in-progress libheif master (future 1.4) does, so I'll add
support when that is released. For now, it always looks like sRGB, and
that seems to be the common case, but it can be incorrect.
Currently, we only will read and write uint8 pixel data. The libheif
authors are working on full support for HDR/extended range, but that's
not even merged into its master yet. I'll add support in the OIIO heif
reader as that gets merged into libheif releases.
Fix some vdb issues, including the fact that we weren't previously
detecting the version, and therefore not properly erroring for vdb
versions too old. Also a variety of other minor fixes and minor
refactorings.
Defining this symbol turns it off. Somewhere on my list is to remove the
OGL dependency from iv, but this band-aid will help until I get some
free time for that.
This patch allows an optional suffix to be specified that will let you
produce something like libOpenImageIO_CIA.so or whatever. Use with
caution, or preferably not at all, if you don't absolutely need it).
Example of how it's used: Suppose you're at a facility that needs (don't
ask why) multiple versions of OpenImageIO to be built. And namespacing
is not enough, because if you have two copies of libOpenImageIO.so, you
can't separately rpath them.
We've found that this is unsafe, dwaa/dwab compression crashes inside OpenEXR for certain tile sizes (only for 1-channel images, it seems). Until there is a fix on the exr end, we'll just revert to zip compression for the problematic cases. Power-of-two tiles larger than 16 seem fine, and that's all maketx uses anyway.
When performing get_image_info() on udim files, we have to try out
lots of the individual tiles until we find one that exists (the set
is allowed to be sparse). We were not properly supressing any error
messages set for the missing files, once we hit one that worked.
For the purposes of deciding whether a `maketx -u` command is an
unnecessary repeat, we compare this command line to the last one to see
if it's the same. In so doing, we strip the directories off any
arguments that look like filenames so that changes is how relative path
names are expressed don't cause us to unnecessarily remake the texture.
It's a little naive to do this for all args. Thiago noticed situations
where color space names (args after `--colorconvert`) might have slashes
and so you don't want to strip them of their "directories."
This patch addresses that, and also several other maketx/oiiotool
commands that might have following args that look like filenames but in
fact are not. Also uniformize the interchangeable -cmd and --cmd so that
those pointless differences don't incorrectly trigger rebuilds by
looking different when they are not.
* Main event: add "nuke-lanczos6" as an alias for our "lanczos3" filter.
(Nuke names it according to the full width, which makes sense but is
different than our more conventional lanczos3 nomenclature. Made the
alias to make it more obvious which to choose if you want to exactly
replicate what Nuke calls lanczos6.)
* Added new flavor of get_filterdesc() that returns a const ref,
instead of copying into a parameter by pointer. (In retrospect, I'm
not even sure why I did it that way.)
* Made `oiiotool --help -v` print the available filters, for handy
reference.
This works the same way as the `force` argument to `invalidate_all()` --
if true, invalidates unconditionally; if false, invalidates only if the
file's modifiation date has changes since the cache first opened it.
The default is true for back compatibility, but probably you will often
want `false`, only invalidating if the file has changed.
It's not uncommon to encounter partially written files. They could be
the partial results of a long render with only some tiles written so
far, or could be a corrupted file or one where the process died partway
through writing the file.
Many formats will consider this hopelessly corrupted, but it's actually
legal to have missing tiles in an OpenEXR file, for example. So
sometimes people want to read this, without the whole file seeming like
an error.
This patch introduces a new configuration hint used for ImageInput
open-with-config: "oiio:missingcolor". Also a "global" attribute (set by
`OIIO::attribute()`) that is equivalent to setting the configuration
hint for all ImageInputs.
If the configuration hint (or global option) is set, it is a (non-required)
request to ImageInput implementations that failed reads of individual tiles
(or scanlines) should not be treated as hard errors by the II::read_blah()
calls, but instead should suppress the error reporting and simply fill in
the pixels of the missing tiles with the color specified.
The hint's value is either an array of floats (one per channel), or a
string containing the text of a comma-separated list of the values. If
it contains fewer values than channels in the image, the last value will
simply be repeated for all remaining channels past the end. As a special
bonus, if the first channel of the missingcolor is a negtive value, it
will draw stripes (of the absolute value) instead of a solid color.
Note that currently, the only format and task that supports this is
reading tiled OpenEXR files. We can try to extend the support as
requested to other formats where it makes sense.
As an example use case, if you have an exr file that is missing some
tiles (say, because it is still in the process of being written by a
long-computing render), ordinarily an attempt to read it such as
oiiotool partial.exr -o out.exr
might get a hard error with a message such as:
oiiotool ERROR: read partial.exr : Failed OpenEXR read: Error reading pixel data from image file "partial.exr". Tile (0, 0, 0, 0) is missing.
But now you could do this instead:
oiiotool -iconfig oiio:missingcolor "-1,0,0" partial.exr -o out.exr
And that will result in a successful copy of the file, with red and
black stripes in place of the tiles that were missing in the original
file.
Akin to how we have `OPENIMAGEIO_IMAGECACHE_OPTIONS` and
`OPENIMAGEIO_TEXTURE_OPTIONS`.
`OPENIMAGEIO_TEXTURE_OPTIONS` is a comma-separated list of name=val
pairs that upon startup will seed the global options that are usually
set by the OIIO::attribute() call.
Along the way, I beefed up the OptParser to be able to handle both
single and double quotes equally well. That lets you set an option that
is a string, that itself contains quotes.
Also document all of these options -- I realized the PDF docs don't talk
about even the existing env variables.
Homebrew updated its copy of clang-format several days ago. Something
in it changed formatting rules and caused our builds to break on the
formatting verification test.
This patch changes the way we look for the clang-format binary -- you
can supply a hint about which directory it's in as
`CLANG_FORMAT_EXE_HINT` cmake variable, or an env variable of the same
name, or it will also try to find a clang-format in your LLVM distro if
you have a LLVM_DIRECTORY env variable set.
This lets us more easily lock down the clang-format to the one with llvm,
not separately installed, so it should get changed less frequenty and is
easier to standardize across users and platforms. Currently, then, we're
trying to tie it to clang 7.
The underlying decoding of pixel value arrays -- used by some of the
ImageOutput methods and ImageBuf.set_pixels -- threw exceptions when
the pixel array didn't conform to expectations. This led to very
confusing behavior, and crashes if the python program didn't carefully
catch them. Instead of throwing exceptions, bubble up an error message
for the usual OIIO error reporting, so wrongly-shaped pixel data arrays
are treated similarly to other kinds of error.
Also, detect with a better error message cases where user tries to write
scanlines to a tiled file or vice versa -- it was handled well in C++,
but the Python would start by parsing the pixel array, then have a
confusing error that it was the wrong shape, instead of the more clear
error message about using tiles instead of scanlines.
They had previously outright rejected if the destination and source
images didn't have the same number of channels.
But actually, it doesn't matter. If they differ, it will only operate on
the minimum channel count in the two images. If dst has more channels
than src, that's fine, it just won't fill in the "missing" channels. If
dst has fewer channels than src, then that's also fine, and in fact it
is totally reasonable to want to resize an RGBA image into an RGB
buffer, resizing while only copying the first 3 channels.
If you are reading a TIFF file with unassociated alpha, but you have
called the variety of read_scanlines or read_tiles that let you restrict
the channel range and your range doesn't include the alpha channel, the
automatic premult will be using bogus values. Oh brother.
Solution is to skip the premult if the alpha channel was not available.
Note that this makes it possible for read_scanlines() of channels 0-2 to
NOT MATCH the first three channels if you had read all 4 channels.
Seems odd, but what else can we expect? You get what you deserve if you
purposely exclude the alpha channel from the read, but it's an
unassociated alpha file that needs the alpha for the premult.
Also noticed that read_scanlines was not quite as careful with
thread safety as it should be, accessing m_spec outside the mutex.
The read_tiles implementation got it right, so do the same thing.
The thread_pool already attempts to keep track of its worker threads,
and if they themselves attempt to submit tasks to the pool, it will just
execute those recursive tasks immediately rather than enqueuing. This
helps prevent deadlock where all the queue workers are running tasks
which themselves are waiting for other tasks, which are so far back in
the queue that nobody progresses.
But, as Thiago points out in #2131, if there are multiple app threads
submitting to the thread pool simultaneously, you can get into similar
trouble, and since those submitting threads are not themselves pool
worker threads, the "am I a worker" test fails.
Moment of honesty: I'm not sure exactly how this can happen. At first
blush, it seems like it shouldn't matter, but nonetheless it does and I
know from experience that reasoning about all the edge cases of queue
submission order is really tricky. The additional check proposed here
appears to solve the problem, so we're going for it.
The proposal here is Thiago's suggestion, but cleaned up a little bit
by LG to make sure it works for both `parallel_for_chunked` and
`parallel_for_chunked_2D`.
The gist is that we use a thread_local variable to keep direct track of
the recursve depth of calls to parallel_for (for each thread) and force
single threaded mode if recursion is detected.
Fixes#2131
Also, we weren't running the webp tests at all on Travis or Appveyor
because they weren't downloading the webp images. Move them to the
oiio-images project and adjust.
WebpInput::read_native_scanline had
if (y < 0 || y >= m_spec.width) // out of range scanline
return false;
when it should have been m_spec.height!
This had the effect of introducing read errors any time a webp image had
a height > width. Worked fine for "landscape" images, which is how it
went for so long without detection.
Fixes#1716.
While I was at it, I also noticed that webp seems to assume unassociated
alpha, so I added the code to premult/unpremult on the way in and out.
webp images seem to be always assumed sRGB, so also add that logic.
JPEG compression was one of the cases where we fell back on the
TIFFRGBAImage interface of libtiff. This worked for some JPEG images,
but crashed on others, inside the TIFF library. Never did figure out
why.
But do we really need to to use the RGBAImage interface? Turns out, no!
At least, not if it's still a spectral RGB image.
So just using the regular interface for ordinary RGB JPEG-compressed
TIFF images is fine, fixes the problem.
Also improved an error message while I was in there.
read_tiles was calling all errors "no support for format", but that's
really weird. If it's a bad tile read, the error will already have
been set by the underlying read_native_tiles.
Allow caller to specify the separation characters instead of hard-coding
to just ' '. There are separate ordinary separators (default " ", break
on the separator), and pre-separators (default: none, break AFTER the
separator).
An example if when you might want to use the pre-separator is for sep=" "
and presep=",". That would mean to split at any blank, but if there are
areas with no blanks separated by commas, it's ok to break immediately
after a comma.
At the suggestion of Thiago Ize: After read errors on texture tiles,
check if the file it's reading from appears to have a different
modification time than when we first opened it, in which case being
overwritten by some other process is probably the source of the error,
so issue a message saying so.
Many of the file format writers have options for compression, expressed
as metadata "Compression" (e.g., "zip"), and a few of them also take
a second control called "CompressionQuality" (some compression methods
have size/quality or time/quality tradeoffs).
As a special case, the OpenEXR writer allowed the compression for dwaa
and dwab to be able to embed the compression amount in the name, like
"dwaa:100".
With some more thought, I realized that this is a superior notation and
that all of the writers should support this. This has two really nice
consequences:
1. Express compression method and amount in a single option.
2. Avoid possible bugs of "CompressionQuality" being inappropriately
passed from one method to another. For example, zip, jpeg, and dwaa have
totally different meanings to their scales. Making the compression scale
part of the name prevents some silly user errors.
For back compatibility, this patch still allows an explicitly set
"CompressionQuality" to function as it did before. But I'm changing all
the docs to favor the new notation.
Our TIFF output doesn't support per-channel data types (because
the underlying libtiff does not).
This patch makes the TIFF writer check that the spec it's passed doesn't
have a filled-in `channelformats` and will refuse to open, with error,
if it does. (Exception: if the vector is filled in but all elements are
equal to spec.format, then it's accepted.)
Get rid of much cruft, install all dependencies with vcpkg. Super easy.
No messing with finding nuget packages and then weeping over their
oddities. And now we have a lot more optional dependencies in place,
which means we're building more of the OIIO code base (fewer parts
disabled for lack of deps), therefore more code coverage in the windows
test build.
There's a trick to this -- it can't actually build all these packages
from source within the 1 core hour it's allotted. It all depends on the
run-to-run caching that leaves the built vcpkg packages in place, and I
had to build it up a few packages at a time. The very expensive optional
dependent packages only build if there's enough time left on the clock.
maketx -u means "update mode", wherein if it looks like there's already
a texture in place that's the result of the same maketx command, it
won't re-make the texture.
The primary way of doing this is to compare file dates: when using -u,
the output texture will be given the same timestamp as the input source
image. So the next time it runs, it checks the two dates, and if they are
identical, the texture is presumed already made.
But wait, what if they used a different version of OIIO, or had
different command line args, such as choosing a different fileter? Then
you DO want to remake the texture. So we also compare the metadata that
gives the command from last time, with the command line from this time.
But... that can have false negatives, because what if they used an
absolute path one time and a relative path another time?
So in this patch, we add a little trick: before comparing the two
command lines, we take each of them and for any argument that looks like
a file path (has a directory separator in it), we strip out everything
before the last directory separator. So
/my/bin/maketx foo.exr -o ./foo.tx
makex /path/to/foo.exr -o foo.tx
Used to look different, but now look the same.
How do we know that the things we strip are actual file paths? We don't,
and we do not care. No arguments to maketx that we care about are going
to have slashes in them. That won't be part of the filter name or any of
the other things we were trying to distinguish between.
Fixes#1472
These are "pseudo-attributes", they aren't metadata themselves, but
aggreagate from x,y,z,width,height,depth (and the full_ versions).
The "datawindow" and "displaywindow" are OpenEXR nomenclature, but
it's likely that people will want to refer to them that way.
Also, minor fix to `ImageCache::get_image_info` and
`TextureSystem::get_texture_info`, to make the right kind of
ImageSpec::find call to retrieve the pseudo-attributes.
Reported in #2106, there was a subtle bug in read_image/scanline/tile
where we fell into the "contiguous and no data type translation"
case when that was not really the case.
The symptom could be triggered by the following case: You want to read a
1-channel float image into a buffer meant to hold a 4-channel UINT8
image with a 4-byte xstride. The correct behavior is to translate the
floats into uint8's and place them in channel 0 of each 4-channel pixel.
But because the buffer stride length (4 x uint8 = 4 bytes) matched the
native stride length (1 x float = 4 bytes), it was copying the floats
instead of translating and depositing them correctly in channel [0]
(leaving channels [1-3] unchanged).
The fix isn't too hard, but we made the same mistake in several places.
Basically, we were comparing whether the strides matched the native
spacing, instead of whether they matched the buffer spacing. Only
particular unusual combinations would trigger the false positive.
Add an explicit test of this case in imagebuf_test.cpp.
Since <windows.h> is included in platform.h, and with great care about
certain symbols being defined beforehand (such as `WIN32_LEAN_AND_MEAN`
and `NOMINMAX`), rely on that one and eliminate other places where we
may include windows.h. This ensures that we don't forget to define those
symbols in other places or get confused by include order.
atoi/atof are frowned upon because they don't have ways to report
errors, so are typically flagged as unsafe by static analysis, they have
the "pipe-fitting" issue of taking only a `const char*`, and for atof in
particular can have is behavior changed by the system's current
locale. Instead, we prefer our Strutil::stoi and stof which take
string_view and are guaranteed locale-independent.
There are just a couple places where we keep the old functions -- one is
pugixml (because it's an import and we don't want to diverge from the
upstream source), and the other is in strutil_test where we are
specifically trying to benchmark our functions against atoi/atof (and so
therefore must call them). In these instances, we at least mark them as
exceptions for the static analyzer to not complain about.
The most important change is that we were missing the __iter__ method.
That was preventing proper iteration over the entries in a
ParamValueList (you could still iterate over the length and index like
an array to retrieve the entries). I did some other cleanup as well, to
conform to the pybind11 examples that handle similar cases.
Iterating still seems to fail for me on OSX, only for some image files,
dumps core after visiting the last item in the list (I guess, as it
iterates past the end). Works on Linux, works on TravisCI (both Linux
and OSX). I've struggled with this for a few days and I'm not sure what
to try next. Maybe the problem is sensitive to particular python
versions?
In any case, if you can't get this to work:
for attr in myparamvaluelist :
print attr.name
Try this, which I'm sure works:
for i in range(len(myparamvaluelist)) :
print myparamvaluelist[i].name
Use of clang-tidy found several instances of "unsafe" string copy or concatenation -- using functions that don't take a buffer length.
In truth, we humans know that these were probably fine in their context, but it's true that those unsafe string functions should be avoided and it's desirable to get clean static analysis on all the security-related tests.
It's worth noting that these changes in rgba.{h,cpp} diverge us from the original sources (they were imported from code by Greg Ward and Bruce Walter). I usually prefer that when we import entire modules of code from elsewhere, we don't make any changes at all, or even reformat to our conventions, so that we can easily integrate future improvements to the upstream sources. But after mulling this over, I think that in this case, it's not actively maintained code, and in 10 years we have never had to integrate fixes. So let's just consider these fully absorbed and able to change at will for our own purposes.
Because what if some loon has curly braces in a filename? It must be
rare because nobody has come across this problem in the years since
OIIO 1.6 added expression evaluation. But it happened today!
So you can do this:
oiiotool -echo {3+4} -evaloff -echo {1+1} -evalon -echo {8*2}
And you will see ouput
7
{1+1}
16
Clang-tidy is part of clang tooling that's a static analyzer of
everything from "unsafe" code that violates security guidelines, to
stylistic fixes. There are many hundreds of tests. Initially we are
just enabling some basic security and correctness tests. Will add more
later.
All you need to do to run it is
make CLANG_TIDY=1
Or if running cmake directly, use -DCLANG_TIDY=1
Warning: it is a long compile.
Set up a Travis test to run clang-tidy for static analysis. But after
much trial and error, I just couldn't get it to run in the 50 minutes
that Travis allows. So for now it's disabled. But if we ever have a CI
solution that executes faster or has longer time limits, I would love to
do a full static analysis on every PR. For now, I will have to be
content running it by hand on my own machine from time to time (and for
every tagged release).
Apparently we had deprecated this file 3 years ago, moved all the useful
content to platform.h. But never got around to removing the header that
was no longer needed.
Once we were fully C++11 and moved MSVS minimum to >= 2015, the contents
of this file were no longer needed as far as I can tell. What few things
were here that anybody might still rely on, are in fmath.h.
Should now build against OpenCV 2.x, 3.x, and 4.x.
Added more modern from_OpenCV and to_OpenCV, converting ImageBuf <->
cv::Mat, and taking the common ROI and nthreads parameters that we use
for the rest of IBA. Deprecate the old IplImage-based functions, which
is an OpenCV 1.x thing.
Add a test of ImageBuf -> cv::Mat -> ImageBuf round trip to make sure
it works.
`convert_image()` and `parallel_convert_image()` had alpha_channel and
z_channel parameters that weren't actually used. I don't even remember
why they were present; I think I imagined that certain conversions might
dither the color channels or something and need to know not to do it to
these channels. But in any case, they were unused, so remove the
clutter.
If no 'from' space name was provided, it tried to look up the
"oiio:ColorSpace" metadata of the source image, and if that also wasn't
found, fell back on "linear."
But the OCIO config may not literally have "linear" as a color space
name. So instead, the fallback should be on whatever color space has the
*role* of linear (aka scene_linear). This in turn makes the default "no
'from' parameter supplied -> use linear" less prone to error.
It came to my attention that IBA.ociodisplay (and also it turns out
ociolook and colorconvert) have parameters named 'from', which it turns
out that if you use the `name=value` named argument syntax in Python
will actually be a syntax error because 'from' is a reserved word!
Rename that parameter 'fromspace', and to be symmetric, also rename
'to' to 'tospace'. And do the same in C++ bindings so that they match.
For many years we have used Chris Foster's magnificent "tinyformat"
package to do a lot of the heavy lifting for our type-safe string
formatting with printf-like notation, which we use for Strutil::format,
nearly every error reporting facility in the major classes, etc. Thanks,
Chris!
The C++ committee has finally caught up to Chris, advancing drafts of a
"string formatting" proposal that now seems very likely to be part of
C++20. The best implementation at this point is the {fmt} package.
Unfortunately, {fmt} and the C++ draft papers differ from tinyformat in
that rather than traditional C printf notation for formatting, they use
a notation very similar to Python string formatting, e.g.,
format ("%d %.3f", 1, 23.4) --> format ("{} {.3f}", 1, 23.4)
This compatibility break is one reason I've stayed away from {fmt},
despite that (no slight to Chris) it's at least twice twice the speed of
tinyformat.
But knowing that this is almost certainly the shape of the future C++
standard, I am confident that the best long-term path is to migrate to
its conventions. And it will be nice to reuse the same cognitive
machinery whether we're programming in C++ or Python, and get the speed
boost.
But how to do this migration without breaking every string format
operation in OIIO, and additionally in all the packages that call OIIO
and use any of its string formatting or error reporting facilities (and
indeed, we use it extensively in OSL)? This PR is the first step of
implementing the transition, which will occur over a couple major
releases. Here we go:
1. Introduce Strutil::sprintf(), ustring::sprintf(), and various
errorf(), warningf(), etc., functions as needed that now and forever
will use the printf notation (and for now are still implemented
primarily with tinyformat, but that's not a detail you need to worry
about).
Neary all internal uses of the format() commands within OIIO internals
have switched to sprintf().
Calling apps that make use of OIIO's string formatting can therefore
also change their calls from format() to sprintf() and everything
will continue working for the forseeable future.
2. Introduce, in a namespace, Strutil::fmt::format() and certain
fmterror(), etc., that use the NEW Python-like notation, implemented
underneath with fmt::format().
Calling apps may start using these now if they prefer the Python
notation...
But BEWARE! I have discovered that the fmt library does not correctly
implement the promised locale-inedependent formatting for floating
point values (it does for integers), and so could cause problems for
those of you who might have apps running on platforms wth certain
locales that, for example, use ',' as the decimal (sheesh). This is a
known problem with fmt, will be fixed soon, and as soon as it does,
I'll update our embedded copy of fmt and give you the all clear to go
nuts with the new python style formatting directives.
3. For OIIO 2.0, Strutil::format, ustring::format, and the various error()
functions will behave identically to sprintf (C printf notation), but
for OIIO 2.1, I am planning on having them all switch to to the new
python (a.k.a. C++20-ish std::format()) format designations.
So, the upshot is that NOTHING should break for your existing code's
format()/error() calls when switching from OIIO 1.x to 2.0.
(Proof: OSL master compiles against it and passes all tests, without a
single change.)
But after you upgrade to 2.0 and before 2.1 is released (I'm assuming in
the latter half of 2019 at the earliest), you'll want to either switch
all those calls to Strutil::sprintf()/errorf() and stick to the
printf-style formatting *OR* change them to
Strutil::fmt::format()/fmterror() using the new formatting specs.
Pretty much all of the OIIO internals (error calls, etc.) have been
changed to use the explicit printf-like forms. So they seem safe.
After 2.1, then, the old names format()/error() will be the new behavior
and you can switch back to those if you want. And by the time C++20
comes out, we'll all be on the same page, and some day OIIO won't even
need to use the fmt library because it can rely on std::format.
There you go. Here is the PR that implements the first phase.
We have put the relevant parts of fmt library in
include/OpenImageIO/fmt. We are currently using commit 01640f44, with
one fix by me on top of it that fixes a Windows compile issue. We will
update the version of fmt as fixes and improvements come.
Sorry for such a weird change in the middle of the beta period. But
sometimes a looming deadline of a code freeze and major release (after
which I wouldn't want to introduce this kind of disrupction) really
makes one focus about doing it right now instead of having to wait
potentially an additional year.
snprintf is a C++11 feature. No need for Windows to try to substitute
_snprintf, which in any case didn't behave quite the same (didn't add a
terminating null character in the case where the alotted space will
filled and the string had to truncate).
I had not correctly understood that __has_attribute is specifically for
GCC __attribute__((attr)) things, and __has_cpp_attribute is the new
C++20 test for standard [[attr]] style attributes. Fix up that use.
Also get rid of __has_builtin and __has_feature, those are gcc only,
not the way the standard works, and we don't use it anywhere.
Some other very minor fixes related to discerning which C++ standard
we're using.
I noticed that an open() configuration hint that causes non-RGB color
channels (such as YCbCr) to not get auto-translated to RGB was
implemented in three different plugins, but not all with the same
name. Settle on "oiio:RawColor" for all of them, and also update the
docs accordingly.
Spurious problems on Windows seem to happen with improperly closed or leaking file handles.
Solved by using gzopen_w on Windows to handle UTF-8 filenames allows us to remove the odd idiom of opening with fopen() then passing fileno to gzdopen (we were botching exactly who was responsible for duplicating or closing those handles).
Also add a zfile testsuite entry.
Fixes#1817
It was reported that if you filled a disk while writing a DPX file, no
errors would be reported, just leaving a corrupted file. (Admittedly,
filling disk volumes is not something I had thoroughly tested.)
The problem turned out to be threefold:
First, inside the DPX library (which was an imported project from
Patrick Palmer, which to the best of my knowledge is no longer actively
maintained outside of OIIO), the underlying OutStream class was used
incorrectly throughout its own library -- in particular, the Write()
method returned the number of bytes correctly written, but the return
results were almost always used incorrectly by merely checking != 0 (as
if it was a bool return for success, when in fact it should check for
any value != the number of bytes being requested to write).
To fix, I added a WriteCheck() method, which returns a simple bool
success/fail, and fixed the call points to use this function. I didn't
replace the original Write() method because there really was one place
where it expected the number of bytes written to be returned.
Second, in our DPXOutput::write_buffer() method, we weren't checking
the return results of the underlying call to m_dpx.WriteElement().
Now we do, and in case of failure try to check `strerror(errno)` in
the hopes of finding a useful system error (and in fact, in this case
of filling the disk, we do!).
Finally, in oiiotool's output_file() function, while we did check the
return codes for the call to out->write_image(), we neglected to check
the return code for out->close(), figuring that nothing could go wrong.
But, oho, the particular workings of our DPX writer are that the
write_scanline calls just batch output, and all the real I/O happens on
the call to close(), so the exact place where the error would occur was
the one place we didn't check. Fun. For most file format types, real
I/O happens in the individual write calls, so this was particular to DPX,
in addition to its other internal bugs described above.
How did I test? On my Mac I created a disk image with the smallest possible
space (around 8MB, I think), then tried to write a big file, like this:
oiiotool -pattern noise 8000x8000 3 -d uint16 -o /Volumes/SmallTest/out.dpx
which now results in the following error:
oiiotool ERROR: -o : DPX write failed (No space left on device)
which is about as good as we can hope for.
I also double checked what happens when we fill the disk while writing
tiff, exr, jpg, and they all correctly reported a write error.
So case closed, and thanks to Pavel Jurkas for reporting the DPX error
when the disk volume is full.
Update CONTRIBUTING to describe the new use of clang-format, and other
minor touch-ups.
Now that we have a LICENSE-THIRD-PARTY file, move some of the explanation
for the bits that originally came from NVIDIA there. Remove them from the
original files in which they appeared, which are in any case no longer
correct since code has moved around and in some cases those files may
no longer have any of the original NVIDIA-inspired code left.
[ci skip]
This fully uses clang-format on the entire codebase (except for a few
files, and a few sections of files, where we explicitly disable it for
reasons.
Henceforth, we will assume that any new code contributions should be
conformant and reject pull requests that are not. The basic idea is:
1. For large code changes, make sure you have a fairly recent version
of clang-format installed on your end, and use
make clang-format
to format the code before sending off the PR.
2. The TravisCI tests include one entry that merely does a 'make
clang-format' and considers it an error if there are any resulting
diffs (i.e. if clang-format needed to reformat anything). In such a
case, it will also print the diffs themselves if you click on the
failed test.
Note that for small patches and users who aren't easily able to have a
local clang-format copy, you can essentially skip step 1, submit your PR
as-is and then examine the failure diffs of step 2, making the requested
changes by hand and then updating your PR.
1. Allow embedded single or double quotes to group arbitrary text in
metadata name. This solves the problem that the default parsing of a
metadata name in the expression syntax uses "C identifier" rules. For
example, suppose somebody has put a metadata attribute in an exr file
named "foo/bar". If you do:
oiiotool file.exr -echo "{TOP.foo/bar}"
it will be wrong, thinking that it's looking up TOP.foo ("foo" attribute,
which doesn't exist) and dividing by "bar", which isn't a value it knows.
Instead, we now allow
oiiotool file.exr -echo "{TOP.'foo/bar'}"
It's not pretty, but it allows arbitrary strings to be used as metadata
lookups, like "foo/bar" or "foo.bar" or other ill-advised names. (Oh,
and for that matter, it was hard to retrieve "foo:bar", which we use all
the time, oops. Now it is possible by quoting.)
2. Ensure that retrieved "timecode" values, when retrieved as strings,
print as human readable timecodes ("00:00:00:00") not the underlying bit
patterns.
The sum of these means that you can set and retrieve timecodes like this:
oiiotool foo.exr -attrib:type=timecode "smpte:TimeCode" "01:01:01:01" -o out.exr
oiiotool out.exr -echo "{TOP.'smpte:TimeCode'}"
Fixes#2063
Also some tweaking to .clang-format rules did some minor changes
in other directories. Still finalizing our choice of values, we'll
lock it down by the last of these big reformat patches.
This didn't work properly. Example:
oiiotool -pattern constant:color=.1,.2,.3,.4,.5 256x256 5 -chnames R,G,B,A,Z -d half -o 1.exr
oiiotool 1.exr -d Z=float -o 2.exr
would end up with 2.exr having all half channels, not Z float as expected.
What can I say? It was a bug.
This first pass only reformats the contents of src/include/*. We will
tackle other directories in later stages. I wanted to start with just
one directory first, to give people a chance to object to the style
it imposes or tweak the format rules before doing the rest of the
code base.
For a minority of headers, I added a `// clang-format off` switch
because I was just too particular about the formatting of those files,
since I consider those public headers to be "documentation" as much as
working code, and therefore am extremely picky about certain ways I
strive to have them readable even when it involves breaking the usual
rules. Note that in some of these cases, even though I added the switch
to turn off future automated reformatting, I nevertheless made extensive
formatting changes to conform to the new prevailing style where I felt
it did not negatively affect readability.
I also turned clang-format off for a couple of files "imported" from
other projects (such as pugixml) in order to preserve our ability to
diff against changes in the original code. In those cases, I made no
other changes to those files except to add that one line to turn off
future reformatting.
No "code" has changed, except for a few very minor bits of clean-up I
performed at the same time:
* Removed a whole lot of `#ifdef OPENIMAGEIO_FOO_H` include gaurds,
changing them to use `#pragma once` instead. In a few cases, I did
leave a `#define` so that certain other source can tell if a file had
been included. (I have reasons.)
* Change NULL -> nullptr in ustring.h
Renamed LICENSE to LICENSE.md.
New file: LICENSE-THIRD-PARTY.TXT houses copies of all the licenses from
3rd party open source projects with code that we have incorporated in
any way into OIIO.
Updated several 3rd party package URLs that had changed over the years.
Break up our massive CHANGES.md file into pieces, storing the old
sets of release notes in src/doc/CHANGES-0.x.md and CHANGES-1.x.md.
Revamp CREDITS.md. It used to be separated into major and minor
contributors, but after years of rewrites and edits all over the code
base, it was no longer accurate (e.g., somebody who was a major
contributor for something in 2009 now may, after 10 years of expansion,
just be one of many people to touch that code). I no longer try to
distinguish size of contribution. If anyone cares to tease apart who
contributed to which parts specifically, they can look at the logs of
the files they are interested in.
Rebuild the PDF docs.
* Add a .clang-format file and a 'make clang-format' target
clang-format is a part of the LLVM/Clang compiler suite that will
reformat your code -- but much better than most code reformatters,
because this one is based on a compiler toolkit and is working on the
fully understood AST, not just text.
The idea here is that we give up a little bit of individual creativity
in code formatting, in exchange for every pull request trivially being
able to conform to the house style and never having to go through
multiple rounds of PR edits for formatting issues, nor ever have any
squabbles about it. I know that nobody will be 100% happy with the
require formatting (even me), but I think if we can make everybody 80%
happy and avoid those other problems forever after, it's worth it.
People I respect running othe projects consistently swear that adopting
clang-format was one of the best decisions they made.
Basically, after the initial cmake configuration, you can reformat
files with
make clang-format
Configuration variables CLANG_FORMAT_INCLUDES and CLANG_FORMAT_EXCLUDES
let you customize the lists of glob patterns that get included, or
excluded, from reformatting. For example, if you only want to reformat
one file (and you should try this to see how the whole thing works!):
make nuke
make CLANG_FORMAT_INCLUDES="exrinput.cpp" clang-format
Currently, I'm thinking that public header files will NOT be reformatted
(i.e. the exclude list will default to including src/include). I just
can't seem to get fully satisfied with the transformations, and I think
that's because I think of public headers as documentation, so I'm
extremely picky about the formatting and think that readability is
paramount, so I make little one-off rule breaks all the time to improve
the flow, readability, and appearance. Of public headers. I think for
everything else, I'm willing to live with a set of rigid rules.
I'm still experimenting with getting the parameters right. I may keep
tinkering a bit before committing any actual reformatting. For that
reason, I'm merely proposing the build script and style configuration
portions now. Only after people have tried it out and we think we can
live with it, will I do the great reformatting checkin.
Also, eventually, we'll make one of the Travis-CI tests do a
clang-format and fail if anything needs to be changed (and print the
diffs), thus making the CI make it very clear when any PR fails to
conform, and we will expect the submitter to run clang-format (or change
the offending parts by hand) and resubmit.
Addenda:
Make one Travis test matrix entry that only configures cmake and then
does a 'make clang-format'. Rig it so that if there are diffs produced
(i.e., the PR was not 100% conformant), it prints the diffs for
inspection and fails the test. If running clang-format produced no
changes to the source, the diffs will be empty and the test passes.
Also discovered the `name:` way of labeling Travis tests! Makes the
dashboard more readable.
Currently, no files are in the CLANG_FORMAT_INCLUDES list, so everything
should pass.
Detecting the OpenEXR version by parsing a C header can be unreliable. Since
we already use pkg-config to detect the presence of the library, we can also
use the version number it gives us, if available, and only resort to parsing
the header file if that failed.
See-Also: https://bugs.gentoo.org/668412
Fixes: ec4f679cd7
A situation came up recently that made me realize it would be good to
have a handy way of knowing whether an image is *almost* a constant
color (e.g., allow the channels in a pixel to differ within some
expected threshold and still be considered monochrome).
So this patch proposes to add an optional `threshold` parameter to
these three IBA functions. The default is 0.0, which is identical to
the old behavior and checks for exact matches only.
While I was in there, I noticed that these three functions were not
previously parallelized/threaded, so I added that as well. That should
speed them up considerably when running on multicore machines.
Mea culpa. Ramon Montoya alerted me to a problem in 2013 and I either
forgot about it, ignored it, or failed to understand it. But I
rediscovered his email while trying to purge my inbox of long-ago
unresponded emails and decided to look into it again.
The problem situation is that we figure out the filter ellipse based on
the derivatives, and then "thicken" it based on the amount of blur
requested. Ramon pointed out that a very uneven amount of blur (very
different in s and t) could thicken the minor axis so much more than the
major axis that, you know, they aren't the proper axes any more. And
that could lead to odd behavior later on, because the samples are laid
out along the major axis and assuming in all the math that majorlegnth
>= minorlength.
I tried a few approaches, and the most streightforward was just to
notice when this happens, and swap the minor/major (rotating the ellipse
by PI/2). This seems to basically work and looks quite good for the
common situations, although it's less than satisfactory for the
combination of very uneven blurs and extreme sheer that you get when the
ds and dt derivatives are nearly colinear (pointed in almost the same
direction or almost opposite, instead of the more typical and expected
nearly-perpendicular configuration). Though maybe it's right then, too,
I dunno, it's very hard to reason about what's "right" in that weird case.
That fix is 3 lines of code in adjust_blur().
The rest of the PR is comments trying to explain what works and doesn't,
and various improvements to the unit test visualizations (including
adding the ability to turn them on and control the amount of blur from
the env OPENIMAGEIO_TEXTURE_OPTIONS, drawing the ellipse axes, and drawing
the individual sample footprints in the visualization).
All my testsuite passes. Lookups should all be identical except for that
problem case -- certain situations of VERY uneven s and t blur amounts --
and in that case it's possible that you might have renders change the
appearance slightly.
We found that certain 'half' pixel values that involved "denormalized"
values (very close to zero) were inexplicably getting squashed to exact
0.0f in some circumstances but not others. Long story short, we traced
this to an older version of libopenjpeg (in turn pulled in by ffmpeg's
libraries) that set the x86 DENORMALS_ZERO_MODE upon its DSO load, and
left it like that, which is a truly terrible thing to do because it can
inject math bugs elsewhere into the app, even in parts that seemingly
have nothing to do with decoding jpeg-2000 images.
The code is only symptomatic in OIIO when we build with SSE support but
not with f16c instruction support -- the symptom is specific to the
series of ops we use to accelerate the half->float conversion with SSE
but without f16c. There could be other effects, but we haven't seen
them.
This seems to be fixed in newer versions of libopenjpeg. But we
encountered it, so others may also. It seems equally evil for our
library to reset the flag and leave it that way. However, for a full
application like oiiotool, we can safely set the flag, and so we do (if
the intrinsic is available, which unfortunately it is not in gcc 4.8).
I think that's the best we can do for now, out of several options that
were less than satisfactory. (A different set of instructions for h->f
conversion also avoided the problem, but was much slower.) We also advise
people to use a sufficiently new version of libopenjpeg to avoid this.
Along the way...
* simd.h methods to get/set the flush_zero and denorms_zero modes.
(Note: known gcc 4.8 bug omits access to those intrinsics.)
* Preemptively turn off the denorms zero mode in oiiotool, in case some
rogue library has turned it on.
* `oiiotool --stats` : Don't force a conversion to float when reading
the full image to compute stats on. This change also avoids the bug,
though in a brittle way. Still probably good to compute the stats on
the native data.
* fmath_test checks EVERY 'half' value and makes sure the h->f->h round
trip works properly.
* Change the behavior of `oiiotool --dumpdata` do try to dump the values
of each data type without conversion, rather than converting all to
float and printing the float values. This is a more true representation
of the raw data in the file, which I assume is why somebody wants to
do the --dumpdata.
* fmath.h changes to speed up bulk h[]<->f[] conversions by using
8-wide simd ops when f16c is available. (This is just incidental,
something I saw along the way.)
* `oiiotool --help` prints the number of cores, main mem size, and set
of processor capabilities (SIMD level, etc.) on the hardware it's
currently running on, and which set of SIMD capabilities was requested
at build time.
As pointed out by Thiago Ize and John Haddon (but unfortunately not
heard by me until literally years later), this is error prone because
there are times when a ustring is converted to an int/bool in unexpected
situations. It even led to verifiable errors in OSL that I only
discovered after commenting out this operator to see what would happen.
Neither std::string nor std::string_view have this operator, bolstering
Thiago & John's arguments.
So we are removing it from OIIO 2.0. Hopefully this won't break too many
people's apps that happen to make use of ustring. The fix is simple, just
change:
if (myustring)
to
if (! myustring.empty())
Add a bumpformat option for the bumpslopes exporter
This option prevents situations where the bumpslopes exporter fails to detect an input bump format as an height map. The typical situation is a 3 channels RGB height map we would expect to be monochromatic (R=G=B) but that has subtle color variation introduced by a texture publication process (typically a precision or rounding error). In such situation the exporter will automatically interpret the bump map as a false normal map, resulting in a corrupted texture output.
The bumpformat option allows the user to bypass the automatic detection and manually specify the input bump format.
For the "null" reader, we were clobbering any configuration hints about
the "full" (i.e. "display window") size parameters. This had bad effects
when using a fake "null" image along with ImageCache::add_file() in
order to set up an in-memory buffer that behaves like a texture, if that
buffer was a crop region or had overscan.
* On OSX, need to install gcc in order to resolve a missing symbol that
numpy needs.
* On Linux, remove the "sudo: false", Travis is phasing this out.
* Make sure that any branch with "travis" in the name runs all travis
tests, not the abbreviated set.
These were never documented, never made part of the test suite,
and have not been maintained to keep up with the evolving APIs.
They serve no real purpose, so just get rid of them.
Fixes#1621
1. Turn join() into a template that can act on any iterable container
of objects that allow stream output.
2. New splits()/splitsv() that directly return a vector of string or
string_view, respectively (more modern C++ idiom than passing a
result vector as an argument).
3. A second version of extract_from_list_string that directly returns
a std::vector (instead of being passed as an argument).
4. Minor fix in unittest.h to allow the OIIO_CHECK_EQUAL macro work to
compare std::vectors. (This utilizes the new join() to turn them into
strings and thus allow them to output with <<).
* Make an official FindOpenImageIO.cmake, borrowed from OSL but with some
minor changes. It will install into INSTALL_PREFIX/share/cmake/Modules
* We hope that downstream projects that have OIIO as a dependency will
use/copy this file, rather than having a dozen or more slightly
different versions that downstream projects all write themselves. And
if there are any deficiencies, they will push fixes or improvements to
this master copy.
* Change places where we expect the cmake and/or environment variable
OPENIMAGEIOHOME to be named OPENIMAGEIO_ROOT_DIR (more standard naming
convention by modern cmake standards).
* Simplify FindOpenEXR.cmake, to conform to conventions as well as to
get rid of some flexibility that is nonstandard and that we didn't use
anyway.
* Some other very minor fixes to the cmake files (INSTALL_DOCS was
declared in two different places, etc.)
If read_scanlines (et al) was called with the 'format' parameter as
TypeUnknown -- which is supposed to mean "use the native format" --
would accidentally and silently fall through the switch statement inside
OIIO::premult(), causing it to do nothing.
If called with an explicit type, it all worked find. Subtle to track down.
The particular symptom was that if you were calling ImageBuf::read()
with `force=true` but relying on the default `convert` value of UNKNOWN
(meaning, no conversion, stay in native format), it would trigger this
bug and neglect to premultiply the unassociated colors.
Allow builds outside the source tree to pass testsuite. Defaults to finding oiio-images, openexr-images, libtiffpic in the usual ".." from the main oiio source directory, but now it can be overridden with the CMake variable OIIO_TESTSUITE_IMAGEDIR.
Also minor change to append to CMAKE_MODULE_PATH rather than clobber it.
* ImageBuf: more careful tracking of allocations, print alloc/free/total
messages when "debug" attrib (or env OPENIMAGEIO_DEBUG) > 1.
* Change interpretation of the debug attribute so that 1 is ordinary
debug, values > 1 are extra debugging for developers.
* Add oiio query for process resident memory in MB (this is helpful
for Python scripts that want to check how each OIIO call they make
might have changed the overall memory footprint).
* Fix crash deleting not owned imagecache
Now that we allow providing an external ImageCache to the texturesys,
destroy handles its destroying and it shouldn't be done in the
destructor.
* Override existing file entries in add_file
Instead of assuming the present file is equivalent, nuke it and reset it
with the new parameters. This fixes a problem adding files after
somebody has asked for a handle. Also allow non mip-mapped files when
coming from user buffers.
* Add an optional argument to TS::create() that lets the caller directly
supply a pointer to an IC (which will not be freed when the TS destroys --
it's owned by the caller).
* Add a TextureSystem::imagecache() method that returns a blind,
non-owning pointer to the underlying IC.
* Clean up the 'destroy' methods, which previously for the sake of not
breaking compatibility mid-release had two separate varieties, but
clearly they can be one signature with an optional argument.
on systems with multiple Python versions installed.
Make sure Travis tests have at least one 3.x test case (OSX, it's easiest).
Additional fixups for Python3.
Some extra SPI specific bending over backwards for our weird install
locations. And uniformize the spcomp2 vs arnold build scripts. And
remove the "lion" cases, we don't use that any more.
We were previously reporting "oiio:ColorSpace" for raw images always as
"linear", but that's not really how the underlying libraw operates. It
converts to one of several color spaces as part of the debayering
process (by default, sRGB), so make sure to report back the correct one,
which should be the one you ask for. Also, no need to set the
"raw:ColorSpace" attribute, that doesn't really tell you anything
useful (it just repeats back the hint you asked for, but that's the same
as the "oiio:ColorSpace").
The point of this feature is that your app has a buffer of pixels, and
you want the IC or TextureSystem to use that buffer directly, without
copying the texels or owning the memory. It should also not count the
memory against the limits of the cache.
See imagecache_test.cpp, the test I added as text_app_buffer() which
pretty much spells out how to do it from the app point of view.
From an API standpoint, all that changes is an optional argument to
add_tile that lets you disable the copy.
You need to make it a tile of an image in the cache, though, and the
obvious no-drama choice is just to use a "null" ImageInput as a
placeholder. In the process, I noticed two helpful enhancement I could
make to the null ImageInput better:
1. Make sure that passing a "config" spec results in the right resolution
and data type -- in fact, there was a bug where the config's res would
be overwritten by a 1024x1024x4xUINT8 default. Oops. So only overwrite
those fields if they are not set in the config.
2. What if you want one of these fake inputs but you don't want the name
of the texture to be "blah.null"? So I added it so that if the config
has metadata hint "null:force" set to nonzero, it will always succeed
in creating a null ImageInput.
* Reorganize how texture3d transforms points to local space of the volume.
What happens when texture3d is called on a point? There was a special
back door for Field3d, but this didn't generalize well. For example, we
want to support texture3d lookups from OpenVDB files (coming soon under
a separate PR -- this is a necessary prelude).
This patch:
* Makes each ImageCacheFile::SubimageInfo contain a unique_ptr to a 4x4
matrix (so it doesn't use as much space except when needed, which is a
small minority of all texture files).
* If reading the spec for that subimage yields a "worldtocamera" or
"worldtolocal" matrix metada, we allocate the space and stash the
matrix there.
* The texture3d() call looks for the matrix, and if found, uses it to
transform the input point to the local volume space. Only if this is
not found does it check for the special field3d back door (which
exists because f3d supports time-varying transforms that aren't a
single matrix). If that, too fails, then no transform is applied.
* Some unused fields that were disabled by USE_SHADOW_MATRICES were
removed entirely, since we have no current plans to implement shadow
maps (and could easily re-add if we change our minds).
No longer need to build an lgritz private one, since 2.3.0 has some
fixes we needed for gcc7 and C++17 compatibility.
Also move the OpenEXR download and installation into the ext/ subdir
where we put other build dependencies.
For SPI, change the site/Makefile-bits files so that OPENEXR_HOME env
variable can override the default system install of OpenEXR (makes it
easier for me to test custom builds).
The bumpslopes exporter originally assumed that a 3 channels bumpmap input corresponds to a
normal map. However, depending on the bumpmap creation process, it's not unsual to end up
with grey RGB heightmaps that we incorrectly interpret as normal map.
To prevent this situation we added a safety check to determine if the input map is
monochromatic or not.
The X_density and Y_density fields in the JPEG header are uint16. We
were botching it if it happened to think the pixel densities were
expressed as values needing > 16 bits. Do some refactoring and handle
this case cleanly.
Fixes#1999
Prompted by Justin Israel's patch 1996
... and in the process of making a good unit test, I found a minor bug
in the ImageSpec ctr from an ROI -- the data window was set properly, but
the display window (aka "full roi") got the proper width/height but not
the right origin. Yay unit tests!
* Fix simd.h warnings revealed on clang 7.0 (1995)
C++17 allows "constexpr if" which would have fixed this. But with C++11/14
and clang 7, we get itcky warnings about some of the simd_test generating
code (that will never run) that overruns some bounds. Solved with some
template ugly.
* Fix clang 7 errors (1995)
It really doesn't like some 'const' issues on boost thread. We only use
it a couple places.
Replace boost::call_once and boost::hardware_concurrency with std versions.
There's no std equivalent for physical_concurrency(), so just the
problematic compiler, punt and implement it as hardware_concurrency.
Like resize, but it preserves the original aspect ratio, resizing to
fill the ROI as well as it can without stretching.
This is really just moving the logic of "oiiotool --fit" into its own
ImageBufAlgo function.
This was very incomplete before. It broke under libraw 0.19, and
investigating I realized that libraw had some recently-added bits
(as of 0.18) that lets met get at a whole lot more metadata. Also, we
now support many camera makes that we didn't before, and have added
new tests to reflect this.
Re-enable Travis testing of libraw on OSX, we had disabled it a few
commits ago, when libraw 0.19 was released and started failing tests.
This fixes it better than it was before.
It's an abomination, but it's one we have to sometimes live with.
This patch has several small goodies for people who are forced to
deal with "unpremultiplied" images.
* A variety of fixes to oiiotool --no-autopremult, which turns off the
automatic premultiplication that is ordinaryily done for
unpremultiplied inputs. It was only working for inputs that were
backed by the ImageCache, which excluded small files read directly,
and also excluded --info results (which bypass the cache and to
straight for an ordinary ImageInput).
* iv: add a `--no-autopremult` flag, much like the one for oiiotool.
This makes it easier to use iv to examine the unmodified input values.
* TIFF read: be sure to set an attribute called "tiff:UnaociatedAlpha"
to indicate that the original input was unpremultiplied in the file,
regardless of whether the usual auto-premultiplication was used upon
input.
refcntptr::release() sets the pointer to null and reduces the ref count,
returns the original pointer, but does NOT free the object even if the
new ref count is 0.
Alphabetically, but case-insensitive, locale-independent, and with
"un-namespaced" metadata appearing before "namespace-prefixed" metadata.
The main purpose is to canonicalize the order of appearance of metadata
in test reference output. This minimizes the amount of reference output
churn that happens when we change code of image format readers, or upgrade
reader libraries, or add support for new metadata.
Add iless(), case-insensitive locale-independent string_view comparison.
Analogous to iequals, etc.
Also augment the StringEqual functor with StringIEqual, StringLess,
StringILess.
These collectively make it a bit more convenient to do certain
case-insensitive string sorting.
We have a growing matrix of tests, and there is development friction
from the fact that every push to a working branch kicks off the whole
matrix, many of which exist to test very specialized things that are
probably not changed by the vast majority of patches.
TravisCI recently started supporting "conditions" that can let you limit
which test cases or stages to run.
I'm proposing that the full test matrix always run when it's:
* a pull request
* a direct push of an updated "master" or "RB" top of tree.
* a nightly cron jobs (which, also, are only for "master" and certain
"RB" branches)
But in other cases -- that is, ordinary work branch pushes while we
develop -- there are many tests that don't really need to run every
time, and just clog up our Travis queue. So I'm adding the following
exclusions:
- The tests that exist solely to make sure we don't break simd.h (builds
that exercise AVX2, avx512, and no SIMD) will now only build for
working branch pushes if the branch name includes the substring
"simd". (Pro tip: if you are editing simd.h or using it heavily in
your modified code, name your working branch to include that substring
to force those tests to run on every push.)
- The tests for gcc8 and/or C++17 (which aren't yet documeted as blessed
by VFXPlatform) only run for working branch pushes if the branch
includes the substring "gcc" or "17", respectively. (Note: since
gcc6 and C++14 *are* VFXPlatform blessed, those tests are never
excluded.)
- The test that runs sanitizers, which takes ~4x longer to run than
other tests, only runs from ordinary working branch pushes if the
branch name includes the substring "san".
The neat thing is that with the exclusions, an ordinary working branch
push that not have its name indicate that it cares specifically about
simd, san, gcc8, or C++17), only has 5 tests in the remaining matrix,
which is exactly the amount that TravisCI will parallelize. Thus, the
new "common case" will complete its CI in roughly the time it takes to
do just one matrix entry (~10-15 minutes), versus the full test suite
that would have taken 3-4x that.
Remember that this is just for working branch pushes. Once you submit
your work as a PR, the full test matrix will be run.
Add a build-time option to select which std::string ABI to use.
Building with GLIBCXX_USE_CXX11_ABI=1 uses the new (C++11 compliant) ABI,
building with =0 uses the traditional gcc string ABI for compatibility.
This only affects gcc builds.
Also automatically define _GLIBCXX_SANITIZE_VECTOR=1 when compiling for
sanitize mode, it turns on some extra instrumentation to detect running
off the end of a std::vector but still within its allocated capacity.
libraw on homebrew just upgraded to the new 0.19.0. This breaks our raw
tests (builds ok, but seems to not find all the metadata). I'm not sure
if this is a bug in 0.19.0, or if the API changed in some way that we need
to change things on our side. But in either case, it's a more lengthy
investigation than I have time for today, so disabling it on OSX Travis
tests seems like the expedient workaround. Libraw (0.15) is still exercised
on the Linux Travis tests.
As soon as I track down the real problems satisfactorally, I'll restore
the tests.
I found that the one we'd been using has some pesky issues with C++17
that sometimes fails to match because of noexcept problems.
The equally short one from LLVM (UIUC license, for all practical purposes
equivalent to BSD) appears to work and does not suffer from these problems.
So just swap out the implementation.
Note the change to test_array_view_initlist... the warning text from gcc
was nearly impossible to understand, but we were indeed violating the
rule about the data needing to have a lifetime that outlasted the
span. Rearranged the code to be compliant, and the warning went away.
Field3DInput::valid_file locked, then called open() which also locked.
How did this ever work? Yet, this was only symptomatic in one internal
renderer test, and only when OIIO was compiled in DEBUG mode.
The simple solution is just to change to a recursive lock. Field3d isn't
an important enough file type to try to be more clever than that.
We make four large changes across the board in ImageBufAlgo.
1. No more raw pointers.
All functions that take per-channel constants, or destinations for
per-channel outputs, in the form of (pointer, length) -- or worse, just
a pointer and an assumed length big enough to hold all channels -- are
being deprecated, in favor of passing span<>. This will let the IBA
function implementations check lengths and deal appropriately, also
makes it very easy for the caller to pass a single float that will be
understood to be replicated for all channels.
2. ImageBuf return results
All existing functions of the form
bool IBA::func (ImageBuf& result, ...input args...)
are getting an additional variant,
ImageBuf IBA::func (...input args...);
For most ordinary uses, this transforms the awkward
ImageBuf Result;
ImageBufAlgo::add (Result, A, B);
into the more compact and intuitive
ImageBuf Result = ImageBufAlgo::add (A, B);
Through the magic of C++11 "move" semantics (aka "rvalue assignment"),
this does not incur an unnecessary copy of the whole ImageBuf.
The old function forms will remain as first class citizens and are stil
useful in cases where you are repeatedly doing operations (so want to avoid
repeated allocation), or you want to use ROI's to write partial results of
the operation into a sub-region of an existing image.
This carries over to the Python API as well, where the old idiom
Result = ImageBuf()
ImageBufAlgo.add (Result, A, B)
was especially "non-Pythonic" looking, compared to
Result = ImageBufAlgo.add (A, B)
3. Reduce combinatorics of Image vs constant.
For some functions, such as mad(), which computes A*B+C, a whole bunch
of clutter is necessary to handle the various combinations of
image * image + image -> mad (const IB&, const IB&, const IB&)
image * const + image -> mad (const IB&, span<const float>, const IB&)
image * image + const -> mad (const IB&, const IB&, span<>)
const * image + const ...
etc.
In fact, we need 7 variants (excluding const*const+const, which is not
useful because you don't know how big the resulting image would be).
These have all been replaced by a single variant,
mad (Image_or_Const A, Image_or_Const B, Image_or_Const C)
using a special new class just used for parameter passing,
Image_or_Const, which can "wrap" either an ImageBuf or a float or a
span<const float>, and let the implementation sort it all out
internally. It really de-clutters the declarations! And it also forces
us to ensure in these cases that all the combinations are accounted for.
4. Some IBA functions that generate structs with data, such as compare(),
computePixelStats(), and histogram(), now directly return those structures
instead of needing them pre-declared and passed as reference paramters.
Thanks to the magic of C++11 'move' semantics, this is no longer expensive,
and it's a more intuitive way to structure the API. (The old versions of
the functions still are available, but are considered deprecated and will
be removed in future releases.)
In the process of these changes, I've done a substantial rewrite of both
the IBA and Python chapters of the docs, as well as all the verbose
comments in imagebufalgo.h itself that explains the functions.
What happens when you use IC/TS to retrieve metadata and get the type
slightly wrong? Like if the actual metadata is an OpenEXR box2f (which
OIIO represents as TypeDesc(FLOAT,VEC2,NOXFORM,2)), but you have asked
to retrieve it as float[4] (OIIO TypeDesc(FLOAT,4)).
It was failing, because the array lengths didn't match. But I think it
should succeed because they are both 4 floats underneath. This is
especially important if the query is coming from OSL, which doesn't have
a vec2 or a box2f type, and the only way you can ask for this metadata
would be into a float[4].
Simple fix: check for compatibility of basetype and basevalues(), not
for matching arraylength().
Add a test case for this. It worked fine on the runtime side. The
maketx side needed a minor fix to know that overscan was ok with clamp
(instead of forcing black wrap for all overscan cases).
Just discovered the --file-line-error option to pdflatex, which causes
error messages to explicitly mention the file and line number in the
.tex. Saves so much work! I can't imagine why that's an option. Who in
their right mind would want pdftex to halt on error, but make it
difficult to debug by not telling you what line and file the error
occurred on?
Also fixed a stray nonstandard character, that I guess was missed by
older TeX but caught now that on my personal machine I upgraded LaTeX
to a more modern TeXLive release.
[ci skip]
Now array_view is a synonym for span, rather than the other way around.
Both will work for legacy client apps for some time, but for OIIO internals
we should start favoring span when convenient.
A few minor fixups to make some existing methods and in-class types
match the C++ standard. One change, as dictated by C++20 std::span<>
specification, is that span.size() returns a ptrdiff_t (signed) rather
than a size_t. This may create some minor build issues with signed vs
unsigned warnings. Something to watch out for.
Spotted an obvious typo in make_numpy_array.
The manefestation of the bug is that ImageInput.read_scanline would
make the wrong "shape" of the numpy array, mashing all channels of all
pixels in the scanline into a 1D array of values, instead of properly
making it a 2D array of pixels which are each arrays of the channels
within a pixel.
Something exactly like our array_view<> template is coming to standard
C++20, but it's named span<>. Slowly, we will migrate our nomenclature
and behavior to match it exactly, eventually (I assume in the late
2020's) just switching entirely to std.
In this first step, we merely establish a span.h that defines span<> as
a template alias for array_view<>. This will let applications (and our
internals) freely use array_view or span as synonyms. I will even
backport this change to older branches, since it's totally harmless to
existing code that uses array_view.
In a later step (not this patch) in master only, we'll swap those roles,
so that moving forward, span<> will be the "real" template, and
array_view will be the synonym for compatibility with legacy apps.
The existing methods invalidate() and invalidate_all() fully invalidate
everything the IC/TS knows about a file (or all files) -- the specs,
the pixels, etc.
But sometimes you just want to close any currently-open handles, but not
necessarily forget about what you know about those images, or any pixel
tiles in the cache. That wasn't possible before, but this patch adds the
functionality as close(filename) and close_all() methods to both IC and
TS.
One interesting use case that this enables is a render-paint loop:
render();
ts->close_all();
... here, it's ok to modify the texture files ...
ts->invalidate_all (false);
render();
... repeat
The close_all(), performed after the render, closes any open file
handles, which is especially important on Windows where the IC holding
the file open essentially locks the files. Then, right before the
next render, invalidate_all(false) will fully invalidate any files that
have been modified since they were last opened, but not any whose
modification time has not changed.
Note that the act of closing means that if additional tiles need to be
read from disk, it will need to re-open the file. But as long as
any pixels needed are already in the cache from last time, it may not
need to re-open at all.
To make it easier for apps to detect if the new methods are available,
we define preprocessor symbols OIIO_IMAGECACHE_SUPPORTS_CLOSE and
OIIO_TEXTURESYSTEM_SUPPORTS_CLOSE in imagecache.h and texture.h,
respectively.
We had set_roi_full() to change the display window metadata, needed an
equally easy way to alter the pixel data window origin. (But note that
for pixel data, it's only safe to change the origin with metadata, not
the size, which would change the whole pixel data layout, lead to
reallocations, etc.)
I really over-engineered the previous commit. The awkward
unique_ptr-with-deleter is not needed after all. I don't need the
deleter to live in every individual plugin, as long as I can ensure that
the allocation and free both happen in the same place -- libOpenImageIO.
And I can do that by overloading operator new and operator delete, then
using the standard unique_ptr flavor.
This patch backs out of much of the previous one (no individual deleter
functions are needed). Simpler. Not sure why I didn't realize earlier
that this would work.
ALERT! This is a breaking change! But I think it's a simplification,
more modern code style, make using OIIO easier and less error prone.
Previously, ImageInput::create(), ImageInput::open() [the static one],
and ImageOutput::create() returned a raw pointer, allocated by the
plugin's create function, and it was up to the client code to properly
manage and free it.
On Windows, it's not safe to delete in one dll when the new came from a
different dll. That's why we had ImageInput::destroy(), but
unfortunately I realized recently that only works properly for the
readers/writers compiled into libOpenImageIO itself. It could
potentially fail with mysterious heap corruptions if you used a true dll
plugin to implement a custom format reader/writer.
This patch proposes to change all of these functions to return a
unique_ptr with built-in deleter, so all resources are managed and will
automatically be released when the pointer goes out of scope or is reset.
Because the deleter is supplied by and in each individual module where
the new occurs, the possible heap corruptions are fixed once and for all
for ordinary C++ ImageInput & ImageOutput usage.
It touches a LOT of files, but it's repetition of conceptually simple
changes:
1. Some new type definitions are added to the ImageInput class definition:
/// Call signature of a function that creates and returns an ImageInput*
typedef ImageInput* (*Creator)();
/// Call signature of an ImageInput deleter.
typedef void (*Deleter)(ImageInput*);
/// unique_ptr to an ImageInput, with custom deleter.
using unique_ptr = std::unique_ptr<ImageInput,Deleter>;
/// Return an empty unique_ptr (no default ctr of unique_ptr with deleter)
static unique_ptr empty_ptr() { return unique_ptr(nullptr, destroy); }
2. For each plugin, in addition to a creator like this:
OIIO_EXPORT ImageInput *tiff_input_imageio_create () { return new TIFFInput; }
the plugin must also supply a deleter:
OIIO_EXPORT void tiff_input_imageio_delete (ImageInput *p) { delete p; }
3. The signature of the static ImageInput::create() and open() methods
change to returning a unique_ptr, for example, from
static ImageInput *open (const std::string &filename,
const ImageSpec *config = NULL);
to
static unique_ptr open (const std::string &filename,
const ImageSpec *config = nullptr);
4. App code therefore gets a lot safer and easier. Old:
ImageInput *in = ImageInput::open (filename);
in->read_image (...);
in->close ();
ImageInput::destroy (in);
new:
ImageInput::unique_ptr in = ImageInput::open (filename);
in->read_image (...);
in->close ();
and if you are a fan of `auto` and take advantage of the fact that
destroying an ImageInput forces a close if not already done, it's just:
auto in = ImageInput::open (filename);
in->read_image (...);
5. Analogous changes for ImageOutput.
6. Note: It's a little tricky to default-construct a unique_ptr that
has a deleter.
ImageInput::unique_ptr A = ImageInput::open (filename); // YES
ImageInput::unique_ptr B; // NO! Compiler rejects!
If you need to do this, use the empty_ptr method:
ImageInput::unique_ptr B = ImageInput::empty_ptr(); // YES
7. The OIIO_PLUGIN_VERSION definition has been changed from 21 to 22.
So if you have client code that must work with new and old OIIO APIs,
it might be handy to do something like this:
#if OIIO_PLUGIN_VERSION >= 22
ImageInput::unique_ptr in = ImageInput::create(...);
#else
ImageInput *in = ImageInput::create(...);
#endif
* unique_path() and temp_directory() were not properly handling UTF-8
results on Windows.
* A whole lot of #if cleanup by adding u8path and pathstr functions.
We already were careful when opening the file to use the
OpenEXRInputStream to ensure proper UTF-8 filename handling.
But valid_file() used Imf::isOpenExrFile, which is not careful enough.
So we instead switch to making an IStream and then calling the version
of isOpenExrFile that takes the stream. Should be correct.
I didn't realize that the prior number I picked inadvertently included
a 4.9 version that did not in fact support this feature.
Changing to the new value will not have ny more "false positives" (error
by incorrectly thinking the feature is present), but does make a "false
negative" (avoid the feature even though it's supported) for gcc 5 and
gcc < 6.2, even though it would work there. That will still behave
correctly, though at a very slight performance penalty, but VFXPlatform
skips from gcc 4.8 to gcc 6.3, thus leapfrogging over those false
negative cases. So I'm going to not care that we get less than maximal
performance from some compiler+library versions that we are unlikely to
use.
ImageOutput and ImageInput supports("ioproxy") will return true for a
format reador or writer that supports the ability to use a
Filesystem::IOProxy object for customized I/O.
For supporting formats, the caller can pass the IOProxy via
a special attribute in the ImageSpec that is passed to
ImageOutput::open() or the configuration ImageSpec passed to
ImageInput::open():
"oiio:ioproxy" (PTR) is a pointer to a `Filesystem::IOProxy`.
One use for this is to read an image "file" from an in-memory buffer
(using the IOProxy subclass IOMemReader), or to write an image "file" to
an in-memory buffer (using the IOProxy subclass IOVecOutput).
The documentation (in the ImageInput and ImageOutput chapters) and
imagebuf_test.cpp contain examples showing how to do this.
It's also pretty straightforward for both ImageBuf and ImageCache to do
this, since in both cases there is a way to declare the configuration
spec to use when the file is opened.
We implement both reading and writing with I/O proxies for OpenEXR,
and writing with prixies for PNG. More format support will be added
as needed.
Turns out it was harder than I originally thought to absolutely ensure
that a config spec is associated with any opening of an ImageBuf,
incuding when it draws on ImageCache.
Along the way, this necessitated a new variant of ImageInput::create().
So be it.
IOProxy is an abstract base class that presents a very simple API that
encapsulates fundamental file methods: seek, tell, read, write, pread,
pwrite, close.
We implement three concrete subclasses of IOProxy:
* IOFile : wraps a FILE*, can be used for either input and output (but
not both).
* IOMemReader : wraps a memory buffer, so it can be read "like a file".
* IOVecOutput : wraps a std::vector<unsigned char>, so writing "like a
file" stores the bytes in the vector.
* Disable SSE for 32 bit Windows patforms
It's unreliable to get clean builds on some of these platforms, and very
hard for us to test thoroughly. We presume that anybody needing high
performance badly enough to want SIMD also is on a 64 bit CPU, so
falling back to scalar on these chips is probably fine.
* Fix up const sloppiness in simd_test.cpp
The previously existing spec() is not thread-safe, since it depends on
retained state of the last seek_subimage. It also returns a const
reference to a single ImageSpec, so it's not even safe to access the
resulting reference if there may be other threads that might change the
current subimage or miplevel.
The new spec(s,m) is thread-safe, and it COPIES the result. The base
class default implementation just locks, does the seek, and
copies. Individual ImageInput subclass implementations may do something
more clever, for example if they are already inventorying and retaining
the ImageSpec's for all subimages, they may not need to lock or seek at
all.
Beware, though, that because it makes a full copy of the ImageSpec, it
may be expensive if the spec in quesiton contains a lot of arbitrary
metadata (which need to be allocated and copied individually).
So there is a second thread-safe method, spec_dimensions(s,m), which
only copies the dimensional and type fields of the spec (just like
ImageSpec::copy_dimensions()), omitting the channel names and the
arbitrary named metadata in the extra_attribs field. This is very
inexpensive copared to copying the rest of the metadata.
Overload these with custom versions for OpenEXR and TIFF. For OpenEXR,
we already collected all the specs, so we can return it without doing a
lock, seek, or read at all. For TIFF, we change to save the specs (not
unlike what we did all along with OpenEXR), and now for the cost of a
shortly-held lock, we can usually avoid the seek and read and just copy.
std::shared_ptr<T> is thread-safe with respect to the "control block",
which means that two threads that each hold their own shared_ptr to the
same object can safely reference copy, or release their shared pointers,
and all is fine. When the very last thread holding a shared pointer to
the object releases it, the object will be deleted.
But a single shared_ptr itself is NOT thread-safe. You have to control
access to a shared_ptr if it may be referenced by multiple threads, as
is the case for the shared_ptr<ImageInput> in the ImageCacheFile. So
the ICF has a m_input_mutex that threads held for quite some time when
they need to access anything about the ImageInput.
This patch is a refactor that eliminates most of those locks. Instead,
when a thread needs to use the ImageInput, it uses a new thread-safe
get_imageinput() method that returns a shared_ptr safely, using the
wonderful std::atomic_load(const std::shared_ptr<T>* p) method that
lets us get a copy of the shared_ptr atomically, with no lock. The
original m_input_mutex is now only held in a few circumstances, and
for less time when it is needed.
The idea is that this should help remove some thread bottlenecking in
the TextureSystem when heavily threaded.
This is also heavily dependent on a previous change that made several of
the thread-safe ImageInput::read_{tiles,scanlines,image} take explicit
subimage and miplevel parameters (which removed the need for any
stateful calls to seek_subimage),
The point of all this is to take client code that looks like this:
lock(); // necessary because of implied state of the current
// subimage/miplevel between the seek_subimage and
// subsequent read.
imageinput->seek_subimage (subimage, miplevel);
read_scanlines (ybegin, yend, z, pixels);
unlock();
and replace it with this:
read_scanlines (subimage, miplevel, ybegin, yend, z, pixels);
with no implied saved state (at least, none that is relevant and visible
to the client), and thus with no client-side lock necessary.
* Change signature of seek_subimage to no longer take an ImageSpec&,
to avoid an obligatory ImageSpec copy that is often unnecessary.
The preferred idiom, if one must call seek_subimage at all, is to
call it and then separately copy the spec(). For back compatibility,
a non-overloadable entry point exists that matches the old call
signature, but should be considered a candidate for deprecation.
* All of the read_native_*() methods take subimage and miplevel
parameters, are expected to do a seek_subimage if necessary.
For the read_native_deep_*() methods, which are the only ones that
user could would ordinarily have called, back-compatible entry points
exist (but should be considered deprecated and not thread-safe).
* New varieties of read_scanlines, read_tiles(), and read_image() are
available that takes explicit subimage and miplevel parameters (and
thus do not require a stateful prior call to seek_subimage).
* All of the read_* calls that take explicit subimage/miplevel parameters
are guaranteed to be thread-safe against concurrent calls to any other
of the read_* calls with explicit subimage/miplevel (but not necessarily
against any other ImageInput methods).
* The ImageInput base class contains a mutex now, to be used internally
by functions that need to lock.
* The generic implementations of read_scanlines/tiles, and the specific
implementations for TIFF and OpenEXR, are very carefully constructed
to hold their internal locks for as little time as possible, only when
required by the underlying format-reading library).
Extend the recent work on reading "raw" strips and then decompressing
them ourselves concurrently, to handle "separate" planarconfig as well
as "contiguous". This required another big refactor of
read_native_scanlines.
With this change, even contig images now take the fast path for some
combiations that it didn't before: uncompressed, and zip compressed but
without the horizontal predictor selected.
Calls to TIFFSetDirectory are redundant if the file was just opened
and you are positioning to the first subdirectory.
Yet, libtiff seems not to notice that, and will happily repeat a bunch
of the header reading that it already did for the TIFFOpen.
So making sure that we only call TIFFSetDirectory when we need to
actually change the current subimage, saves some work (and in fact
can approximately double the speed of reading the header).
When we write an IB, it presumably changes the pixels, so we should
invalidate the file we're writing if it's in an ImageCache. Also, be
careful that were not writing an "in-place" file that hasn't been read
fully yet (because it's IC-backed).
The recent patch that greatly speeds up reading whole TIFF images,
where we do our own decompression, forgot to consider the case where
the TIFF was written with the opposite byte order.
We realized that for the error condition of the optional "subimage"
specifying a nonexistant subimage would issue an error and return
without writing any values into the location of the color result.
The proposed patch treats the nonexistant-subimage case just like we
always treated the nonexistant-texture case: issue an error and store
the "fill" color in the result, unless the "missingcolor" optional
parameter is supplied, in which case return the missingcolor and do
not issue an error.
This patch stores slopes values in i,j texture space rather than normalized s,t space to prevent
floating point overflow with half precision float format.
testsuite update
This sets us up for planned changes to use the ROI class more extensively
in other imageio.h declarations.
Add roi(), roi_full(), set_roi(), and set_roi_full() methods to
ImageSpec.
Try to constexpr everything possible in ROI.
Add HOSTDEVICE decorators to some inline hash functions. Guard some color conversion functions when compiling for CUDA devices. Use the CUDA funnel shift intrinsic for left rotate when it is available.
The C++11 `override` keyword is helpful in preventing coding errors by
ensuring that your subclasses aren't using virtual functions with the
wrong signature (and thus making a new, useless methods rather than
correctly overloading the method you intended).
Remove some outdated material that implies that dither and quantization
values are stored in the ImageSpec. They have been gone from there for a
long time.
[ci skip]
Expand the set of files to all the imagebufalgo_*.cpp, which are the
most likely files to need it because they tend to have template expansion
for big sets of data type pairings.
FIXME: Does anybody know if there is an advantage to limiting it to
just the files that most need it? Or is it better to make it totally
foolproof by using /bigobj for all our modules on Windows?
OpenEXR's library has a neat feature where it maintains a thread pool
and if you are using the multi-scanline or multi-tile I/O API calls, it
will do the compression/decompression of the scanlines or tiles in
parallel, even though the disk I/O itself is serial (but the codec work
accounts for most of the time, so this is a big speed win). The result
is that with many threads, a 5-7x speedup in I/O is typical, versus the
single threaded case.
libtiff doesn't do this -- it is not thread-safe or reentrant (for a
single file), and lacks API calls to write multiple tiles or scanlines
at once (small exception: it will output a "strip" of several scanlines,
but not in a flexible way like OpenEXR's). So any straightforward use of
libtiff means using one core to do all the work writing (or reading) a
file, and all the other cores just waiting around, unable to contribute.
So with this patch, I take an end run around these libtiff limitations.
Instead of using the TIFFWriteScanline or TIFFWriteTile calls, I use the
alternative TIFFWriteRaw{Scanline,Tile}, where you have to hand it a
"raw" buffer (already compressed, ready for direct write to disk), which
means we have to compress it ourselves. But for the very common case of
"zip/deflate" compression, we can do this with our own call to zlib and
get the same compressed data blob that libtiff would have produced. And
in doing so, we can do that compression, and a bunch of other data
wrangling and format conversion, in parallel using our thread pool! We
can also pipeline the operation, having the serialized write happen even
while later chunks are still in the queue waiting to be compressed.
And of course, the same process in reverse for reading.
Some limitations: for now, this fast path is taken only for uint8 and
uint16 images, only deflate/zip compression, only "contig" planarconfig,
no weird modes like cmyk or palette images, etc. But these are the
most common (nearly exclusive) settings for TIFF files, as we use them.
Note also that this ONLY applies when writing multiple scanlines or
tiles at once. It can't parallelize this work if the calling app is only
reading or writing (from the OIIO API perspective) a single scanline or
tile at a time. But again, a very common use pattern is to read or write
the whole image at once. (Important exception: ImageCache/TextureSystem,
which is inherently single-tile based).
The results are stunning: I'm easily getting a 5-10x performance
improvement in TIFF input or output time, sometimes even much more. This
translates directly into large performance improvements for oiiotool as
well as maketx texture creation (of which by far the biggest portion of
runtime is the output of the texture, and most of that was due to the
compression, which is now multithreaded).
As a specific example, when doing "maketx" of an 8192 x 8192, 4 channel,
uint16 data TIFF input file on my 16 core (32 hyperthread) workstation,
Old: 28.2s total (1.13s read, 23.3s write)
New: 1.6s total (0.13s read, 1.0s write)
At least on the system I have, some of the masked partial load/store are
actually slower than the non-avx512 fallbacks, so disable the fancy one
until I find hardware where it truly runs faster.
Fixes#1889, which reported that frame sequence wildcards involving
negative frame numbers were not correctly supported. For example,
oiiotool --frames 1-5 ... # 1, 2, 3, 4, 5 (as usual)
oiiotool --frames -3-3 ... # -3, -2, -1, 0, 1, 2, 3
oiiotool --frames -3--1 ... # -3, -2, -1
In addition to handling negative frame numbers and ranges, I also made a
change to facilitate testing and benchmarking, which is that the
presence of `--frames` is now itself enough to trigger the looping +
wildcard expansion. Previously, if there were no numeric wildcards in
any of the filenames, it would not execute the frame loop (even if there
was a --frames).
* Add CUDA-friendly __device__ specializations for a subset of functions in fmath.h. Add preprocessor guards in fmath.h to allow for device compilation.
* Add HOSTDEVICE macro to platform.h, used to decorate functions in fmath.h. Move CUDA specializations into the original function/template definitions, instead of placing them in a block at the end of the file. Use anonymous namespace for sincos to avoid infinite recursion. Add note and TODO about fast math.
* Fix some wrong infdefs (inverted sense). Fall back to std math functions where no CUDA intrinsic is available, leaving it to the compiler to select the appropriate version.
Fall back to some std math functions.
* Remove #ifndef __CUDACC__ guard from fast_pow_pos. Use std::exp in fast_correct_exp.
1. Move the body of parallel_for_chunked and parallel_for_chunked_2D
from an inline in parallel.h to being non-inlined in thread.cpp.
The main advantage is so that as I tinker with the internals of these
functions, it's just one module that needs to recompile, rather than
the fact that an altered parallel.h forces a rebuild of most of the
project.
2. Implement a thread_pool::jobs_in_queue() that reports how many tasks
are waiting to execute (but haven't started yet) and a
thread_pool::very_busy() method that basically tells whether the number
of tasks waiting in the queue is significantly larger (currently: >4x)
than the number of threads in the pool. The idea here is that clients of
the thread pool could use this to avoid putting things onto the end of a
greatly oversubscribed queue and therefore avoid a long latency waiting
for other tasks to be completed.
3. And so the parallel_for loops use this -- before adding tasks to the
queue, it checks if it's very_busy() and if so, will execute the
sub-task directly. This helps reduce latency in situations where the
queue is heavily oversubscribed, which can happen if several threads are
each adding lots of jobs, oblivious that all the others are also piling
tasks on. This shouldn't come up commonly, but it makes things more sane
for this particular edge case.
4. Add an optional name field to parallel_options. This make it vastly
easier for me, when debugging, to have a particular launching of a
parallel loop have a unique name, and then trigger various debugging
scaffolding inside the execution of the parallel loops, only when
certain ones are called.
* Remove unneessary gcc5 test
* Temporarily disable osx static linkage test, which is broken after some
recent Travis+homebrew package upgrades, and I don't have time to fully
track it down at this time.
* Option to time all IBA functions
All ImageBufAlgo functions (for now; other functionality may be added
to logging down the line) are outfitted with optional timers to count
the number of executions and total time spent for each.
This is enabled by a new global OIIO attribute "log_times", which can be
enabled and disabled at any time (for example, to only log a certain
phase of execution of an app).
The default is 0, meaning not to log -- and I believe that when turned
off, there is no added runtime cost (except for checking the flag once
per IBA function).
The initial default can be changed with the OPENIMAGEIO_LOG_TIMES
environment variable.
When log_times is nonzero, logging will occur, with a slight increase in
runtime expense to check the times and to lock and access the map that
stores all the running totals (stored separately, indexed by name). This
extra cost is probably not significant for whole-image operations such
as IBA calls, but it would not be an especially accurate way to time
fine-grained operations that work on one or a few pixels at a time.
The global getattribute() call may be used to retrieve "timing_report",
which will return a string containing a list of all the timing totals,
alphabetized by name, like this:
IBA::computePixelStats 2 0.003s (avg 1.34ms)
IBA::make_texture 1 0.074s (avg 74.05ms)
IBA::mul 8 0.002s (avg 0.30ms)
IBA::over 10 0.024s (avg 2.38ms)
IBA::resize 20 0.240s (avg 12.18ms)
IBA::zero 8 0.001s (avg 0.08ms)
If the log_times option has a value > 1 at the time that the program
terminates, the destructor of the timing data structure will
automatically print the timing report to stdout.
So, even for a program (that uses OIIO ImageBufAlgo internally) that has
no exposed option to retrieve and print the report, you can still find
out how it's spending its time by launching as
OPENIMAGEIO_LOG_TIMES=2 my_application
Some caveats to keep in mind:
* It's possible that in situations where one IBA function ends up
calling another, the time could be doubled-counted.
* This measures "wall clock time", so interpret carefully in the
presence of multithreading.
Convert the (float[]) values to the data type of the buffer once, rather
than building float->buftype data conversion into every channel of every
pixel.
Fill of float and half buffers speed up by about 1.5x.
Fill of uint8 and uint15 buffers speed up by about 2.5x.
There was no reason for task_set to be a template -- the templated type
was no longer used for anything. Changing it from template to normal
class also means that it was convenient to move task_set::wait() and
wait_for_task() from inline to be in thread.cpp. This makes it so that
tinkering with these methods just rebuilds one file, whereas when it's
in thread.h, any changes to it forces a recompile of most of OIIO.
Be sure to disable task_set operator= and copy ctr.
Let the pixeladdr also take a channel offset.
Clean up some legacy versions of pixeladdr(); only one version is
needed with parameter defaults properly used.
Simplify the implementation of pixel_stride, scanline_stride, z_stride
to be based on the already-computed byte counts (I just didn't remember
that they existed when I exposed those functions).
With the new pixeladdr method that takes a channel offset, I can
lift the restrictions on special cases of copy and crop that require
starting at channel 0.
Only one of the two read methods was properly exposed.
This could be really bad if you tried using the other one, as the
default arguments could kick in and make the interpretation quite
different. In particular, your 'force' and 'convert' hints may be
misinterpreted, ugh.
Both of these were threaded, but used iterators to be fully general.
Much like the recent ImageBuf::get_pixels fix, using
parallel_convert_image (which has been heavily optimized) speeds up the
common situation where the source image is already in local memory and
the ROI being copied is fully contained in the pixel data window.
I'm seeing speedup of copy about 35-45% when copying a 4 channel float
image to either a float buffer, 2.5-3x speedup when a data conversion is
involved (copying a 4 channel float to a 4 channel uint8 buffer), and
2-4x improvement copying 4-chan half to 4-chan half (because the general
code did an expensive half->float->half set of conversions in the
process). Note: the uncertainties are from trying on a few different
machines / processor revisions / number of cores.
Fixes#1854
This was quite tricky. A while back, #1741 was intended to do the
"unpack" lazily, so that opens that only needed to read metadata didn't
take several seconds to do the full image nupack. But it inadvertently
broke vertical rotated images.
I went round and round on this, unable to make it work with a single open.
I finally just settled on opening it the first time without the unpack,
and if the pixels are needed, close and re-open with the full pack. Seems
to work, so taking this victory where I can get it. It also needed
the m_processor to be changed from a member to a pointer, as I couldn't
seem to fully re-initialize it properly without starting with a new one.
Also added a testsuite/raw that tests a couple raw images. Somehow we
previously had no testsuite verification of raw at all! The source
images are stored in the oiio-images project, originally from
https://www.rawsamples.ch (big stash of raw images from many cameras).
I realized/remembered that the "general" version that uses iterators is
not only necessary for the ImageCache-backed case, but also handles wrap
modes and samples outside the data window. Thus, it is not safe to use
the fast parallel_convert_image unless the ROI we are dealing with stays
strictly within the pixel data window of the image. So add that check.
* Forgot to update C++ and Python docs about the change to default value
for the `unpremult` optional parameter.
* Forgot to actually change the default value in the Python binding setup.
The fully general case uses IB::iterator element by element and can
handle ImageCache-backed images.
For the common case of the image being fully in memory, it reduces to
a call to parallel_convert_image, which is about 3x faster.
This speeds up the general case by about 30%, and the special case of
4 channel float -> 4 channel float by 50%.
Note that all the IBA::ocio* functions also use colorconvert underneath,
so these will all speed up proportionally.
Run testsuite unit tests with a fixed sample OCIO config.
This makes it easier to have unit tests for the wrapped OCIO functionality
without worrying that the user's system might not have an OCIO config, or
might have one with differing transformations than we used when making the
testsuite reference images.
Move color-related oiiotool tests into separate testsuite/oiiotool-color,
and add a few basic test the OCIO functions.
Make src/build-scripts/build_ocio.bash, run it on Linux for Travis so the
CI unit tests have a fully working OCIO.
After some deep thinking and discussion with people who know more color
than me, I have become convinced that you *almost always* want to make
sure to un-premultiply colors (i.e. divide by alpha) before doing color
space transformations with OCIO, and then re-multiply by alpha (to get
back to "associated alpha") immediately after the color transformation.
To support this,
* For the ImageBufAlgo color transformation functions, change the default
value of the `unpremult` parameter from `false` to `true`.
* `oiiotool --autocc` will do the unpremult/premult steps surrounding any
automatic color transformations that it does upon input and output.
* `oiiotool --colorconvert` and the various `--ocio` commands now take
an optional modifier `:unpremult=1` which will cause the conversion to
be internally bracked by a unpremult/premult step (when you do this,
you no longer need an explicit --unpremult and --premult on the
command line).
* oiiotool now issues warnings for cases where it has a good reason to
think you might be double-unpremulting (for example, if you do both
--unpremult and follow it by a --colorconvert:unpremult=1).
Later, we'll stage a second part of this where we change the default
value for unpremult= to be 1, so that the modifier is not needed (at which
point you REALLY don't want the explicit --unpremult/--premult, or it will
be wrong).
On the whole, I think this makes the situation less error prone for
casual users. Most oiiotool command lines should just use --autocc and
then not need to worry about any explicit use of `--colorconvert` or
`--unpremult/--premult`. For the rare case when more control is needed,
the individual commands and modifiers can be used.
Note that for oiiotool, `--autocc` is not on by default. If you want
images auto-converted to linear space and ouputs auto-converted from
linear to the desired output space, you need to use `--autocc`.
ImageBufAlgo::computePixelStats was completely serialized. Turns out that
this was a big chunk of "unaccounted" time in maketx.
So parallelize it in the usual way, with parallel_for_chunked_2D,
and some other cleanup to the PixelStats struct.
Also modify the stats keeping in make_texture() to account for and
report the computePixelStats time separately.
I think it's possible to deadlock when doing certain parallel IBA
functions in combination with autotile. When the deadlock possibility is
fixed, maybe we'll turn it back to on by default, but for now this seems
safer. Users can always turn it on with the --autotile command line
flag to oiiotool.
* Add a few places (mainly in boost) to the nosanitize.txt exception
list for lsan.
* Make runtest.py skip any test directory that contains a file called
TRAVIS_SKIP_LSAN, but only if it's running the sanitize test case on
Travis. On local machines, or for all other test matrix entries, it
will run as usual. So we set up this file for 4 testsuite directories
where it's just not practical to fix the extremely minor leaks.
This cleans up a handful of tests that recently started to fail the
Travis test that runs the sanitizer. (Nothing changed on my side; I
suspect some minor compiler upgrade on the Travis side started catching
additional cases.) It all passes with this patch, now that I'm ignoring
a few specific cases where I've verified that the "leaks" are small
one-time allocations that are just too much trouble to clean up.
This lets you wait for just one task in the task set to finish. Versus
the usual wait() that waits for all tasks in the set to be completed.
This is handy for "pipelining", where you are using the tasks as a
queue, and as they are done, you can pass them off to the next
stage. There's no reason to wait for all to complete to begin processing
the first-completed tasks coming out of the queue.
* Strutil: use the right variable
* Strutil: use strtod_l/strtod_l on GNU libc
Since they are available on any platform that uses GNU libc, enable it
directly.
The EXR spec allows for channels to be subsampled, but we don't handle
this properly in the exr ImageInput, and it can lead to a crash. Since
it's a rarely used feature (if ever), I'm not inclined to fix it for
real until somebody says they actually need it. Instead, let's just
detect this unusual case and correctly issue an error saying that we
don't support it.
This required a bit of housekeeping to allow the errors to be propagated
up from the place where it was convenient to notice this.
Field3D is supposed to just throw an exception that we can catch if it
can't open the file, but under some circumstances it just plain crashes
when asked to open a file that isn't a f3d file at all.
The reason this comes up is that OIIO's behaveior when the usual reader
can't open a file is to try ALL of them (hoping that maybe the file is
just mis-named) or the hint heuristics are wrong.
I'm not inclined to try to fix the field3d internals myself at this
moment, so for now I'm just making the OIIO field3d reader fail if the
file is not named ".f3d". This violates our intended rule of "it doesn't
matter what the file is called", but I can only rely on that if the
underlying format library is rock solid when presented a file of the
wrong type.
* Minor improvements to rcp_fast, rsqrt_fast for AVX512.
* Maked load/store tweaks:
* Remove some concern comments on 4-wide masked load/store for AVX
because benchmarks show it to be no concern (the lines of potential
concern were indeed faster on AVX).
* Change implementation of 4-wide scatter and masked scatter for avx512
after benchmarks show that the non-SIMD way is faster. Come back to
retest on future hardware.
* Speed up reduce_and / reduce_or for vbool16
* Fix typo
* Fix some type mismatch issues on 32 bit platforms. Fixes#1841
* DCMTK version enforcement
Have CMake figure out the version of DCMTK found.
Reject anything older than 3.6.1.
Older versions botched some preprocessor symbols in ways that are painful
to deal with, so since the ones that work (>= 3.6.1) date from 2011 (!),
I don't think it's too stringent a requirement.
* More conservative goards on certain symbols.
EVR_OD, EVR_UC, EVR_UR were added part way through 3.6.1 releases.
So test for them with 3.6.2 minimum not 3.6.1.
(Thanks, Richard Shaw, for the tip on this one.)
Instead of relying on GLEW for resolution of GL extension function entry
points use QOpenGLFunctions, which provides the same, but also allows to
run on both OpenGL desktop and GLES platforms.
Also drop extension based shader support. Shaders are supported since
OpenGL 2.0, so ARB shaders are likely no longer used and make the code
harder to read. Assume PBOs are available, which is true since OpenGL 1.5.
Text rendering in the pixelview overlay was apparently broken (no text
drawn at all), switch to QPainter based text drawing, as
QGLWidget::renderText() is not supported by QOpenGLWidget.
Signed-off-by: Stefan Brüns <stefan.bruens@rwth-aachen.de>
* For more realism in the "thread_workout" tests, which I noticed had
filter shape with exactly 1.0 aspect ratio, add a --anisoratio option
that lets you set anisotropic shape. By making a 2.0 or 4.0 aspect
ratio, that's a bit more realistic in terms of what a 3D renderer will
see for average aspect ratio.
* For clarity, change the older --aniso option to --anisomax, to emphasize
that it's setting the maximum allowable anisotropy.
* Make test thread_working samples twice as big, so it's sampling between
levels 1 and 2 instead of between levels 0 and 1, that's more realistic,
I think. The old way biased the tests much more to bicubic sampling which
is probably less common than the tests made it out to be.
* Stats were printing only for the ImageCache, not for the TextureSystem.
Fix by getting the stats explicitly, instead of relying on verbosity
to print them when the TS/IC is destroyed (because the way testtex is
written, the IC destroys but the TS doesn't because it's the default).
* Replace awkward mutex locking for stderr output with Strutil::fprintf,
which automatically keeps threads from clobbering each other's output.
This could make crashes with certain filenames on Windows. And even
though zfiles are rare, when OIIO is not sure which plugin to use, it
tries them all, so when given a bogus filename or a file that isn't
really an image, it will be called sooner or later.
TIFF didn't have write_scanlines() of its own, just depending on the
default implementation, which calls write_scanline repeatedly. So we
give it a proper write_scanlines implementation, which tries to speed
things up by first doing all the conversions and data shuffling on the
whole block (which can be parallelized by our code), before writing the
individual scanlines.
Slightly adjust the chunk size for write_image (it was a little too
conservative).
In parallel_convert_from_float, instead of a hard-coded block size, let
parallel_for_chunked make the choice on its own, which will be better
because it takes into account the number of threads in the pool
These changes, cumulatively, speed up whole-image writes of scanline TIFF
files by about 10%. I was hoping for more than this, but the bottleneck
is the unthreaded code in libtiff that's doing the compression encoding.
This wasn't a problem on the specific images I was testing, but I also
noticed an odditity in the way we did image checkpointing for TIFF
output -- it tries to only checkpoint every 5 seconds (which is longer
than most image writes, so it usually doesn't happen anyway), but the
timer started running when the ImageOutput was created. Seems to me now
that it's better to start the timer when the output opens, not merely
when the ImageOutput is constructed (in case there's a delay between
the two).
Adding fasthash64, a high quality 64 bit hash which happens to be both portable and quite short. It works particularly well when hashing a sequence of 64 bit values
Revamp the hash_test program to benchmark a bigger variety of sizes, and do so on a larger buffer of random values to test both aligned and unaligned reads and reduce the compiler's temptation to optimize the loops away
The hash_test also contains falkhash which is not quite viable on current hardware, but interesting for benchmarking purposes.
Interesting edge case I came across:
oiiotool rgb.exr rgba.exr -add -o out.exr
The logic was that the result had 4 channels (the greater of the two
inputs), but the channel names were just copied straight from the first
input, and it was (coincidentally, in this example) the one with fewer
channels, so it did not know that channel 3 was alpha, it would just get
"R", "G", "B", "channel3".
The new logic is that for all channels, it tries to get the name from
the first image, and if it doesn't exist, then it gets it from the
second. Also it's smarter about setting alpha_channel and z_channel in
the spec of the output, even if those didn't come along with the first
input.
* Only print the detailed stats if -v is used.
* New option --noinvalidate instructs it to not invalidate the cache
when starting each threadtimes trial. Combined with setting --cachesize
very large, this makes it possible to time the texturing math and
resident cache behavior without any disk access (because after the
first trial, all the needed tiles will probalby be resident).
Inspired by Matplotlib, which has added new color maps called inferno,
magma, plasma, and viridis. These color maps, developed by Nathaniel
Smith & Stefan van der Walt, are public domain, and were specially
designed to be (a) perceptually uniform, (b) strictly increasing in
luminance, (c) looking good when converted to grayscale for printing,
(d) usable by people with various forms of color blindness. They are
therefore superior to most of the ad-hoc visualization color maps used
elsewhere, including the original ones used in OIIO, which have none of
these desirable qualities.
The authors cite studies (in diagnostic radiology) showing that older,
ad-hoc non-perceptually-uniform color maps have empirically worse
performance, i.e. a lower rate of correct diagnoses. Bad color maps
literally have blood on their hands!
I took the tables for these new maps (https://github.com/bids/colormap)
and converted the values from sRGB to a linear response (since that's
how OIIO wants to operate), and also decimated from 256 entries to 17
entries (good enough, since we interpolate rather than needing a value
for each of 256 integer levels).
It seems that the clock sweeping algorithm for finding cache pages to
evict (i.e., the uts of ImageCacheImpl::check_max_mem) relied on the
fact that it could erase entries from the cache and continue sweeping
with the iterator.
This worked fine for the original unordered_map, and as well for the
unordered_map_concurrent as long as the individual shards were also
implemented by unordered_map -- but only because unordered_map used
chaining and thus an iterator would still be valid after an erase
(assuming the iterator wasn't pointing to the erased item itself).
But when Chris tried using a different underlying hash table which
didn't have this property, things broke.
Some refactoring of check_max_mem fixes the problem. Basically, I just
save the ID of the advanced iterator, allow the iterator to be
invalidated by the erase operation and then re-establish a new iterator
by using the ID we saved in the previous step. Some other cleanup of the
loop also helps make the logic of the whole affair a little more clear.
The main purpose of this option is to export additional gradient
informations along bump maps to ease the implementation of bump to
roughness techniques.
The resulting output is a single 6 channels bump map containing the
height, the first and second slopes moment values stored in subsequent
mip-map levels.
Channel data layout:
channel quantity channel name
------- -------- ------
0 h "b0_h"
1 dh/ds "b1_dhds"
2 dh/dt "b2_dhdt"
3 (dh/ds)^2 "b3_dhds2"
4 (dh/dt)^2 "b4_dhdt2"
5 (dh/ds) * (dh/dt) "b5_dh2dsdt"
Excuse the odd channel names -- because OpenEXR is obliged to re-sort
channels into their alphabetical order, we had to make absolutely sure
that the channels were named in such a way that would not be reordered.
Note that if the input texture is a normal map, then channel[0] = -1.0f
and the corresponding normal can be simply retreived as follows:
n = normalize( -(dh/ds), -(dh/dt), 1.0 )
The normals are assumed to be defined in the tangent space.
Subtle point: on MIP level 0, channels 3-5 are simple multiples of
channels 1 & 2, but as we filter to get the higher level (lower res)
MIP levels, this is no longer the case, and in fact we depend on this
fact to give us the right standard deviations.
References:
The technique we're using that requires information is based on this
technical report:
Christophe Hery, Michael Kass, and Junhi Ling. "Geometry into Shading."
Technical Memo 14-04, Pixar Animation Studios, 2014.
https://graphics.pixar.com/library/BumpRoughness/https://graphics.pixar.com/library/BumpRoughness/paper.pdf
And the following references may also be useful in understanding:
Mark Olano and Dan Baker. "LEAN Mapping." Proceedings of I3D 2010.
https://www.csee.umbc.edu/~olano/papers/lean/https://www.csee.umbc.edu/~olano/papers/lean/lean.pdf
Jonathan Dupuy, et al. "Linear Efficient Antialiased Displacement and
Reflectance (LEADR) Mapping." ACM TOG 32(6), 2013 (Proceedings of
Siggraph Asia 2013).
https://hal.inria.fr/hal-00858220/enhttps://hal.inria.fr/hal-00858220v4/document
I'm worried there are some edge cases where the hashes aren't quite as
deterministic as we want, across all platforms and use cases.
I decided to also get rid of the option to use OpenSSL for its hash, the
different for the two code paths, we could have different hashes for
people who build with OpenSSL versus those who build without it.
I also beefed up the code in ImageCacheImpl::verify_file where we check
for duplicates using the hash fingerprints. I became worried about what
could happen if you had two textures that were different "shapes" but
the same hashes -- such as a 64x64 4 channel texture and a 128x128
single channel texture, where the "unwrapped pixel sequence" (if you
know what I mean) was the same, they would end up with the same hash.
So I modified the check to only count identical hashes as a match if the
width, height, depth, channels, number of subimages, and miplevels also
were the same for the two images.
For compute_test and filter_test, switch from time_trial to Benchmarker.
This reduces the code complexity, and because Benchmarker figures out
on its own how many iterations to do in order to get good timings, it
runs MUCH faster overall (we were using too many iterations), so this
will speed up testsuite runs.
Also reduce time spend on atomic_test by lowering iterations, raising trials.
* Move parallel_image_options to OIIO level in parallel.h as parallel_options.
* Add field to parallel_options to allow for recursion within the
thread pool (use with caution to avoid thread explosion!).
* Use parallel_options as a control parameter to the various parallel_for
family of functions as a way of communicating preferences for maximum
threads, custom thread pool, and recursion.
* Some minor logic changes in chunking logic.
* More careful tracking of which threads are participating in the pool --
not only pool threads, but also the *calling* threads that opportunistically
execute tasks from the queue while they are waiting for the rest of the
task set to be done... those also need to be protected from inadvertent
recursively adding things to the queue.
Note that this should be source back-compatible, since the places we add
parallel_options to functions all are either (a) adding it to the end of
the parameter listings, with a default value, or (b) putting it in place
of a `nthreads` parameter (a parallel_options struct can be implicitly
constructed from an int, which does in fact initialize the maxthreads
field).
A strange edge case turned up: when parsing a file with XMP data block,
and the XMP contained metadata "Exif:PixelXDimension" (or PixelYDimension),
a cascade of failures eventually led to a crash in `oiiotool -info -v`.
Fixes herein include:
* Add those two entries to the table of xmp fields we expect to see, in
xmp.cpp.
* In xmp.cpp's add_attrib(), if something is not found in the xmp table
(which among other things translates the XMP name into an OIIO name),
stick with the xmp name it came with. We were actually adding it as an
attribute using 'oiioname', which for that case was uninitialized!
* Finally, in ImageSpec::attribute(), check for empty names, which are
obviously errors, and don't add those attributes.
Most notable is that `oiio --help` now prints some important usage tips
that explain command parsing, syntax of optional modifiers, and the path
where you can find the full PDF docs.
This lets you provide callbacks that will print extended information
into the verbose usage output: one that prints before all the options
are detailed, one that prints after all the options are detailed.
ParamValue holds "small" data locally (in the struct), and only
allocates when above that threshold. Previously, the threshold was 8
bytes, the size that the allcoated pointer would be anyway. But it seems
to me that the way we us ParamValue frequently must contain a color (3
floats), so I bumped the local space from 8 to 16 bytes (meaning it can
hold up to 4 floats or ints before a malloc). This increases the total
size of a ParamValue from 32 to 40 bytes, but I think it will make it
fairly rare to allocate, so we'll come out ahead.
This should all be inconsequential from an API perspective.
Python bindings for ImageBuf.get_pixels() and set_pixels(), when passed
an ROI without a channel range specified -- which should assume you mean
to include all the channels of the image -- calculated that number wrong,
making it one channel too many.
If you pass an ROI that is fully constructed (with all of x, y, z, chan
ranges explicitly specified), it worked fine, which I guess is why nobody
noticed this problem before.
Break the longest-compiling file, imagebufalgo_mathops.cpp, into
imagebufalgo_addsub.cpp, imagebufalgo_muldiv.cpp, imagebufalgo_mad.cpp,
and a teensy bit of refactoring within those files to reduce compilation
effort.
Also changed the src/libOpenImageIO/CMakeLists.txt to list the most
expensive files first, in the hope that if it does that first, it can
load balance easier by having lots of inexpensive files to work on with
the other cores.
The net result is that it saves another 15 seconds of wall clock time
when building with 24 cores, down from about 2 minutes (so around another
12%).
Some source modules took minutes to compile, mostly because they involve
lots of template instatiation for many type combinations.
So, first, some of the functions that had a very wide cross-product of
type combinations could be changed to the variety that only handles the
common types and uses float as an intermediate format for the outlier
cases. That reduces total compile work substantially.
But also, particularly long-compiling modules can screw up parallel
builds, because that all the other modules for a library may get done
compiling while that one with an extra long compile is still going, but
nothing else can proceed until that's done, so only one core is doing
work while the others wait. So to combat this, I took the two files with
the very longest times (imagebufalgo_pixelmath and imagebufalgo_copy)
and split them up into multiple pieces. That doesn't reduce the total
amount of compilation work, but it does make it easier to parallelize it
over multiple cores.
The net result can be seen by these stats for fresh full builds (after
clearing ccache) on my 4-core laptop:
before: 315.85 real 850.53 user 42.60 sys
after: 200.44 real 725.09 user 36.21 sys
The "real" time is the important one -- reduce wall clock time by ~35%
when doing a full parallel build on a 4-core machine.
pybind is a great project by Wenzel Jakob, very much in the spirit of
Boost.Python (and with substantially similar syntax), but is
C++11-based, light-weight, header-only, and compiles in a fraction of
the time as code that uses Boost.Python.
So the primary change here is to switch underlying implementation of our
Python bindings. But in the process, I made a number of improvements and
changes to the Python bindings, some of which are not strictly backwards
compatible, so this is a BREAKING CHANGE that may require some (hopefully
minor) changes to your OIIO-using Python code.
* All of the functions that read and write blocks of pixels (such as
ImageInput.read_image(), ImageOutput.write_tile(), or
ImageBuf.get_pixels()) no longer use old-style Python arrays, but now
use NumPy ndarray objects. I'm guessing that NumPy is such a common
Python package that it's not an onerous dependency to add, the n-D
array syntax is really handy for images (e.g., pixel regions are
naturally structured as [y][x][channel] multidimensional arrays and
very easy to use that way), and pybind11 makes it very easy to
manipulate NumPy ndarray objects but comparitively difficult to use
old Python arrays.
* We now transparently accept strings anyplace that a TypeDesc is used
as a function parameter. For example,
`ImageBuf.set_write_format(TypeFloat)` can alternatively be called as
`ImageBuf.set_write_format("float")`.
* Some specialized enum types (such as ImageInput.OpenMode) have been
replaced by strings: `ImageInput.open (filename, ImageInput.Create)`
should now be called as `ImageInput.open (filename, "create")`
It should be encouraging that of all the Python test files in our test
suite, `testsuite/python-*/src/test-*.py`, a very small number of source
lines needed to be changed.
If pybind11 is not found on the system, we just git clone as needed, with
the following logic:
* If BUILD_PYBIND11_FORCE is true, download a local copy (regardless of
any system install).
* Otherwise, look for a system-installed copy, and use it if found.
* Otherwise, if BUILD_MISSING_PYBIND11 is true (default), download
a local copy.
So, the default behavior is to use an installed copy, or auto-download
if that's not found. But it's possible to force a download (ignoring system)
or to disallow a download (making missing pybind11 simply an error).
Also, none of this happens if the build is USE_PYTHON=0.
maketx (really, the underlying IBA::make_texture() call) writes to a
temporary output file, then after it's done, atomically moves the file
to its requested destination. This ensures proper operation if the user
foolishly specifies that output should overwrite input (same file), and
also is a guard against a crashing program leaving a corrupted
half-written file behind that could be mistaken for proper output.
This patch employs the same trick to oiiotool. It was "safe" to have
oiiotool in.exr ...blah... -o in.exr
if in.exr was small enough -- in which case it would be fully read into
its ImageBuf when first encountered. But if it was a large file, it would
rely on ImageCache, so it was possible that when the output started to
overwrite it, trashing the file, not all the parts of the image had been
read into memory. With this patch, that should no longer be a danger.
We also do an ImageCache invalidation of the file that was output, in case
an portions are being used somehow.
Increase the number of shards in the unordered_map_concurrent used by
the ImageCache for file and tile caches. As we get machines with larger
core counts and thread them more heavily, the previous limits were causing
bottlenecks.
This solution is not perfect nor complete. But it does push the
bottleneck from ~16 to more like ~32 threads. I'm separately working on
some other strategies to reduce the amount of locking, with an eye
toward every-growing core counts. But this is a solid step forward for
our newer machines.
This refactor comes after a belated realization that our numeric
conversion routines (principally from_string<> and format) are
"locale-dependent" via their use of strtod, as well as atof() scattered
through the code.
In particular, this means that when software using this code is running
with a locale in which the decimal mark is something other than '.'
(e.g., many European locales that use ',' for decimal mark), these
functions will incorrectly parse text floating point numbers that use
the standard period as decimal mark. For example, if the locale is
"fr_FR", atof("123.45") will return 123.0, rather than the correct
123.45.
After much debate, and a few implementations, we are declaring that
since the whole point of OIIO is about reading and writing persistent
data (image files!), that in order to be consistent, ALL of OIIO's
string parsing and formatting, including that done by utility library
functions, will be locale-independent by using the "C" locale (in
particular, we will use the usual North American default of '.' for
decimal separator).
Additionally:
* Added a new stof(), stoi(), stoul() to replace the clunkier template
syntax of from_string<>. As explained, these are hard-coded to "C" locale,
not the native/global locale.
* Much more attention to string_view safety, finding a number of subtle
bugs (including Strutil::parse_int, etc.) where it was definitely not
safe to assume that the string_view passed was the null-terminated edge
of the string, and so, for example if a string_view pointed to the
characters "12" but the very next characters were "345" parse_int would
return 12345 instead of the correct 12.
* Strutil::format and ustring::format (and our copy of tinyformat
underneath) were touched up so that the versions of format() that return
strings will use the "C" locale, versions added that return strings and
use an explicitly-passed locale, and the versions that append their
results to existing streams continue to operate as before by honoring
the existing locale conventions of the streams they are using.
We may revise this later depending on what the upstream tinyformat
chooses to do about the issue.
* Several places where we assembled strings using std::ostringstream,
I forced the stream to use classic "C" locale in cases where it was
likely to have any floating-point data output.
* For maketx and oiiotool, globally set the locale to classic. In these
cases, we control the whole app, so this is safe to do.
If we ever decide that we want versions of these functions that take
explicit locales, it's not hard to add.
It previously worked for the OIIO tests that all seemed to have a
'using namespace OIIO', but failed for external projects that just wanted
to use unittest.h and weren't inside namespace OIIO.
You can see the performance in the following two tables. It's a
benchmark using our unit test spin_rw_test.cpp. We vary the ratio of
writers to readers from 1:9 (one write lock and modify for every 9 read
locks) to 1:9999.
All times are in seconds for this workload (smaller is better), and
they were executed on Linux on a machine with 32 physical cores (64
hyperthread cores).
Old code:
threads 1:9 1:99 1:999 1:9999
--------------------------------------------
1 0.3 0.3 0.3 0.3
2 0.5 0.5 0.6 0.5
4 5.4 3.4 3.2 2.8
8 9.9 9.5 10.6 9.0
12 12.3 13.1 11.8 12.1
16 13.7 14.3 14.0 14.8
20 15.4 16.2 16.1 17.5
24 17.9 16.9 18.3 18.1
28 20.0 21.0 21.2 20.9
32 21.4 22.2 22.8 20.9
64 20.9 22.4 22.2 21.4
New code (this patch):
threads 1:9 1:99 1:999 1:9999
--------------------------------------------
1 0.2 0.2 0.2 0.2
2 0.9 0.7 0.5 0.5
4 1.4 1.0 0.8 0.8
8 3.6 1.5 1.1 1.0
12 5.1 2.4 1.4 1.2
16 6.0 2.8 1.8 1.4
20 6.8 3.6 2.1 1.7
24 8.4 4.4 2.4 2.0
28 9.1 5.0 2.8 2.2
32 10.8 5.3 3.2 2.4
64 11.8 5.8 4.2 3.0
So the performance of the new code has three interesting properties:
(a) For every thread count, and every write-to-read ratio, it is
superior to the old code (only exception: 2 threads, heavily weighted
to writers). (b) For every workload, the new code scales better,
versus thread count, than the old code did. (c) Whereas the old
code has similar performance regardless of workload, the new code
gets remarkably more efficient as use is dominated by readers --
there is VERY little interference between simultaneous readers.
I don't expect much in OIIO to speed up *today* as a result of this,
because there are only a couple places where we use the spin_rw_mutex.
But I'm laying the groundwork for some improvements I'm doing to the
ImageCache/TextureSystem, which currently doesn't use rw locks but I'm
trying out an improvement that will utilize them, and I think this is
going to be a key component to making it scale better with larger number
of cores.
* Change ColorConfig to return shared ptrs to ColorProcessors. This frees
the caller from having to own the resource and remember to delete it.
It also makes leaks impossible.
* Establish an internal cache of ColorProcessors, so that repeated requests
for the same processor (i.e., same color space transformation) doesn't
need to do any allocations or trigger anything expensive on the OCIO
side.
* Added new create...() methods that take ustring arguments, in addition
to the existing ones that take string_view. That version is faster
if you already have ustrings on the app side -- especially the part of
looking it up in the cache. The string_view ones are still fine to use,
they just build ustrings on their own.
* Move ColorProcessor class declaration to color.h so client code that
gets a CP handle can actually apply the transformation to colors with it.
* Allow "rgb" and "default" to be additional alias for the "linear"
role. (This really helps OSL, where "rgb" is a longstanding name for
'the default color space'.)
* A little bit of C++ modernization.
NOTE that this is a breaking change, although a minor one. Any applications
that had explicitly asked for a ColorConfig and used it to request a raw
ColorProcessor will need to make some minor changes:
- The return type of the ColorConfig::create... has changed from
ColorProcessor* to ColorProcessorHandle.
- You don't need to free the handle, it's a shared pointer now and will
clean itself up.
"Maker notes" are extra camera manufacturer-specific metadata stored as
opaque blobs inside Exif records. The manufacturers don't seem to
publish the specs for these blobs, but they have been reverse engineered
and with some hunting you can find tinkerer-prouced documentation (at a
range of levels of completeness or correctness).
With this patch, we do a substantial refactor of exif.cpp (and a bit of
work in formatspec, rawinput, and other places) to incorporate full
awareness of maker notes into our metadata processing. At this stage, we
only read Canon makernotes, and quite incompletely. But we get a lot
more of the metadata than we used to (previously we just ignored any
makernotes we found), and the structure is now sound for adding more
complete support for this and other cameras.
This will be an ongoing project to support more cameras, but the foundation
is now set for this work.
There was a --handle flag that lets you test performance differences
between the texture calls that take handles, versus the ones that take
filenames and figure out the handles by looking it up in the file
cache. But it turns out I had implemented that option fully for the
usual "make a texture image" tests, but not for the special thread
workload test you get with the --threadtimes option.
This I/O plugin for "null" images are a file "format" where output of a
null file does nothing at all (no I/O, not even any data copying), and
input of a null file just fills the result buffers with black pixels (0
values), with no I/O or data copying.
What's this good for?
* Benchmarking, if you want to have OIIO's input or output truly take
as close to no time whatsoever.
* "Dry run" of applications where you don't want it to produce any real
output (akin to a Unix command that you redirect output to /dev/null).
* Make "fake" input that looks like a file, but the file doesn't exist
(if you are happy with constant-colored pixels).
The filename allows a REST-ful syntax, where you can append little doodads
that specify things like resolution (of the non-existent file), etc.
For example,
foo.null?RES=640x480&CHANNELS=3
would specify a null file with resolution 640x480 and 3 channels.
Token/value pairs accepted are
RES=1024x1024 set resolution (3D example: 256x256x100)
CHANNELS=4 set number of channels
TILES=64x64 makes it look like a tiled image with tile size
TYPE=uint8 set the pixel data type
PIXEL=r,g,b,... set pixel values (comma separates channel values)
TEX=1 make it look like a full MIP-mapped texture
attrib=value anything else will set metadata!
See the comments in the patch for details. It's been like this for a
long time. I think in practice, we'll never hit the bug. But we'll
probably phase this method out anyway, since it's not part of the C++17
std::string_view.
These are handy for extracting values from aggregates or arrays.
They still do the conversion to int or float, even if the real data
in the ParamValue is stored in a different (vaguely compatible) type,
such as converting a uint16 data to an int when requested.
So, oiiotool reads one or more images, might combine them, might
generate some from whole cloth. And the ones read from disk might not be
stored internally with the same data format as it existed on disk (it's
biased to store in float internally). How does it decide what data
format to write to disk for output?
We thought we were doing something sensible -- which is to allow an
explicit override (via `-d`), but in other cases to write using the
same data format as whatever was the FIRST file input. That at least
guarantees that a basic read-write cycle preserved the format.
But Chris Allen pointed out an interesting corner case, which is that
when you use `--siappend`, it will write out all subimages using the
first format it encountered. But it seems like in this case, it ought
to be able to keep each subimage in the format it started in.
This sent me down a bit of a rabbit hole.
The logic is hereby modified to:
1. Explicit format requests (`-d`) are honored.
2. Otherwise, a file that appears to be an unmodified copy of a file read
from disk will write using its native (i.e., from disk) format.
3. Otherwise (for a modified or generated file), it will write using
a data format taken from whatever was the first file read.
4. If all those fail, it will just use the format of the buffer.
Note that (2) is the new bit of logic.
Turned out to take a fair bit of refactoring to really track this
through all the corner cases, and also make it understand that the
"subimage append" operation does not lose the "it's a directly read
image" property.
I implemented similar logic for tracking tile sizes as well. So output
tiling should more accurately reflect input tiling, except when overridden.
Some other miscellaneous things that came along for the ride:
* New testsuite entry: oiiotool-copy to test these cases.
* A bit more care internal to ImageBuf::copy() of keeping track of the
"native" format info.
* Some C++ modernization of some ImageRec and Oiiotool internals.
* Minor fixes to how regex matching is done for `oiiotool -info` when
`-metamatch` is used.
* The `-o` command now takes additional arguments: `scanline=1`,
`tile=XxY`, which can override the tile size or scanlineness
on a per-output basis (just like how it could override data type
or contig/separate).
We pasted the result into the dest to be aligned with the dst's data
origin, instead of the correct source origin.
While I was there, I slightly simplified the error checking by using
existing IBAPrep functionality to check for the same edge cases.
Previously, we had pre-constructed TypeDescs of common types as static
data members of TypeDesc itself. This presented two (minor) problems:
1. Because their full definitions were in typedesc.cpp, all other modules
did not know what was actually in the fields, so optimization opportunities
were potentially missed.
2. There existed the potential for the static initialization order fiasco,
if the compiler could not figure out that the definitions in typedesc.cpp
could be expressed as POD and baked into the module without running the
constructor at module load time.
So we are deprecating these, though they will stick around for another
major release in order to not break any apps that are depending on them.
In their place, we are making OIIO-scoped 'static constexpr' versions.
These serve the same purpose, but by being constexpr (yay, C++11!),
they are should be turned into compile-time constants everyplace they
are used, which is what we ideally wanted all along.
It came to my attention that somebody could take a maketx-generated (or,
`oiiotool -otex`, same thing) texture, load it in Photoshop or other
utility, change it, and save it, and it just might still have some of
the maketx-specific metadata in the altered file, such as the SHA-1 hash
that allows us to find duplicates, and those data may be totally wrong
in the new file.
So this patch is intended to do a sanity check: if there is no
"textureformat" metadata present, or if it's not tiled, or if the
"Software" metadata doesn't say maketx or OpenImageIO, then it can't
have come straight out of maketx/oiiotool, so erase any of the
problematic attributes that maketx writes, because they are very likely
wrong and no longer match the potentially modified pixel values in the
file.
Photoshop always writes scanline images, so the check for tiling works
for that case. It will catch others as well. Not perfect by any means,
but it's better than before and I'm not sure if we can ever be certain.
We still rely on users not being foolish -- please only use textures
that have passed through `maketx` or `oiiotool -otex` and don't modify
the pixels after that step without re-maketxing it.
* Fix fmath.h's ifloor() turned out to have an icky bug where it gave the
wrong value for negative integers! It was because of the bit fiddling trick
we were using to avoid the call to floor(). Worked fine for positive values,
but failed for exact negative integers. The new formulation is a little
slower, but is always correct.
* The simd versions have the same problem. Fix them there as well, and
also rename (floori -> ifloor) so that the simd and fmath versions
have the same naming convention. Add unit tests and benchmarks, and in
light of that, tighten up some of the SIMD code paths a bit.
* floorfrac uses the same trick. I realized that a slightly different idiom
was much faster -- even slightly faster than the "fast but buggy"
version we used previously. So in that case I managed to fix without
any performance penalty.
* Added simd versions of floorfrac, in fmath.h (where the float version
was defined).
* texturesys.cpp: use the fmath floorfrac (remember, we put simd versions
there now) instead of defining it locally, which also lets us remove
a redundant quick_floor() that was identical to the old buggy
ifloor anyway.
* Various unit tests for both the scalar and simd cases of these related
functions, which led to...
* unittest.h: added an "OIIO_CHECK_EQUAL_APPROX" unit test macro,
basically it checks a result and considers it ok if it's within
0.1% of the larger of the quantities being compared. Some extra
trickery makes it work with either scalar or simd values.
Templatize based on the stride units. The default strided_ptr<T>
is equivalent to strided_ptr<T,sizeof(T)>, which means that strides
are measured in terms of the size of the element type. But, if you
really want to, you could make strided_ptr<T,1> which would be a
strided pointer where the strides are bytes rather than T's.
Note that this is a compatibility break -- previously, strided_ptr<>
used byte addressing all the time. This was counter-intuitive in many
circumstances, so now the default is to use array-like addressing where
the strides are measured in units of sizeof(T)>. But we have the ability
to still use byte addressing when really needed.
While I was in there, I changed a lot of the declarations to use
constexpr and noexcept very liberally, to maximize the chances that the
compiler can exploit certain optimizations.
Change all hard coded paths to GNUInstallDirs varibles.
Remove variable setting in install.cmake file since
GNUInstallDirs does all this for you.
Added new search paths for fonts.
Changed location of INSTALL_DOCS test to stop all the document
building when it would never be installed.
Signed-off by: Jonathan Scruggs <j.scruggs@gmail.com>
VFXPlatform has dictated a minimum OpenEXR version of 2.0 since 2014,
and 2.2 since 2015.
Upgrading our minimum to 2.0 eliminates a lot of #if nonsense that was
necessary to continue supporting OpenEXR 1.x. There's no real burden
in continuing to support 2.0 & 2.1 for now. But we will continue to
re-evaluate the appropriate minimum in future releases.
Fixes#1750
These two spots were implicated in the initialization order fiasco.
(See: https://isocpp.org/wiki/faq/ctors#static-init-order )
The solution for maketx.cpp is that since we only are using the
ColorConfig for the sake of generating a help message, there's no reason
it can't be an ordinary local variable in the one (rarely called)
function where it's used.
For the std::locale involved in strutil.cpp, use the old trick where
instead of a file static, we have a function returning a reference,
which itself has a function-static pointer that's allocated the first
time it's needed. This leaks (a single std::locale), but who cares?
* Move "benchmarking" utilities -- time_trial, timed_thread_wedge,
DoNotOptimize, clobber_all_memory -- to a separate benchmark.h
and benchmark.cpp. The stuff in timer.h is JUST the Timer and
directly related material.
* New Benchmarker class tries to do micro-benchmarking more cleverly.
Wrap up some stuff we had done elsewhere (often redundantly).
Benchmark multiple trials, report the statistics. Select number of
iterations automatically (!), also try to correctly compute and
subtract all the loop overhead. Read the comments in benchmark.h for
explanation.
Alas, the new Boost 1.65 moves some boost python material from one
namespace to another in a way that breaks compatibility related to
support for numpy arrays. This breaks the build, including all of
our MacOS-based TravisCI tests, which of course pick up the latest
Boost via Homebrew.
If a file is "broken", remember the error message that caused us to
consider it broken, so it can be properly reported later.
This is particularly helpful for files that are rejected for being
untiled or not mip-mapped (if "accept_untiled" or "accept_unmipped" are
false). It's confusing to see "invalid file" as if the image file
itself was corrupted, much more helpful to see a specific error message
about the reason why it was rejected.
My stupid bug, introduced during a minor refactoring related to the
static analysis from two commits ago. The static analysis was right,
by my fix to an apparent possible null ptr dereference was flawed. I
avoided the null dereference, alright, but broke some other logic in
the process.
There was previously a set of TextureSystem API calls (with multi-point
TextureOptions struct and a fancy VaryingRef helper class) that was a
multi-point interface, but we never did much with it and in retrospect,
it does not use a data organization that is good for future-looking
architectures. We are therefore depcrecating it, it will remain for 1.8
in case anybody is using it, but will thereafter be removed.
But we are adding a new set of batched routines that are organized
differently, we think in a way that will be well-suited for better
parallelism.
The new API assumes that the inputs and outputs have a SOA rather than
AOS layout, and the arrays are a fixed width, defined by
Tex::BatchWidth, which is a constant and set to 16. Build-time overrides
let us change this for experimentation and future expansion. You can
texture fewer points than this using the RunMask, which is a bit field
saying which "lanes" should be used. Inputs will not be read, or outputs
written, for the lanes where the run mask is 0.
The goal is for this to really fly on chips with ISA that includes
AVX-512 support (naturally 16 wide), and to also perform very well for
AVX2 class machines (emulating 16x operations with pairs of 8x SIMD
instructions). I'm not very hopeful that old SSE hardware (4x wide) will
have a high payoff using the batched functionality rather than the
single-point texturing, but we'll see what happens. We'll also
experiment with 32 wide (there's no true 32 wide hardware, but pairs of
16-wide instructions may dual-issue in nice ways that provides an
advantage, we'll see).
I'm not planning to ever remove the single-point scalar API calls. There
will always be a use for them and they'll certainly be more efficient
for applications that don't have a good way to batch the calls.
This pull request only implements the batched APIs -- the internals are
still scalar, they just loop over the enabled lanes and call the scalar
functionality.
I'm actually working on the true batched reversions right now, but they
aren't ready to review yet. Expect a second PR for that as soon as I
have good results. But in the mean time, this allows applications to
start experimenting with the batched interface.
The testtex program now has modes that exercise the batched interfaces,
and the testsuite has been doctored so that every test whose name
contains the substring "texture" will run twice: once using the old
scalar mode, and once using the batched API calls. This means that
starting now, we will immediately notice any breakages or regressions in
the batched versions, as well as have a convenient way to benchmark the
scalar versus batched calls.
Among other things, this helps with some clang tidy warnings that the
virtual destructors of these classes should not be calling the virtual
close() methods, as is common. Its safe for a final class, but in the
general (potentially sub-classable) case, it's considered unsafe.
The TIFF spec says these (which we use to generate the ImageSpec x,y
fields giving the data window origin of the image) should be relative
to the "resolution units" described by the XRESOLUTION and YRESOLUTION.
But OIIO has always just stored the data origin directly in this field,
without regard to any scaling factor of the [XY]RESOLUTION fields.
Now, let's realize that it's not very common to have files with nonzero
data origin (crop windows) in TIFF files, also not common to have any
RESOLUTION tags in the files. So the problematic combination, which
would be interpreted differently by versions on either side of this
patch, is rare.
But one way that we can get into trouble is the following sequence: (a)
a JPEG from a camera (or any number of other cases) has resolution or
density data (common for cameras); (b) that JPEG is converted to TIFF,
thereby copying the fields; (c) the TIFF file is cropped, and written by
a version of OIIO prior to this patch. Oops.
Mitigation strategy: OIIO (at least insofar as oiiotool and maketx)
likes to store its name and version in the "Software" TIFF field. So
upon input, we check this, and if the file seems to have been written
by OIIO with a version prior to this patch, use the old interpretation.
Fixes#1623
These were templated on dimensionality of the array, and that made the
implementation much more complex and quite hard to read and
understand. But we never used anything but the 1D case. The reason it
was templated on rank before was to emulate the array_view that was
proposed for C++17, but that has since been scrapped, so keeping
compatibility with it is no longer a priority.
When we remove "Rank" from the template and take away all the machinery
that isn't needed for the simple 1D case, a lot of the complexity goes
away, and these templates will be a lot easier to understand and modify
in the future.
Also finally remove the long-deprecated OIIO_USING_CPP11 macro.
Because (1) C++11 is the minimum now anyway, and (2) we switched
some time back to using OIIO_CPLUSPLUS_VERSION.
Obsolete -- this is the standard noexcept declaration in C++11.
We don't seem to use it anywhere, so I'm going to take a chance and
just remove it straightaway, rather than deprecating it.
Upon reviewing parallel.h, it was apparent that there was a cut-and-paste
error that resulted in parallel_2D() having extra unused parameters.
Put it like I clearly meant it to be.
When merging, high number of splits was crippling performance due to
constant realocation.
New strategy more carefully pre-computes the NUMBER of splits expected
in the merged pixel, and allocates enough storage for the result pixel
to hold the number of samples in both of the input images, and also the
expected number of new segments that will result from splitting.
This speeds things up for very complicated frames by several orders of
magnitude.
We had load_mask and store_mask for the 8x and 16x classes, but
neglected to add them to the 4x classes.
Also, add scatter/gather operations for all classes (not expected to
be efficient until AVX2), both regular and masked.
* Move the SIMD-oriented comparisons from simd_test.cpp into unittest.h.
* Use Strutil::Term() to use color in the unit test error message macros,
it makes it a lot easier to spot the assertion messages in the middle
of a long console output. I had used this in the simd_test.cpp ones
and really liked it and wanted to propagate the practice to all
unit test checks.
* Change the global unit_test_failures from a regular int to a special
counter class that will automatically print an OK/ERRORS message at
the end of execution when it destructs, and also automatically call
std::exit() with a nonzero return code if errors occurred, making it
impossible for a file using the unit tests to accidentail forget to
return an error code if there were errors.
For 2-channel images where the first channel is intensity and the second
is alpha, TIFF output was neglecting to set the TIFF "EXTRASAMPLES" tag
to mark the alpha channel. In particular, it just didn't consider
extrasamples for images that had fewer than 4 channels.
This was in turn causing the file, when re-input, to not realize that
the second channel was "A" or that there was an alpha channel at all
(since TIFF does not store channel names, we infer it all from the
EXTRASAMPLES tag).
One consequence of this mistake was that a grey-alpha image (say, coming
from OpenEXR or PNG or some other format that allows it) and run through
'maketx' (converting to TIFF if not otherwise specified) would lose this
knowledge, which in turn would affect the ability of the TextureSystem's
"gray_to_rgb" mode, which only applies to certain channel combinations.
Also noticed that for TIFF input, 2-channel images where the second is
alpha were still being named "R","A", but we actually changed the convention
some time ago for a single intensity channel to be named "Y", no "R", since
it's almost certainly representing luminance and not "red" per se.
Background: OpenEXR forces channels to be stored (or at least presents
it through its API) in strictly alphabetical order. But OIIO expects
uniform channel orders for all the common cases such as RGBA.
For a 2-channel image that's intensity and alpha, we name the channels
"Y" and "A", and expect it to be presented with alpha second, as usual.
But OpenEXR was flipping the channel order because "A" comes before "Y"
alphabetically. Sheesh. It was because we neglected to put "Y" in the
list of special channel names that have order imposed. Easy fix.
Somehow float4 and float8 never bothered me, and seemed obvious that it
menat 4 or 8 float (what else could it mean?).
But after adding the 16-wide classes, it seemed we'd backed ourselves
into some confusion: float16 could be confused for a 16-bit float (aka
half), and int16 looks confusingly like int16_t (aka short). If some day
we ever have 32-wide or 64-wide simd classes, it's only going to get
worse.
So to reduce future confusion, we're renaming to vfloatN, vintN, vboolN.
For a couple major releases, we will maintain typedefs so that float[48],
int[48], bool[48] continues to work for any 3rd party apps that use
OIIO's simd.h.
* Add float16, int16, bool16 classes
* Preliminary support for masked loads & stores (including propagating those
to 8x and 4x, which admittedly will not be very efficient for older HW).
* When 16x HW support is not selected, just fall back to 8x[2] style
implementation. The aim is to make it convenient to code AS IF we were
on AVX-512, but have pretty decent performance for AVX.
* Lots of miscellaneous fixes to the 4x and 8x classes and cleanup of
this big header file (lessons learned, and conform to more clear
semantics apparent when expanding to 16x).
`getattribute("oiio:simd")` retrieves the comma-separated list of
mostly-SIMD-related capabilities that were enabled at build time (e.g.,
"sse2,sse3,sse41,sse42,avx").
`getattribute("hw:simd")` is a similar list of the capabilities detected
on the actual hardware at runtime.
Note that the two don't need to match -- it's very common to compile
with build time flags set to use only the least capable hardware you
expect to run on. But this can help you detect if you are leaving
important hardware facilities unused by your build flags.
The layer mask channel has independent rectangle position from the regular channels. Mostly, they are same position, but sometimes not. PSDInput::open fails by reading wrong data as ChannelInfo::compression when the pixel size of the layer mask channel is different from the image pixel size.
After upgrading to Qt5, display scaling was all wrong on my Mac laptop
with retina display. Some web searching revealed how to retrieve the
device pixel ratio and use it as a scaling factor for GL.
This seems to mostly fix things.
The int& operator[]() was always sketchy, and really isn't going to work
well as we look forward to avx512. So let's set up a safe version and
call it setcomp. It is "safe" because it sets the bit you want, and
doesn't claim you can get a reference to an underlying integer. That
gives us freedom to implement it with an underlying bitfield, among
other helpful things.
- switch the cmake search from Qt4 and Qt5, making use of the components
for the Qt5 libraries
- adopt the automoc handling provided by cmake, so there is no more need
to manually run & add moc sources
- make use of the cmake imported targets for the Qt5 libraries, so there
is no need to manually care about variables for libraries and include
directories
Building on Joseph Goldstone's adding support for "rational" metadata type,
* Allow setting of rational data with oiiotool like this:
$ oiiotool foo.exr --attrib:type=rational onehalf "50/100" -o rat.exr
* Allow nice human readable printing from iinfo or oiiotool --info, like:
$ iinfo -v frame.0.exr
...
captureRate: 24000/1000 (24)
framesPerSecond: 24000/1000 (24)
onehalf: 50/100 (0.5)
...
* Change the definition of the "FramesPerSecond" standard metadata from
float to rational. This only affects GIF and ffmpeg-based movie files,
and if you retrieve it with get_float_attribute(), it will auto-convert
anyway. So we hope this will not break anybody.
* Add test cases
* Artifact: removed some code redundancies between formatspec.cpp and
paramlist.cpp.
* Add support for OpenEXR rationals, implementing TypeDesc::TypeRational along the way. Comment out some unused typedefs in simd_test that were stopping the compile.
* Change (1 << 31) to (1UL << 31) for the sake of GCC
* Remove unused typedefs in simd_test that clang found problematic
Safely means, even if the channelnames is not filled out or you ask
for an out-of-range index, it will not crash or overrun the array, but
rather just return an empty channel name.
See #1700, at some point when we made some ref images, it was doing
something on tiff write that was making images that some readers would
give warnings about. All seems fine now, so just fix the refs.
Sometimes you really, REALLY just want closest pixel blocky appearance
rather than bilinear sampling. ImageBufAlgo::resample() had that option,
but `oiotool --resample` didn't expose it.
For several ImageBufAlgo functions related to color conversion:
* Deprecated in 1.7 -> turn on the OIIO_DEPRECATED attribute.
* Deprecated in 1.6, attribute turned on in 1.7 -> remove them.
In issue #1239, @lgritz described a situation in which the assertion in `check_max_mem()` could be triggered because `m_tilecache` could be empty, but `m_mem_used` could still be over the limit due to "dead man walking" tiles still being held in per-thread microcaches. A temporary fix was created in PR #1240, simply replacing the ASSERT with a DASSERT so that release builds wouldn't trigger the assertions. But this just allowed release builds to execute exactly the code path that the assertions were intended to protect - leading to the dereferencing of an invalid TileCache iterator. Here we simply replace the DASSERT with an early return if we detect this unexpected (but still entirely possible) situation.
The deep holdout function is one that culls the deep samples in one deep
image (A) that are behind the opaque depth of a second deep image (B).
No samples of B are included in the output, it is only used to mask
parts of A. Samples of A that straddle the opaque frontier are split.
This is implemented as ImageBufAlgo::deep_holdout(), and also in
oiiotool --deepholeout.
Push a bunch of the logic previously in ImageSpec, to retrieve typed
attributes and do type conversions in the process, down to the ParamValue
and ParamValueList levels. Makes it a little more generally useful.
Some (older, not very color-smart) image formats store a gamma exponent
-- neither defining a good color space nor being in a standard like
sRGB. We were previously haphazardly labeling these as color space
"GammaCorrected", storing the gamma value in the "oiio:Gamma" metadata,
and hoping for the best. But the best, unfortunately, was total
ignorance by our standard color conversion functions (including the
likes of `oiiotool --autocc`). And we just have too many places where
simply the name of the color space itself (without access to other
metadata) is assumed to be enough to perform color conversions.
So this patch does the following:
1. Adopts a new convention of calling the color space of such images
"GammaCorrectedX.Y", where the X.Y is the decimal gamma value. As an
example, we might now set the "oiio:ColorSpace" metadata for such files
to "GammaCorrected2.2".
2. Follow this convention when reading and writing the formats
supporting an explicit gamma exponent, including Cineon, DPX, HDR/RGBE,
PNG, RLA, Targa.
3. Augment the color transformation engine to recognize these names,
decode the exponent from the name, and thus be capable of (at the very
least) transforming between linear and gamma space. This means that
`oiiotool --autocc` should now be able to properly linearize input files
of this kind (previously it would give a warning and fail to linearize).
And also, it means that you can apply gamma corrections using commands
such as `oiiotool in.exr -colorconvert linear GammaCorrected2.2 -o out.png`
4. Makes `iv` recognize plain old gamma corrected images and not impose
a second correction on it. This is a band-aid, and what it really needs
is a full-blown OCIO support. But it's still a small step forward in
seeing some images properly that previously were incorrect.
When using --autocc, previously having the deduced color space be one
whose transformation was unknown was a hard error that would exit
oiiotool. This patch changes an unknown color space be a warning, with
the rest of the oiiotool commands proceeding despite not transforming
the colors (this is just for the automatic transformations triggered by
--autocc).
1. If pugi load_buffer finds syntax errors in the XML it is parsing, it
returns an error code, which caused us to ignore the XMP block entirely.
It turns out that's a poor strategy, because lots of files have XMP that
is ever so slightly malformed, so we weren't picking up their XMP at
all! So, just ignore the fact there was a parsing error, go ahead and
try to make sense of whatever portion (maybe all?) of the block was able
to be parsed. It works!
2. Added a additional XMP tags for GPano, Google's Photo Sphere metadata
schema.
3. Replace linear search of the XMP tag list with a flat_map::find
(which underneath is a sorted vector with binary search).
4. Also add a bunch of camera raw (crs: prefix) XMP tags to the table.
I had been assuming that for TIFF photometric mode "separate",
it would always be 4 channels and they would be CMYK. Not so!
Turns out that there's a tag called InkSet which says if it's really
CMYK or not. If it's not, check the InkNames to provide channel names.
Also, the auto-conversion of CMYK to RGB should only happen if it's
truly CMYK, because the conversion wouldn't be correct for arbitrary inks.
On output, also, if the request is made to output color separated mode
(by setting the attribute "tiff:ColorSpace" to "CMYK" or "color
separated"), set those attribute as correctly as we can -- to true CMYK
if it appears to be so (4 channels, named C,M,Y,K or Cyan, Magenta,
Yellow, Black), but otherwise say it's custom inks and output the
channel names as ink names.
Aside: It has always bothered us that TIFF doesn't support channel names
(like OpenEXR does). But it does support the InkNames tag. What if we
always output the channel names as InkNames, even for non-separated
photometric mode. Would there be any negative consequences? Would apps
ever misbehave in the presence of an InkNames tag for an RGB photometric
image, or would it just be ignored?
searchpath_find had a couple calls to the underlying is_directory
and is_regular_file without catching any exceptions they might throw.
Those should be changed to Filesystem is_directory and is_regular
which are our existing wrappers that already catch any exceptions.
We can remove most of the complexity of this class by making
ParamValueList inherit from std::vector<ParamValue> and just add the few
extra methods we want (like a version of find that is case-insensitive
on name, but excludes potential matches that are the wrong data type).
In other words, a change of "has-a" to "is-a" makes most of the code go
away.
A finger malfunction ages ago botched the list of channel names that
help us sort channels into canonical order. Where I wanted to say
RA, GA, BA, somehow I said RA, RG, RB. Oh boy.
I realized that if intrusive_ptr<T>::reset(T*) is called with a pointer
to the same object it already refers to, it could be erroneously freed
because it was decrementing the existing refcnt before incrementing the
refcnt of the new value. So rewrote reset() to be safe under that
circumstance.
Simon Smith pointed out on oiio-dev a flaw in TS::get_texels(filename,...)
where it called itself recursively instead of correctly calling
get_texels(handle,...).
This patch contains the fix (the 2-line change in texturesys.cpp).
I also added a new testsuite entry, and needed to beef up testtex.cpp
a bit to make it work smoothly to test get_texels().
1. Some complaints (including failures in OSL TravisCI builds) led me to
believe that the 'cmake_policy (SET CMP0042 OLD)' was not correct and
that we should remove that and set MACOS_RPATH ON. This seems to clean
up some rpath troubles on OSX. It should not have any effect on non-OSX
builds.
2. For the Makefile wrapper, allow INSTALL_PREFIX to be set, which is
passed along to cmake as CMAKE_INSTALL_PREFIX. It defaults to the usual
of ./dist/PLATFORM, but this lets you override to, for example, instruct
the build to put the full installed files in another location. You could
always do this if invoking cmake directly, but it was tricky for those
who preferred to use the make wrapper.
It was not working, had an assertion.
Remove the assertion, fix the logic to make it work, add a test.
While I was there, I moved a histogram test from imagebuf_test.cpp to
imagebufalgo_test.cpp, it seemed to be more on-topic there.
Remember that "resample" is a resize with no filtering or interpolation:
for each pixel in the destination, is merely copies the source pixel whose
center is closest to the corresponding spot in the image.
This patch extends the functionality of ImageBufAlgo::resample() and
oiiotool --resample, to happily do the obvious thing for "deep" images.
This lets you quickly (but inaccurately) resize or create different-sized
proxies for deep images as you would for regular images.
Some day, it would be nice to extend a true resize() to work for deep
images, so we can have high-quality deep resizes. But that is going to
take some head-scratching about how to properly interpolate or filter
deep images.
The 'size' specification argument to oiiotool --resize and --resample
had very unintiutive behavior for images where the display and pixel
data windows were not the same, especially if the data window had a
nonzero origin (such as crop or overscan cases).
The underlying ImageBufAlgo::resize is mapping a display window and
is more straightforward. So in this patch, we take the oiiotool --resize
(and --resample) and interpret it as the desired display window size,
and then compute the approprate pixel window from it, rather than the
other way around (which was really strange).
So now you are saying: "oiiotool, here is the size and offset of the
display window I want for the output. Resize the input image so that its
original display window maps exactly to the new display window I've told
you about. Figure out what pixel data window you will need in the output
to ensure that all of the data pixels in the input will be represented."
For the usual 99% case where display == data and origins are 0, the
behavior does not change. But this is definitely easier to control for
the oddball cases.
Mimics recent fix in OSL:
* Explicitly and automatically build the vMAJOR_MINOR versioning into
the namespace.
* Simplify double nested namespace to just one level.
* Change OIIO_NAMESPACE_USING with `using namespace OIIO` (the macro
was obfuscating without being short or helpful).
1. atomic_min/max: Some web reading makes me think that our
implementation of atomic_min and atomic_max ought to use
compare_exchange_weak rather than compare_exchange_strong. As I
understand it, the stricter requirements of the strong versions
(basically, guaranteeing to never have a "false negative" where it
thinks it needs to try again but it's actually unnecessary) can on some
architectures lead to much worse performance.
2. New atomic_fetch_add for std::atomic<float> and <double>. They
exist, but don't in the current standard support atomic add (so we
implement them in terms of compare_exchange, which is supported).
It's for helping with benchmarks where you want to see how something scales
with number of threads.
Example output:
threads time speedup efficient its/thread range (best of 1)
1 2.3 1.00x 100.00% 20000000 0.00
2 1.2 1.97x 98.50% 10000000 0.00
4 0.7 3.59x 89.86% 5000000 0.00
New file, function_view.h, contains the definition for a function_view<>
template that is a lightweight, non-owning wrapper for a callable
object. It has much less overhead than a std::function<>, but at the
expense of being non-owning and needing to ensure that its use does not
outlast the lifetime of the callable entity it wraps (very analogous to
the limitations of string_view and array_view).
Seems that MSVC will give errors if the simd types are passed on the
stack by value, since they can't guarantee proper alignment.
Changing a few instances of these from, say, `float4` to `const float4&`
fixes it, and should make no performance difference for small inline
functions.
Using vector::push_back when constructing a new object can cause
extra copies or allocations/deallocations. The new (C++11) emplace_back
grows the vector first and then constructs the object in place, avoiding
any unnecessary moves, copies, or allocations.
Thanks, clang-tidy, for finding all of these for us with no hassle.
At the end of the IC/TS stats, there's a list of "broken" files, and
what printed there were those files which were either broken() or
!validspec().
Two circumstances lead to the spec being not valid:
(1) Calls to invalidate() mark the spec as invalid, of course! But just
because the app invalidated the file (perhaps because it thought the
file might be subsequently updated on disk and wanted a fresh look at
it) doesn't mean the file was ever broken.
(2) When the app requests an ImageCacheFile handle up front (as OSL
does) but then never opens the file. That also doesn't mean it was
broken. You just never read from it.
For both of these circumstances, it's misleading to list the file among
those that are known to be broken (presumed corrupted or missing).
Previously we used a mix of quotes, and sometimes not having the
OpenImageIO/ qualifier.
This is helpful in clang-tidy use, because we're including headers
based on matching OpenImageIO/*.h, as a way to have it analyze all our
own headers, but no system headers.
The first review of clang-tidy support neglected to use the --header-filter=
command, thus excluding headers from analysys.
Also, when tidying headers, if you use -fix, it's very important to
ensure that only one module is compiled at a time, or two processes
might try to modify a header at the same time, garbling it.
All off by default, but allows really easy use of clang-tidy while building.
For example,
make CLANG_TIDY=1 CLANG_TIDY_CHECKS="-*,modernize-*"
or if you are really adventurous
make CLANG_TIDY=1 CLANG_TIDY_CHECKS="-*,modernize-*" CLANG_TIDY_FIX=1
1. Extend ImageBufAlgo::render_text in several ways:
(a) Text color is now passed as an array_view<const float>, which is
safer than passing a raw pointer and hoping it points to enough
data.
(b) New parameters: alignx and aligny, which can be used to control
the alignment of the text with the (x,y) coordinates passed.
Options for alignx are ImageBufAlgo::TextAlignX::Left (default),
Right, and Center, and options for aligny are TextAlignY::Baseline
(default), Top, Bottom, Center.
(c) New parameter: int shadow, default to 0 (old behavior), with nonzero
values causing the rendered text to be dilated with black halo to be
more visible on backgrounds with similar color.
(d) The destination image no longer is required to be initialized. If not,
it will be initialized to be just large enough to hold the rendered
text. Pre-initialized images will continue to draw the text overtop
of the existing pixel values (as if an "over" composite).
2. Add a new ImageBufAlgo::text_size() function, which is related to
IBA::render_text(), except that rather than modifying an ImageBuf, it
merely computes the bounding box (relative to the left edge of the
baseline of the first character) as a ROI. But with this knowledge, it
becomes easy to do things like align or center text in tricky ways.
3. oiiotool --text has also been extended to have optional xalign= and
yalign= modifiers to align the text to the coordinates
passed. Horizontal alignment choices are "left" (default), "right", and
"center", and vertical alignment choices are "baseline" (default),
"top", and "bottom." Also, an optional modifier shadow= can control the
drop shadow.
So, for example, to make your text centered along the horizontal axis of
an image at scanline 100, and use some shadowing to clarify the text,
you could:
oiiotool in.exr -text:x={TOP.width/2}:y=100:xalign=center:shadow=1 \
"Hello, world" -o out.exr
On oiiotool command line {expressions}, FRAME_NUMBER is replaced based
on the numeric wildcard iteration, and FRAME_NUMBER_PAD is the same
value but as a string that appears zero padded to the number of digits
specificed by --framepadding.
So, for example, to burn in frame numbers to every image in a sequence:
oiiotool -framepadding 4 -frames 1-100 in.#.exr \
-text:x=50:y={TOP.height-50} "Frame {FRAME_NUMBER_PAD}" \
-o out.#.exr
Just like the existing "format_list" returns a comma-separated list of
all formats supported by any built-in or found plugins, the new queries
return the lists of input and output formats, respectively. An example
of how they may differ is if a particular format is available for
reading but not writing.
At the time of this writing, cineon, dds, dicom, ffmpeg, psd, ptex, raw,
and softimage are input formats but not output formats. Presently, there
are no output-only formats.
In organizing this, we removed entirely the *output.cpp files for
several formats that did not support outputs, with those now-unneeded
files containing only nonfunctional stubs.
They are very similar. From time to time I do two-directional transfers
to migrate the best practices of each to the other.
I also try to nudge them even closer in various ways, such as using
${PROJECT_NAME}_BLAH in both instead of OSL_BLAH in one and
OpenImageIO_BLAH in the other.
JFIF files, as you know, only support 1 or 3 channels (grayscale and
RGB, respectively).
Previously, we accepted the special case of 4 channel output (presumably
RGBA) by silently dropping the last channel (A, not representable in
JFIF files), but for any other channel count (2, or >= 5) would just
allow open() to fail.
With this patch, we make the JPEG writer even more forgiving. For
nchannels > 3, we truncate to just the first three channels, and for 2
channels, we truncate to just the first channel (outputting grayscale).
The reason why we want this extra flexibility is that JPEG is such a
common format for "proxies", "thumbnails", and other kinds of embedded
or web-facing images, where we might want to just quickly generate such
a proxy for arbitrary images without wanting to think hard about the
logic of how to conform unusual channel counts to JPEG's limitations.
And let's face it, if the user really wanted high fidelity, they would
not be using JPEG -- a usually-lossy, 8-bit-only, 1-and-3-channel-only
format -- and so silently dropping a channel that can't be represented
is probably not the biggest compromise being made to the original data
when saving to this format.
Also support 32 bit RGB and grayscale. We thought we had 32 bit working
before, but we were wrong -- PhotoShop stores linear float for "32 bit",
not uint8 as we assumed.
There are two main things changed here:
- First we check whether requested filename points to a regular file.
This solves the issue of ftell() returning really huge value for
file stream opened for a directory.
- Second we check ftell() output and compare it to -1 to catch possible
acess issues.
In theory it'll be nice to use Filesystem::get_size() which does all
the checking already, but leaving it as a TODO. Also the thing to note
here is that Filesystem::get_size() returns 0 for directory, which
wouldn't let us to report more meaningful error to the user about
what's exactly wrong with file one tries to open.
---
WebP: Use Filesystem::file_size instead of ftell()
This allows us to keep all file-size related magic in a single place
and simplifies the code.
Additionally made m_image_size an explicit 64bit type to avoid possible
narrowing type casts and possible issue loading big images on 32bit
platforms.
---
WebP: Perform preliminary header check prior to full file read
This way we can avoid read of unnecessary big file just to realize
it is not a valid WebP image.
New validation code goes as following:
- We check that regular file was passed.
This way we can give reasonable error message to user. Perhaps
this could happen on a higher level so we always know regular
files are coming to the image readers. On another hand, one
might want to implement an image sequence reader where we pass
a directory containing images.
- We check that file size is 12 bytes at least.
This goes according to the file specification
https://developers.google.com/speed/webp/docs/riff_container
- We read 64 bytes at a max which we use for WebPGetInfo()
This 64 bytes constant comes from Blender, which seems to be
safe for all the file formats we support.
If decoding of this chunk fails we avoid reading the whole file.
Otherwise we read the whole file and decode that.
Things which can still be improved:
- There is some duplicated code around reading file data and verifying
it's size. While it's possible to de-duplicate it within this particular
file it probably makes more sense to have common utility which does
error() on the case of fread() failure.
- There is still some place in OIIO which causes slow behavior of iinfo
when trying to get info of 1Gig file filled with zeroes, but this does
not come from WebP codec. While it's something to be investigates still
it does not really belong to this commit.
- Not sure it's more clear to re-set file position before reading the
whole buffer or resize the buffer and read the remaining part with
a header offset.
Never liked the offset magic, so for now used simple file position
reset.
1. Wasn't setting the C++ standard flags properly for the test, so it failed
on some platforms and didn't use std::regex when it could.
2. Revised test -- the prior C++ did not properly discriminate between
what was implemented vs unimplemented on the broken gcc 4.8 (at least
some subsets of regex_match work, but regex_replace does not, until
it was all fully implemented in gcc 4.9).
Extend ImageSpec::erase_attribute so that the name of the attribute to
erase is a regex, and any matching attributes will be erased.
Add oiiotool --eraseattrib, taking an argument with regex pattern,
and it will strip any matching metadata from the top image.
Example uses:
oiiooool in.jpg --eraseattrib smpte:TimeCode -o no_timecode.jpg
oiiotool in.jpg --eraseattrib "GPS:.*" -o no_gps_metadata.jpg
oiiotool in.jpg --eraseattrib ".*" -o no_metadata_at_all.jpg
This is an attempt to solve #1635. This avoids the crash by ignoring
any directory entry which is outside of the parsing buffer (that would
be really weird if there is a legit memory pointer to an outside memory
block).
There are much more places where there is potential crash is burried
in here, but those i don't feel like going into.
For such things it's really good idea to trust no input data and verify
it, otherwise it's really easy to run into crashes and even possibly
security issues.
Maybe it's also not a bad idea to add an option tyo ignore exif headers
reading for the applications which doesn't need exif.
In various contexts (such as oiiotool -v -info) when printing metadata
that consists of an arbitrarily long array of values, it will print
the first several values and then "..." to avoid going haywire if there
are 100000 values, say.
But, hmmm, in such a case you just might want to know how many values
there are and their type. So now instead of
Foobar: 0, 0, 0, 0, 0, 0, 0, 0, ...
it will more helpfully say
Foobar: 0, 0, 0, 0, 0, 0, 0, 0, ... [557168 x uint8]
And then you know who you can yell at about what.
This adds an --echo MESSAGE command to oiiotool, which will print the
message to the console at that point the left-to-right evaluation of
command line arguments.
For example:
oiiotool test.tif --resize 256x0 --echo "result is {TOP.width}x{TOP.height}"
This will resize the input to be 256 pixels wide and automatically size
it vertically to preserve the original aspect ratio, and then print
a message to the console revealing the resolution of the resulting image.
For the particular cases of "open with config" (triggered by the
--iconfig option), we had a different code path that goes through the
ImageCache and were not properly propagating any errors that were
encountered upon opening the file.
The "raw" reader by default will convert its input to sRGB color space,
but allows you to request that it be converted upon read to other
color space (or not alter the values) by setting the "configuration"
option "raw:ColorSpace" to the name of the color space you want.
This patch adds "ACES" to the list of color spaces it understands,
at least for the versions of libraw that support it (version 0.18 and
newer, it seems).
While I was in there, I changed the color space names it recognizes
to be case-insensitive.
Fixes#1628
1. The default value for missing "raw:use_camera_matrix" attribute should
have been 1, not 0, in order to match the default behavior of libraw/dcraw.
2. The demosicing control logic wanted to erase the "raw:Exposure"
attribute, but incorrectly specified the search to be constrainted to a
string attribute (thereby failing to erase the attribute which is
actually a float). The simple solution is that those erase_attribute
calls really don't need to constrain the type at all.
3. Add support for newer (as of LibRaw 0.16) demosaic algorithms "DMT"
(mode 11) and "AAHD" (mode 12). Also allow the demosaic algorithm names
to be case insensitive.
4. Rename "Modified AHD" to "AHD-Mod", to simplify the name and
match libraw terminology.
OK, this is subtle. When decoding XMP, I originally thought that it
wouldn't include Exif data, but sometimes it does. But the XML is just
string data. So this patch will look up the name to see if it's a known
Exif field, and if so, use the type we know it ought to be to correctly
decode from a string.
Also in exif.cpp, changed the implementation of the tag map used by
exif_tag_lookup so that it's case insensitive (because, ugh, some
raw images have their Exif data in XMP as "exif" rather than "Exif").
The bottom line is that these changes improve proper recognition of
certain metadata when reading RAW files.
* Found a bug in Srutil::format and ustring::format that was introduced
recently when we switched it to use variadic templates -- when passed
just a single bare format string (with no arguments), it was just
passing back the original string, without doing correct conversion of
%% to % as it should to match printf-like behavior. We need to run
even the format-string-only case through tinyformat.
* Convert all the other places that used TINYFORMAT_WRAP_FORAMT into
much more straightforward use of variadic templates, now that we are
C++11 minimum.
2017-02-22 13:39:54 -08:00
1258 changed files with 367224 additions and 127212 deletions
Please ask questions ("how do I...?") on the [oiio-dev](http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org) mail list, not by filing issues.
Make concrete suggestions for fixing the code ("here is a patch") by filing
a pull request, not an issue.
### Problem
Describe the problem or suggestion here.
**Expected behavior:** [What you expected to happen]
**Actual behavior:** [What actually happened]
If your problem is a broken build, please create a [gist](https://gist.github.com)
that contains the _full_ verbose build log, which you can create like this:
```make nuke ; make VERBOSE=1 > build.log```
If you are reporting a problem about a release branch, have you already
tried the current head of `master` to see if the problem has been fixed?
NOTE: delete this section after reading, it is just to give guidance for new contributors.
<!-- This is just a guideline and set of reminders about what constitutes -->
<!-- a good PR. Feel free to delete all this matter and replace it with -->
<!-- your own detailed message about the PR, assuming you hit all the -->
<!-- important points made below. -->
Please read the [contribution guidelines](https://github.com/OpenImageIO/oiio/blob/master/CONTRIBUTING.md), and be sure we have a Contributor License Agreement ([individual](https://github.com/OpenImageIO/oiio/blob/master/src/doc/CLA-INDIVIDUAL) or [corporate](https://github.com/OpenImageIO/oiio/blob/master/src/doc/CLA-CORPORATE)) on record with the project.
## Description
Please provide a description of what this PR is meant to fix,
and how it works (if it's not going to be very clear from the code).
<!-- Please provide a description of what this PR is meant to fix, and -->
<!-- how it works (if it's not going to be very clear from the code). -->
## Tests
Did you / should you add a testsuite case (new test, or add to an existing
test) to verify that this works?
<!-- Did you / should you add a testsuite case (new test, or add to an -->
<!-- existing test) to verify that this works? -->
------
## Checklist:
<!-- Put an 'x' in the boxes as you complete the checklist items -->
- [ ] I have read the [contribution guidelines](https://github.com/OpenImageIO/oiio/blob/master/CONTRIBUTING.md).
- [ ] I have updated the documentation, if applicable.
- [ ] I have ensured that the change is tested somewhere in the testsuite
(adding new test cases if necessary).
- [ ] If I added or modified a C++ API call, I have also amended the
corresponding Python bindings (and if altering ImageBufAlgo functions, also
exposed the new functionality as oiiotool options).
- [ ] My code follows the prevailing code style of this project. If I haven't
already run clang-format before submitting, I definitely will look at the CI
test that runs clang-format and fix anything that it highlights as being
OpenImageIO has been accepted to be an ASWF-administered project, by vote of
the ASWF Technical Advisory Council (TAC) on May 3, 2023.
At this time, the ASWF subdirectory contains *proposed* (and sometimes TBD/WIP
placeholders) project organization and policy documents that will be formally
adopted as the OpenImageIO project is formally turned over to ASWF and moved
to the AcademySoftwareFoundation GitHub account. Please file issues or PRs
or review comments against these documents to make any suggested changes while
we are revising the drafts.
Summary of the current state of documents:
| File | Status | Notes |
| ---- | ------ | ----- |
| CLA-corporate.md | Final | Standard ASWF corporate CLA, approved by the TSC. Will take effect when the repo moves. |
| CLA-individual.md | Final | Standard ASWF individual CLA, approved by the TSC. Will take effect when the repo moves. |
| CODE_OF_CONDUCT.md | Final | Links to the standard LF CoC, approved by the TSC and now in effect. |
| CONTRIBUTING.md | Preliminary | Modified version of our existing CONTRIBUTING guidelines, lightly amended to reflect the changes that will come along with moving the repo, the new CLA/DCO rules, etc.
| GOVERNANCE.md | Final | Approved by the TSC and now in effect. |
| project_intake.md | Ongoing | This is the latest ASWF project intake checklist. We will check off items as we get them done. |
| Technical-Charter.md | Final | Approved by the TSC. |
This is a checklist for TSC's to review as part of the intake process. The TSC should review this entire list during the kickoff meeting. For anything outstanding, create an [issue](../issues) to track and link to it in the list
- Existing Project Governance
- [X] A [LICENSE.md](../LICENSE.md) file in every code repository, with the license chosen an OSI-approved license.
- [X] Any third-party components/dependencies included are listed along with thier licenses ( [THIRD-PARTY.md](../THIRD-PARTY.md) )
- [X] A [README.md](../README.md) file welcoming new community members to the project and explaining why the project is useful and how to get started.
- [X] A [CONTRIBUTING.md](../CONTRIBUTING.md) file explaining to other developers and your community of users how to contribute to the project. The file should explain what types of contributions are needed and how the process works, along with how to disclose security issues responsibly ( may also point to a [SECURITY.md](../SECURITY.md) file ).
- [ ] A [CODEOWNERS](../CODEOWNERS) or [COMMITTERS](../COMMITERS.csv) file to define individuals or teams that are responsible for code in a repository; document current project owners and current and emeritus committers.
- [X] A [CODE_OF_CONDUCT.md](../CODE_OF_CONDUCT.md) file that sets the ground rules for participants’ behavior associated and helps to facilitate a friendly, welcoming environment. By default, projects should leverage the Linux Foundation Code of Conduct unless an alternate Code of Conduct was previously approved.
- [X] A [RELEASING.md](../RELEASING.md) file that provides documentation on the release methodology, cadence, criteria, etc.
- [X] A [GOVERNANCE.md](../GOVERNANCE.md) file that documents the project’s technical governance.
- [ ] A [SUPPORT.md](../SUPPORT.md) file to let users and developers know about ways to get help with your project.
- Infrastructure/Assets
- [X] Technical Charter Approved
- [ ]**(Awaiting scan from LF)** License scan completed and no issues found
- [ ] Code repository imported to ASWF GitHub organization or ownership of current GitHub organization given to `thelinuxfoundation` user
- [X] Developer Certificate of Origin past commit signoff done and DCO Probot enabled.
- [X] Issue/feature tracker established (JIRA, GitHub issues)
- [X] Mailing lists ( one of )
- [X] Setup new lists ( -dev@ and -tsc@ ) on [ASWF groups.io](https://lists.aswf.io) ( create [issue on foundation repo](https://github.com/AcademySoftwareFoundation/foundation/issues/new) to setup )
- [X] (In progress) Move to [ASWF groups.io](https://lists.aswf.io) ( create [issue on foundation repo](https://github.com/AcademySoftwareFoundation/foundation/issues/new) to transfer )
- [X] Slack ( create [issue on foundation repo](https://github.com/AcademySoftwareFoundation/foundation/issues/new) to setup project channel on [ASWF Slack](https://slack.aswf.io) )
- [ ] Website
- [X] CI/build environment
- [X] Trademarks/mark ownership rights ( complete 'LF Projects - Form of Trademark and Account Assignment' - create [issue on foundation repo](https://github.com/AcademySoftwareFoundation/foundation/issues/new) - only needed if project using existing name )
- [ ] Domain name ( create [issue with the LF IT staff](https://jira.linuxfoundation.org/plugins/servlet/theme/portal/2/group/19) to setup/transfer )
- [ ] Zoom account access ( create [issue on foundation repo](https://github.com/AcademySoftwareFoundation/foundation/issues/new) to get access to credentials )
- [X] (N/A) Social media accounts or other project accounts ( create [issue on foundation repo](https://github.com/AcademySoftwareFoundation/foundation/issues/new) to transfer )
- [X] Logo(s) ( create [issue on artwork repo](https://github.com/AcademySoftwareFoundation/artwork/issues/new) to add in SVG and PNG format and color/black/white )
- New Project Governance
- [X] TSC members identified
- [X] First TSC meeting held
- [X] TSC meeting cadence set and added to project calendar
- [X] CLA Approved ( if used ) ( [CCLA](CLA-corporate.md) and [ICLA](CLA-individual.md) )
- Project charter ( [Technical-Charter.md](Technical-Charter.md) )
- [X] [Core Infrastructure Initiative Best Practices Badge](https://bestpractices.coreinfrastructure.org/) achieved as the 'Passing' level.
- Outreach
- [ ] New project announcement done ( create [issue on foundation repo](https://github.com/AcademySoftwareFoundation/foundation/issues/new) to trigger )
- [X] Project added to ASWF website and ASWF landscape
- Incubation Stage requirements
- [X] Have achieved and maintained a [OpenSSF Best Practices Badge at the passing level](https://bestpractices.coreinfrastructure.org/en/criteria).
- [ ] Have had a successful license scan with any critical issues remedied.
- [ ] Have a defined project mission and scope
- [ ] An overview of the project’s architecture and features defined.
- [ ] A project roadmap defined, which should address the following questions.
- What use cases are possible now?
- What does the next year look like in terms of additional features and use cases covered?
- [ ] Community and contributor growth assessment
- The current number of contributors and committers, and the number of different organizations contributing to the project.
- Demonstrate a sustained flow of commits / merged contributions
- A credible plan for developing a thriving user community, in particular expanding the number of committers and contributors?
- Outline of the plan for the project to complete the requirements for Adopted Stage
- [ ] Obtain an affirmative vote of the TAC.
- Adopted Stage graduation requirements
- [ ] Demonstrate a substantial ongoing flow of commits and merged contributions, authored by a healthy number of diverse contributors.
- [ ] Demonstrable roadmap progress.
- [ ] A healthy number of public adopters that are identified within the project ( using an ADOPTERS file or showcased on the project’s website ).
- [ ] Have achieved and maintained an [OpenSSF Best Practices Badge at the gold level](https://bestpractices.coreinfrastructure.org/en/criteria/2).
- [ ] Have a technical lead appointed for voting representation of the project to the TAC.
- [ ] Be deemed by the TAC to add value to the mission of ASWF.
- [ ] 2/3 supermajority vote of the TAC
- [ ] Affirmative majority vote of the Governing Board
# "Vendor" is only used in copyright notices, so we use the same thing that
# the rest of the copyright notices say.
set(CPACK_PACKAGE_VENDOR"Larry Gritz et al.")
set(CPACK_PACKAGE_DESCRIPTION_SUMMARY"OpenImageIO is an open source library for reading and writing image file formats, a nice format-agnostic image viewer, and other image-related classes and utilities.")
Code contributions to OpenImageIO are always welcome. That's a big part of
why it's an open source project. Please review this document to get a
briefing on our process.
> NOTE: This is the proposed post-ASWF-move version of CONTRIBUTING. After the
> project is legally transferred and moved to the new repo, this file will
> replace the one at the project root.
>
> TO DO:
>
> - [X] Update the mail list sign-up page after the mail list moves.
> - [ ] Update the repo URL
> - [ ] Double check the security and info email addresses.
>
Code contributions to OpenImageIO are always welcome, and [nearly 200
people](https://github.com/OpenImageIO/oiio/blob/master/CREDITS.md) have done
so over the years. Please review this document to get a briefing on our
process.
Mail Lists
----------
Mail List and Slack
-------------------
There are two mail lists associated with OpenImageIO development:
Contributors should be reading the oiio-dev mail list:
* [oiio-dev](http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org) For developers of the OpenImageIO code itself, or users who are
really interested in the OIIO internals. This is where we discuss the code
(including bug reports).
* [oiio-dev](https://lists.aswf.io/g/oiio-dev)
For developers of the OpenImageIO code itself, or users who are really
interested in the OIIO internals. This is where we mostly discuss the code
(including bug reports), but are also happy to answer user questions about
use or working of OIIO.
* [oiio-announce](http://lists.openimageio.org/listinfo.cgi/oiio-announce-openimageio.org) For announcements about major OIIO releases or other important news.
You can sign up for the mail list on your own using the link above.
You can sign up for these mail lists on your own using the links above.
The [ASWF Slack](https://slack.aswf.io/) has an `openimageio` channel. Sign up
for the Slack on your own, then under "channels", select "browse channels" and
you should see the openimageio channel (among those of the other projects and
working groups).
How to Ask for Help
-------------------
If you have trouble installing, building, or using OpenImageIO, but there's
not yet a solid reason to suspect you've encountered a genuine bug, start by
posting a question to the [oiio-dev mailing
list](https://lists.aswf.io/g/oiio-dev).
This is the place for questions such has "How do I...".
Bug Reports and Issue Tracking
------------------------------
We use GitHub's issue tracking system for bugs and enhancements:
https://github.com/OpenImageIO/oiio/issues
We use GitHub's issue tracking system for reporting bugs and requesting
1. Check out the trunk or a branch of your choice. The remainder of
these instructions assume that you checked out the trunk to the
D:\OIIO\trunk directory.
You will need to have Git, CMake and Visual Studio installed.
2. Download the package with precompiled external libraries from
http://www.openimageio.org/external.zip
The minimal set of dependencies for OIIO is: Boost, zlib, libTIFF, OpenEXR, and libjpeg or libjpeg-turbo. If you have them built somewhere then you skip
the section below, and will only have to point OIIO build process so their locations.
Next, unpack it. The directory with downloaded code from the repository
and the directory with unpacked libraries should be siblings. For example,
* Boost: get the boost source archive, extract into `{BOOST_ROOT}`.
```
cd {BOOST_ROOT}
bootstrap
b2
```
* zlib: this will build it, and then delete the non-static library, so they don't get picked up:
* OpenEXR: you'll have to point it to your `{ZLIB_ROOT}` location from the above. If copy-pasting the multi-line command (with lines ending in `^`) into
cmd.exe prompt, make sure to copy all the lines at once.
Also, if cmake won't find GLEW set up GLEW_INCLUDES and GLEW_LIBRARIES
in cmake-gui. Don't add them (they are already added), just set them.
1. Visit Microsoft's vcpkg GitHub page: https://github.com/Microsoft/vcpkg. Also note that the openimageio package is located here: https://github.com/Microsoft/vcpkg/tree/master/ports/openimageio
2. Follow vcpkg installation instructions and complete the install. Please note vcpkg has its own list of prerequisites listed on their page.
3. Execute the PowerShell command from where vcpkg is located in directory. ``vcpkg install openimageio``
Test Images
-----------
We have yet another SVN project just for containing a set of sample
images for testing OpenImageIO. We split test images into a separate
SVN project in order to make the main source code tree smaller and
simpler for people who don't need the test suite.
We have yet another project containing a set of sample images for testing
OpenImageIO. We split test images into a separate project in order to make
the main source code tree smaller and simpler for people who don't need the
| master | :white_check_mark: :construction: ALL fixes immediately, but this is a branch under development with a frequently unstable ABI and occasionally unstable API. |
| 2.4.x | :white_check_mark: All fixes that can be backported without breaking ABI compatibility. New tagged releases monthly. |
| 2.3.x | :x: Only receives occasional critical fixes, upon request. |
| <= 2.2.x | :x: No longer receiving patches of any kind. |
## Reporting a Vulnerability
If you think you've found a potential vulnerability in OpenImageIO, please
report it by emailing security@openimageio.org. Only the project administrators
have access to these messages. Include detailed steps to reproduce the issue,
and any other information that could aid an investigation. Our policy is to
respond to vulnerability reports within 14 days.
Our policy is to address critical security vulnerabilities rapidly and post
include ("${CMAKE_CURRENT_LIST_DIR}/@PROJECT_NAME@Targets.cmake")
check_required_components ("@PROJECT_NAME@")
# Set a CMake variable that says if this OpenImageIO build has OCIO support
set (@PROJECT_NAME@_HAS_OpenColorIO @OpenColorIO_FOUND@)
Loaded 100 of 1258 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Larry Gritz
