There turns out to be a way to get wedged into a mutual deadlock on
the file handle cache, with how get_texture_info on a udim pattern
checks to see if the attribute requested is present and equally valued
in all constituent files of the udim set. This hits the handle set
pretty hard, and if multiple threads are hammering the same sets and
rapidly cycling textures out of the cache, there can be trouble, it
seems.
Remedies in this patch:
* When it's trying to close files because the number of handles is too
high, we already had it so that if we only exceeded the max_files
limit by a little bit, we would simply try to get the lock, and
abandon the attempt if somebody else is holding it, but if we are
quite a bit over, then stay and wait for the lock to be released.
In this patch, we make the that default behavior -- if we are over
the limit and need to free some handles, simply try to get the lock,
and if somebody else is holding it, leave it all up to them and
return immediately. The pros of this are (a) snappier response by
never waiting at this point, and (b) removing the possibility of a
deadlock, because if you can't acquire the lock, you just move
on. But the con is that lots of threads adding new files may run
faster than the one thread who has the file and is trying to release
files, so it's possible that the number of handles will exceed the
limit by a wider margin now, but hopefully only momentarily. (But
note that this can be overridden with a new ImageCache attribute
"max_open_files_strict", which if set to nonzero revert to the old
behavior. At least for now, in case I am neglecting to understand
some bad consequence of this change.)
* The recursive_mutex controlling access to the ImageInput in each
tile cache record is changed to a recursive_timed_mutex, and when
closing a handle by calling release(), instead of an unconditional
lock, try acquiring a lock for 100ms. If it fails to do so, it may
be in a possible deadlock, so just give up and move on. The only
consequence is that maybe that handle isn't freed after all. It will
then move on to the next one, not the end of the world, but at least
it can't get stuck there.
* In find_file, eliminate the call to check_max_files -- it's not
needed here, since it's also called by open(), which is the only
time that new file handles might be opened. We just don't also need
it again simply because we're adding a file to the cache.
* test: Add a corrupted test file
* fix(IC): upon tile read error, mark the file as broken
The idea is to keep fruitlessly reading from a corrupted file.
* fix: race condition in TIFF reader
TIFFInput::read_native_tiles had a very subtle race condition. There
is a section where it has a task_set of decompression tasks that are
sent to the thread queue. When the method ends, the task_set goes out
of scope, and its destructor will wait for all the remaining tasks to
finish. All fine, right? Except that other variables in the function
include unique_ptr's with allocated memory that the tasks need. So
what can happen is that if the compiler generates the code such that
the destructor of the unique_ptr happens before the destructor of the
task_set, the memory that the unique_ptr held may be deallocated
before some of the tasks still running in threads have finished.
In practice, we seem only to hit this race condition for particular
cases involving corrupted files, because the early `return` seems
quite likely to run those destructors before all the threads are done,
but it was possible to encounter it for valid files as well, if the
timing is just right. Shocking that we never had it reported.
The solution is just to have an explitit tasks.wait(), which will
block until all the tasks have finished before hitting the actual end
of the method and letting the other destructors to run. And for the
"early return" case, change it to a break so that we go through the
wait call at the end.
I believe that this fix addresses TALOS-2023-1709 / CVE-2023-24472.
* Make lookups of const unordered_maps thread-safe
* added back deleted end brace
* clang-format
---------
Co-authored-by: Larry Gritz <lg@larrygritz.com>
For non-0-origin images, we were checking the resolution range
incorrectly (`>=` vs `>`).
Also were printing the error message with the wrong notation, should
have shown a half-open range as `[beg,end)` and not `(beg,end]`.
When we recently added IOProxy support to the Maya iff file reader in
PR 3647, something broke for certain (but not all) iff files, and I
have productions stumbling over this.
It's proving tricky to track down and I have shows needing a fix
immediatly, so this change sets up a CMake variable
ENABLE_RLA_IOPROXY, default to ON, but when set to OFF, it switches
back to the old "no proxy" versions of the iff reader.
Disabling this will get me out of my current emergency -- at the cost
of not having IOProxy for iff, which I don't need at this moment
anyway -- and give me some breathing room to fix the problem for real,
at which point I will re-remove the old code.
There's a recurring idiom spread all over the python bindings for
ImageBufAlgo, which I encapsulate in a new local helper template
vecresize(). This ensures that the logic is identical in all the
places it's used, and also improves our coverage analysis (because it
was tricky to make tests that exercise the uninitialized-handling
cases separately for each of the many different places the idiom was
used).
Tweak the heuristics for trying to determine which arbitrary-named
color spaces are equivalent to certain known common names.
For OCIO 2.2+, some of this is easier and more reliable by using the
new built-in color spaces -- for example, we can transform a set of
sentinel colors from an arbitrary cs into the known built-in acescg,
or lin_srgb, etc., and if the values don't change, we know that cs
*is* the canonical one, even if named something weird.
But one problem is that for older OCIO, the built-in config is not
present, so if you have a color space named, I dunno, "linear", it's
hard to know what that is. Is it what we now call lin_srgb? Is it
ACEScg? Something else? So the hack is: first, we try to identify a
space that we think is sRGB (almost all configs have something called
"srgb", because everybody needs it, and we take it at face value that
if they call a cs "srgb", it's the real "srgb", fools who use the name
for something else get what they deserve). We transform the sentinel
colors from the srgb into the cs we are interrogating, and if they
match what we expect to see from an srgb->lin_srgb transformation,
then guess what -- the unknown color space must be lin_srgb!
We may, in the future, expand this to deduce other spaces. But this is
a start.
There are other changes I made here to make this whole process much
less expensive than it had been getting, there was some redundant
work, and some work that only was useful for 2.2 or only useful for <=
2.1, so I only do those in the right circumstances now.
Also along for the ride: improve test coverage for maketx features of
colorconvert, unpremult, attrib, attrib.
Examining some timings and function call counts, I saw that a typical
oiiotool command was spending a surprising amount of time in
ImageBufAlgo::zero(), not just more runtime overall, but also calling
it many more times than I thought it should. For example,
oiiotool -create 4096x4096 3 -colorconvert srgb acescg
called ImageBufAlgo::zero()... 4 times, totaling as much time as the
colorconvert operation.
The idiom used in `IBA::op(dst, src, ...)` which initializes the `dst`
IB if it's not already initialized will call the IB constructor that
zeros out its contents (so you don't have every new IB full of garbage
pixel values). But for IBA functions, this is almost never necessary,
since the IBA function itself will tend to overwrite all pixel values.
Not always, of course -- since IBA functions let you restrict the
spatial area or channel range -- but in the cases where the
destination starts out uninitialized and is allocated to exactly mtch
the ROI, I *think* you can count on the calling IBA function filling
in all those values.
So I changed IBAPrep() to call the IB constructor that says not to
zero out the pixels.
Just in case we ever need the ability, I made a new flag value that
lets you call IBAPrep and tell it, yes, do zero out any newly
allocated buffers. But we pass the testsuite without needing that flag
anywhere, so I think we're ok.
Also, the --create command itself would make new buffers (with the
implied zeroing), and then explicitly zero it again!
So after these changes, the oiiotool example command line above only
calls IBA::zero() ONCE (correctly, in the initial new image --create
makes), eliminating three redundant calls.
I also hypothesized that each call to zero was taking too much time,
because underneath it was calling IBA::fill which laboriously walked
through the IB with iterators and copying a fixed pixel value that
happened to contain 0 values. So I rewrote a special case where if the
buffer had contiguous strides, it would just use memset instead.
Surprisingly, it was not appreciably faster, and I'm guessing that the
memory bandwidth necessary to touch every pixel in a 4k buffer is the
limiting factor, and the other pixel-by-pixel iterator diddling was
inconsequential compared to waiting for the memory bus. I'm still
going to leave this "optimization" in, even though it was barely
measurable for the case I was trying.
Timer:
* Fix goof where m_printdtr was a bool but was tested for equality to
an enum. It happened to work like that, but only coincidentally.
* Add another enum value that allows a debugging message to be printed
upon construction of the timer, in addition to its destruction.
* Switch from printf style to std::print style formatting of the
debugging messages.
LoggedTimer:
* Modify to allow modifying the call count. This turns out to be
helpful for times when one logged function calls another logged
function (like if a special case is broken out) and they both log to
the same timer -- it makes it look like just one invocation instead
of two.
Currently, --resize takes a single argument giving an "image geometry"
(like "640x480") that designates the size of the result image, and the
rescaling specifically maps the "full size" (a.k.a. display window) of
the input image to the output.
This experimental change extends --resize to accept additional new
optional modifiers:
- ":from=<geom>" Overrides the default of the full window of the
source.
- ":to=<geom>" Overrides the default of the full window of the
destination.
- ":offset=+x+y" Adds an additional offset transformation to the
destination.
- ":edgeclamp=<0|1>" Turns on and off some special edge sample
clamping to try to reduce certain ringing artifacts. (Experimental.)
Unlike the output resolution, which like all image sizes must be an
integer resolution, the overrides on the source and destination
windows and the offset is allowed to be floating point, that is,
allowing for fractional pixel sizes or offsets.
The from and to windows do not restrict the boundaries of pixels
computed, but rather, simply define the transformational mapping
between the two images: the transformation will be chosen such that
the float-valued coordinates of the source image correspond exactly to
the float-valued coordintes within the destination image. In both
cases, coordinates refer to the true continuous coordinates in the
image plane, with (0.0, 0.0) being the "upper left" corner, and pixel
centers being at half-unit positions such as (ipixel+0.5, jpixel+0.5).
For example,
oiiotool in.exr -resize:from=100x50.3+50+20 1000x503 -o out.exr
creates a 1000x503 pixel image where the full output image is a
resized version of the 100x50.3 pixel window starting at coordinates
(50,20) in the input image.
This is mostly implemented (when it doesn't end up equivalent to the
old style resize) in terms of the existing IBA::warp() function. I needed
to add a new variety of warp() to avoid some artifacts, but I do this
tentatively and so currently do not expose it as part of the public API.
I want to gain some experience with the oiiotool usage by the users who
requested this, then will return later to exactly how it should be
exposed directly as IBA functions.
Some background: TIFF allows for pixel values to be encoded in some
alternative color spaces, like YCbCr, CIELAB, etc., and this is
indicated by the TIFF PhotometricInterpretation tag. Since in practice
we almost never encounter these encodings in the wild, and they
require a lot of custom code to handle (for example, YCbCr also comes
along with the ability to subsample the chroma, and options for
different placements of the subsampled positions, yuck), for most of
these cases we just lean on libtiff's ability to convert the whole
thing to RGBA for us, and so in that case, we return to the user the
pixels as RGBA, and set the "tiff:ColorSpace" metadata just as an
advisory to indicate how it had been encoded in the file. But we don't
support writing any of those. (One exception is that we do allow
reading and writing of CMYK files.)
OK, but now what about a case where the app already has the data in
one of these spaces (such as YCrCb; and let's ignore the whole
subsampling miasma and assume we don't want to support that), and all
we want to do is write that data to a TIFF file -- without any kind of
data conversion -- but to set the TIFF tags properly to indicate that
the data in the file is what we know it to be?
A related special case of this is: what if we have a TIFF file that
has YCbCr data but neglected to mark it as such (I'm staring daggers
at a certain widely used compositing package right now), and to fix
this, we want to use OIIO to just copy the file and set the right tags?
It turns out that in the status quo, it is surprisingly difficult to
do this. There is just nothing in our existing TIFF output code that
will write any of those photometric tags for those spaces, and no way
to communicate that this is what we desire.
Whew. Now here's what this patch does:
* If the "oiio:ColorSpace" attribute (our usual way to indicate
presumed color space of the pixel data) says that the pixels are
already in one of those color spaces (specifically, "YCbCr",
"CIELAB", "ICCLAB", "ITULAB", "LOGL", "LOGLUV"), then we set the
PhotometricInterpretation tag as such in the TIFF file that we
write.
* We still DO NOT interpret that as a request to convert pixel data
from any other space into this one. In other words, the TIFF output
does not convert color spaces (except for RGB <-> CMYK); this merely
lets us output a file that correctly says the data are what the app
thinks they are.
* A bunch of minor refactoring was necessary because of some unobvious
coupling and order dependencies. For example, the sgilog compression
techniques MUST be used with the LOGLUV color space, and vice versa.
You have to know how both should be set before checking that they
are compatible, and if they are not, adjust something that it's not
too late to change.
So the punchline at the end is that if you had a TIFF file that has
YCbCr encoded pixels, but for some reason it's marked as ordinary RGB,
you can fix it as follows:
oiiotool ycbcr_disguised_as_rgb.tif --iscolorspace YCbCb -o real_ycbcr.tif
Refactoring in #1641 incorrectly caused this to be set on the last layer
rather than the composite image as before. Now set it on the composite
image and every layer, as all of them use unassociated alpha.
* Tests for maketx -u, --attrib, --sattrib, --cdf
* maketx was documeted to take --cdf as a simple flag, but was
incorrectly implemented as requiring a (pointless) nonzero argument.
Fix it and hope nobody was depending on the undocumented behavior.
* maketx -v arg never had any effect. Add it back.
* Better Typedesc testing of 'tostring' with fmt style formatting
commands, and fix a bug that uncovered.
* Refine 'make CODECOV=1' behavior
* Beef up python-imagecache test to test getpixels, more attribute,
invalidate.
* Better testing of python IB construction from numpy array.
* testtex: Use correct guard to avoid compiling unused code for
coverage analysis testtex: get rid of unused 3d warp code.
* Add texture-threadtimes test
* Eliminate redundant code in exroutput.c by implementing
write_scanline using write_scanlines.
* Add Exif write support to PNG output.
Stores any Exif data from the given ImageSpec into the PNG eXIf chunk.
* add test for writing Exif data in png
* add test reference output for libpng versions not supporting the eXIf chunk.
* Update testsuite/png/ref/out-libpng15.txt
Co-authored-by: Joris Nijs <joris.nijs@esko.com>
Sets color space metadata. While conceptually, the main job is to call
`attribute("oiio:ColorSpace",name)`, this is a central spot to add
logic that adjusts or removes other contradictory metadata.
Note that these intrinsics don't seem to be available for gcc.
They are available on clang, but are defined in terms of the memcpy
trick. But in Intel, they seem to actually help.
The new 2023 one on the Intel site seems to require a glibc too new to
be compatible with the one on the ASWF CentOS-7 based containers, so
has been breaking our CI for a couple weeks.
It should not have been set in platform.h (inside the cpuid function!)
in the first place. And apparently, though this was ok for Windows ARM,
on Mac ARM it's an error because that symbol is already defined.
So only redefine it if it's currently undefined. And move the whole
check and definition into simd.h where all the other ARM_NEON stuff
lives.
We never had a test for `maketx --lightprobe`, so add it to
testsuite/maketx.
Immediately discovered that it only worked correctly for float images,
fails for half or other types. Fix.
We didn't have thorough tests of pbm/pgm/ppm files, add some.
That immediately uncovered a problem with pbp files.
By spec, pbm (1 bit) files use 1 to mean black, 0 for white.
Confusing, because pgm (grey) and ppm (rgb) use 0 for black.
* When checking code coverage, don't abort for failed tests
* Rejigger imagebufalgo_test a bit for better IBA test coverage
* Test zover
* Have testsuite/oiiotool-fixnan test with deep images
* Add to the extisting --fixnan tests a test for 'error' mode.
* Make testsuite/oiiotool-color test all colormap names
* Test 2D filters in filter_test
* Exclude python scripts in src/doc from coverage analysis
* Remove redundant attribute-seting code from the python internals
* Remove unused python_array_code function
* Exclude some imported code from coverage testing
* Re-enable and improve coverage testing from 'make CODECOV=1'
* Exclude unused/unexposed parts of xxhash.cpp from compilation
Turns out the clang/gcc intrinsics themselves compile to the same
memcpy idiom that I was using for the general case. So there is no
need for the clutter of the specialization.
libtiff 4.5 was just released, so confirm that we build against it,
update the "latest versions" CI test to use 4.5.
More careful sorting out of libtiff version numbers for conditional
compilation.
Get rid of some ancient tiff version guards that are related to
libtiff versions that are older than we currently support.
When building against libiff 4.5, use the new open calls that allow
per-file error handlers, which is better than the old global error
handlers for thread safety of multiple threads reading from the same
ImageInput if there are errors.
Convert error messages to std::format form.
We had no testsuite exercise of --pdiff, add it.
Along the way, I saw tha we were getting the "you didn't have any
output" warning when using pdiff, so fix that.
And some other minor changes: formatted output update, some other
tightening up of things.
A little bit of jiggling the contents of XYZtoLAB_color to work around
what I think is a compiler error with icx -- it was generating NaNs
for reasons I was unable to understand or fix, but not if I rearranged
the code slightly. All the other compilers and versions had correct
behavior.
Fixes#3721
Please read the comments in #3721. This is the "local" fix for the
build break due to the type mismatch. It is a band-aid. And it may be
the best solution for the 2.4 branch if we don't want to potentially
break ABIs by changing the definition of any public types.
Still pending is to examine the issue of whether it was a mistake to
define vbool4 storage for neon as uint32x4_t or if we should change it
to int32x4_t to better match the non-simd reference implementation.
After debating that (and identifying somebody with access to an
ARM-based machine to test the solution for us), we may return to
tackle this more fundamental change.
A few minor fixe to the term code, add a "term:filename" optional hint
to redirect output rather than go to stdout, and also change to the
new std::format notation for output.
* OpenColorIO improvements, especially for 2.2
* When OIIO builds against OCIO >= 2.2, in the event that there is no
config supplied (e.g., specified by env `$OCIO`), use he new
build-in "cg" config to supply a basic set of color spaces and
transformations. Yay for OCIO 2.2 and its built-in configs!
* New OIIO-understood aliases (regardless of whether they were in the
config or not): "scene_linear", "srgb", "lin_srgb", and "ACEScg".
It will do its best to know which, if any, supported color spaces
are equivalent to these, even if arbitrarily named. For OCIO >= 2.2,
some trickery is employed to use the new features that can convert
to built-in config color spaces, transforming several colors in this
manner and seeing if it gets what would be expected for various
canonical color spaces.
* `oiiotool --help` now a much abbreviated set of color information
(just the OCIO version, and the config, and a hint about how to get
more info).
* A new command `oiiotool --colorconfiginfo` prints the FULL
inventory of color management information, which now has been
expanded to reveal the aliases of each color space, print the roles,
and generally spruce up the output to be much more readable (though
longer) for large configs.
* New methods for our ColorConfig class:
- `resolve(name)` turns the name, which could be a color space, an
alias, a role, or an OIIO-understood universal name (like "sRGB")
into a canonical color space name.
- `equivalent(name1,name2)` returns whether or not the two names
refer, ultimately, to equivalent color spaces (to the best of its
ability to figure out).
* Some minor changes (including additional `const` qualifiers)
on the ColorConfig class.
* Make testsuite tests use the built-in OCIO config if OCIO 2.2+ is
found.
* Build sonar against ocio2.2 so we are primarily testing the new
code paths.
* For OCIO >= 2.2, because we can count on the built-in configs,
conditionally compile a bunch of hard-coded color transforms. These
will eventually disappear completely, someday, when OCIO 2.2 is the
minimum supported version.
I think the behavior should all be largely unchanged when running
OIIO in a facility that has a proper OCIO config pointed to by the
`$OCIO` env variable and using color space names contained therein. So
studio users will (I hope) see no changes. For users that have no
OCIO config, or that are writing software that they want to work
reasonably well for common cases no matter what config they encounter
at runtime, this should give more helpful and consistent behavior when
OCIO 2.2+ is used (which I now strongly recommend for anybody not
constrained to 2.1 for compatibility reasons.
But because of the risk and the possible behavioral changes, this is
all currently envisioned as OIIO 2.5 features, and I am not planning
to backport it to the 2.4 release branch. Additionally, a lot has
changed and I expect some additional tweaking and identification of
edge cases I may have missed.
* docs, etc
IOMemReader::pread catch out-of-range read positions.
Fixes#3711
Along the way, I noticed that the system pread & pwrite return a
ssize_t rather than a size_t as our IOProxy methods do, so I need to
catch the negative value returns that they use to indicate errors.
Remove the xxhash stuff from hashes.cpp -- it long ago moved to
xxhash.cpp. Also, comment out the bjhash implementations that OIIO
itself doesn't use or expose publicly, since there's no point
compiling it (or having it count against us in coverage analysis).
Seems like a reasonable assumption for BMP files. It's not strictly
correct -- there are header fields on V4 and V5 BMP files that
communicate color primaries and gamma, but honestly, I do not have any
idea if anybody uses them or whether they're more misleading than
helpful. For now, assuming that a BMP file is meant for output on
an sRGB display seems like it's likely to be true.
* More extensive testing of texture3d: handle vs filename, subimages,
>4 channels, closest sample interpolation.
* Fixes for tex3d > 4 chan that turned out to be broken.
* Some rearrangement for the >4 channel case
* Refactor and simplify accum3d_sample_bilinear so multiple data types
all use the same templated code path.
* This all reduces the amount of untested code in OIIO by another 0.4%.
* Build: fmt external version flexibility
The current default behavior is to
1. Check if there's already a copy of fmt found.
2. If not (or if BUILD_FMT_FORCE is ON), download it locally.
3. Copy the relevant fmt .h files as part of the OIIO install in
OpenImageIO_ROOT/include/OpenImageIO/detail/fmt
This patch adds a new INTERNALIZE_FMT option, which defaults to ON,
whch is the old behavior. But if set to OFF at build time, the fmt
headers will NOT be copied into OIIO's install area (instead, it will
have dummy files that in turn include the real ones).
Set INTERNALIZE_FMT to OFF if you are really sure you want users of
OIIO to use the system install of fmt and not have OIIO install a new
copy.
When an externalized fmt is used, also make sure our exported cmake
config files cause the downstream user to find fmt and pass on the
dependency properly (necessary because strutil.h actually requires
properly including fmt headers).
Eliminate extraneous include_directories of fmt
Bump the auto-download version of fmtlib to 9.1
Many ImageOutput::open implementations start out with various validity
checks -- such as whether the ImageSpec requests a resolution, number
of channels, or other features simply not allowed by that file format.
This is not only repetitive, but which checks are done by each file
type is somewhat haphazard, error-prone, hard to get full test
coverage (must we separately test each possible failure case for every
file format?), and the error messages are not always consistent across
formats. Frankly, most of the format writers assume that the caller
has a full understanding of supports() and will never send requests
for features that are unsupported.
This PR proposes adding a utility method ImageOutput::check_open(),
which performs all validity checks on the ImageSpec that can be
deduced from supports(). The various ImageOutput::open()
implementations may call this function and avoid having to replicate
(or forget) all the checks done in other output formats. This is a
protected method that is used by ImageOutput implementations
internally, but has no use for client-side code.
I've implemented check_open, and also used it here for all the output
formats. If this seems ok to everyone, I also plan to do an analogous
helper for ImageInput common validity checks (which will be an even
bigger reduction of redundant code and greatly increase the safety
against corrupted or nonsensical input files).
Along the way I also instituted a new supports() query: "noimage".
That is to indicate a file for which it is permitted to have a pixel
data window consisting of 0 pixels (i.e. an "image" that is only a
container for metadata). Currently, FITS is the only format we support
that allows this, though it's something we occasionally talk about for
OpenEXR. Allowing an explicit way for formats to say they allow no
pixels is what allows check_open to issue error messages for all the
other formats if it is passed zero-sized width, height, depth, or
nchannels.
Would you believe we never had a testsuite test for environment
mapping? Well, it wasn't totally untested, in the sense that OSL
testsuite had one, and obviously it's used in production all the
time. But OIIO's testsuite didn't have any tests of
TextureSystem::environmemt(). We now remedy that.
* Add testsuite/texture-env, contains several environment tests,
including handles vs filenames, subimage selection, different interp
and mip modes, ask for >4 channels.
* New CMake build option OIIO_TEX_IMPLEMENT_VARYINGREF can be used for
testing to disable implementation of the old 1.x style of batch
texture calls. We don't use them, they are deprecated. This option
give us a way to exclude them from code analysis, because we really
don't care.
* Fix bug with >4 channels for environment batch shading!
* Avoid unnecessary spec lookup in TileID constructor.
* Avoid building testtex test_hash during code coverage -- it is an
extremely specialized test, not run in the testsuite, it shouldn't
really count againt good code coverage.
When creating a texture, the old logic was to set the file metadata
"DateTime" to the current time, unless update mode was used, in which
case it was set to the file's write timestamp of the input file.
But that made it impossible to explicitly set that metadata, and also
a big PITA for testing because it meant the metadata of any written
texture would be the time of its writing, and so appear different
every time.
This change alters the logic: update mode still sets DateTime to the
last update time of the input file, as before; other cases will only
set DateTime if it isn't already set, but not alter it if the input
carries that metadata already.
* There was lots of duplicated code between iinfo and `oiiotool -info`.
Refactor much of that into a new src/libOpenImageIO/printinfo.cpp.
These are internal functions, no change to public API.
* Test iinfo --hash, --stats, and with nonzero origin and full size
* Test iinfo and oiiotool --dumpdata of deep image
* Test iinfo for subimages, mips, -f, -s
* Some conversion to new print instead of printf
There is probably more such refactoring to do later, I did the easy
parts for now and that's a start.
When reading PSD file, if reading the thumbnail doesn't pass basic
sanity checks, we consider that a hard error reading the file. One
check that we do is to make sure that our computation of the byte
width of the uncompressed thumbnail (`bpp/8 * width`) matches what the
file says the `widthbytes` is.
BUT... it seems that it might round up to an even (or maybe 4-multiple?)
of columns, presumably for scanline alignment reasons. And so we were
considering whole PDF files unreadable if they fell into this case.
This PR primarily loosens the test to allow some padding up to a
multiple of 4.
Fixes#3658
* Maya IFF documentation says IFF file has max resolution of 8192, and
our implementation only allows writing RGB and RGBA, so check these
limits when an IFF file is opened for output.
* Make sure scratch space allocates enough for tile padding.
* Check for non-zero image origin coordinates (which are not allowed
in IFF files).
* Change some 16-bit loop variables to uint32_t to avoid possible
overflow.
Fixes TALOS-2022-1654, TALOS-2022-1655, TALOS-2022-1656
In open(), check for non-zero image offsets, which is not supported by
BMP, and issue an error if found. If the spec passed in has nonzero
spec.y, other code in the write functions would fail because they
assumed spec.y == 0 and could therefore access memory incorrectly.
More generally, also put checks in the write functions that issue
errors and take early outs if they find that they are called on a
BMPOutput that is not currently open, for example if open() fails but
the user doesn't check the return codes and proceeds to call the write
functions anyway.
This addresses TALOS-2022-1653 / CVE-2022-43594-CVE-2022-43595
There was a subtle error where we had a reference to an ImageSpec for
one of the subimages, confusingly called m_spec, which we altered, but
then didn't make the same change to the this->m_spec, and in a later
spot we referenced the `m_spec` meaning the one in `this`. This could
lead to a crash.
Fix, but not only that, also rename to avoid the confusion. `spec0` is the
alias we us for the spec for subimage 0, and `spec_s` is the alias we
use when referencing subimage S, and we really never use m_spec
anymore to avoid any confusion. This should make it easier to reason
about this code in the future and avoid these kinds of mistakes.
A second error resulted from the situation where a failure in open()
would return before one or both of m_buf or m_tilebuf were allocated.
If the return status of open() were ignored and the caller
subsequently attempted calls to write_scanline or write_tile, invalid
memory could be accessed. Fix this with a variety of guards not only
based on the allocation status of these buffers, but also issuing
errors (and exiting early) from write_scanline or write_tile if they
are called on an ImageInput that is not in fact open.
Addresses TALOS-2022-1651 / CVE-2022-43592 and
TALOS-2022-1652 / CVE-2022-43593
Fixes#3660
In investigating this issue, I also noticed that now that IOProxy
support can be relied on, the old code that in the psd reader that
directly fumbles with libjpeg is not needed and can be replaced by
just a couple of lines to make an ImageBuf read from an IOProxy.
And it fixes the bug, too. So I'm not even going to try to figure
out what was going wrong with the old code.
When writing a zfile, anything that causes open() to fail will leave
m_tilebuffer at 0 size. Subsequent calls to write_tiles, and also
to close (which cleans up unwritten parts of the buffer at the end)
did not check for this condition and would access out of bounds memory.
While I was in there, I converted the zfile.cpp error messages to the
new formatting convention.
Fixes TALOS-2022-1657
I uncovered a very subtle problem that can happen with the interaction
between ImageBuf, using an IOProxy, and the underlying ImageCache.
Remember that when using an IOProxy, it is critical that the lifetime
of the proxy outlasts any ImageInput, ImageBuf, or ImageCache that was
told about it.
Consider the following code:
IOProxy proxy(...); // temporary proxy in this local scope
ImageBuf A ("foo.exr", 0, 0, proxy);
We've set up an ImageBuf that will read from an IOProxy. Behind the
scenes, it made an ImageCache entry that remembers the IOProxy, as it
has to, since it needs to use that proxy for subsequent reads.
Now we read the image, and we use the `force=true` parameter to
force an immediate read of the file:
A.read (0, 0, true);
It reads from the proxy. But the IB and the underlying entry in the IC
still know about the proxy. But in the user's mind, they have fully
read the image, will never read from it again, so of course (they
think) it is safe to close or destroy the proxy, knowing that they
made it last past its use:
proxy.close();
But -- oops! -- the cache entry still has a pointer to the proxy, and
even though it won't need to read from it again, it still will access
it, close it, etc., but the user's proxy may by that point be out of
scope.
This is *technically* not a "bug" in the sense that it is the user who
violated the contract by closing or destroying the proxy before its
last use by anything they passed it to. But in context, it's very easy
to be confused by this -- the user thinks that the only observer of
the proxy is the ImageBuf, and they're done using the ImageBuf at that
point, but because they don't explicitly mention an ImageCache, they
don't realize that the ImageBuf passed the proxy reference along to
the underlying IC that could be backing any ImageBuf.
It's the user's fault in some sense, but let's just say that we have
given them an "attractive nuisance."
That's a long bit of background explanation, but I wanted to force
myself to explain it clearly and make sure it was spelled out in the
commit comments. Now here is the fix:
* When IB::read does an immediate forced read involving an IOProxy,
after the read is complete, tell the cache to invalidate the entry
for the file so that it no longer retains a pointer to the proxy.
* When an IB itself is destroyed or cleared, if it used an IOProxy,
also invalidate the IC entry for that file.
* Unrelated cleanup: avoid a redundant invalidate call in init_spec().
* Make it so that when we build against OCIO 2.2, it is possible to
specify the built-in configs. The logic had previously precluded
this, because at one point it checks to see if the passed config or
contents of env $OCIO was an existing file. Now you can set OCIO to
"ocio://default", for example, and it will be used by OIIO.
* Expose getAliases (I think that has been there for a while but we
didn't expose in in our ColorConfig wrapper).
* Expose isColorSpaceLinear (new to OCIO 2.2)
* New OIIO global getattribute query: "opencolorio_version" returns the
human-readable version string (such as "2.2.0").
* Some miscellaneous other cleanup in color_ocio.cpp
* Change in color methodology for test suite: Previously, for every
call to oiiotool in the testsuite, we passed a `--colorconfig`
argument to oiiotool pointing it to the sample colorconfig that we
ship in the testsuite. Here, I'm changing it so that the cmake setup
for tests sets the OCIO env variable for all tests, so no explicit
--colorconfig is necessary, and thus you won't see a path to the
config in any of the test output files if they need to echo the
command line arguments (and you'll see several test outputs altered
in this way). This is a prelude to later changes that will use the
built-in configs themselves for tests, when available.
More interesting changes coming in subsequent PRs.
Lately, it has been experiencing intermittent availability problems
that causes our CI to fail while setting up dependencies.
What happens if we don't use the problematic ubuntu-toolchain-r at all?
Nothing, it seems. So eliminate it.
I think that once upon a time, it was necessary to add this repo to
get certain new packages? But maybe they are mainstream enough now
that it's not necessary to add this repo? In any case, it doesn't seem
to hurt anything, it's one less moving part that can't fail if it's
not there.
We use this all the time in our internals, and I'm tired of having to
write Strutil::print. Too verbose. By putting it in the main OIIO
namespace, OIIO internals can reference as simply print(), and
external users now have a choice of saying Strutil::print() or
OIIO::print().
Kai Pastor (dg0yt) made an important observation that the way I
recently made OpenImageIO_VERSION be a cache variable might lead to
unexpeted behavior by inadvertently retaining a version number upon
recompiles if the cache itself was not cleared, and this can happen
unexpectedly even when the user isn't trying to override the version.
This patch changs the behavior as follows:
You must use the special `-DOpenImageIO_VERSION_OVERRIDE=...` to
override the version.
* if you do so, you are playing with fire and get what you deserve
if you misuse it or change branches and don't remember to clear
the cache.
If you do a normal build (without _trying_ to override the version),
* you will get the real version number because it IS NOT cached
* if you switch branches, you'll always get the right version number
* ...but switching branches without clearing the cache can still, as
always, have other subtle behavior differences versus doing a
truly fresh build from scratch.
So in short, it still allows a way to override the version if you
really want to, but if you are not trying to do that, you can no
longer get yourself inadvertently into unexpected trouble.
* testtex: better control over temp file creation
* Additional texture tests to improve coverage of udim generation,
and deadling with cache limits.
* Better ImageCache unit tests for perthread create/destroy and
individual tile retrieval. Fix a bug in get_tile for nonexistant
subimage or miplevel.
* Dead code removal: ImageCacheFile::udim_to_concrete() and
find_file_no_add(), which were no longer called anywhere.
* IC stats: fix messed up 'slowest' list ordering when there are broken files
* IC stats: when stats level is high, enable printing of some stats that
we previously skipped over when they were "uninteresting" (e.g., too
little time spent on something, no problematic files, etc).
* IC testing: test a variety of get_pixels error conditions
* IC testing: more tests of IC::imagespec() and get_imagespec(), incluing
error conditions.
* More get_texture_info query testing
* Minor bug fix for int-to-float data conversion in get_image_info
* Test IC query for resolution[3]
* Test IC::reset_stats()
* Fix ImageCache::reset_stats in cases where threads ended
* Add casual tests of oiiotool with no args (print brief usage message)
and when an unknown command.
* Also changed the meaning of -q (quiet mode) to be more quiet -- in
particular, not to print the full help message for errors when quiet
mode is used, only print the error.
* oiiotool test sequence errors.
* Test (and fix) --selectmip and --subimage with out of range indices
Add IOProxy support.
Fixes#3623
Along the way, noticed an important bug at line 209 of this change --
the header "imagename" field should be fully zeroed out even if there
is no ImageDescription metadata!
Changes:
* Scorecards and analysis -- allow newly launched job to cancel
running old job from the same PR.
* Scorecard only runs on main repo, no need to run on forks.
* Nightly CI only runs on main repo, no need to run on forks.
* Test oiiotool control flow syntax mistakes
* Test oiiotool --set with array of strings
* oiiotool: better adjust_geometry unit testing
Helper utility additions:
* oiiotool --no-error-exit
`--no-error-exit` causes an error to not exit immediately, but
rather to try its best to continue execuing command line
operations. Use with caution! This is mainly for unit tests and
debugging.
`--unittest` also turns this on automatically, so that tests that
are probing failures don't cause an exit.
* unittest.h: If OIIO_UNIT_TEST_QUIET_SUCCESS is defined, don't print
anything extra for unit test success
It surprised me a little to realize that --printinfo prints the
current spec of the top image (I mean, duh), but that may not match
what you'd get with `oiiotool -info -v`, which prints the data about
each file as it's read from disk, inasmuch as the "current" spec
reflects the in-memory representation. So, for example, it might say
the image is float, because oiiotool tends to store everything
in-memory as float, rather than uint8 of the disk file.
So these changes help fix this oddity and other ambiguities I realized
follow:
* Give --printinfo an optional modifier `:native=1` that will print
the native (from the original file) info, if that's what you need,
rather than the up-to-the-moment in-memory data.
* Also add --printinfo optional modifiers `:verbose=` and `:stats=`.
* Expression substitution gets some new names it knows: "nativeformat"
is the data type in the file, versus "format" which I now see would
return the in-memory version; METANATIVE and METANDATIVEBRIEF, which
are the native versions of META and METABRIEF.
Make extra sure to clear the oiio:ConstantColor, oiio:AverageColor,
and oio:SHA-1 metadata when writing textures, if those things are not
going to be set.
* Add OIIO::getattribute() queries for "font_list", "font_file_list",
and "font_dir_list", which are, respectively, the lists of fonts
that were found, the list of font files (full paths), and the list
of directories it searched. All return a single string that is a
semicolon-separated list of the items.
* As a consequence of the new attributes, if you are curious to see
the full inventory of fonts it can find, you can just query the
attribute:
oiiotool --echo "{getattribute(font_list)}"
* Some refactoring of the font inventory process (in
imagebufalgo_draw.cpp) was necessary to achieve this.
* We now search for ".ttc" (TrueType collection) files, which we didn't
notice before.
* We now search explicitly in share/fonts/OpenImageIO, which we
neglected to do before, but found those fonts because we searched
/share/fonts recursively. In exchange, we now do not search the
directories recursively, because that seems unwise for performance
reasons.
* We pass the full testsuite, but it's possible that this
change may fail to find fonts that were previously found because of
the recusion. If that is the case, you should fix it by explicitly
adding the deep directories with fonts to the search path specified
in the environment variable `$OPENIMAGEIO_FONTS`.
* Unit test adjust_geometry
* Tests of --set for various implicit and explicit types not
previously covered in the testsuite
* Comment out Oiiotool:get_position, which seems to be unused
* Exercise --iconfig for explicit types
* Add test for --origin and --originoffset
* Add test for --ociodisplay
* Add test for --unmip and --selectmip
It's similar to trim_whitespace, but returns the trimmed result, rather
than operating in-place on a `string_view&`.
Also add unit testing of trim_whitespace and trimmed_whitespace
* Add an optional size parameter to read_text_file to limit the
maximum size that will be allocated and read (default to a limit of
16MB).
* New read_text_from_command, similar use to read_text_file, but reads
from the console output of a shell command.
Along the way, improve the runtest.py script by giving the
info_command() fuction the same silent, concat, and failureok
parameters that the oiiotool() function has. (This lets individual
tests assemble iinfo command lines with a little more flexibility.)
* In the case of DDS cubemap files where a cube face was not present,
we were filling m_buf with 0 bytes before allocating it.
* Add check for invalid bits per pixel.
* Add regression test for another bug that was perviously fixed in
2.4.3.
In one case, we actually had a check for this, but an assignment to an
int made the nonsensical offset appear negative, and we only tested
whether the necessary offset it was bigger than the buffer size.
Keeping it as (unsigned) size_t makes the test work as intended.
In another case, there were several places where we never checked that
we were staying within the exif block, and here we address this by
changing the utility decode_ifd so instead of passing it a pointer to
the ifd, it passes the offset (the pointer turned out to always be
inside the buffer) so it can check the extent for subsequent accesses.
Also some fixes related to squashing undefined behavior sanitizer
cases.
Also, change `operator<<` to output the string, not the hash itself.
These changes help push us more toward the original intent of
ustringhash being a drop-in replacement for ustring, any time it's
more desirable to be using around the hashes rather than the pointers.
* oiiotool tests for expressions var() and getattribute()
* Make sure we test oiiotool IMG[] expressions and several specific
special attributes to retrieve.
* Add coverage of oiiotool --keyword and --clear-keywords
When reading the image (which can have signals to jump around in the
array), we checked for out-of-bounds x coordinate, but not y, and some
corrupted files could overrun buffers as a result.
The existing syntax `TOP.name`, if the name wasn't found, produced
an error.
This PR adds another syntax: `TOP[name]` which is identical if the
named metadata is found. But if not found, this syntax just evaluates
to the empty string, with no error.
This test was taking almost 30 seconds to run, without a lot of images
being read or written. Turns out it was just a large number of separte
oiiotool invocations, and the overhead of all those shells and
programs launched adds up. Most of them were just firing up oiiotool
for the sake of doing an `--echo` to print things. A little bit of
restructuring to make fewer oiiotool invocations (each one doing a
bunch of echo commands, each testing a different thing) cuts the time
for the whole test by 50%.
I also made the runtest.py script allow certain test debugging by
letting a test set the `wrapper_cmd`, and if so, will prepend every
oiio utility launch with it. For example, a test could say
`wrapper_cmd='time'` to time every command.
Add a hidden --unittest command to oiiotool, only functional for Debug
builds, which enables additional testing of internal utility funcions
and other things that are difficult to fully exercise in the testsuite
from normal oiiotool commands (could be done, but would take tons of
additional tests to get into particular deep edge cases).
Prototype by adding unit tests for scan_box. More will be added later
now that the mechanism exists.
Simplify string_to_dataformat
Make sure to use --list-formats and --runstats during testing.
ustring::hash() and the representation of ustringhash were size_t,
which might be 32 bits on some platforms. But we really would prefer
ustrings hash to 64 bits, to greatly reduce the chance of hash
collisions.
* Add strhash64() which is like strhash() but guaranteed to be a 64
bit value (returning uint64_t instead of size_t -- though note that
these are the same on 64 bit CPUs).
* ustring::hash() and ustringhash::rep_t are now uint64_t.
* Note that std::hash(ustring) and std::hash(ustringhash) are still
size_t, per the C++ definition of std::hash. So ustring::hash() may
have more bits than std::hash(ustring) on 32 bit platforms.
This will NOT be backported to the release branch, since it changes
ABI on 32 bit platforms.
* We mis-named color_test, and that caused it to never get run!
* imagespeed_test is meant for benchmarking only, run by hand.
We don't make a testsuite test, so don't even build the source
(which would never be counted for code coverage because it doesn't
run) during CI.
* Add a test for cineon -- we never directly tested it before
* Don't compile code used only for writing. We don't supporting
writing cineon, so there is no point compiling the cineon module
used only for writing.
* And of course, once cineon was in the testsuite, we immediately
found a problem via the sanitizers! So fix buffer overflows when
reading FilmEdgeCode in Cineon. (And clean up the equally sloppy,
though not technically buggy, code in DPX as well.)
On any one platform, our testsuite was only testing the old C++ and
the new C core exr implementations.
Rig it to run the openexr-oriented tests twice -- once in each state
(when using OpenEXR new enough to support the core library).
Doing this cleanly required adding SUFFIX and ENVIRONMENT optional
parameters to the oiio_add_tests utility macro. Once that was added,
the texture batch testing could be implemented that way as well,
removing a bunch of hackier special purpose code.
* Protect against 0-sized allocation
* Exception safety in IffOutput destructor
* Using sonar.coverage.exclusions as a better way to exclude files
from coverage analysis
* Null pointer safety
I realized that the recently added ustringhash class is confusingly
similar to the (obscure, barely used, mostly forgotten) ustringHash
functor. But also ustringHash doesn't need to exist, because it's
equivalent to `std::hash<ustring>`, if we define it. So now we do define
it, and also `std::hash<ustringhash>`, and deprecate ustringHash, to
be removed entirely for 3.0.
Also, add a ustring ctr from ustringhash (explicit only for now),
and `operator<` for ustringhash.
And add from_hash(size_t) method to ustringhash, analogous to
ustring::from_hash().
Modernize a few places we still used `(void)` to just `()`.
Give ustring and ustringhash a `using rep_t =`
rep_t is the type of the underlying representation (const char* for
ustring, size_t for ustringhash).
Modernize some typedef into using.
Make `==` and `!=(ustringhash, const char*)` be constexpr for C++ >= 17
Including windows.h leads to a lot of possibly common names being
"taken", mostly by preprocessor defines. Some of these are possible
to turn off (like "max" and "min" by defining "NOMINMAX" before
including <windows.h>). But many others are not possible to turn off,
e.g. "near" and "far" just get defined to nothing, preventing from using
variables named like that.
Additionally, includes have a cost, and while <windows.h>, being a C
header does not have expensive language constructs that would slow down
compilation a lot, it's still something close to 100k lines of
preprocessed code to wade through.
Now, only the OIIO implementation .cpp files that actually need
<windows.h> include it (all of them under "libutil").
There were two exceptions:
1) thread.h pause() implementation uses YieldProcessor() macro when on
a Windows system. Replaced it with manually "re-implemented"
things that YieldProcessor expands to. Right now made the
implementation for all 4 platforms that Windows normally supports
(x64, x86, arm, arm64); the latter two OIIO does not support though
when on Windows so it's untested.
2) timer.h Timer::now() inline function used QueryPerformanceCounter
and LARGE_INTEGER. These are quite hard to somehow "forward-declare"
or similar, without resulting in duplicate type definitions. So on
Windows I moved the now() function to be non-inline.
In particular, defining NOGDI before including <windows.h> achieves
several things amongh others:
- No longer causes ERROR symbol to be defined to 0. ERROR being defined
prevents using e.g. google/glog library, since it wants ERROR to mean
something else.
- No longer defines some possibly widely used names e.g. GetObject.
16 bit per pixel (e.g. 555, 565, 444) formats probably have never
worked. One part of the code was assuming 5 bits in the mask, another
was shifting by 4 bits, and the 16-bit pixel fetch code was reading just
one byte.
The code assumes "char" is "signed char" (as on x86), but on ARM "char" is
"unsigned char".
Co-authored-by: Joachim Reichel <joachim.reichel@posteo.de>
In some cases (at least on Windows, with static linking) the oiiotool
main returns zero, but the full process returns 3 error code due to
abort() being called. And that is called because a global shared
image cache is already destroyed, but the image references from another
global (Oiiotool itself) try to use it, get a call into a "pure virtual
function" stub that is put by MSVC C runtime into a destroyed object
vtable, and that ends up in an abort().
Work around that by making sure various member variables holding
ImageRecRef are cleared before main() finishes and global destructors
are called.
Most of the cost in HDR file reading was converting from RGBE to floats,
inside ldexpf(), which at least on Windows/VS2022 spends most of the
time inside _ctrlfp(). But we only have 256 possible exponents in total,
so just use a precomputed table.
On my PC (Windows 10, VS2022, Ryzen 5950X) this change gets file read
time for an 8x resolution .HDR image from 1.10s down to 0.27s
HDR code overhaul to support IOProxy (#3218) changed all reads into
a `pread()` with explicit file position tracking. Turns out, that is
quite a larger overhead compared to simple sequential reads; even more
so on Windows.
On my PC (Windows 10, VS2022, Ryzen 5950X) this change gets file read
time for an 8x resolution .HDR image (with RLE compression) from
8.28s back to 1.10s just like it was in OIIO 2.3.
On Windows the extra cost when using pread() is a bit more extra mutex
locks, but the real cost is in enventual ReadFile; looks like any seek
operations make it take some sort of "way slower" path with various
callbacks and whatnot inside the kernel.
---
Note repoduced from discussion #3588 from LG:
Switching from IOProxy::pread to using IOProxy::seek + IOProxy::read makes the IOProxy stateful and not thread-safe without an external lock (which this HdrInput has, so it's not going to break anything), precluding any future removal of the locks from read_native_scanline to make the whole ImageInput able to read concurrently. That's probably not important for this file format, but for others (especially crucial those usable as texture), it would be desirable to rely on those stateless pread calls and remove the lock from the ImageInput.
I think this patch is fine. We don't expect a lot of concurrent reads from hdr like we do for tiff and openexr that we use extensively for highly threaded on-demand reading of texture. It fixes the recent performance regression, and it doesn't really cost us concurrency because we already had the lock in HdrInput. But in general, we want to prefer eschewing read/seek and using pread instead, so we can expect maximum concurrency from multiple threads using an ImageInput (as we do in ImageCache/TextureSystem).
So I believe that a revised and more complete diagnosis is that the problem isn't the pread vs read per se, but that for RLE-compressed images we were calling pread separately for every 4 bytes! The calls to read are faster simply because they are buffered (in FILE) and require fewer OS system calls. But the real sin is that we call it way too many times.
I think that the rle case could go back to pread if we want, but do just one pread per scanline (rather than one per pixel value run), asking for the maximum amount that will be needed to read the scanline for the worst case rle representation (it's ok if it's the end of the file, pread will just read up to the end and return the true number of bytes read). In other words, we'd be doing the buffering ourselves.
But anyway, the real misdesign of this HDR reader is ancient, and consists of doing many many tiny reads. Even the buffered fread would be sped up a lot by not doing it separately for every 2-4 bytes of the file. I don't know if it's worth fixing for this format. Depends on whether anybody really needs to maximize its performance or expects multiple threads reading the same hdr file to be fully concurrent.
Something changed in libtiff master and it doesn't build shared libs
by default, breaking our "bleeding edge" CI test. So when we build it,
be sure to ask for it. It can be overridden in build_libtiff by
invoking it with env variable `LIBTIFF_BUILD_SHARED_LIBS=OFF`.
Comparing the time it takes to load DXT/BCn compressed DDS images via
OIIO, I noticed that it's several times slower than in another codebase
(Blender). So I started profiling, and fixing some low hanging fruit.
- After decoding a BCn block (always 4x4 pixels), it was written into
destination image using quite many branches in the innermost loop.
Now it still takes care of not writing outside of a destination image
that might not be multiple-of-4 in size, just with way fewer branches.
- BCn decompression is multi-threaded via parallel_for_chunked, using
at least 8 block rows per job (i.e. 32 pixel rows). This is
similar to internal multi-threading done by EXR or TIF readers.
- Avoid one case of std::vector resize (which clears memory) followed
by immediate overwriting of that buffer with bytes read from the file.
Switched to using unique_ptr for that array, similar to how other
input plugins do it.
Performance data: I'm timing how long it takes to do "iinfo --hash" on
24 DDS files, each 4096x4096 in size. Quite a lot of time there is taken
by just the SHA hash calculation which is not related to loading, so
also have timing info for just DDSInput::read_native_scanline where
all the actual DDS loading/decoding happens.
Timings on PC (Windows 10, VS2022 Release build), AMD Ryzen 5950X.
- BC1: total 3.68s -> 2.76s, read_native_scanline 1.54s -> 0.59s
- BC3: total 3.74s -> 2.84s, read_native_scanline 1.83s -> 0.66s
- BC4: total 1.26s -> 0.82s, read_native_scanline 0.74s -> 0.24s
- BC5: total 2.29s -> 1.57s, read_native_scanline 1.31s -> 0.46s
- BC6: total 7.50s -> 4.30s, read_native_scanline 4.67s -> 1.60s
- BC7: total 6.18s -> 3.09s, read_native_scanline 4.34s -> 0.84s
Possible future optimization: implement DDSInput::read_native_scanlines,
that could avoid one extra temporary memory buffer copy. This extra
buffer clear & eventual copy from it into final user destination pixels
is non-trivial cost from profiling. But it's also quite involved
to handle all the possible edge cases (e.g. user requests scanlines
not on 4-pixel row boundaries); the code would have to do similar
juggling as EXR reader does to handle "not on chunk boundaries" reads.
It could potentially also interplay with DDS cubemap/volume texture
reading ("tiled") code paths, for which there is no test coverage at all
right now. Maybe for some other day.
Suggested by SideFX: explicitly shut down the OpenEXR thread pool
before exiting oiiotool, because sometimes on Windows it seems to hang
otherwise. A better solution is probably possible on the OpenEXR side,
but I don't know what it would be. (And in any case, we'd still need
to do this be be well behaved when using OpenEXR versions earlier than
where a fix might land.)
Solution tracked down by Edward Lam.
* DDS: fixes for A8, L8, A8L8 formats
While developing #3573 at one time I had them working properly, but then
while fixing the address sanitizer failure for 3-channel formats I
botched them again. Note to self: never again do a "yeah I'll add tests
later", since if I had all of them running all the time this would
not have happened. 🤦
* DDS: extend test coverage for more formats & channel size cases
With more test images recently added (https://github.com/OpenImageIO/oiio/pull/3459),
start using them for DDS tests. This covers now:
- Alpha, Luminance and Alpha+Luminance formats,
- RGB formats with rarer channel sizes (rgb10 a2, r3g3b2),
- Several "broken" DDS file cases
- Removed usage of sample-DXT1 since it is well covered by others.
Add a new build-time cmake cache variable: FORTIFY_SOURCE
Default to empty, but can be set to 1, 2, or 3, corresponding to
setting the _FORTIFY_SOURCE macro available in recent versions of
clang and gcc. (I'm not sure exactly which minimum compiler version is
needed for each fortification level, except that for level 3, gcc must
be 12+.)
Fortification involves replacing several "unsafe" memory-related
functions such as memcpy, memset, strcpy, etc, with special versions
that do bounds checking, aided by some compiler smarts for
understanding the likely bounds of different buffers. If I understand
correctly, at level 2 it can figure out bounds of constant-sized
arrays, and at level 3 it can figure out certain dynamic cases as
well.
There are two use cases for this:
1. For our own CI, this is yet another bit of static and dynamic
anslysis to enable (currently, just in the gcc12 test) to possibly
catch bugs.
2. Users who are building OIIO to deploy it in security-sensitive
environment may wish to build with some fortification level enabled to
help prevent certain memory errors or security issues, understanding
that it may slightly impact performance.
- If a DDS pixel format header contains "this is a normal map" flag
set, then handle that properly (#3525):
- For a DXT5, this means it's a "DXT5nm" format. Normal X & Y
components are in green and alpha channel respectively. Compute Z
component from those, and the result is a 3-channel tangent space
normal map.
- For ATI2/BC5 format that also has a "normal map" flag set, compute
the Z component automatically too, and result is a 3-channel image.
However a bunch of ATI2/BC5 normal maps out there in the wild do
not have that bit set, so also have a global "dds:bc5normal"
attribute to optionally turn this on.
- Fix support of R10G10B10A2 format - was producing garbage data due to
code assumption that truncation down to 8 bits never needs to happen.
- Fix bit expansion ("too dark") of very low-bit formats, e.g. R3G3B2.
Code was simply shifting the bits, instead of properly repeating
the high bits into new low bit positions (i.e. a 3-bit "111" needs
to become an 8-bit "11111111" and not a "11100000").
- Support "RXGB" (DXT5, with red & alpha swapped) as well as "BC4U"
and "BC5U" formats (the latter two are just BC4 and BC5, produced by
some older tools).
- Support alpha-only (e.g. A8) formats.
All this mostly prompted by doing DDS improvements in Blender, and
wanting to get Cycles renderer (which uses OIIO) to the same feature
parity. See https://developer.blender.org/T101405 and
https://developer.blender.org/D16087.
* Refactor the clang-format job to be part of the aswf batch of jobs
rather than separate (reduces redundant logic in the workflow file).
* Give proper names to distinguish the two Mac jobs' artifacts for
broken builds and their caches.
* Revise scorecard workflow triggers: no need to run weekly, and run
on push only to the main repo, no reason to run on forks.
Cmake-generated visual studio project files were using just one CPU core for all the compilation, which is not terribly fast. Add the `/MP` option so that it compiles separate .cpp files in parallel. On my (16 core, 32 thread) machine this makes a Release build of OpenImageIO.lib go from 136 sec down to 30 sec.
Also remove three raw test reference output files that we apparently
no longer use (corresponding to platform + libraw combinations that
aren't in our current set of CI configurations).
This was an experiment that was added to master at one point but
removed before we branched for a release. But we forgot to remove a
few headers. Removing them now is not a compatibility break, since we
never shipped a supported release that contained the library that
implemented the functions declared in these headers.
In addition to address and leak sanitizers, also turn on the undefined
behavior sanitizer in our CI run. We can do it in the same job as
asan/lsan.
Some things that come along for the ride, including fixes for things
that ubsan uncovered:
* src/build-scripts/ubsan-suppressions.txt can hold suppression list
(though we don't have any now, it just sets up the space for it for
future needs).
* Split the "targa" test so that the parts that directly need the OIIO
python module (which are all related to thumbnails) are now in
"targa-thumbnails" test. That has to be disabled for the sanitizers,
since we can't arrange for python to b linked to the sanitizer
libraries.
* Found vint4::store has a potentially unaligned integer store.
* Found Strutil::concat is not necessarily safe if either of the
strings are empty.
This was part of a set of obsolete calls where we recently removed the
implementations, but for link compatibility, made matching shim calls
and wrap the modern versions. But I missed one case.
Adapted from the similar document I wrote for OSL.
This serves both as a checklist for me to make sure I don't miss any
steps during a release, but also so that other people could take over
release procedures if necessary or desired.
I also rearranged the order of some lines in CMakeLists.txt for
clarity (but did not change the logic).
ImageSpec and ParamValueList have had (since 2.1) a getattributetype()
method, which simply returns the TypeDesc of the named attribute, or
TypeUnknown if no such attribute exits. This goes along well with the
related getattribute() and attribute() family of methods.
The getattribute() method was not present in ImageCache and
TextureSystem, despite their having very similar
getattribute()/attribute() interfaces.
A recent issue made me realize that in the Python bindings, the IC and
TS bindings for `getattribute(name)` did not work properly, always
returning None, and *required* you to use the full
`getattribute(name,type)` variety where you have to supply -- and
already know -- the type. This isn't especially Pythonic, where you
should just be able to ask by name and get a polymorphic result. And
when I went to look at how to repair this, it was going to be very
awkward without getattributetype(), but trivial with it.
So this patch sets up getattributetype() for both IC and TS, and
exposes them to both C++ and Python APIs, and also fully fixes the
getattribute() so that it works properly with the name only.
Unfortunately, because getattributename is a new virtual method, it
changes the vtable, breaking the ABI, and thus cannot be backported to
any supported release that makes ABI compatibility promises.
Currently when cross-compiling for Android build fails due to usage of
TRY_RUN(). In the particular case it tries to figure out whether atomics
can be used without the libatomic library.
Fortunately we don't need this check because on Android NDK atomics are
built-in and libatomic.so is not present. Only static dummy libatomic.a
exists for compatibility with -latomic flag.
This patch allows ParamValue, ParamValueList, and ImageSpec to support
attribute/getattribute of metadata consisting of uint8[] arrays, which
is how we store certain binary blobs or raw byte arrays. We translate
this to Python as a numpy array of type uint8 (dtype='B').
This fixes a limitation where it was not possible to directly retrieve
or set the "ICCProfile" metadata (which is stored as a uint8 array)
from the Python APIs, as you could easily do from the C++ APIs.
In the process, I found similar bits of code in several different
places, in various states of incompleteness. I refactored to make a
single make_pyobject() utility and then used that call in place of
several other redundant pieces of functionality.
TBB::tbb is added as a library dependency via TARGET_NAME_IF_EXISTS.
Therefore if an executable links OpenImageIO static library without the
executable itself depending on TBB, then TBB will not be linked in and
result in link time errors.
Just a couple missing pieces were preventing getattribute() from
retrieving attributes that were stored as int64 or uint64. That
doesn't come up very often, but one place is retrieving certain
statistics that use 64 bit counters. (This comes up in ImageCache,
among other places.)
While we're at it, there was no existing test dedicated to the python
binding of ImageCache. So to add these getattribute calls, I made one.
There is an important caveat about ImageCache.getattribute(): In this
process, I discovered that the flavor of IC.getattribute(name) where
you only pass the name is BROKEN, and will always return None. You
MUST use the form where you pass the type, like
x = ic.getattribute("foo", "int")
This is unfortunate and not very Pythonic, but it is the workaround
for now. I will addess getting the name-only form working in a
separate PR.
Closes#3537
For a while, we have found embedded ICC profiles in TIFF, JPEG,
JPEG-2000, PNG, and PSD (PhotoShop) files and store them in the
metadata as a byte array with the name "ICCProfile".
With this patch, we further add the facility of parsing the binary ICC
profile data to extract a number of other fields that we add to the
ImageSpec metadata so that "oiiotool -info -v" or "iinfo 'v"
will print this info.
Important caveat: You can read this for informational purposes, but
changing it (e.g. with spec.attribute()) does not change the contents
of the ICCProfile byte array, which is still the canonical ground
truth of the profile. You have to replace the "ICCProfile" byte array
to actually change the profile.
* Add `Strutil::utf16_to_utf8(const std::u16string& utf16str)`.
This is subtly different from our existing utf16_to_utf8() that
takes a std::wstring, because wchar_t is not the same on all
platforms. If it really and truly is utf16-in-char16_t, you need
this and can't use the utf-in-wchar_t which may be 32 bit chars.
* Add `Strutil::utf8_to_utf16wstring()` (identical to the old
utr8_to_utf16) to make it more self-documenting as returning a
utf16-encoded wstring and NOT a u16string. Note that wchar_t is not
necessarily char16_t on all platforms (it is on Windows, but it's 32
bits on Linux).
* Deprecate the ambiguously named `utf8_to_utf16()` for OIIO >= 2.5 and
remove it for OIIO >= 3.0.
This fixes builds in projects that compile OpenImageIO as static
libraries and don't themselves depend on the image libraries used by
OpenImageIO such as PNG, JPEG or TIFF.
When OpenImageIO is built as a static library its targets have e.g.
PNG::PNG among INTERFACE_LINK_LIBRARIES. If the including project does
not itself depend on PNG, the following error would be observed:
Target "<...>" links to target "PNG::PNG" but the target was not found.
Perhaps a find_package() call is missing for an IMPORTED target, or an
ALIAS target is missing?
Rather than specify the version in the usual project() statement,
project (OpenImageIO VERSION 2.5.0.0 ...)
this patch decouples it
set (OpenImageIO_VERSION "2.5.0.0" CACHE STRING "Version")
project (OpenImageIO VERSION ${OpenImageIO_VERSION} ...)
thus allowing the possibility of overriding the version number at
build confuguration time (`cmake -DOpenImageIO_VERSION=3.1.4.1`).
This should be used with extreme caution. Probably never used by most
people. I don't need to describe all the ways that things can go
sideways if you make your build of x.y.z.w actually be arbitrarily
different from the x.y.z.w that everyone else uses and that is
documented.
That said, there are two scenarios I have in mind for which this
ability is very useful:
1. Sometimes, we add or deprecate API features "in advance" by having
an `#if` enable or disable code based on the version, essentially
putting in place changes that can't be enabled yet because they
would break compatibility rules, but we want to integrate the code
now and have it become active upon reaching a particular future
version in which that incompatibility will be allowed. The version
number override makes it easy to test this by quickly doing a build
"as if" we had already bumped the version number to see what will
happen when we actually do.
2. Every once in a while we need to do an internal build that, to
conform to version compatibility rules, needs to appear to still be
part of a past version family. This lets individual sites play some
games with version number overrides rather than have to create new
branches and tags for certain one-off internal releases.
--iccread reads a named file and directly adds its contents as the
"ICCProfile" metadata of the top image (as a byte array).
--iccwrite takes the "ICCProfile" metadata of the top image and
writes it to a named file.
Oops, at some point, something about the way we set up the CI test
that runs the sanitizers, and ever since then (embarrassingly, perhaps
as much as a couple years ago), the "sanitizer" test has not in fact
been running the sanitizers. Here we fix that up, get sanitizers
running again.
* Minor png refactoring to fix some memory leaks for broken png files.
* Remove unused string variable from read_info() and simplify the
surrounding "if"
* Split png test into png and png-damaged, because I just can't seem
to squash some very minor memory leaks for broken png files, I think
it's not worth worrying about and I'll just exclude the broken case
from testing for the address sanitizer.
Reading uncompressed scanlines would copy 4 bytes from each pixel into
an int and then decode the bits, incrementing where it was reading
from by Bpp bytes each time. But when Bpp == 3, the last pixel of
every row would therefore read an extra byte past the end of the heap
storage. Allocation padding would in practice keep it from crashing,
and that meaningless extra byte was not used in the decode because it
knew it only needed the first 3 of the integer. But found by address
sanitizer!
When enforcing the max open files limit, there's a heuristic where we
only "try" to get the lock, and if we can't, rather than block, we
just defer the cleanup step until the next time around. So what if we
go slightly over the limit temporarily?
Interesting edge case: what if max_open_files is relatively low, but
the number of threads is very high and you have a high contention rate?
Then you can end up *significantly* over the limit, because almost all
the threads say "somebody else has the lock, I'll leave it to them to
clean up" and blithely open new handles.
So this patch modifies the logic so we do the defer trick only if
we're only a little over the limit. If we've exceeded the limit by
16 handles, grab the lock unconditionally and do the cleanup.
* ArgParse::add_version(str) tells ArgParse the version string, and
automatically adds an option --version that prints the version string
and exits.
* Add this to oiiotool, iinfo, igrep, iv, maketx.
* OIIO::getattribute("version") retrieves the version string.
JFIF/JPEG files only support 1 and 3 channels. For > 3 channels, we
had just been discarding the extra. For 2 channels, we had been
discarding the second, but maybe that's not always right. Try to
distinguish between files whose names appear to be YA -- in which case
output the Y as grayscale (JFIF has no way to represent alpha anyway),
and other cases which we now promote to RGB and just make B be black.
* For 2-channel DDS files, label them as Y,A if the flags indicate
luminance and/or alpha, otherwise label them as R,G.
* Do not set "oiio:BitsPerSample" for cases where the dds.fmt.bpp field
is not assumed to be valid. MS docs say it's valid only if the flags
field indicates RGB, LUMINANCE, or YUV types.
The implementation of boost::filesystem::path::extension() and the
equivalent for std::filesystem disagreed about the result of
extension(".foo"). C++17 correctly considers ".foo" to have no
extension.
This was envisioned to be useful in the very earliest days of OIIO,
but we never really finished it or used it in any helpful way.
In addition to being essentially dead and untested code, it ties us
to boost.asio, which we use nowhere else.
So just get rid of it this vestigial piece.
* Any use of `-i:ch=...` would result in a crash because of
dereferencing an empty shared pointer.
* In the case of `-i:ch=` specifying missing channels, improve the
warning message to make it clear which channels were actually present.
Extend ImageBufAlgo::compare() to take relative metrics, which if
nonzero check the symmetric mean relative error, in addition to the
current absolute error. To be formal, this means that a value
comparison will fail if
abs(A-B) > failthresh && abs(A-B)/((abs(A)+abs(B))/2) > failrelative
Setting the failthresh to 0 is equivalent to only checking relative
error, and setting failrelative to 0 is equivalent to the (previous
behavior) of checking only absolute error.
The idiff utility is extended to take new `-failrelative` and
`-warnrelative` command line arguments.
The hope is that there are various kinds of image comparisons within
tolerances -- like we frequenty use for unit tests, shader
verification, etc. -- where this metric is more helpful than absolute
error.
Set up SonarQube to run weekly (seems unnecessary to do on every PR,
it's a LONG run). Also required some build system fixes to enable code
coverage analysis.
This is handled by the "analysis.yml" workflow (into which I also
merged the scorecard analysis we were doing). This workflow runs once
weekly or on only those pushes/PRs with specially named branches.
There's no reason to run this expensive analysis on every last PR or
push.
A number of fixes come along related to things found by Sonar as I was
setting this up:
* Fix buffer warnings
* Address sonar warning: was reading wrong size into buffer
* warning reduction in dpxoutput, simplify string ops
* Clean up dubious uses of sprintf
* Add assertions
* Save a needless string construction in ParamValue::get_string_indexed
* Fix string safety warnings
* iptc: simplify encoding string data and improving safety
Stranely, this made slightly numerically different texture test
results on icc. Switching its math mode to precise brings it back in
line with the other compilers. The differences aren't visible, so this
isn't an issue in practice, but for tests it made it hard to compare
to reference images.
Hide nearly all of the implementation of the parallel_for family of
methods rather than expose them as inline. This gives us more freedom
to change the implementation in the future without breaking ABI.
Deprecate the parallel methods that take task functions whose first
parameter is the thread ID. The only reason they existed is because
our cobbled together thread_pool used that in its inner workings. But
there are good reasons why we should not expose that:
* We never used it anyway.
* It was not very useful, since sometimes it was a real thread ID, but
other times it was -1 in cases where the calling thread executed the
task directly.
* It is inconsistent with other thread pool implementations that we may
wish to try in the future.
So better just to not expose those thread IDs at all. Mark those
versions of the parallel loops as deprecated and schedule them for
removal in OIIO 3.0.
If TBB is detected at build time, support will be built in that allows
either the old OIIO built-in pool, or TBB, depending on the setting of
a new global OIIO attribute, "use_tbb" (default 0), if set to nonzero
will use the TBB thread pool.
In my experiments, the TBB thread pool seems slightly better -- I
think because there is less overhead, and maybe the clever
work-stealing it does elps to load balance. It's not used by default,
set the attribute if you want to use it (assuming the build
included. After we've had a chance to test more thoroughly, we may
change to use TBB by default. I'm interested to hear people's thoughts
on the matter.
One case where you almost certainly will want to use the TBB thread
pool is if you're using libOpenImageIO from within an application that
uses TBB for its own threding -- that way you're using one pool for
everything, rather than have two separate thread pools that don't know
about each other (and probably twice as many threads as you have
cores)..
Incorrect use of reserve(), then writing past the end of the vector.
It should have been resize(). But it can be cleaned up even further by
using assign().
If udimfile is null, it would get dereferenced anyway. Was missing
a `return` in that case.
While I was there, I realized that ImageCacheImpl::inventory_udim and
resolve_udim had no reason to be virtual, so I removed that
qualifier. (It was probably a cut/paste error, as the
identically-named methods in TextureSystem are virtual, on
purpose. But they don't need to be for this class.)
All read_native_* methods of ImageInput are required to be thread-safe
with any other read_native_* call or with the varieties of read_tiles()
that take an explicit subimage and miplevel parameter.
JpgInput::read_native_scanline() does a seek_subimage() call internally,
so it's obviously not thread safe without additional locking.
The current description of IOProxy::pread() and IOProxy::pwrite() is a
little ambiguous. This commit expands the description to explicitly tell
what the functions are and are not required to do.
Should not have passed an argument. Went undetected for a long time
because I guess we didn't previously call the erase() method. (Until
I did when working on something unrelated.)
Virtual destructors don't significantly differ from regular virtual
functions, so it is customary to add override specifier when virtual
destructor is overridden.
Current code assumes that CMAKE_INSTALL_INCLUDEDIR and
CMAKE_INSTALL_LIBDIR has exactly one filesystem component. This is not
the case on e.g. Debian, where standard paths are as follows:
CMAKE_INSTALL_INCLUDEDIR=include
CMAKE_INSTALL_LIBDIR=lib/x86_64-linux-gnu
Fortunately the Config.cmake file is installed into a path that is
itself relative to CMAKE_INSTALL_LIBDIR, so we can get
CMAKE_INSTALL_LIBDIR from CMAKE_CURRENT_LIST_DIR and then get
CMAKE_INSTALL_INCLUDEDIR by applying relative path from
CMAKE_INSTALL_LIBDIR to CMAKE_INSTALL_INCLUDEDIR on top of that.
Fixes: c8dfbd7066
The current limit has been already hit when AMD released
second-generation Epyc processors (codename Rome) back in 2019. Top
models have 64 cores per CPU, or 128 threads. This means 2 socket
configuration has 256 threads.
Fourth generation Epyc processors (codename Genoa) are rumored to have
96 cores per CPU, or 192 threads. This means 2 socket configuration has
384 threads. While the CPUs have not been announced yet, it makes sense
to prepare for them in advance.
Revealed by clang-tidy. Use length-safe versions.
Also get rid of some dead code by disabling some memory-clearing
code that we don't need (we're not using SHA-1 for cryptographic
or security reasons, just for comparing images).
Python 3.8 stopped loading DLLs from PATH on Windows, meaning anything that relied
on that environment for shared library resolution breaks. This PR works around that
behviour by manually adding PATH back to DLL resolution using os.add_dll_directory()
in a new __init__.py which just re-exports everything in the current .pyd.
The new behaviour is enabled by default, but is easily disabled at runtime if the
environment variable OIIO_LOAD_DLLS_FROM_PATH is equal to "1".
Found this coincidentally during some address sanitizer runs.
Some tiff files out in the wild that have RichTIFFIPTC tags
inexplicably mark them as if the data was LONG instead of BYTE or
NOTYPE. I guess early in our history, we came across one of those and
thought it was correct, because we were always treating it as if it
came in the form of longs. But that's not always true, and in
particular, the size in the tag I think is always expressed in
bytes. So this could make for buffer overruns (generally harmless, but
caught by the sanitizer!).
So this patch cleans this all up.
And also we update a few sample images with bad RichTIFFIPTC tags to
strip out the in the incorrect form of the tag.
Recently, we added the ability to do stochastic sampling in texture
lookups, by choosing the MIP level stochastically rather than reading
two MIP levels and mixing them.
This PR adds the ability to also stochastically sample the anisotropic
filter footprint -- instead of multiple anisotropic samples, choose
just one sample well distributed over the footprint axis. On real
scenes, this seems to speed up overall rendering (not just texture) by
up to 2-3%, for scenes where average anisotropy is around 3-4.
It's much less dramatic a speedup than the stochastic MIP selection,
which sped up texturing by around 40% and in practice sped up overall
rendering by around 15% for us. Why doesn't cutting the number of
texture samples by another factor of 3 only speed up by 3% instead of
another 20%? Basically, it's because the stochastic MIP mapping *by
definition* means that each texture call touches half the number of
tiles as before. In contrast, those multiple probes along the
anisotropic axis are almost always within the same tile, and
frequently touch a very nearby pixel within the tile, so not only are
we not needing to look up a new tile from the tile cache, but also the
very memory we need to read is highly likely to be in L1$. But still,
we'll take 2-3% rendering gain any time we can get it, that's
equivalent to a lot of money if you have a really large compute farm.
Some other changes that come along for the ride:
Add a --bluenoise option to testtex (makes for prettier 1spp results),
and add tests for the aniso sampling.
Some internals changes to precompute min w,h of mip levels so there is
less per-texture-call rifling through the ImageSpec of each MIP level.
Sadly, this did not produce any consistently detectable speedup in
overall rendering.
And I changed the interface for stochastic texturing a bit. The old
way involved asking for new MIP modes, StochasticAniso or
StochasticTrilinear, in addition to being sure to set TextureOpt::rnd
to the stratified random number you are using for sampling. Now we are
deprecating StochasticAniso and StochasticTrilinear. Just use the same
old mip levels you always did. So the new interface can be summarized
as follows:
* On each texture call, setting opt.rnd >= 0.0f is itself a request for
stochastic texturing on that call (opt.rnd will default to -1.0, and
any negative value does the old fashioned non-stochastic filtering).
* A ShaderSystem-wide option "stochastic" (int) indicates which things
should be sampled stochastically, for those lookups that have rnd >= 0.
It's a bit field: 0 means no stochastic sampling (even if rnd >= 0),
1 means to do stochastic MIP level selection 2 means to do stochastic
anisotropic sampling (so 3 means to do both, and higher bits are reserved
for future extensions). The default is 0, so renderers need to enable
it to get any stochastic texturing.
Finally, let me give a reminder that you ONLY want to do stochastic
texturing for shading that involves multiple stochastic samples per
pixel, like orinary surface shades. For displacement (shade once per
mesh vertex) or "imager" (shade once per pixel), you just have a
single shade that you want perfectly filtered, so you probably don't
want to set opt.rnd >= 0 in those cases.
Previous code was doing ioread one pixel at a time. Change it to
read one scanline at a time, similar to e.g. how BMP reader does it.
On a 8192x8192 rgba8 image (256MB), on Apple M1 Max "iinfo --hash"
time goes from 2.19s down to 0.65s, and is now similar to the time
it takes on an uncompressed TIF or BMP image.
When viewing this .md file in Github UI, everything seems to work. But when creating an actual PR, the links to CONTRIBUTING and CLAs point to nothing that would exist. Looks like they try to be relative links from the "create a new PR" page. Workaround seems to be to use absolute links instead.
Implements support for BC4, BC5, BC6H (unsigned and signed) and BC7
GPU texture compressed formats in DDS reader (#2338).
- BC4: loaded into 1 channel uint8,
- BC5: loaded into 2 channel uint8,
- BC6H: loaded into 3 channel half,
- BC7: loaded into 4 channel uint8.
This is implemented by replacing whole of libsquish (~4700 lines of
code) with bcdec.h (https://github.com/iOrange/bcdec, ~1300 lines of
code). bcdec.h has an advantage of being faster, smaller and supports
all BCn formats for decoding.
bcdec.h license is MIT or Unlicense (aka public domain), updated all
the relevant third party / license files that I could find.
* Add a SECURITY.md detailing our security policy and the private
email address to contact to report vulnerabilities.
* Create scorecards.yml, to run OSSF scorecard to check supply chain
security issues. It has some false positives, but it's only advisory.
* Make sure actions we use don't get write perms on our repo.
* Use exact SHA's for actions versions to guard against hijacked
actions repos that might try to substutite something else at the
version tag we use.
* Tried to use harden-runner action to ensure no actions on the runner
overwrite source code or have unauthorized outgoing net connections
during the GHA run, but currently leaving it disabled because of too
many false positives -- it flags every dependency download as a case
of altering source code. Which it technically is, but it doesn't
matter for any jobs that aren't producing any artifacts that will be
consumed or relied on by users.
Learned about this from a security talk.
On Linux, we'd been using gettimeofday, which gives results with usec
resolution. Switch to clock_gettime, which has nsec results.
In practice, the round trip time of a pair of "get time" system calls
is about 30-50ns in either case. But the new call at least nominally
returns ns resolution values. Another way of saying that is: the
answers we get should be more accurate with this call, but the call
itself is the same expense as before, so we should still be just as
cautious about not inserting timers around events that are so brief
that the timers themselves are going to be a significant addition to
the cost.
This only effects Linux (and maybe BSD?). On MacOS, we use
mach_absolute_time(), and on Windows we use QueryPeformanceCounter(),
so neither of those platforms is affected by this patch.
Oops, calling it "m_max_tile_channels" (the name of the member
variable) instead of "max_tile_channels" (the name of the attribute)
actually prevented properly setting or retrieving the attribute by
name.
ustring is a unique string (that lives in an internal hash table) whose
representation is the pointer to the characters that live in the table.
ustring hash is its alter ego -- it also refers to a ustring in the
internal hash table, but the representation is just the hash itself.
It presents an interface that is a subset of that of ustring, and in
many ways they can be used interchangeably. For ustringhash,
accessing the characters requires using the hash to find the string in
the table -- which is not hard, but more expensive than accessing the
charcters with a ustring, where the representation is the pointer
directly to the characters.
For some uses where you don't need access to the characters in any
performance-critical paths, this may be a more convenient
representation. In particular, it's well suited to a GPU that doesn't
have access to the character memory. Another interesting difference is
that from run to run, a ustring may have a different literal value,
since there's no reason to expect that the pointer to a string like
"foo" will refer to the same memory location every time the program
executes, but in contrast, the hash is guaranteed to be identical
from run to run.
In the process of adding unit tests for ustringhash, I ended up
overhauling all of ustring_test.cpp, as I saw that the ustring tests
were extremely cursory. Did a tiny bit of (non-breaking) touch-ups
on a few ustring methods as well.
If the BMP image we're reading declares a certain sized palette, but
then has pixels that contain values outside the range of that palette
size, we would read past the end of arrays.
With a 1x1, 2bit image of a Targa 1.0 file, the entire image was only
22 bytes, so the seek to 26 bytes before the end to look for the
Targa 2.0 data... obviously failed.
Don't look for the Targa 2.0 block if the file itself is not long
enough to hold both the header (18 bytes) and the block (26 bytes).
This tiny change is a quality-of-life fix for me, addressing the fact
that there are two different simd.h files in our code base. I never
would have done that on purpose, but one of them is part of the
imported "squish" source code that we use for dds encoding.
For years I've been frustrated that reflexively typing `CMD-P simd.h`
in the editor has a 50/50 chance of pulling in this file, when in fact
there is literally a 100% chance of my wanting
src/include/OpenImageIO/simd.h.
So finally put my editing brain out of its misery by renaming the file
that I never want to edit.
When doing thread "wedge" tests, most of the time spent is for the
1-thread case. Who cares about 1 thread anyway? So we make two
improvements with new command line options:
* --minthreads sets the minimum number of threads that will be
used. For example, `--minthreads 2` will skip the single-thread case
and start the wedge with the 2 thread case.
* --lowtrials is an optional maximum number of trials just for the 1
or 2-thread case (overriding --trials just for those thread counts,
if it is a lower value).
By default, the old behavior is still the default. But these allow a
bit of customization in the tradeoff of how much time to invest in the
very low thread counts in a wedge of thread timings.
You'd think there would be no way that basic failure like this could
occur after 14 years, right?
But the TIFF reader code was a little mixed up about the fact that the
internal m_next_scanline field is relative to the starting scanline of
the image, not an absolute value. But in the case of the code that has
to deal with handling the stray scanlines at the end of a strip-read
file neglected to set it properly and the end of the stripped part.
Obviously, for TIFF files with a first scanline of 0 (the vast, vast
majority), failing to adjust by this offset doesn't matter. And also
as long as the reading continued to be strictly sequential, it also
was asymptomatic. But in the presence of certain multithreaded random
access patterns, it makes a difference. Oops.
The udim_lookup_mutex_pool is not necessary. Just using an atomic
pointer for udiminfo.icfile is enough.
I'm seeing about a 5-8% overall speed improvement in testtex-based
intense udim benchmarks under heavy threading, a ~2% overall
improvement in a real rendered scene that uses a bunch of udim
textures (I think there are bigger bottlenecks elsewhere that I'll
move on to next, but every little bit helps).
Some testtex enhancements coming along for the ride:
* UDIM-oriented tests (automatically decuded if the test textures
contain `<UDIM>` in their name.
* --maketests will construct random test textures and use them (handy
if you need lots of textures for a big test, but don't want to store
them). The tests are based on a file pattern using std::format-like
replacement with an index. Also, if the pattern contains `<UDIM>` in
the name, it's understood to be a udim pattern and many individual
udim piece files will be created.
An example of these two items in action is:
```
testtex --maketests 200 "tmp/foo{:04}_<UDIM>.exr" --threadtimes 8 -iters 100000000 -runstats -v --handle -maxfiles 1000 -trials 5 -wedge
```
fmt master seems to have just deprecated the ability for
fmt::sprintf("%s") to default to printing anything that allows stream
output. This breaks use of Strutil::sprintf formatting ustrings.
This isn't yet in any tagged release of fmtlib, but I'm trying to get
ahead of it.
The quick fix is to just change those handful of places to use
std::format style calls, which we are slowly converting to anyway.
Also throw in a fmt::formater for ustrings for good measure.
It's a lot of data, you don't need it if you don't want to run the
full testsuite, and a lot of places and distros have policies against
live downloads happening in the build step. Also, downstream projects
that need to build OIIO for their CI or other reasons don't want to
accidentally download a lot of test data they won't need.
So change the default of OIIO_DOWNLOAD_MISSING_TESTDATA to OFF.
Build with -DOIIO_DOWNLOAD_MISSING_TESTDATA=ON to turn on auto
download of the data images, or else run
src/build-scripts/install_test_images.
Now, this does have the following danger: That people who build OIIO
and do want to run the tests -- and don't know to turn this option
back on -- will end up with very few tests able to run.
We largely address this by observing that quite a lot of our tests
that depended on oiio-images actually depended only on one file
(grid.tif, which is not especially large). So by making a copy of that
file in testsuite/common, quite a few additional tests can run even
without oiio-images.
Though, to be clear, there are still many tests that want oiio-images,
particularly tests that validate support for various image file
formats, which often depend on large collections of images that are
stored in oiio-images. So remember, if you are developing OIIO, the
best policy is to run the whole testsuite. But on the other hand, if
you submit a PR, they'll all be run in CI as well.
For failed CI builds, save the cmake caches in a safe place, don't zip
them up in situ, because then when I unpack them in my build area
locally, they overwrite my working cache.
The Intel OneAPI LLVM-based C++ compiler, as we download it from Intel
for our CI, bumped from 2022.0.0 to 2022.1.0.
New warning popped up -- newest Intel LLVM-based compiler warns about
overriding certain floating point math flags.
Also had to adjust thresholds on a couple texture tests that have a
few pixels change slightly due to numerical differences.
Another round to hide Imath externally to OIIO public headers as well
as speed up our own builds.
* Remove the external reference to Imath::M44f from color.h, and instead
use M44fParam.
* The `<OpenImageIO/Imath.h>` includes the most commonly used Imath
headers including half.h. But this is unnecessarily expensive (and
presumptuous about wanting other Imath types as well as fmt.h) if
you just need to know about `half`. So we create a new
`<OpenImageIO/half.h>` that only includes what you need for `half`
(as before, with logic to handle the changing location of the half.h
header depending on whether you're using Imath 3.x or OpenEXR 2.x).
* Several files that included Imath.h, but in reality only needed half
all along, change to including OpenImageIO/half.h.
* simd.h leans more on vecparam.h and other template trickery to
reduce exposure of methods that only work in the presence of Imath
headers, and make fewer use cases that would care whether the Imath
headers are included before or after simd.h (which is less
controllable for unity builds).
* Vec3Param gains a `cast<V>()` templated method that allows easy
casting to foreign vector types even if Imath headers are not
included prior to vecparam.h being included (which is less
controllable for unity builds).
We have a template OIIO::bit_cast which has served us for a while, but
it turns out that C++20 has std::bit_cast which operates ALMOST the
same -- unfortunately, the tempalte arguments are reversed from the
way we did it!
So here we:
* Set up a new `bitcast<>` that works exactly like C++20 std::bit_cast.
* Implement our old bit_cast in terms of the new one, for now.
* Set up `#if` trick so that the confusing OIIO::bit_cast will be
marked as deprecated beginning with OIIO 2.5, and will be removed
entirely from OIIO 3.0. (Farther out than that, we may choose to
re-introduce bit_cast as truly being identical semantics to
std::bit_cast. But before that is possible, we need to completely
ween downstream users off the old one.)
* Took the opportunity to expand the set of compilers for which
certain bitcast overloads use Intel intrinsics. Previously, we only
did this for the Intel compiler, but it seems to work for modern
versions of clang and gcc as well. Do a little bitcast testing in
fmath_test.cpp to verify that the use of these intrinsics doesn't
break for any of the compilers we use for CI.
Over the span of several individual PRs, we've made the code base safe
for "unity" (a.k.a. "jumbo") builds -- a feature of CMake that
automatically combines multiple .cpp files into a single compilation
unit (basically making temporary .cpp that `#include`s each original
cpp) in order to, allegedly, speed up compilation by not re-parsing
headers multiple times, and other sources of per-module overhead.
We discovered early on that on a modern computer where modules are
built in parallel (another automatic feature of CMake), jumbo builds
are simply not an overall performance improvement for OIIO. Basically,
the efficiency gain from removing redundant parsing is mostly offset
by the loss of parallel efficiency -- fewer and larger modules means
there are periods where all cores are not able to find work to
do. (Reductive example to illustrate the basic problem: If you have 8
independent modules to compile, and each takes 1 second, 8 cores can
compile the entire collection in parallel in 1 second. But a unity
build that batches those files in groups of 4 and achieves a 2x
speedup, thus compiling each set of 4 modules in 2 seconds instead of
4, only can do two of these compilations in parallel, so it takes 2
seconds. We've cut the total work in half, and yet it takes twice as
long in real time to complete, and 6/8 of your cores are idle.
OK, but that's when you have several cores. If you had only one core
(or maybe two) so you weren't using much parallelism in the first
place, then the unity build would look like a win, since you'd really
see the improvement from the amortized overhead of the combined
modules.
Most of us don't have single core machines on our desks, or laps, or
even in our pockets (or on our wrists?).
But we do have single core machines (or close to it) in our CI runners!
We get 1.5-2 cores, roughly, in the slice of VMs we get to run CI on
GitHub Actions. So maybe this is the one place where a unity build would
be of benefit.
That's the theory anyway. Does it pan out? I tested it, and the answer
is basically yes. I can speed up the "build" step of our CI by
somewhere in the vacinity of 30% (varies a LOT from run to run). I
tried a whole bunch of different approaches, summarized in the table
below. I picked the best basic strategies revealed by those tests and
set up the CI to use them. (I didn't preserve all the other
permutations.)
| Name | try 1 build | try 2 | try 3 | try 4 |
|:----------------------- | ----------- | -------------------- | -------------- | -------- |
| **linux gcc9 BASELINE** | 14:19 | 11:46 | 11:56 | 13:32 |
| linux gcc9 group | 9:39 | 8:06 | 9:58 | **7:58** |
| linux gcc9 batch 8 | 8:06 | **7:17** | **8:02** | **7:59** |
| linux gcc9 batch 16 | 8:29 | 7:32 | **7:56** | 9:35 |
| linux gcc9 batch 4 | 8:39 | 8:38 | 8:52 | 10:31 |
| linux gcc9 batch 2 | 13:34 | 12:43 | 11:22 | 9:32 |
| linux gcc9 batch mix 4 | 7:22 | 9:14 | 9:01 | 9:39 |
| linux gcc9 batch mix 2 | 9:11 | 7:45 | 10:55 | 9:36 |
| **linux icx BASELINE** | | 14:39 | 15:07 | 15:06 |
| linux icx group | | 11:59 | **10:36** | 10:26 |
| linux icx batch 8 | | **9:40** | 11:31 | **9:14** |
| linux icx batch mix 4 | | 10:06 | 11:58 | 11:16 |
| **macos11 BASELINE** | 25:41 | 27:53 | 26:30 | 28:32 |
| macos11 group | 10:56 | 20:24 | 19:28 | 19:24 |
| macos11 batch 8 | 16:43 | 17:38 | **18:34** | **17:42** |
| macos11 batch mix 4 | | **9:22** | 19:12 | 18:26 |
| **windows BASELINE** | 20:17 | 23:27 | 28:43 | 19:13 |
| windows group | 23:05 | 15:57 (tests broken) | 21:57 (broken) | 15:33 |
| windows batch 8 | 21:16 | 15:57 | **17:38** | **15:08** |
| windows batch mix 4 | | **15:30** | 23:19 | 15:16 |
Notes:
* Timings vary wildly from run to run. You never know which VMs you'll
get or how overloaded they are. But over 3 trials, the trends were
clear.
* I rigged it to clear the ccache, so these are FULL uncached builds.
* This is only "build" time. The CI run also includes time to set up the
VM, download docker images, build or install dependencies, and of course
run the testsuite. So the overall speedup of the entire CI run will not
be as improved overall as these numbers indicate for the build step.
But still, saving several minutes per job means the CI tests turn
around faster, and that's what we want.
For C++17 and beyond (and maybe earlier in some compilers), we already
used std::gcd. Now for C++14 compilers without std::gcd, we roll our
own simple gcd implementation instead of using boost::gcd. Maybe it's
slightly slower, but this is nowhere near a performance-critical
path. It's worth it to take one more step at eliminating boost.
Reminder: When opening a file by name, first it tries to open assuming
that the file's extension indicates the format. If that doesn't work,
it tries all known format readers to see if one will work.
This patch includes several improvements to this process and surrounding
code in imageioplugin.cpp:
* When trying all formats because the extension was wrong, instead of
traversing the input_formats map in alphabetical order of the list
of all format names + extensions, try the three most common formats
we encounter (openexr, tiff, jpeg) FIRST, then try the other formats
in alphabetical order. We do this by traversing the
format_list_vector, which contains all format names in the order
they were declared, and ensure that those common formats are
declared first. Basically, this means that if the mislabeled file is
one of those most common formats, the correct reader is found after
only a couple tries, instead of having to try a dozen or more bad
guesses of rarely encountered formats.
* Debug mode messages that make it easier to understand which readers
are being tried for which files.
* Comments clarify that input_formats and output_formats hold not just
the map from format names to creators, but also map extensions to
creators. (That confused me while I was debugging this, so I figure
the clarification is needed.)
* Establish a new extension_to_format_map that lets you get the format
name if you know the extension.
* Get rid of input_extensions and output_extensions maps, which weren't
ever used.
* Some other minor refactoring.
If you know the hash of a ustring, this tells you the ustring.
(Caveat: in the very unlikely chance of two ustrings having the same
hash, this will return the first one it finds in the table.)
I realized that --attrib (and several related actions) only set the
attribute as directed for the first subimage if it's a multi-image
file (such as a multi-part exr), or all of them if the -a flag is
used. But there was no ability to set it for a particular (non-first)
subimage, like many other oiiotool actions have the ability to do.
So in this patch, I refactor the following commands to use the
OiiotoolOp and related macros, which automatically make it responsive
to the `:subimages=` optional modifiers:
--attrib
--sattrib
--eraseattrib
--caption
--orientation
--clear-keywords
--iscolorspace
As a bonus aside, one thing this lets you do is set a different
compression method per-subimage using, e.g.,
`--attrib compression:subimages=3 dwaa:100`.
But if you set compression with `--attrib compression`, do not also
use `--compression`, which will override it and also apply to all
subimages.
When this symbol is defined, this marks as deprecated the ErrorHandler
class methods that use sprintf formatting notation. It's not enabled
by default, but a downstream project can define the symbol in order to
help root out any uses of the old style formatting functions, to aid
identifying and changing those spots to the new style, long before
they are truly deprecated or removed from OIIO.
This change also sets up an OIIO_INTERNAL symbol that will be present
when OIIO itself is compiling, but not a downstream project using the
OIIO headers.
This is the first tiny step to fixing all the little things that break
in a unity build. This one is pretty small fry -- the oiiotool set of
modules only needed some minor rearrangement of header file include
order.
* Initial implementation of `ImageBufAlgo::st_warp`
* Simplify default filter logic, and share it with `ImageBufAlgo::warp`
* Add st_warp to oiiotool
* Add tests for bespoke `st_warp` validations that are done before IBAprep
* Add st_warp test to oiiotool-xform testsuite
* Address review notes
- Remove ST buffer datatype guard
- Simplify additional ROI prep
- Remove unnecessary buffer reallocation
* Use sample accumulator type for source/dest iterator USERT
* Add `--st_warp` section to `oiiotool.rst`
Cut/paste error inadvertently deleted a critical cmake line that I
wasn't intending to alter or remove, that resulted in the command line
utilities not being tagged for part of the install step. Oops!
Oh boy, never leave anything unbenchmarked.
Turns out the boost::algorithm functions we were relying on underneath
many Strutil "case-insensitive" comparisons were ridiculously slow.
We thought they were good enough because they allowed specification of
locale, so we could just pass the static classic locale, and so they
would be inexpensive because they didn't have to query the current
locale. But this is wrong, they were still ghastly.
So here I rewrite Strutil::iequals, iless, starts_with, istarts_with,
ends_with, iends_with in terms of a new (internal) Strutil::strcasecmp
and strncasecmp (which underneath handle differences in platform, and
use the locale-independent versions). The net result is that most of
those case-independent comparisons speed up by a factor of
50-100x. Wow.
I still need to tackle the family of 'ifind' related functions. They
are a bit trickier. But I'll leave them for another time, because I
need to roll this present fix out right away to fix a real production
bottleneck.
(Worth noting: iequals is instrumental when you're searching a
ParamValueList for a particular name, which is what happens when you
look up attributes from an ImageSpec, which is what happens when you
call get_texture_info(), which is what underlies OSL gettextureinfo()
calls in the cases that they cannot be constant-folded during runtime
optimization. So this came to my attention when analyzing a slow scene
whose shaders had a pathological explosion of gettextureinfo calls that
couldn't be optimized away.)
Reminder: isConstantColor() finds out if an ImageBuf is constant
color. It uses the thread pool to check several areas in
parallel. Once any task finds that its subregion is not constant, it
sets a flag so that as other subregions are pulled off the task queue,
they will take an early out if one of their siblings already knows the
image is not constant.
Turns out this isn't much of a big savings in practice, because lots
of regions may be in flight by the time one region realizes it's
non-constant, and once a region task starts, it doesn't check the flag
again.
This patch speeds up isConstantColor by a pretty big factor for
non-constant images (it can't really help for images that ARE
constant) in the following ways:
1. Before doing anything, it checks to see if the FIRST TWO pixels of
the image are unequal. If that's the case -- and it's a fairly
common case for non-constant images -- then it immediately knows
the whole image is not a constant and doesn't need to look at
anything else or launch any of the threads at all. (If it is a
constant image, or if the first two pixels are just coincidentally
identical, this quick check of only two pixels didn't really cost
us any masurable time.)
2. In addition to checking the result flag at the start of each
parallel job, now we also check if the flag has been changed as we
iterate to each pixel. So pretty much as soon as one thread finds
unequal pixels, the other threads can spot this and stop their
tasks early, no need to finish looking at their entire region. (To
make the more frequent checks of the flag inexpensive, I changed it
from an atomic to just a regular bool. It doesn't need to be
atomic, because the strict ordering is not important at all here
and it doesn't change the result if the threads' access to the flag
is unsynchronized.)
When outputting OpenEXR files, in a case where the "compression"
request was for "dwaa" (no compression specified in the main name,
like "dwaa:100") we accidentally forced a default quality level of 4,
which is the default we wanted for zip compression. But that's not a
good default for dwaa!
As I learned recently, a "unity" (aka "jumbo") build is where multiple
.cpp files are combined into one translation unit -- imagine a unity.cpp
that simply has
#include "file1.cpp"
#include "file2.cpp"
...
#include "fileN.cpp"
and so you compile unity.cpp instead of the separate file?.cpp files
individually.
Turns out that CMake understands this concept and can do it for you
automatically!
The benefit of a unity build is that file1...fileN probably include
most of the same headers, expand the same templates, etc., so a bunch
of the per-file work of the compiler can be done once rather than
redundantly for each file.
There are two potential downsides, however:
1. It may not be safe to concatenate your cpp files! For example, if
both file1.cpp and file2.cpp each contain a `static int foo;`, that
may have been safe when compiled separately, but is not allowed to
happen twice in one compilation unit. Similarly, if you have headers
that don't have proper guards against multiple includes, etc. So one
should expect a whole lot of little fix-ups are needed for this to
work properly. (We'll come back to that topic.)
2. Combining source file into these "jumbo" modules can make heavily
parallelized builds on many-core machine not be able to load balance
or keep all the cores busy. (Simplified examples: if you have 16 .cpp
files on a 16 core machine, each core can compile one cpp file in
parallel with the others. But if you mash the modules into just one
huge cpp file, give that to one core, and your other 15 cores are
idle, so the full build probably takes much longer.)
I tried this out, including the many fixups implied by (1) above, and
at first the unity builds were slower on both my laptop (8 cores) and
workstation at work (32 cores), because of downside (2) explained
earlier -- harder to take advantage of parallel builds when there are
fewer, bigger, compilation units. Some tweaking of strategy got me to
the point where I could always get the unity builds to go faster, but
not by a whole lot when many core were available. Slighty faster, but
not worth the trouble. A bit disappointing, nearly abandoned the whole
idea.
HOWEVER, in situation where you are limited to very few cores (like in our
CI, which allocates 2 cores, and it sure seems more like 1-1.5 for the
Windows and Mac CI runners), the unity builds are substantially faster --
there's already not much parallelism to exploit, so you come out ahead
with the savings of that redundant per-file work we talked about.
So my current thinking is to go ahead and make the changes that allow
unity builds. I don't particularly recommend them when highly parallel
builds are available to you, but it might help to cut our CI
turnaround time down on the GitHub runners. And maybe it will help in
other situations for other people.
Ok, then. The present patch introduces these concepts and makes the
CMake and other build system changes to allow unity builds. (N.B. It
won't WORK yet, so don't try it!) After we get that out of the way, in
subsequent PRs I'll show all the changs to the code that were
necessary to fix all the little things that went wrong when source
files got combined.
* Fix warning from duplicate define
* Fix warnings about bool/int expressions
* Fix discarded result warning in PSD input
* Fix codepage warning from unicode symbol in string constant
* Ensure the term object is initialized with the stream we intend to output to
* Fix pointer casts in paramlist test
Signed-off-by: Chris Kulla <ckulla@gmail.com>
Co-authored-by: Larry Gritz <lg@larrygritz.com>
In FFMpegInput::close(), some of the tear-down calls would happen
regardless of if their pointers were already null. Seemed safe, but
maybe not always? So check the pointers.
Fixes#3371
When trying to break filenames into the base name and any "RESTful"
arguments (such as "/foo/bar.exr?name=value&dog=cat"), we were
just looking for presence of the "?" to determine if it was a RESTful
thingie and using that as the split position.
But this is thrown off by an odd Windows notation where names that
start with (get this) "\\?\" somehow designate long filepaths. See
See: https://docs.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation
So the solution is to change Strutil::get_rest_arguments to explicitly
check for this pattern at the start of the string.
In OpenEXROutput::write_deep_scanlines and write_deep_tiles, don't
just pass the DeepData along -- handle the case of where the DD (the
buffer the caller is presenting) does not match the data types that
you said the file is going to hold. In this case, we need to make a
local temporary DD that holds a copy of the passed-in data, but with
the formats transcoded.
Some changes to DeepData were done to support this.
GHA is retiring the windows-2016 image, we won't be able to use it soon.
Also bump openexr versions in a couple places.
There's some in-progress work on a VS2022 test, but ignore that for now,
it's commnted out.
I'm a total convert to fmt (and/or std::format, std::print) and slowly
converting all prior use of sprintf and iostream to it.
If I print an Imath::V3f or a simd::vfloat3,
fmt::print ("v = {} {} {}\n", myvec[0], myvec[1], myvec[2]);
That's ok, I guess, but it's clunky and a big pain to put the format
specification for every element separately if it were a vfloat16,
say. I would like to be able to do this:
fmt::print ("v = {}\n", myvec);
and in fact, it does print all elements (thanks to some nice behavior
by fmt)! But this does not work as expected:
fmt::print ("v = {:.3f}\n", myvec);
fmt's tricks will print all elements, but only using the default format
spec, and the spec above applied so the "whole thing", not each element,
if you see the distinction.
So this patch sets up the custom formatter infrastructure to be able
to print array-like things (and in particular, the Imath and
OIIO::simd vector and matrix classes) by specifying the format spec
used for the elements.
The important elements to note:
* detail/fmt.h adds custom formatter helpers for (a) any class with
operator[] and size() ("index_formatter"); (b) any class that lacks
these but nonetheless has an array-like memory layout and a known
length ("array_formatter"). I also added a nonstandard extension to
the format spec for these, which optionally lets you use a comma
separator rather than just the space between elements.
* These building blocks are used to specify custom formatters for
the commonly-used Imath types (in our Imath.h) and our internal
SIMD vector types (in simd.h).
* simd.h also adds a few other goodies, including an `is_simd<>`
template, and `size()` constexpr methods to each vector type. These
make certain template metaprogramming tasks easier for these types.
* A new "type_traits.h" header for collecting a bunch of little utility
templates, such as `has_subscript<>` and `has_size_method<>`.
JPEG expresses pixel aspect ratio implicitly as header fields
"xdensity" and "ydensity" (which we call "XResolution" and
"YResolution" in our metadata after the TIFF names). We also have
"PixelAspectRatio", so the three of them, if they are all used, form
an over-constrained system. Plus, they may come both in the JPEG
header and also (not necessarily having the same values) in any Exif
block.
And on top of that, the way Nuke and PhotoShop infer the aspect ratio
is BACKWARDS! (If pixelaspect = width/height of a displayed pixel,
then it's also = ydensity/xdensity. But Nuke is the other way
around. Only for JPEG, it gets it right for TIFF and OpenEXR.)
Anyhoo, there were some edge cases in which we were still getting
confused (specifically, that we were making some files that Nuke was
not properly recognizing as anamorphic). This patch tightens up
these cases.
Create a small shim class template called
Vec3Param (and its specialization, V3fParam) that does nothing but
implicitly convert from and to things that looks like 3-vectors
(thanks to some template magic I developed a while back for the Imath
project). And similar for template MatrixParam (and specializations
M33fParam and M44fParam) to pass a square matrix.
Basically the idea is that we can replace the public API function
parameters where we take an Imath::V3f and instead use a
V3fParam. User code that passes an Imath::V3f still works because
V3fParam accepts and converts it. Implementation code can implicitly
convert the V3fParam to an Imath::V3f, so it doesn't need to be
changed, either. Only the function declaration itself. (Same basic
situation for MatrixParam and Imath::M33f/M44f.)
So in the process, now we have laundered our public API so that it
doesn't reference the Imath types and headers any longer. This confers
two big advantages:
1. It allows OIIO itself and any downstream client code calling OIIO
to use *different* versions of Imath if they want.
2. It allows client code to not need Imath at all, for example if they
have their own proprietary vector type that they prefer to use
internally.
I'm not necessarily 100% sold on this approach, but I think it's got
promise and am seeking feedback. It seems to work, and I was able
to compile OSL against this altered OIIO with *no changes*.
The use of ends_with instead of iends_with made it so that
capitizalized files (especially on case-independent filesystems like
Windows or some Mac filesystems) were rejected.
(And yes, I did grep the whole code base for ends_with to be sure that
this was the only place we made this mistake.)
Fixes#3362
icx is the *new* Intel LLVM-based compiler, and it behaves a lot like
clang (versus icc, which is the "classic" Intel compiler).
* An additional CI test in ci.yml for icx.
* Tweaks in compiler.cmake and platform.h to detect and set options for
this compiler.
* A variety of warning fixes encountered for this compiler.
* Add a new cmake build-time option `ENABLE_FAST_MATH` (default: OFF)
for enabling fast-math optimizations that trade IEEE compliance for
speed. The Intel compiler enables it by default (!), so this build
option makes icx behave like the other compilers (not fast math
unless explicitly asked for), which makes the CI tests and other
behavior more consisent across compilers and platforms.
* Note: The bug that prevents some versions of fmtlib from working
with icc does not appear to be a problem for icx.
I finally learned how the "strategy" stuff works and refactored our
CI workflow based on it.
In short, we previously treated each CI test as its own universe, with
a huge amount of repeated build logic. These changes cut down on
redundancy by, for each set of related tests, collapsing them into
just one recitation of the build script logic and a handful of
variables that create the different test variations.
Currently, the categories are (a) all tests based on ASWF containers and
corresponding to VFX Platform specs, (b) "off-year" combinations based
on the native Ubuntu installations, (c) Mac tests, (d) Windows tests,
(e) the clang-format checker.
This refactor cuts the size of the ci.yml file by more than 50%, and
makes it much easier to add or change individual test configurations
in the future.
Strutil::print wraps fmt::format to render the formatted output into a
std::string, then sends the string to Strutil::sync_output(), which
ensures that every string printed (or sent to a FILE or stream) is
done so atomically, using an internal mutex, and with a stream flush
after each string. But this makes it a lot more expensive than it
needs to be, when you don't need such care -- either because buffered
output is ok, or because you don't need the thread safety.
This patch:
* Adds an optional `flush` parameter to sync_output(), defaulting to
true, which controls whether sync_output flushes or not.
* Introduces Strutil::sync::print, which is the same fully
synchronous, thread-safe, non-buffering behavior that Strutil::print
has traditionally had. Use this when you need those features.
* Redefines Strutil::print to be an alias for fmt::print, which means
- There is one fewer layer of template expansion and generated code,
and possibly fewer internal string copies, so that should make for
both faster compilation and faster runtime performance.
- Results are buffered, no sync after every output, so much faster
when you don't need that.
- No internal mutex.
On the platforms we use -- and with the current implementation of
fmtlib -- fmtlib's underlying use of stream and FILE output appears to
be adequately thread-safe. But we can't 100% guarantee this on all
platforms or all future versions of fmtlib, so users who need the
stronger guarantees (and are willing to sacrifice performance) can
simply use sync::print.
* New CI test downloads the Intel icc compiler and builds with it.
* A variety of minor fixes to address warnings from icc that were
not apparent from the other compilers.
* Some minor changes to just a few tests and reference output where
the icc-built compiler produces slightly different (usually LSB)
results.
These are cases where we declared a template, then later gave its full
definition, and we marked the declaration of the OIIO_FORCEINLINE but
omitted it from the later definition.
This addresses some compiler warnings from icc.
Slight reworking of -Werror logic to let the default behavior be
changed by setting a cmake variable. In particular, this allows us to
make a CI build disable -Werror when we need to.
Previously, having the "OpenmageIO_CI" env variable set trumped all
else and forced -Werror. Now it just contributes to the default behavior,
but setting `-DSTOP_ON_WARNING=OFF` and cmake config time can still
disable it.
Seemed like a good idea when we wrote it, but subsequently,
std::string_view was accepted into C++17 without a str() method. So
for the sake of future full interchangeability with std::string_view,
stop using this method. We will eventually deprecate it.
Special case some methods of ParamValue, ParamList, and
ImageSpec::attribute for ustring and string_view data, to reduce some
redundant copies and/or ustring conversions.
The internal method init_noclear() is augmented with a flag that allows
the caller to specify that any string data it might point to is already
ustrings, and thus does avoid the costly default behavior of converting
strings in the data block to ustrings. (This is not publicly exposed, so
users can't screw it up.)
So the new ParamValue constructor directly from a ustring can use this
flag to avoid passing the already-ustring through the string-to-ustring
conversion a needless second time. But also, the copy constructor, move
constructor, and assignment can use the flag and avoid the extra
conversion, since by definition they are copying from the contents of an
existing ParamValue which is guaranteed to be already ustring-ified.
While I was at it, I turned the optional arguments of ParamValue
constructors into "strongly typed" flags, deprecating the ones that
take a raw bool, which might be confusing.
c_str() was cute, but it's a little dicey and it's not present in
the C++20 std::string_view. We didn't use it in a ton of places,
so replace with other idioms so that in the future we are better
able to interchange with std::string_view or replace our own entirely
once we have flying cars and C++20 minimum.
* Allow ustring and OIIO::string_view to auto-cast to fmt::string_view
(helps some fmt formattting of types without custom formatters).
* When std::string_view (C++17) and/or std::experimental::string_view
(some compilers pre-C++17) are available, add OIIO::string_view
auto-construction from and auto-casting to those types.
...and stop using them internally as well.
For some of the IBA functions, we were unable to fully mark them as
OIIO_DEPRECATED (so instead that hint is commented out) because the
function signatures of the deprecated and new ones can be
ambiguous. We're just going to have to delete those when we are at a
compatibility break point, without going through a period where they
are present but marked as deprecated.
Just like we had done for the other libraries recently. And need
to make sure that the imath libs are added explicitly to the python
module's needs, since for hidden static libraries, they will not get
picked up merely by linking to libOpenImageIO.
Add an OIIO build-time option for forcing our exported
OpenImageIOConfig.cmake to skip the find_dependency of Imath and
OpenEXR.
This is helpful if you're at a site that is trying to build OIIO
against statically compiled Imath/OpenEXR, and want to be super sure
the downstream projects are definitely not going to encounter that
find_dependency(Imath) when the read the OIIO config files.
(They could do their own `-DOPENIMAGEIO_CONFIG_DO_NOT_FIND_IMATH`,
but I want even that to be unnecessary.)
There was that one pesky place where we were still including Imath
headers, from simd.h in the case where OIIO_SIMD==0 and we wanted to
fall back on some Imath::Matrix methods.
Ugh, but this backfired because when including this file from Cuda,
OIIO_SIMD==0 and it included the header. But what happens if a
downstream app (a) includes this file from Cuda, and (b) uses a
DIFFERENT Imath (so names clash) or doesn't want to use Imath at all?
So one more change:
* Remove the include of Imath.h from simd.h entirely.
* Most places where we fell back on the Imath::Matrix methods, just
replace it with scalar code, it's not complicated. (For some technical
reasons, this required some minor surgery on the matrix44 class, but
not anything that changes anything from a caller's point of view.)
* The one complicated case is matrix44::inverse(). Punt! Just let that
only be implemented if either (a) there is SIMD available, or (b)
there is Imath available (i.e., included by the downstream code
prior to this header). That's it. If you want to invert a 4x4 matrix
without SIMD, you need to arrange inclusion of the correct
ImathMatrix.h yourself before including this header.
For vector displacement maps or normal maps that are in tangent space,
it is heplful to be able to record whether the data are intended to use
a local left-handed or right-handed coordinate system, and for that to
be querably from the shader accessing the map.
The `maketx --handed` option, or `oiiotool -attrib -otex:handed=...`,
or adding "handed" metadata to the configuration ImageSpec being
passed to `IBA::make_texture()` does this. It's easy for texture
formats like openexr that allow arbitrary metadata. For TIFF textures,
we embed it in the ImageDescription tag (as we do for several other
texture-specific hints).
Skip 3 bytes between "software" and "background (key) colour" for the "software
version" (was simply missing). Avoids that m_alpha_type is overwritten with
wrong values.
The very latest fmt master changes a behavior: fmt functions no longer
automatically know how to format any type that can output to an
ostream, and instead need custom formtter objects. This breaks A LOT.
Fix by defining the preprocessor symbol `FMT_DEPRECATED_OSTREAM`,
which restores the old behavior (for now, at least).
While I was at it, I moved the fmt inclusion logic out of strutil.h
and into a new separate header, detail/fmt.h. This is in turn included
from strutil.h, so nothing in user code needs to change.
It turns out that this can crop up all the time, sometimes from
malformed metadata inside image files. We weren't noticing it because
in many cases, things ended up being turned into ustrings twice, and
the sequence string-with-nuls -> ustring() -> c_str() -> ustring()
will launder it. As I was tracking down and eliminating some of the
double ustring constructions, the metadata strings that were malformed
all along started becoming symptomatic.
Bad paradoxes follow from a ustring contain embedded NUL characters,
including ustring.c_str() and ustring.string() being different
character sequences, as well as ustring.hash() not being equal to
strhash(ustring.c_str()). One should definitely be able to count on
ustring(cstring) == ustring(ustring(cstring).c_str()).
So this patch checks for embedded NUL characters at the appropriate
time of ustring construction, and truncates the character sequence at
the point of any internal NUL character.
* `OPENIMAGEIO_IMATH_DEPENDENCY_VISIBILITY` is a new CMake cache
variable at OIIO build time that controls whether the Imath library
dependencies will be declared as PUBLIC (default) or PRIVATE target
dependencies of libOpenImageIO and libOpenImageIO_Util.
* For *downstream projects* that consume OIIO's exported cmake config
files, setting CMake variable `OPENIMAGEIO_CONFIG_DO_NOT_FIND_IMATH`
to true will skip the find_depencency() calls for Imath and OpenEXR.
To clarify, this is not a variable that has any effect when building
OIIO, it's something set in the downstream project itself.
* Add a `OIIO_IMATH_H_INCLUDED` header guard in OpenImageIO/Imath.h,
which allows something downstream to pre-define it to prevent OIIO
headers it needs to read from using the contents of that header.
Most builders of OIIO won't care and shouldn't change the defaults or
use any of these. But here's the specific case it addresses: If you
are building OIIO against static libraries of Imath & OpenEXR, but the
libOpenImageIO will be consumed by a downstream project that wants to
use a different Imath version, this can help you avoid some build-time
conflicts in the downstream project.
PPM by specification is Rec709! Make it so.
PNG files were getting marked as sRGB or Gamma if certain header
attributes were found in the file, but if they were absent, we didn't
designate a color space at all. Change so that the default assumes
sRGB, in the absence of any metadata telling us for sure.
Fixes#3084
One new method that should have been inline and non-virtual (so that
this whole set of changes does not change ABI by shifting the vtable
around) was accidentally marked virtual -- cut and paste error. This
patch fixes it to restore ABI compatibility.
IPTC data blocks seems buggy, and I don't know why. Our code?
Something in libtiff? Since nobody is depending on it and none of our
testsuite appears to fail when disabled, only write the IPTC block if
the user explicitly asks for it. Also, when reading them, don't
overwrite other metadata of the same name.
We wrote our string_view before C++17 std::string_view was finalized,
and the std one doesn't have a c_str() method. For conformance with the
standard, I want to remove this over time from ours.
Part 1: There were some places we did not need to call c_str() at all,
and some places where it was much more appropriate to just use a
std::string.
As Doug Rogers points out in Issue 3304, it is customary for Windows
applications to use wstring extensively for Unicode support. But it's
especially critical for unicode filenames, because unlike Unix-like
OS's where fopen() and similar functions support UTF-8, on Windows
those functions are strictly ASCII, and Windows users are used to
calling _wfopen() and other nonstandard functions that expect
filenames to be std::wstring if they want to support Unicode.
This patch:
* More clearly documents through all of our classes when parameters that
represent filenames or searchpaths are assumed to support UTF-8.
* Augments ImageInput, ImageOutput, and ImageBuf so that their methods
that take file names or path lists have additional flavors that take
wstring (always implemented as an inline shim that translates the
UTF-16 wstring to a std::string with UTF-8 encoding to pass on to the
usual method).
* Augments ImageCache and TextureSystem in just a couple places, the
functions where a filename is turned into a handle. All the many
other functions that take ustring for function names were left alone,
because they are called very frequently and are performance-sensitive,
so we don't want any translation from wstring to string happening on
every call.
We do not attempt to add wstring-based versions of functions that take
strings for other purposes -- that is, that are not intended to
represent filenames.
C isspace() takes an int, but on Windows when building in debug mode,
the implementation of isspace() can hit an assert if the value is < 0,
which of course it can be if you're looking at a character in a string
whose high bit is set (because char is signed).
Easiest solution is to make our own Strutil::isspace that does exactly
what we want and is safe.
Fixes#3300
This makes sure that two new preprocessor symbols are defined in
oiioversion.h: `OIIO_USING_IMATH_VERSION_MAJOR` and
`OIIO_USING_IMATH_VERSION_MINOR`, revealing the version of Imath being
used by the build of OIIO (which may not be the same as the version
found or being used by the app that's calling OIIO). They also get
defined as CMake variables in our exported cmake configs, allowng
downstream cmake build systems to be able to easily test whether the
Imath they found is compatible with the version that we use in our
APIs.
This patch sets out to minimize the extent to which Imath is a
necessary source dependency for using OIIO C++ header files and
APIs. The major changes are:
* Remove the unconditional `#include <OpenImageIO/Imath.h>` from
fmath.h and simd.h. Those forced Imath headers on nearly all all
code that uses OIIO's public APIs, whether or not they needed the
very few calls where Imath types are used as parameters.
* Make sure all public OIIO non-virtual methods and freestanding
functions that have Imath types as parameters are enclosed in header
guards like `#ifdef INCLUDE_IMATHVECTOR_H`. Downstream client code
needs to include Imath headers before including OIIO headers, if
they want to call those few particular functions.
This means that client applications who don't need those particular
functions, but call other parts of the OIIO APIs, don't need to
`#include` any Imath headers (or pay the extra compilation time for
doing so), and don't need any knowledge of which specific version of
Imath is used by the OIIO they are using.
The small number of public API calls that will no longer be visible
downstream unless the client code also include Imath headers (before
the OIIO headers) are:
- color.h: ColorConfig::createMatrixTransform (needs ImathColor.h)
- fmath.h: type conversions involving `half` (needs half.h).
- imagebufalgo.h: warp(), colormatrixtransform() (needs ImathMatrix.h)
- simd.h: conversions between our SIMD types and Imath types (Vec3,
Matrix33, Matrix44) and vtransform() calls involving those types
(requires ImathVec.h and ImathMatrix.h), and SIMD load/store
involving `half` data (requires half.h).
These changes necessitated adding including of Imath.h in a number of
internal .cpp files, the ones where those types are actually used.
The following header files do still unconditionally reference Imath
types or headers:
- texture.h: Some virtual methods of TextureSystem use Imath types
as parameters.
- imagebufalgo_util.h: For `half` support. We use this a lot
internally, but presume most client code never uses this header.
- simd.h, ONLY IF USE_SIMD=0 (which is generally only used for
development debugging and if porting to obscure platforms that are
neither x86 nor ARM).
The bottom line is that the vast majority of downstream callers of OIIO
APIs who need ImageInput/ImageOutput, ImageBuf, ImageBufAlgo, and
generally everything in the OpenImageIO_Util library, do not need to
use Imath types, include Imath headers, or even be aware of which Imath
release OIIO is using.
Why would we do all this?
Because this paves the way for optionally building OIIO to link against
static libraries for OpenEXR and Imath, essentially embedding and
completely hiding an OpenEXR/Imath inside libOpenImageIO.
That allows libOpenImageIO to use the very latest Imath (3.1, say,
with the new faster core API), even though that libOpenImageIO is
being consumed by an application that for whatever reason is forced to
use an entirely different (older) OpenEXR or Imath library. In other
words, it lets OIIO use OpenEXR/Imath fully embedded without
transitively passing that OpenEXR version as a downstream dependency
(except for the few use cases that need to call the tiny number of
OIIO API methods that require Imath types.
There is a small chance that client code that use Imath types and
SHOULD have been includeing Imath headers themselves actually omitted
those includes and dependd on the fact that certain OIIO headers
included them. Now that they don't, it may require the client-side
code to add those includes. I'm going to claim that this is a "bug" in
the client code (they didn't include a header that defined a type they
used), and not a breaking incompatibility in OIIO's public calls
themselves. Thus, I feel comfortable with these changes being in OIIO
2.4 (today's master) and not necessitating a 3.0 release (in which
arbitrary loss of backward compatibility would be allowed). So watch
out for the occasional need for OIIO-calling code to need to add an
`#include` for Imath types the code was always using, but neglected
to explicitly include.
Our Strutil utilities for utf8 <-> utf16 conversion were based on
Windows specific calls that we had added even before C++11 was our
minimum. Replace the Windows-specific code with std calls, and
expose them in strutil.h for all platforms (why not?).
We were passing along, in the exported cmake config files, a demand
for downstream clients to use a C++ standard version equal to that
which we used to build OpenImageIO itself. But that's not
necessary. We may choose to build with C++17 (for example), but all of
our public APIs are safe to use for C++14.
Split the build-time options to CMAKE_CXX_STANDARD for the standard we
build this project with, and DOWNSTREAM_CXX_STANDARD for the standard
we are declaring is the minimum to *use* the library.
The main event:
* Assume that for a TGA 1.0 file (too old to have the header field
that says if alpha is associated, unassociated, or meaningless), if
the alpha is zero everywhere, it means that it's meaningless, not
unassociated, so in that case we should skip the
auto-premultiplication. (See issue 3277)
Public-ish stuff:
* Be more consistent with Targa-specific attributes all being called
"targa:foo" (most were, but one was "tga:alpha_type").
* Document all the targa-specific attributes (a few were not).
* Add "targa:version" to reveal whether the file was a TGA 1.0 or
2.0 version of the format.
Internal stuff, opportunistic:
* Some more minor conversions of string fomatting to fmt style.
* Convert some std::cerr debugging code (commented out) to the DBG macro
we use anywhere.
* The `std::unique_ptr<uint8_t> m_buf` should always have been
`unique_ptr<uint8_t[]>` because it's not a pointer to a single value.
I realized recently that if you said something like
oiiotool foo.exr --ch R,G,B,A -o out.exr
and there was no "A" channel... it would just silently fill the channel
with 0 in the output.
This patch adds a warning if --ch is told to copy a channel that doesn't
exist in the input.
Ugh, there's a situation where PhotoShop is misinterpreting an alpha
channel for certain OIIO-written TIFF files, but it's fine with some that
came from elsewhere, and the only thing that appears to be different is
that the non-OIIO one entirely lacks the EXTRASAMPLES TIFF tag.
So I'm adding an option (use with extreme caution) allowing you to
tell the TIFF output to not output this tag, even though clearly it
should be.
Some rogue app out there is writing 4-channel TIFF files without the
ExtraSamples tag in the header that explains that the data is a valid
alpha channel.
We were using TIFFGetFieldDefaulted to retrieve this tag, and guess what?
The defaulted behavior changed between libtiff 4.0 and 4.3!
Changing to TIFFGetField actually turns out to fall back to the right
logic on our side, which boils down to assuming a 4th channel is alpha
if there's no other information. That seams reasonable -- RGB + mystery
channel is almost certainly RGBA.
* Bugs fixed in build_opencolorio.bash, it botched some of the directory
choices when customizing where to place and build it. Also, switch to
the more standard BUILD_SHARED_LIBS cmake variable to control whether
to build dynamic or static libraries.
* Correctly handle the cmake variable VISIBILITY_MAP_FILE being set to
"" on the command line as a way to disable using the visibility map file
entirely.
* FindOpenColorIO.cmake changes:
- Prefer and use an exported cmake config for OCIO if found (is should
be present for OCIO 2.1+), unless caller set OPENCOLORIO_NO_CONFIG=ON.
- New hint OPENCOLORIO_INTERFACE_LINK_LIBRARIES lets caller force a
list of interface link libraries. This is helpful when purposely using
a static OCIO library, which itself has static dependencies, and you
need to name those specific library filepaths.
* Simplify some of the logic for setting extra link flag arguments for the
two main libraries.
* On Linux, also set link flags `--exclude-libs ALL` to hide symbols from
any embedded static library dependencies, to prevent clashes with other
versions of those libraries that may be compiled into a DCC app.
I realized that a call to make_texture() with the "maketx:ignore_unassoc"
attribute set did not properly set the configuration attribute up when
opening a new ImageBuf.
```include/OpenImageIO/typedesc.h:620:10: warning: 'const' type qualifier on return type has no effect [-Wignored-qualifiers]
auto parse(format_parse_context &ctx) -> decltype(ctx.begin()) const // c++11
```
Co-authored-by: Thiago Ize <thiago@solidangle.com>
The docs said that passing a tuple for min or max param of the clamp()
function gave per-channel values, but passing a single float gave one
value for all channels.
In fact, the implementation used the float to clamp only the first
channel. This patch fixes the behavior.
Fixes#3264
In addition to C++ code examples for each IBA function, work toward
also having equivalent Python code. Also, where applicable, show the
equivalent operation using oiiotool.
We had forgotten to mark --colorcount and --rangecheck as commands
that intentionally outputs to the console. This omission meant that an
oiiotool command consisting only of a those would warn about no
output. But we didn't really want it to.
How do you craft an oiiotool command that will examine a file and add
an alpha=1 channel if it's a 3-channel file, but leave the existing
alpha alone if it's already a 4 channel file?
This patch adds control flow, so you could express the above like this:
oiiotool in.exr --if "{TOP.nchannels < 4}" --ch R,G,B,A=1 --endif -o out.exr
The following slate of new fetures is added:
* `--set name value` sets a "user variable".
* Expression evaluation will expand identifiers that are the names of
user variables. For example, `{i}` is replaced by the value of user
variable `i`, and `{i+3}` returns 3 more than the current numerical
value of `i`.
* Expression evaluation has been extended to support the following
numerical comparison operators: `<` `>` `<=` `>=` `==` `!=` and
`<=>` (the last one being the 3-way comparison, like in C++20).
Also logical operators `&&` and `||`. All return 1 for true, 0 for
false. Note that the comprison and logical operators have the same
precedence as `+` and `-`, so use parentheses liberally to
disambiguate complex mixed expressions.
* New functions recognized within expressions:
- the value held by a named user variable: `var(name)`
- logical negation: `!`, `not()`
- string comparison: `eq()`, `neq()`
* `--if condition commands... [--else commands...] --endif`
* `--while condition commands... --endwhile`
* `--for variable range commands... --endfor` where the range may be,
like Python, any of `end` (implied begin=0, step=1), `begin,end`
(implied step=1), or `begin,end,step`.
Also needed to modify ArgParse so that it has the notion of whether
it's "running" -- actually calling the actions for each argument, or
if it's merely parsing but taking no action -- and to be able to set
and retrieve which command line argument index an action is responding
to.
This little change makes this work:
ImageBufAlgo::add (buf, { 0.2f, 0.2f, 0.2f });
instead of the more awkward
ImageBufAlgo::add (buf, cspan<float>({ 0.2f, 0.2f, 0.2f }));
and for anyplace tht has Image_or_Const parameter types.
New OpenColorIO and fmt versions are out.
* Bump "latest releases" test to fmt 8.1.0, and OCIO 2.1.1.
* Re-adjust CI test fmt versions across many tests to give a better
spread over the supported releases.
* Bump the default version to download and use if none is found to 8.0.0.
We had --line (wrapping IBA::render_line) and --box (wrapping
IBA::render_box).
This adds the obvious --point, wrapping the existing IBA::render_point.
The Python bindings for ParamValueList and ImageSpec allow a "dict"
style (associative array) interface:
imagespec["key"] = value
val = imagespec["key"]
This patch makes some minor improvements:
* Clarify in the docs that like a regular Python dict, the associative
retrieval will raise a KeyError if the key is not found.
* Add `__contains__` python method as a synonym for contains(), thus
implementing `key in dict`.
* Add `__delitem__` python method as a synonym for erase(), thus
implementing `del dict[key]`.
* Add an `ImageSpec.get()` method that is a synonym for getattribute
(which retrieves the item or returns None or a default if not
found), mirroring the behavior of Python's `dict.get()`.
Somehow I never implemented a copy() method for ROI, which of course is
super useful if you want to make an actual copy of an ROI and modify
the copy.
I also realized that ImageSpec.copy() unnecessarily/incorrectly set
return_value_policy to reference_internal, but actually it should be a
real copy, which it has to be anyway because the implementation
creates a separate copy.
* Docs: we previously only documented the "width=" optional modifier.
Add height, color1, color2.
* Change behavior ever so slightly so that if the "height=" modifier is
not supplied, the height defaults to the "width".
* Use the new pybind11 v2.9.0 for 2022 and "latest" tests.
* Bump the default version built by build_pybind11.bash to 2.8.1.
* Adjust the pybind11 version used in many of the other CI tests, to
give an even spread across versions, and to make each of the
"vfx platform" CI tests use the pybind11 version that was available
near the start of its repsective year.
For each ImageSpec ctr that takes a TypeDesc, add another one
that's the same but takes a string_view (which is just a simple
pass-through of the TypeDesc-from-string_view).
Also tagged Strutil and ustring::fmtformat as 'nodiscard', to produce
warnings if people call it and ignore the results. (As was the case in
my own bug that I tracked down in the course of this patch.)
When doing operations with filters that have negative lobes, you can
sometimes get visible ringing artifacts in very high contrast regions
of HDR images (you tend not to see these artifacts in images that have
range [0,1] because the ringing is very low amplitude unless the input
is very large).
You can do "highlight compensation" to reduce the artifacts when they
occur by doing --rangecompress, which does a log transform on the
input, then the filtered op, then --rangeexpand to transform back to a
linear space. But that makes for kind of verbose and clumsy command
lines, and makes it easy to err by doing the transforms in the wrong
sequence or omitting one.
The -otex command takes a highlightcomp=1 modifier, which does this
bracketing automatically, foolproof, with a compact notation.
This patch adds support for optional highlightcomp=1 modifier for all
of the oiiotool commands that let you specify a filter= option:
--rotate, --warp, --resize, --fit, and --pixelaspect.
* Clean up intra-document section references.
* New explanations about input and output configuration hints.
* More code examples in both C++ and Python.
* Fix some formatting flubs.
This was an obsolete experiment in adding a C binding. We abandoned
this effort (a replacement will happen eventually), but this one file
was left behind when we intended to remove it all.
Errors encountered while reading tiles were getting lost as we used
ImageBuf::Iterator to iterate over an image. Have the iterator keep a
flag for whether it encountered an error, set to true if the
ImageBuf::retile indicates an error occurred during the read.
Also, some ImageCache errors might get lost on the way to
TextureSystem calls, fix that as well.
TextureSystem is designed primarily for tiled images (typically 64x64
pixels each), but to handle untiled images, it just treats the whole
image as one big tile. That means for very large untiled images, you
might end up with one VERY large tile, and if the image is large
enough, the size and/or offset computations therein may overflow a
32 bit int.
This patch attempts to make all the offset computations in the texture
system be 64-bit safe.
After setting up the new IOProxy helper methods for ImageInput and
ImageOutput, here we go back to the readers/writers that had used
proxies for a long time, and switch them to using the new helpers.
Add two new global OIIO attributes: "limits:channels" and
"limits:imagesize_MB" which are imposed limits for validating a
reasonable number of channels and total uncompressed image size,
respectively. We presume that an image file header describing an image
that purports to exceed these is most likely corrupted or even
maliciously crafted. The limits, therefore, are designed to avoid
crashes, absurd memory allocations, or other cascading errors that
would result from taking such a claim at face value (for example,
a file that appears to have tens of thousands of color channels).
The limits default currently to a maximum of 1024 color channels, and
a maximum uncompressed data size of a single 2D image to 32GB. These
seem reasonable today, but if such large sizes become likely to be
seen in the wild in legit files in the future, we may change the
defaults. They may be adjusted now simply by setting the attributes,
of course, as should be done by applications that legitimately expect
to see input that exceeds them. (Applications also may make the
validation limits even smaller if they wish.)
I've modified the TIFF reader to check these limits as a first step.
We'll expand to doing such validation checks on other readers over time,
as we have opportunities to visit them.
This string utility computes the "edit distance" between two strings.
Currently it only uses the Levenshtein metric, but it has room for
future expansion by taking a parameter that can specify which metric
to use. Levenshtein is an algorithm for computing how many single
character insertions or deletions are necessary to turn one string
into the other.
I have a few ideas for where this can be useful, but as one
demonstration, I also modify the ArgParse class internals so that if a
command is not found, it will print a suggestion naming the valid
command option that has the closest edit distance (but only if one
choice is <= 2 edis away, and only if it's not a single-character
option, in order to reduce the number of useless suggestions). Here's
its behavior in action:
$ iinfo --stat foo.exr
iinfo error: Invalid option "--stat" (did you mean --stats?)
...
* The main event here is that we're now telling OpenJPEG to use
threading when compressing/decompressing JPEG-2000 files. This
improves the speed of JPEG-2000 reading on my laptop by 2-3x! The
threading feature was added to OpenJPEG 2.2 for decode, and 2.4 for
encode.
* CI: Build OpenJPEG 2.4 for the "latest releases" test, and build
their master for our "bleeding edge" test.
* Fixes to deal with the in-progress 2.5 deprecating a struct field.
* Generally clean up our detection of what OpenJPEG version we are
using.
* Some clean-up of the FindOpenJPEG.cmake file, use PATH_SUFFIXES.
Also fix longstanding bugs in PNM input:
* We weren't correctly emulating random scanline access, as is expected
(but in practice rarely used) of all ImageInput implementations.
* We weren't marking the color space in any way, now we do mark as
"Gamma2.2", which is the correct/preferred interpretation according
to the PNM spec.
* Add full IOProxy support to both input and output.
* This necessitated a major refactor, including sucking the I/O specific
code from rgbe.{cpp,h} into hdr{input,output}.cpp and getting rid of
the old files.
* Add a testsuite entry for hdr (shockingly, there was none before)
As well,
* Add a DDS test, apparently we never had one.
* Remove DDS setting of "texturetype", we don't ever use that, it's only
"textureformat" that we use. Not sure how that crept in there years ago.
Helper functions ioproxy_opened() and iotell() should have been const
all along, since they just return back information and should not
change state of the proxy, and also ioproxy() should have both const
and non-const versions.
* On Mac CI, use openvdb from Homebrew now that it's OpenVDB 9, which
properly supports OpenEXR/Imath 3.1.
* Update our FindOpenVDB.cmake target names to match the names used in
OpenVDB's own FindOpenVDB.cmake (even though we don't use their module,
may as well stick to their convention, especially if they eventually have
a properly exported config file).
For some reason, in the early days I got in the habit of having OIIO
input readers call their color space "GammaCorrected2.2" (for example)
when they appeared to be a straight-up gamma. (Generally only used for
certain fairly old formats that date from before sRGB existed.)
But there appears to be a widely used convention of naming OpenColorIO
spaces "GammaX.Y", so I'm switching OIIO to conform that
convention. It's more straightforward and increases the chances that
it will match the names of color spaces that already exist in people's
OCIO configs.
As we move toward fully equipping all image format implementations
with full IOProxy support, it's clear that a lot of repeated code is
error-prone. I first tried to collect this in a mixin class, but then
re-implemented after realizing that we can use the existing PIMPL
ImageInput::Impl and ImageInput::Impl classes, without changing any
ABI or making new headers.
This patch add a couple data members to the Impl data -- an
`IOProxy*`, (the current proxy, used for both passed-in and created
proxies) and also a `unique_ptr<IOProxy>` used when we have to create
our own and want to make sure it's destroyed with the ImageInput or
ImageOutput.
Then we add a number of protected member functions to ImageInput and
ImageOutput, not meant to be ever called by "users" of the OIIO APIs,
but used internally by II's and IO's to encapsulate several common
tasks related to managing IOProxy's. This coalesces redundant code
that would need to appear separately in most of the plugins that
support IOProxy, helps to reduce cut-and-paste errors from having to
make these changes separately in many places, and centralizes the
logic so that as our ideas of how to implement IOProxy evolve, we can
easily change it in just one place and have it apply to all plugins.
GIF revise with new IOProxy utility methods
JPEG simplify with new IOProxy utilities.
To try to combat long template instantiation times, did some refactoring
on the ImageBuf iterators:
* Pushed more functionality from the templated Iterator and
ConstIterator to the non-templated base class IteratorBase. This
should reduce the amount of code that is separately instantiated for
every data type, in every module that uses iterators.
* Pushed many implementations from inline defined in imagebuf.h to
separately compiled in imagebuf.cpp. The most important methods that
are called on every pixel are still inline, but we are switching to
separate compilation for all the methods that are only called upon
construction (once per image being iterated over) and most of the
code that is called only when we wrap to the next scanline.
* Some other miscellaneous refactor to remove redundant computation in
those classes.
Maybe somebody else is building this as a subproject or has a varible
called "CI"?
* For our CI runs, set an env variable "OpenImageIO_CI" (not just
generic "CI").
* Make sure STOP_ON_WARNING is true for our CI runs, even for release
branches -- but now check OpenImageIO_CI, not CI, in case somebody
else has set that.
This was deleted a few months back, but I think I did not realize
it still served a purpose. We don't use JasPer directly, but LibRaw
does, so if we are using static LibRaw, we still need it here.
Fixes#3209
When the make_texture option monochrome_detect is used to collapse 3
identical channels to 1, it was leaving the first channel name in
place (for example, "R"). That was confusing, because at this point,
we're not trying to represent a red image, we're trying to represent a
grayscale image. A strategic call to `default_channel_names()` clears
it up, imposing the standard "Y" we use for a single channel luminance
image.
Also amended the documentation of IBA::channels(). It turned out not
to be the way I called channels() that was the problem here, but by
the time I figured that out, I had already seen that its docs could be
improved.
Fixes#3204
* --onlyformat or env IMAGEINOUT_TEST_ONLYFORMAT to test only one format.
* --no-delete to preserve the temporary files for inspection
* other debugging checks
* Add a DDS test, apparently we never had one.
* Remove DDS setting of "texturetype", we don't ever use that, it's only
"textureformat" that we use. Not sure how that crept in there years ago.
The previously existing max and min take 2 images and compute the
pixel-by-pixel, channel-by-channel max or min of the two images.
The new maxchan/minchan take one multi-channel image and produce a
single-channel image where each pixel contains the max or min of the
*channels* of the corresponding pixel of the original image.
It is equivalent to this (for an RGB image):
oiiotool in.exr -ch R in.exr -ch G in.exr -ch B -max -max -o max.exr
Except, of course, that --maxchan is a single command, is much more
efficient, and doesn't require you to know the number or names of the
channels of the input.
This was a specific user request.
Note: in the course of making the tests, I noticed that we did not
have documentation or testsuite coverage of the python bindings for
IBA.min() an max(), although they were implemented. So I added that as
well.
* Make PROJECT_VERSION_RELEASE_TYPE be a cache string, so individual
build configs or sites can override it.
* Change OpenImageIO_IS_SUBPROJECT to PROJECT_IS_TOP_LEVEL (and with
the opposite sense), to match CMake 21+ behavior. And no need to set
it ourselves if using cmake 21+.
ImageBufAlgo has both premult and repremult (as well as unpremult).
But oiiotool only exposed unpremult and premult, but not repremult.
This little patch rectifies that oversight.
Just to remind everybody of the difference:
`premult` just multipies the color channels by alpha, like it says.
Note that for a pixel that has Alpha = 0 but nonzero color, this will
crush the color to black.
That's fine if you have an *original* image that's unpremultiplied and
you want to do a one-time conversion to premultiplied.
But there's another use case, which is when you start with a
premultiplied image, you need to temporarily unpremultiply it (like
for a color space conversion), and then re-premultiply it. In that
case, you may have those glowing pixels in the original image, the
unpremult will preserve them, and you DO NOT want to crush them to
black with the subsequent premult. So there is a "repremult"
operation that's like premult, but preserves the color values when
alpha == 0. This is the preferred operation when you are doing the
sequence of "unpremult, do something, re-premult" so that the glowing
pixels make it through the round trip.
When I implemented `oiiotool --invert`, I knew that the default should
be to only invert the first three channels (presumably R,G,B) and not
anything beyond, because we wouldn't want to invert the Alpha channel.
But I botched it, and instead what happened is that if given an RGBA
image, it would just leave the A black in the result image.
So this patch has two improvements:
1. If there are more channels in the source image than we are
inverting, make sure to copy the other channels unmodified, don't
just leave them blank.
2. Allow optional modifiers chbegin= and chend=, so users can change
the default if they want a different channel set to be inverted.
This new IBA/oiiotool command adjusts the saturation of a color image,
with a scale factor. Scale = 0 makes the image gray scale of
equivalent perceptual luminance, scale = 1 leaves color unchanged,
scale > 1 increases color saturation.
This adds a new cmake variable in our build system, TIME_COMMANDS,
that when set ON will adjust the compile and link rule launch to
include a time command. This is probably only useful when also setting
`CMAKE_BUILD_PARALLEL_LEVEL=1` so that build are happening serially.
The use case is to investigate how long each module takes to compile,
for optimizing the build.
* Add a test for MacOS 11
* Updated supported versions of dependencies
* Move linux clang test to clang13 on vfx2022 container
* Switch clang-format test to use clang12
In the early days of OIIO, it was 100MB. Ten years ago, we bumped it
to 256MB. I've heard a rumor that computer memories have grown in the
last 10 years, and so have the size of images some people deal
with. So it seems reasonable to raise it to 1GB.
`sscanf()` still has locale-dependent behavior that we wish to avoid
because it can lead to subtle bugs for people whose machines are
configured for certain locales.
This patch removes from the OIIO codebase almost every remaining
sscanf use (and definitely every one that could harbor locale bugs).
Instead, we use the rich set of parsing functions we've added to
Strutil over the years, which are locale-independent, and in most
cases are also faster than the standard C/C++ equivalent.
The official names we use are "box2" and "box3", but users of Imath
know these as box2f/box3f, so accept these as synonyms when
constructing a TypeDesc from a string.
* Typo in webp section -- said "gif:LoopCount" when we meant
"webp:LoopCount".
* In the code, did not actually set the "oiio:LoopCount" metadata
that the docs said would be set.
* Opportunistic fix: TIFF docs said we only suported IOProxy for input,
but we have supported it for output for a while now.
* `parse_values()` and `scan_values()` both can parse a series of
values in a string, with optional prefix, suffix, and separator. The
scan versions just figure out the values, the parse versions
optionally can update a string_view& like the other parsing
functions in Strutil.
* `scan_datetime` parses the int components of a string that looks like
"YYYY:MM:DD hh:mm:ss" (and a few minor variants).
* Fix stoui -- it was previously casting stoi results, which is not
correct.
* Add `from_string<>` specialties for `double`, `int64_t`, `uint64_t`.
When outputting to formats that don't support negative pixel origins
(such as JPEG), we crop away the negative areas. But we did not
consider the case where the entire data window was in the negative
region!
TIFF has an absolute limit of 4GB file size due to use of uint32
offsets within the file. If you try to write a big enough TIFF image,
it will fail. This is not a software problem, it's a limitation of the
original TIFF file specification itself.
The underlying libtiff supports a nonstandard "bigtiff" extension that
uses 64 bit offsets. This patch switches to this automatically for
images whose uncompressed size would get too close to (or exceed) the
4GB limit. Any such files can be read by OIIO-based software, as well
as anything else using a relatively modern version of libtiff, but may
not be readable by independent TIFF reading implementations.
This is a mixed bag, though. On one hand, this at least lets us write
files that would have been an error before. But note that some files
that would have worked as classic TIFF before -- because their
*compressed* size was still under the 4GB limit, even if their
uncompressed pixel size was not -- will now be written as bigtiff
because we have to commit at the time we open the file, before we've
seen the pixels and know the final compressed size. We just can't
know for sure, for an image with > 4GB of pixels, if the file will get
lucky and be below the limit by the time the scanlines or tiles are
compressed. So for those files, not all apps will be able to read
them (though anything using libtiff >= 4.0 or OIIO will be fine).
Fixes 3156
When ImageInput::create() or open() is called, it first uses the
extension of the filename to guess which reader plugin to try (for
example, "foo.jpg" will try the jpeg reader). If that fails, however,
the default behavior is to then try EVERY reader to see if any will
open the file, in case it guessed wrong, or the extension is used by
more than one format, or the file was simply given the wrong
name. Thus, ultimately, OIIO will never fail to open a valid image
file in a known format, even if the file is wildly misnamed. However,
it's a lot faster to open a correctly named file, because it will try
that one first, before trying a long succession of readers in the hope
that eventually one will work.
There are three cases where this is not the desired behavior:
1. You are confident that files are named with the correct extensions,
and you want a corrupt or invalid file to fail QUICKLY, without trying
every format reader.
2. If the file is corrupt, rather than see a generic "Could not find
any format reader that will open the file" because they all failed,
you want to see the more specific error message for why the (presumed)
correct reader could not succeed.
3. When doing regression tests to make sure that sample corrupted
files are correctly reporting the right errors, we really do want to
see error messages from that one reader we are testing.
This PR exposes a global OIIO attribute, "try_all_readers", which
defaults to 1, but when set to 0, changes the behavior so that ONLY
the one format implied by the file extension is tried, and OIIO does
not subsequently try every known reader.
This new oiiotool command lets you set global OIIO attributes, i.e.,
making calls to OIIO::attribute(). It allows oiiotool command lines to
set some additional OIIO-wide options for which no specific oiiotool
command flags could otherwise access.
When building against OpenEXR >= 3.1.3, exr output now allows zip/zips
compression to take a quality metric (such as compression
"zip:4"). The range is from 1-9, 4 is the default, and it's a control
over speed of compression vs compression ratio (higher number is more
expensive but may result in smaller files). This only affects
compressed output -- it's lossless and will be read to the same pixel
values upon input.
This is also incentive to upgrade to OpenEXR >= 3.1.3, because its new
default (and ours) is 4, versus the old (implied, not directly
controlled) 6. This change was made because compared to 6, it's
significantly faster to write exr files with zip compression 4 (tens
of % faster), while only increasing file size by 1-2%, so that seemed
like the right tradeoff. Lower numbers make much larger files for
little speed gain, and larger numbers than 6 make for much slower
writes of exr files for very tiny compression improvements. But YMMV
and you may want to experiment to find the right trade-off for your
use cases.
OIIO 2.3 marked it as deprecated. This new change is for master (2.4+)
only -- removing all support for Field3D.
F3d is effectively unmaintained, is known to be incompatible with
the latest versions of OpenEXR/Imath, and we haven't found anyone
prepared to speak up as a user of it via OpenImageIO support.
One more boost extraction: tokenizer.
The only place we used it is Filesystem::searchpath_split, which can
easily be rewritten in terms of the Strutil parsing functions.
While I'm at it, add a new version that returns the vector instead
of needing to pass in a ref.
Suggested by Luke Emrose, the idea here is that we introduce two new
MIP interpolation modes: StochasticTrilinear and StochasticAniso.
These work similarly to Trilinear and Aniso, respectively, except that
instead of sampling both of the bracketing MIP levels and blending
between them, in stochastic mode we use a random variate to select ONE
of the two MIP levels with probability proportional to the computed
blending weight.
If these mip modes are used, the caller needs to pass a
well-stratified value in the new "rnd" field of TextureOpt. Also note
that the method only really makes sense in the context of a renderer
that is using many samples per pixel. As a single texture lookup, it
looks horrible and noisy, as you can see by the testshade-based tests
we do of this, which are 1spp and looks pretty rough.
My benchmarks (in a hard stress test of the texture system using
`testtex --threadtimes 8`) show that this approach has speedups of
35-40% when using between 1-4 threads, dropping to more like 25-30%
speedup as threads are in the 12-32 range (that makes sense, it's
doing the same amount of total I/O).
In this first implementation, in order to preserve ABI compatibility
if we decide to backport this feature to the release branch,
TextureOpt::rnd is added as a union co-member with the unused "bias"
field. For TextureOptBatch, we had to add it outright, but it's
guarded by an `#if` and will not be backported to 2.3. (We may change
our minds and not backport it at all, I just wanted to leave the
option open with this initial patch. Also, there will probably be
additional further ABI-breaking changes coming later that will
definitely not be backported.)
In this first phase, we only use the stochastic component to turn the
2-MIP-level lookup into one level lookup. It has been further
suggested that we could apply stochastic techniques to anisotropic
filtering (instead of many samples spaced across the major axis of the
filter footprint, take one sample distributed across the axis), and
even for the ordinary bilinear or bicubic within each level (instead
of 4 or 16 weighted texels, just retrieve one texel chosen with the
weights as probabilities). But these deserve to be benchmarked and
have their quality evaluated separtely and deliberately, and so will
be relegated to a later investigation and implementation.
We will have a C API eventually, but it will be implemented
differently, and this cruft will probably get in the way. If the new
code needs any of this (particularly the test cases), it's easy enough
to revive it from the project history).
When requesting that raw image be returned as "ProPhoto" color space,
we were using the wrong gamma. Also, at user request, add
"ProPhoto-linear", which is the same but with linear response. (This
is analogous to the existing "sRGB" and "sRGB-linear".)
Fixes#3152
Thiago Ize points out that TextureSystemImpl::fill_gray_channels
assumes that the `float*` addresses it is passed are big enough for
a vfloat4, and also properly aligned as such. Not true.
This is an edge case anyway, only called in particular cases where
we are doing 3/4 channel lookups on 1/2 channel files with the option
turned on to expand single channel gray to RGB gray, and also doesn't
do enough SIMD to have a perf impact. So just recode as scalar and
have no more alignment mismatches to worry about.
Also make sure at least one of the Windows CI tests is explicitly
turning SSE on.
Fixes#3128
Field3D is essentially unmaintained, and is currently incompatible with
OpenEXR/Imath >= 3.
In this first stage (which will apply to OIIO 2.3):
* By default, Field3D support is disabled. To enable it, you must
invoke CMake with `-DENABLE_FIELD3D=1`. No warning or other messages
are printed by the build configure step when not enabled.
* If requested, but an incompatible OpenEXR is found, it will be disabled
with a clear error message.
* If requested and enabled, a warning will be printed indicating that
Field3D support will be removed from future versions (2.4 and beyond).
* Add a new test, texture-texture3d, that provides the same testing that
texture-field3d did, but with openvdb.
We have options that allow adding random dither to images before
output, which when converting float images to 8 bit or less, can
greatly reduce the appearance of banding artifacts. This PR makes a
number of improvements:
* When the option is used, it should apply dither any time you are
reducing an image with > 8 bits/channel to an output with <= 8
bits/channel. Not just from float/half, but also from uint16, for
example. (When outputting > 8 bits per channel, there are enough
gradations that you tend not to see banding artifacts, so dithering
is not necessary for 16 bit output, even if coming from a float data
source.)
* Change the actual dither we use to blue noise. The old dither just
used a hash to add uniformly distributed random numbers to each
pixel value before quantization, completely uncorrelated from pixel
to pixel. Blue noise is specially constructed to still have a
uniform distribution of the output range, but it ensures that
spatially, similar values are distributed as far away from each
other as possible, rather than having similar values sometimes
clumped together. The result is a MUCH NICER appearance.
* Of course, this means the we needed to store a blue noise table in
the library, and some access routines for it. We incorporate a
public domain blue noise texture as a table.
* ImageBufAlgo::noise() and `oiiotool --noise` now take "blue" as a
noise name to return blue noise. Also, make "white" the (preferred)
synonym for "uniform" (to disambiguate the fact that both white and
blue noise are uniformly distributed; the difference is that white
is independently random for each pixel, whereas blue shapes the
spectrum to favor high frequencies, making it better suited for
dithering and sampling purposes.
* ImageBufAlgo::bluenoise_image() returns a reference to a stored
periodic blue noise image.
* oiiotool -d would let you say "uint10" or "uint12" to indicate 10 or
12 bit output (for formats that support it), but had no way to ask
for less than 8 bit output (for example, TIFF supports that). So we
add "uint6", "uint4", "uint2", "uint1" as recognized names for the
`-d` option in those rare cases when you specifically want to
produce a low bit depth file. And furthermore, if you ask for
`--dither` when using lower bit depths, the dither amplitude is
adjusted to be right sized for the bit depth you are using.
* Many format output modules that support 8 bit output honor the
option to dither the output, but in the "built-in plugins" chapter
of the docs, most did not list this option in the section that
describes which special output options it knows. So now we've added
this to the docs for all the output plugins that did support it.
Turns out that heif files that contain alpha can have either
associated or unassociated alpha. We had been assuming
associated. With LibHeif 1.12 and beyond, you can ask, so now we
do. Thanks Frédéric Devernay for pointing this out.
Similar to other formats that optionally have unassociated alpha:
* The default behavior is to associate/premultiply automatically when
reading, but set metadata "heif:UnassociatedAlpha" to indicate that
it was unassociated in the file.
* Open configuration hint "oiio:UnassociatedAlpha", when nonzero,
means to please keep the data unassociated and not premultiply. In
that case, metadata "oiio:UnassociatedAlpha" will be set in the
ImageSpec to indicate that the returned image (not just the file)
consists of unassociated values.
I did a couple other minor docs touch-ups while I was at it.
Fixes#3129
New ":C=name" modifier for --dumpdata will cause the dumped image data
to be formatted with the syntax of a C array, that can be dropped right
into source code.
$ oiiotool --dumpdata:C=foo image.exr
// image.exr : 256 x 256, 4 channel, float openexr
float foo[256][256][4] =
{
{ /* (0, 0): */ { 0.517036676, 0.261921108, 0.017822538, 0.912108004 },
/* (1, 0): */ { 0.653315008, 0.527794302, 0.359594107, 0.277836263 },
...
};
Without this modifier, --dumpdata will keep the same formatting it
always had.
This works pretty well for ordinary boring flat, single subimage,
non-MIPmap images. The option should probably not be used (or at
least is very unpolished) for multi-subimage, MIP-mapped, deep, or
other oddball cases (but you probably don't want to shove those into
a single C array anyway).
We had long ago established that the one in fmath was a slightly
faster formulation, as well as having the property of squashing
NaNs. There were a few remaining places where we used a different
clamp so fix them.
The DSLR raw format reader has the ability to pass hints to the
underlying LibRaw requesting that the pixel data be returned to us in
particular color spaces.
* Add new supported color spaces "DCI-P3" and "Rec2020" when using a
LibRaw new enough to understand them -- which means the upcoming 0.21.
* A new name to request the data be returned using sRGB color
primaries but a linear response curve: "sRGB-linear".
* The old name we used for sRGB-linear, "linear", is now considered
deprecated since it's ambiguous -- since its first use, new color
spaces "ACES", "DCI-P3", and "Rec2020" have been added, all of which
are also linear response.
When reading tga files, make sure that every seek and every read
checks results and issues error messages. Also double check that
large allocations succeed.
Also improve error propagation in several spot of iinfo, oiiotool
Owing to our C++11 focus in previous years, we had a number of places
were constants were declared as `static constexpr`. When done in
headers, this potentially causes separate copies of those constants to
be instantiated in each module that includes the header. It's not
really harmful the way we're using it for small constand objects, but
it's wasteful to potentially have dozens of copies of each those
constants by having it repeated in each module.
C++17 has a new idiom, `inline constexpr` that is more like we want --
makes it a true compiler constant that doesn't get instantiated in the
memory of each module.
We still have to cater to C++14, and in platform.h we already had a
C++-standard-dependent definition of OIIO_INLINE_CONSTEXPR that was
almost correct. Fix it (it wasn't previously used, but I must have
seen this coming at some point) so that it expands to
`inline constexpr` for C++17 and beyond, but stays as
`static constexpr` for C++14, and then use it for these constants.
Fixes#3117
If think that our inventory of UDIM tile files won't find names with
wildly out-of-range tile indices, you haven't been working in a
production studio! When those indices imply tile numbers < 0, we were
running off the beginning of the vector when filling in the udim
table, to hilarious results.
Oiiotool::extract_options was kind of fishy and brittle before --
splitting the string at ':' characters, then split each piece (which
should form a `name=value` pattern) at the first '=' character. There
are probably many malformed option strings that would have had weird
results with the prior naive parsing method. And it has been noted
that it can't handle a value that is intended to be a Windows filename
(like "C:/foo"), nor a searchpath (like "/usr/bin:/usr/local/bin").
Rewrite to parse it more carefully as name=value pairs delimited by
':'. In the process, we now have the ability to have the value be
enclosed by a single or double quoted string, in which case the
contents of the quote may themselves contain a colon, for example, the
following command and modifier is now no longer a problem:
`--foo:filename='C:/blah':count=3:`
We were already passing this on to libtiff when it handles the
compression. But for the case where we were calling zlib ourselves
(to facilitate multi-thread compression, which libtiff doesn't do),
we were just using the default compression.
This is inspired by a great [blog post](https://aras-p.info/blog/2021/08/05/EXR-Zip-compression-levels/)
by Aras Pranckevičius in which he made a similar change to OpenEXR
and benchmarked the results. He reported approximately a slightly more
than 2x write speed increase, while getting files that were only around
2% larger.
I tried replicating his experiment for TIFF, and while I don't quite
get results as good as he did, it still is really interesting. I
tested with 100 images randomly selected from the [Animal Logic Lab
USD scene](https://animallogic.com/usd-alab/) (they are almost all
4k^2, a mix of 1- and 3-channel images), converted from half exr to
uint16 TIFF images. I ran the tests for both scanline and tiled
images.
Compared to the default zip level 6, using level 4 resulted in 3%
bigger files, and around 40% faster write times. That seems like it
is a potentially very helpful tradeoff.
I tested the same set of images also with zip levels 8 and 2. For
level 8, the files were only another 2% smaller than the default level
6, but the write speed was 45-50% slower, so there is little value to
asking it to work harder on compression -- it just burns up time. For
level 2, it went somewhat faster (maybe another 10% faster than level
4) but the files started getting bigger, level 2 was 12% bigger than
level 6, that's definitely getting into territory where people might
mind the larger files.
So I want to add the control now, though I'm deferring a decision on
changing the default value until I can do some more extensive
tests. In particular, I would like to assess how it behaves with 8 bit
files (I was testing uint16, specifically), how well it works with
rendered images and with natural/photographic images (I was testing a
set of textures, which may not have the same characteristics), and
perhaps also see if results are better or worse for lower resolution
images.
The old libOpenEXR/libIlmImf (default) implementation handled
multithreading on the exr side of things, inside the readPixels method.
The new (experimental, enabled only by option) OpenEXRCore
implementation leaves the threading to the caller, and we weren't
doing it, so the performance was unsurprisingly awful.
This patch turns the implementations of the exrcore-based
read_native_scanlines and read_native_tiles (and also the deep
versions thereof) to use parallel-for loops, restoring parallel
processing to the exr reader when using OpenEXRCore.
Reading of whole scanline and tiled images now is on par or slightly
faster (a few %) than when using the main OpenEXR library. Reading of whole
scanline and tiled deep images is substantially faster (2-3x!) than
before.
I did some extra refactoring while I was there, simplifying and
eliminating some redundant error handling and using a new
DecoderDestroyer utility class.
STILL TO DO:
* See the long comment at the end of read_native_scanlines -- the error
message strategy implemented here is temporary, I'd like to improve it,
will probably involve both some work here as well as on the openexr
side.
If set to ON, will include the testtex program in the install.
The default is OFF, which builds it for tests, but doesn't install it
with the real command line utilities.
In forward looking code (current master means OIIO 2.4, which won't
be released until mid-2022), get rid of some stray logic catering to
OpenEXR releases 2.2 and older.
Background: Old style CMake was to use FindFoo.cmake modules, which
set FOO_INCLUDES and FOO_LIBRARIES variables. Best new cmake practices
are for the upstream packages themselves to export a FooConfig.cmake
and friends, which set up cmake targets. Of course, many packages don't
provide exported configs, so we use the old Find modules for many of
our dependencies.
I've slowly been migrating our Find modules to set up targets in addition
to the variables, and favoring use of the targets over the variables.
But I was wrong about this. It's more than just a choice of style.
When targets are used, use of transitive linkage (especially for static
library versions of dependencies) make the dependency targets show up in
our own exported OpenImageIOConfig.cmake. But to a downstream package,
now our config talks about a target like Libsquish::Libsquish, but it
doesn't know what that is, and it doesn't have the FindLibsquish.cmake
module that we used (internally) when we found that dependency.
This was causing particular problems for Libsquish on Windows with
vcpkg, but I see now that this could be a problem elsewhere as well:
field3d, libheif, openvdb, and webp.
So this patch:
* Reverts from the ill-advised targets back to the old style
variables, for the dependencies where we use *custom* Find modules.
(We presume that a dependency's exported configs, as well as any
Find modules that are distributed as part of CMake itself and define
targets, are done correctly, and will be available to downstream
packages, and so we continue to use targets in those cases.)
* Beefs up our FindLibsquish.cmake module to correctly find debug
versions of the library, to use in debug builds.
* Alters build_webp.bash to allow WEBP_BUILD_TYPE to override the build
type, which had previously been hard-coded to Release.
* Removes search for HDF5, which as far as I can tell was unused
directly by OIIO.
Instead of a build-time option for enabling (and always using) the
experimental OpenEXR "core" library...
* Build support for it by default any time we're using a new enough
OpenEXR version (>= 3.1). You can still hard-disable it being compiled
in at all with `-DOIIO_USE_EXR_C_API=OFF`.
* Select whether it's actually used with the global OIIO attribute
"openexr:core" (default to 0, disabled, for now).
This easily lets an app select support at runtime, and also may be
selected by user with environment variable
`OPENIMAGEIO_OPTIONS="openexr:core=1".
This all is intended to make it easier to benchmark use of this new
feature, switching back and forth without a rebuild.
* Honor env $OPENIMAGEIO_FONTS as an additinal font searchpath.
* Global attribute "font_searchpath" as an additional font searchpath.
* Refactor and simplify the code that assembles the list of directories
to search.
When $OCIO was not set at all, the call to OCIO::GetCurrentConfig()
was checking it again and writing the info message that none was found
to stdout, which we really don't want. And as Patrick pointed out
recently, asking for the "current" config is suspect, since who knows
exatly what the app has done.
So we're making the following executive decision: At least from OIIO's
viewpoint, if you want an OCIO color config, you either have to name it
expliticly, or have it determined by the `$OCIO` env variable. That makes
cleaner logic, and also rids us of the info messages.
I also now have the exported cmake config set
`OpenImageIO_HAS_OpenColorIO`. This makes it easier for downstream
consumers of our exported OpenImageIOConfig.cmake to know if the OIIO
build they have found has build-in support of OpenColorIO.
When it comes time to read_image() into the ImageBuf's internal
memory, we should have said to convert to `m_spec.format`, which
describes the buffer we allocated. Instead, we asked for `convert`,
which is the user request, which at that time can still be UNKNOWN,
which indicates to preserve the file's data format, which in fact does
not necessarily match the internal buffer.
This becomes especially troublesome for images with mixed per-channel
data types, which will obviously never match the internal buffer since
ImageBuf only internally stores one data format for all channels.
This is a very low priority change, but every little bit helps when it
comes to limiting boost's footprint as thing we used to require it for
become available in the standard library.
Add tests to be sure we have reasonable error reporting for at least
two important cases: a specified OCIO config file is not found at all,
and an OCIO config is present but cannot be read (in particular, if
the config is newer than the library). Because the exact error spewage
is different for each OCIO version, and apparently also depending on
the exact version of the yaml library that OCIO used, we have a few
different possible output references.
* oiiotool --nostderr (new hidden command)
This hidden flag causes oiiotool errors to go to stdout instead of the
default stderr. It only does this for oiiotool's own error and warning
calls; if there are other places within library functions that
directly output to stderr, they will not be redirected by this
command.
The main purpose of this is for testsuite and debugging. I don't
expect ordinary users to have any use for this, and so it is currently
hidden from the oiiotool --help explanation. If it seems useful for
something other than testsuite, I'll expose it in oiiotool --help.
It seems that for the case of OCIOv1 reading a v2 config, OCIO itself spews a bunch of warnings to stderr. And apparently the order is nondeterministic, because I just could not get any set of reference error output to match stably from run to run in CI. But the problem is, if I don't capture stderr, I also don't get the oiiotool error messages, which are the very thing I'm trying to test!
So what I did is add a new flag to oiiotool, --nostderr, which causes oiiotool's error and warnings to get sent to stdout rather than stderr. This lets me capture the usual output and the error messages from oiiotool, for this test, while allowing OCIO's stray stderr output to go into the ether where we don't care about its order.
I'm not sure if there's any non-CI, non-debugging application for oiiotool --nostderr, but maybe it'll turn out to be generally useful.
When OIIO that was built against OCIO 1.x, but encounters an OCIO 2.x
config file, OCIO could throw an exception that was not caught by
OIIO. The problem was the call to OCIO::GetCurrentConfig(), it should
have been inside the try/catch block.
Calling close() in open will forget too much state. All we want to do
here is make sure the TIFF file itself is closed at that point (so it
can be re-opened in append mode to add a subsequent subimage).
* CI test for VP2022, fix revealed python binding bugs
* CI test against ASWF's latest VFXPlatform-2022 docker container,
including Python 3.9.
* Pesky crash resulted in testsuite/python-imagebufalgo, laboriously
traced to IBA_color_range_check releasing the GIL too liberally. The
scoped release cannot contain the tuple construction!
* Audited the rest of py_imagebufalgo.cpp and found the same mistake in
only one other place: IBA_histogram. Was asymptomatic so far, but fixed
now.
* Presumably we just got lucky and these bugs were never symptomatic
until we started building against Python 3.9.
* Noticed that the recent change in build tree layout where we now
copy the test images into the build/testsuite area, resulted in HUGE
GHA build artifacts for failed tests. Modify ci.yml use of
upload-artifact action to exclude those directories.
* more tweaks
oiiotool --runstats tries to report how much time was taken by each
operation. But it has always had a major flaw: Because we try to read
input files lazily and use ImageCache underneath, only in certain
circumstances do the true cost of reading the files show up in the
entry for "-i"; often the cost gets wrongly credited to the other
command (like --add or -o) whose operations triggered the I/O.
This patch addresses the problem, and I think correctly credits I/O
time where it belongs.
The workhorse here is a new helper class within oiiotool,
OTScopedTimer, which does some tricks to try to account for both
direct read time and ImageCache I/O time, subtracting those from the
op it was instructed to time and instead adding it to the counter for
"-i". Because the destructor apportions time to the right commands,
this eliminates a whole bunch of spots where we did that by hand.
Also fixed a few places where ops directly called ImageRec::read()
instead of Oiiotool::read(), which would also result in bypassing some
timers we relied on. I made the ImageRec one be private, so harder to
make that mistake in the future.
Use a cmake idiom so that you need an explicit `-DCMAKE_INSTALL_PREFIX=`
in order to write over /usr/local or any potentially sensitive area.
In the absence of this being set explicitly, it will configure the
cmake build so that an "install" ends up in ./dist (which is what the
Makefile wrapper did all along).
Before doing all the work of copying the image, first see if the
`--ch` is actually specifying any change versus the image as it
stands. If not, skip all the copying and just stick to the image we
have.
The main benefit here is if you have a --ch primarily for "conformance",
e.g. you want to reduce to a known subset of channels and remove any
extra that may be unanticipated in the input, but it's possible that
the input will come to you already with only the desired channels. As
an example,
oiiotool in.exr --ch R,G,B ...more ops... -o out.exr
Previously, if in.exr aleady had channels R,G,B (and no other), there
would be some unneceessary expense of creating (in memory) a new image
for the output of --ch copying data to fill it in. With this patch,
the --ch would very nearly turn into a no-op. In both cases, if
in.exr's channels were something other than R,G,B, channel copies
would need to be done identically in both old and new code.
I realized after this was merged that I often share and copy a lot of
these little dependency build scripts across multiple projects, so I
wanted to remove the OIIO-specific naming.
To help make testsuite pass even with nonstandard relative path to the
build area, slightly change how we reference the OCIO config that is
in the testsuite.
Fixes python error about OIIO_TESTSUITE_ROOT not defined in the case
where the testsuite dir wasn't found in the standard places. This can
happen if you put the build directory in an unusual location that's
not right in the main project area.
* Ensure that "CMAKE" built "OpenCV" libraries can be found.
* Fix incorrect url to "openexr-images" repository.
* Fix typo in "FindTBB.cmake" module.
Co-authored-by: Thomas Mansencal <tmansencal@wetafx.co.nz>
* New SETVARIABLES option sets named cmake variabes to true if the
package is found.
* DEBUG should print variables even if not found. Duh, failing to find
the package is the most important debugging situation.
For historical reasons, our wrapper makefiles assume that we build in
`build/ARCH[.variant]`, so maybe build/linux64, build/linux64.debug, etc.
I think this is a throwback to literally decades ago when I would
build multiple architectures from different machines that viewed a
common shared directory of code, this allowing them to build
simultaneously and/or erase one without needing to rebuild the others,
etc.
But now I find that:
* It's contrary to modern cmake practice, and there are many benefits
to having our build directory structure identical to the vast majority
of other open source projects.
* I never want or need those simultaneous builds in practice, haven't
for a long, long time.
* ccache means that there is no longer an onerous cost to, for example,
having to blow away a release build and do a debug build and then
blow that away and do release again. It's just a few seconds to do a
ccache-based full rebuild of a recently-built tree.
So this patch switches all our assumptions to having a single level
hierarchy to the build and dist areas.
Also, I'm changing the assumption that the oiio-images (and other data
sets used by the testsuite) live "up a level", but instead added code
that either copies it from there (or another area settable with the
cmake variable OIIO_LOCAL_TESTDATA_ROOT) or downloads it, into the
build/testsuite directory itself. This simplifies a great number of things
in the test scripts as well as the test reference output (a lot of which
changes slightly to accommodate the new paths to test files).
* raw: Fixed output when raw:Demosaic == "none"
* raw: Include metadata regarding filter pattern
* raw: Added missing call to set unpacked status
* raw: Implement "balance_clamped" mode to help HDRI processing
* raw: Implement raw:apply_scene_linear_scale feature
* Fix up formatting with clang-format
* Update testsuite reference output
* raw: Fixed error message format handling
* raw: Updated documentation to include new raw features
* Ensure that all the `build_*.bash` scripts don't download anything if the packages appear to already be there.
* Env variable `OIIO_DEP_DOWNLOAD_ONLY` can be used to do the download only (if not present) and skip the actual build.
This combination allow the scripts to be run twice: once to do all the network I/O to download the packages we'll need, and a second time to trigger the builds.
The existing --info and --stats seem a little confusing and
inflexible. Unlike the majority of oiiotool arguments which are
evaluated in order, these merely set flags which apply to the entire
oiiotool command (specifically, they cause all image inputs to print
the metadata or image statistics at the time that the input are read
from disk). There isn't a way to make it apply to a single image, nor
to print the metadata or stats of any *computed* image, leading some
people to awkwardly save a computed image and re-read it merely to
print information about it.
So in this patch, we add new commands
* `--printinfo` which is like `-info -v` but is done immediately for
the "current" image on the top of the stack (even if it's a computed
image and does not correspond to any disk file).
* `--printstats` which is like `--stats` but is done immediately for
the current image on top of the stack.
Furthermore, `--printstats` takes an optional modifier `:window=`
which can be given a rectangular region in either of the notations
we use ("WxH+X+Y" or "xmin,xmax,ymin,ymax"), for example:
oiiotool blah.exr --printstats:window=30x30+1000+500
prints the statistics of the 30x30 region whose upper left corner
is pixel (1000,500).
The code to implement this is actually pretty short and straightforward,
but this patch turns out to touch a lot more. I needed to plumb
some of the info-printing internals for this support and did some
refactoring along the way.
Also, upon adding these to the docs, I realized that it would be
better to reorganize a bit to separate descriptions of the "flags"
from the "actions", and so many of the oiiotool arguments got shuffled
between different subsections, and in a few cases the descriptions
themselves got rewritten for clarity.
To help applications that need to deal with UDIM textures, the
TextureSystem now exposes the following public API calls:
* is_udim() : is a filename (or a `TextureHandle*`) referring to a
UDIM pattern / virtual file?
* resolve_udim() : given a filename (or handle) and s,t coordinates,
return the handle for the concrete file corresponding to those
coordinates.
* inventory_udim(): retrieve the full list of concrete filenames, as a
vector indexed by `(utile + vtile * nvtiles)`, and the nutiles,
nvtiles counts. Empty filenames in the vector indicate UDIM tiles
that are not populated.
Some extra functionality falls out naturally from these and existing
ingredients:
* inventory_udim + get_texture_handle together can get you the list of
handles corresponding to each found udim tile.
* resolve_udim + get_texture_info together can retrieve the attributes
of the udim tile corresponding to a particular texture coordinate.
Additionally, I have done some refactoring and performance
improvements of the internals:
* The internal UdimInfo that's stored inside the ImageCacheFile record
is now a vector, not a flat_map, so conversion of texture
coordinates to a handle is now O(1) rather than O(log n).
* Per Luke Emrose's suggestion, use `rcontains()` when searching for
UDIM pattern markers, since they will almost always occur close to the
end of the full pathname.
* Added TextureSystem and ImageCache methods that lets you find out
the filename that corresponds to a handle. (That's bound to be handy
for somebody.)
* Beefed up the udim tests to be sure to test every udim notation that
we support.
* At the suggestion of Luke Emrose, add rcontains() and ircontains(),
which are like contains/icontains, but start the search at the end
of the string rather than the beginning. This is measurably faster
than contains() for the case when you are reasonably certain that if
the search pattern is found at all, it is highly likely to be close
to the end of the string. (They're approximately the same speed when
the pattern is not found.)
* To verify this, I added a bunch of benchmarking to strutil_test.
* The benchmarks also revealed that contains was about 2x slower than
calling find! It should be the same. I don't know what the underlying
boost::algorithm::contains is doing differently than find, but whatever.
Changing our implementation of Strutil::contains to simply call find
doubles its speed.
* The new implementation of contains has an extremely minor behavior
change for the degenerate case of `contains("", "")`, which used to
return true and now returns false. Since the right behavior here is
debatable at best (is it false because the first empty string
contains nothing? or is it true because containts(a,b) should return
true if a==b?), I'm going to accept this behavior change as well
worth the 2x speed improvement (besides, "contains(a,b) should have
the same result as find(a,b)!=npos" is also a reasonable
expectation).
* The benchmarks also reveal that the case-insensitive functions (like
ifind) are not just a little bit slower than their case-sensitive
siblings (like find), they are 20x or more slower! I don't think
there is anything to do about that, but I did note it in the
documentation of those functions so that we are disuaded from using
them except when a case-insensitive test is really necessary.
The primary issue being addressed here was that get_texture_info() on a
udim pattern filename (like "foo_<UDIM>.exr") was unable to retrieve
info, except for an extremely narrow range of queries, because of course,
different UDIM "tiles" might have different attributes or be missing.
So in consultation with Lee Kerley, we decided that get_texture_info should
work as follows:
* If the attribute exists and has the same value for all of the
matching UDIM file files, then succeed and return the value.
* If any of the tile files does not contain the attribute, or if the
attribute doesn't have the same value in all found files, then it's
a failure and the value cannot be retrieved.
This necessitated a more rebust accounting of all the tile files
comprising a UDIM pattern, by doing a full inventory the first time
the UDIM pattern is encountered, rather than (ick) iterating through
all possible tile files every time we retrieve an attribute, as we
were apparently doing before. And so the following improvements were
also made:
* get_texture_info now works for more queries -- in fact, any query
that would be possible to ask about on an individual file, without
restriction. (Though, as I said, the query will fail if the value
isn't identical across all the found tile files.)
* The info queries (even the few that were previously supported) will
be MUCH FASTER than before, because we are doing the tile file
inventory once per udim pattern, rather than on every query.
* That inventory will also pre-build the association between u,v tiles
and the concrete filename (and texture handle!), so that each
individual texture call does a simple map lookup to get all the way
to the texture handle, without any string manipulation to first
construct a concrete filename, then look that new filename up in the
cache to get a handle, etc. I'm expecting this to reduce some of the
overhead of using UDIM files with the TextureSystem.
Note: I added an ImageCacheImpl::find_file_no_add at some point in
this work, eventually not needing it by the time I got to the final
version of my implementation of this PR, but I'm going to keep it
because I can forsee its usefulness and I don't want to delete it only
to write it again some day.
We now think that this C API should be removed, and replaced by a
different one coming soon.
Merely commented out the option and other lines in the top-level
CMakeLists.txt, since almost the exact thing will be needed eventually.
Also kept libOpenImageIO-C/CMakeLists.txt since we'll eventually need
one much like it. But deleted the .cpp and .h files, since those will
be totally replaced.
For the upcoming OCIO 2.1, the library names will incorporate the
major/minor version numbers, so our Find module needs to take that into
consideration.
Work towards eventually using OCIO exported configs (though still not
100% working now, may need fixes on their end).
Change CI to use the OCIO's that come in the aswf containers instead
of building from scratch in many cases.
Sometimes we use a filename, or pieces of it, as a regex.
But filenames can have characters that are interpreted differently when
part of a regex pattern (like `.`, which in a filename is literal, but
in a regex means to match any character!).
So this handy utility inserts the right backslashes so the filename will
end up as a literal match when part of a regex.
Furthermore, there's an optional parameter `simple_glob`, which when
true, will also substitute `?` with `.?` and `*` with `.*`.
We then use this to fix a latent bug in scan_for_matching_filenames():
it could have incorrectly matched files during the search.
I noticed when working on an ImageCache thing that even the smallest
change to imagecache.h was causing a substantial amount of the
codebase to needlessly recompile even though very few places really
need to use the IC APIs directly.
It took a little bit of juggling of some declarations, but this patch
allows imagebuf.h to be able to treat ImageCache and ImageCacheTile as
opaque, forward-declared types, without the need for every file that
needs imagebuf.h to also have to parse imagecache.h and everything it
might pull in.
This speeds up my development cycle substantially in many cases, and
it also speeds up compilation of user code (by a small amount) by
avoiding parsing of imagecache.h in circumstances where the user code
only needed to know about ImageBuf but not ImageCache.
There's a downside to this -- it's possible that user code that truly
needs both imagebuf.h and imagecache.h only includes the former
(inadvertently relying on the fact that the latter was transitively
included). These apps may need `#include <OpenImageIO/imagecache.h>`
to be added after upgrading to OIIO 2.3. Is that really really bad?
Added this handy function that figures out if the first string contains
any character present in the second string.
Also rewrite the comments describing parse_until and parse_while for
better clarity and correctness.
Some of the OpenEXR sample files had an attribute "version".
All along, I assumed this was telling me the version of the OpenEXR file
itself, so I was passing it along and also renaming "openexr:version".
Turns out I was wrong! This is just arbitrary metadata. Who knows why
it's in some of these files? Maybe nobody. But it's not meant to indicate
the file format version, and indeed it does not correctly reflect it.
In fact, there is no simple API call in the OpenEXR API that tells you
the file format version, you have to infer it from the features
present (multi-part or deep or long attribute name indicate changes
that came along with OpenEXR 2.x, but there is no version number in
the file per se).
So I'm just going to stop doing this rename, let "version" be "version",
it's just some user's metadata, I guess.
A few image formats (but not most) support "thumbnails" -- low-res preview
images.
Traditionally, we have shoved them in the ImageSpec metadata as a byte
array (and ints for the width, height, nchannels). This is less than
ideal: It makes the metadata very bloated, taking more memory and
being more expensive to copy metadata, and potentially wastes time
doing I/O on this data that most applications don't want or
need. (Remember that any alterations to the image will make the
thumbnail "wrong", also. So it's rarely useful except for applications
that specifically want a preview image.)
Discussion in Issue 2902 led us to think that thumbnails should be
treated explicitly by the API, rather than shoved into the metadata in
this way. This patch is the attempt to overhaul thumbnail handling:
* ImageInput has a new `get_thumbnail()` method that can retrieve the
thumbnail (if it exists) into an ImageBuf. The base class / default
implementation does nothing; the image formats that support
thumbnails should have their readers override this method. They are
encouraged to read the thumbnails "lazily" if possible -- only
if/when this method is called, and avoid the cost, memory, and I/O
if not requested.
* ImageOutput likewise has a new `set_thumbnail` method.
* These are implemented for Targa and PSD, the only two formats for
which we previously had implemented thumbnail functionality. (There
are probably other formats that support thumbnails, but for which we
never added OIIO support. Leave that for another day.)
* ImageBuf has new `has_thumbnail`, `get_thumbnail`, `set_thumbnail`,
and `clear_thumbnail` methods.
* ImageCache has a new `get_thumbnail` method.
* oiiotool and iconvert attempt to preserve a thumbnail from input to
output (but of course, only if both file types support it).
* ImageBufAlgo::IBAPrep clears the thumbnail in the destination image,
on the theory that pretty much everything IBA would ever do will
likely invalidate any preview image that it started with. This gets
us to the point with oiiotool where a straight copy (like a format
conversion) `oiiotool in.tga -o out.tga` will preserve the thumbnail,
but just about anything else `oiiotool in.tga --stuff -o out.tga`
will not transmit the original thumbnail to the output (which would
almost certainly be wrong).
There is more work to do in the future. Some file formats that support
thumbnails don't yet have OIIO support for it. I'm not sure we ever
really want (except in the case of a truly straight copy) for oiiotool
to keep any input thumbnails; maybe if a thumbnail is wanted for
output, it should always be created anew (by downscaling) so that it
definitely matches the main image? Maybe also oiiotool needs a command
to explicitly extract and save the thumbnail (is that needed for any
legit use? or only as a debugging tool?).
Imagecache repeatedly has to determine whether a filename is a type
of file that's "procedural" and does it expensively -- every time, it
creates the type of ImageInput in question just to ask it if it
supports("procedural").
We can ask this of each plugin type once, when we first register it.
Then just look it up when we need it.
This is strictly internal, has nothing to do with public APIs.
OpenCue-launched apps are encouraged to honor the setting of
CUE_THREADS to limit the number of threads they use. This change makes
oiiotool (and other OIIO-based tools and scripts) better citizens when run
as part of an OpenCue job.
CUE_THREADS has lower precedence than the more OIIO-specific
`OPENIMAGEIO_THREADS`.
For ordinary command line usage, this should not change behavior, since
you wouldn't expect CUE_THREADS to be set.
(Also fix docs which incorrectly named `OPENIMAGEIO_TEXTURE_OPTIONS`.)
And make sure to switch all of our internal uses of those to the
preferred new equivalents!
Also, hide some deprecated things from the documentation system so that
new people won't be tempted to use them.
These are all things that have been deprecated starting with OIIO 2.0
(or in some cases, even earlier). Multiple releases (2.0, 2.1, 2.2)
have gone by with them commented or documented as deprecated, but
without our adding the warnings. It's about time. We're still not
taking them away entirely, I usually don't do that until we've had at
least one release where it's been marked as OIIO_DEPRECATED, and also
only do true removals of API at major releases. But we're laying the
groundwork for being able to totally remove these when we eventually
have a 3.0 release.
This patch modifies the build system to make it possible to control
whether OIIO should link Boost in the form of static or dynamic
libraries, independently from the LINKSTATIC setting, without
having to use the fully custom route (BOOST_CUSTOM).
This is done by preserving the value of the Boost_USE_STATIC_LIBS
variable, if it has been defined by the user. If the value
of Boost_USE_STATIC_LIBS has NOT been explicitly defined, it is
set to match the value of the LINKSTATIC variable, which should
result in Boost being linked in the same way as other dependencies.
* For a full list of supported build features: `.\vcpkg search openimageio`
*`vcpkg install openimageio [tools]`
* For a full list of supported build features: `vcpkg search openimageio`
* Instructions for building a Python wheel on Windows:
https://github.com/Correct-Syntax/py-oiio
* homebrew (https://github.com/Homebrew/brew)
* https://formulae.brew.sh/formula/openimageio
*`brew install openimageio`
@@ -153,7 +162,7 @@ file format (jpeg, fits, png, etc.). This works both as a CMake variable and
also as an environment variable.
`ENABLE_PkgName=0` : Disables use of an *optional* dependency (such as
FFmpeg, Field3D, Webp, etc.) -- even if the dependency is found on the
FFmpeg, OpenVDB, Webp, etc.) -- even if the dependency is found on the
system. This will obviously disable any functionality that requires the
dependency. This works both as a CMake variable and
also as an environment variable.
@@ -259,8 +268,92 @@ Building on Windows
**Method 1 - from source**
I really need someone to write correct, modern docs about how to build
from source on Windows.
You will need to have Git, CMake and Visual Studio installed.
The minimal set of dependencies for OIIO is: Boost, zlib, libTIFF, OpenEXR, and libjpeg or libjpeg-turbo. If you have them built somewhere then you skip
the section below, and will only have to point OIIO build process so their locations.
* Boost: get the boost source archive, extract into `{BOOST_ROOT}`.
```
cd {BOOST_ROOT}
bootstrap
b2
```
* zlib: this will build it, and then delete the non-static library, so they don't get picked up:
* OpenEXR: you'll have to point it to your `{ZLIB_ROOT}` location from the above. If copy-pasting the multi-line command (with lines ending in `^`) into
cmd.exe prompt, make sure to copy all the lines at once.
| master | :white_check_mark: :construction: ALL fixes immediately, but this is a branch under development with a frequently unstable ABI and occasionally unstable API. |
| 2.3.x | :white_check_mark: All fixes that can be backported without breaking ABI compatibility. New tagged releases monthly. |
| 2.2.x | :x: Only receives occasional critical fixes, upon request. |
| <= 2.1.x | :x: No longer receiving patches of any kind. |
## Reporting a Vulnerability
If you think you've found a potential vulnerability in OpenImageIO, please
report it by emailing security@openimageio.org. Only the project administrator
has access to these messages. Include detailed steps to reproduce the issue,
and any other information that could aid an investigation. Our policy is to
respond to vulnerability reports within 14 days.
Our policy is to address critical security vulnerabilities rapidly and post
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Larry Gritz