Virtual destructors don't significantly differ from regular virtual
functions, so it is customary to add override specifier when virtual
destructor is overridden.
Current code assumes that CMAKE_INSTALL_INCLUDEDIR and
CMAKE_INSTALL_LIBDIR has exactly one filesystem component. This is not
the case on e.g. Debian, where standard paths are as follows:
CMAKE_INSTALL_INCLUDEDIR=include
CMAKE_INSTALL_LIBDIR=lib/x86_64-linux-gnu
Fortunately the Config.cmake file is installed into a path that is
itself relative to CMAKE_INSTALL_LIBDIR, so we can get
CMAKE_INSTALL_LIBDIR from CMAKE_CURRENT_LIST_DIR and then get
CMAKE_INSTALL_INCLUDEDIR by applying relative path from
CMAKE_INSTALL_LIBDIR to CMAKE_INSTALL_INCLUDEDIR on top of that.
Fixes: c8dfbd7066
The current limit has been already hit when AMD released
second-generation Epyc processors (codename Rome) back in 2019. Top
models have 64 cores per CPU, or 128 threads. This means 2 socket
configuration has 256 threads.
Fourth generation Epyc processors (codename Genoa) are rumored to have
96 cores per CPU, or 192 threads. This means 2 socket configuration has
384 threads. While the CPUs have not been announced yet, it makes sense
to prepare for them in advance.
Revealed by clang-tidy. Use length-safe versions.
Also get rid of some dead code by disabling some memory-clearing
code that we don't need (we're not using SHA-1 for cryptographic
or security reasons, just for comparing images).
Python 3.8 stopped loading DLLs from PATH on Windows, meaning anything that relied
on that environment for shared library resolution breaks. This PR works around that
behviour by manually adding PATH back to DLL resolution using os.add_dll_directory()
in a new __init__.py which just re-exports everything in the current .pyd.
The new behaviour is enabled by default, but is easily disabled at runtime if the
environment variable OIIO_LOAD_DLLS_FROM_PATH is equal to "1".
Found this coincidentally during some address sanitizer runs.
Some tiff files out in the wild that have RichTIFFIPTC tags
inexplicably mark them as if the data was LONG instead of BYTE or
NOTYPE. I guess early in our history, we came across one of those and
thought it was correct, because we were always treating it as if it
came in the form of longs. But that's not always true, and in
particular, the size in the tag I think is always expressed in
bytes. So this could make for buffer overruns (generally harmless, but
caught by the sanitizer!).
So this patch cleans this all up.
And also we update a few sample images with bad RichTIFFIPTC tags to
strip out the in the incorrect form of the tag.
Recently, we added the ability to do stochastic sampling in texture
lookups, by choosing the MIP level stochastically rather than reading
two MIP levels and mixing them.
This PR adds the ability to also stochastically sample the anisotropic
filter footprint -- instead of multiple anisotropic samples, choose
just one sample well distributed over the footprint axis. On real
scenes, this seems to speed up overall rendering (not just texture) by
up to 2-3%, for scenes where average anisotropy is around 3-4.
It's much less dramatic a speedup than the stochastic MIP selection,
which sped up texturing by around 40% and in practice sped up overall
rendering by around 15% for us. Why doesn't cutting the number of
texture samples by another factor of 3 only speed up by 3% instead of
another 20%? Basically, it's because the stochastic MIP mapping *by
definition* means that each texture call touches half the number of
tiles as before. In contrast, those multiple probes along the
anisotropic axis are almost always within the same tile, and
frequently touch a very nearby pixel within the tile, so not only are
we not needing to look up a new tile from the tile cache, but also the
very memory we need to read is highly likely to be in L1$. But still,
we'll take 2-3% rendering gain any time we can get it, that's
equivalent to a lot of money if you have a really large compute farm.
Some other changes that come along for the ride:
Add a --bluenoise option to testtex (makes for prettier 1spp results),
and add tests for the aniso sampling.
Some internals changes to precompute min w,h of mip levels so there is
less per-texture-call rifling through the ImageSpec of each MIP level.
Sadly, this did not produce any consistently detectable speedup in
overall rendering.
And I changed the interface for stochastic texturing a bit. The old
way involved asking for new MIP modes, StochasticAniso or
StochasticTrilinear, in addition to being sure to set TextureOpt::rnd
to the stratified random number you are using for sampling. Now we are
deprecating StochasticAniso and StochasticTrilinear. Just use the same
old mip levels you always did. So the new interface can be summarized
as follows:
* On each texture call, setting opt.rnd >= 0.0f is itself a request for
stochastic texturing on that call (opt.rnd will default to -1.0, and
any negative value does the old fashioned non-stochastic filtering).
* A ShaderSystem-wide option "stochastic" (int) indicates which things
should be sampled stochastically, for those lookups that have rnd >= 0.
It's a bit field: 0 means no stochastic sampling (even if rnd >= 0),
1 means to do stochastic MIP level selection 2 means to do stochastic
anisotropic sampling (so 3 means to do both, and higher bits are reserved
for future extensions). The default is 0, so renderers need to enable
it to get any stochastic texturing.
Finally, let me give a reminder that you ONLY want to do stochastic
texturing for shading that involves multiple stochastic samples per
pixel, like orinary surface shades. For displacement (shade once per
mesh vertex) or "imager" (shade once per pixel), you just have a
single shade that you want perfectly filtered, so you probably don't
want to set opt.rnd >= 0 in those cases.
Previous code was doing ioread one pixel at a time. Change it to
read one scanline at a time, similar to e.g. how BMP reader does it.
On a 8192x8192 rgba8 image (256MB), on Apple M1 Max "iinfo --hash"
time goes from 2.19s down to 0.65s, and is now similar to the time
it takes on an uncompressed TIF or BMP image.
When viewing this .md file in Github UI, everything seems to work. But when creating an actual PR, the links to CONTRIBUTING and CLAs point to nothing that would exist. Looks like they try to be relative links from the "create a new PR" page. Workaround seems to be to use absolute links instead.
Implements support for BC4, BC5, BC6H (unsigned and signed) and BC7
GPU texture compressed formats in DDS reader (#2338).
- BC4: loaded into 1 channel uint8,
- BC5: loaded into 2 channel uint8,
- BC6H: loaded into 3 channel half,
- BC7: loaded into 4 channel uint8.
This is implemented by replacing whole of libsquish (~4700 lines of
code) with bcdec.h (https://github.com/iOrange/bcdec, ~1300 lines of
code). bcdec.h has an advantage of being faster, smaller and supports
all BCn formats for decoding.
bcdec.h license is MIT or Unlicense (aka public domain), updated all
the relevant third party / license files that I could find.
* Add a SECURITY.md detailing our security policy and the private
email address to contact to report vulnerabilities.
* Create scorecards.yml, to run OSSF scorecard to check supply chain
security issues. It has some false positives, but it's only advisory.
* Make sure actions we use don't get write perms on our repo.
* Use exact SHA's for actions versions to guard against hijacked
actions repos that might try to substutite something else at the
version tag we use.
* Tried to use harden-runner action to ensure no actions on the runner
overwrite source code or have unauthorized outgoing net connections
during the GHA run, but currently leaving it disabled because of too
many false positives -- it flags every dependency download as a case
of altering source code. Which it technically is, but it doesn't
matter for any jobs that aren't producing any artifacts that will be
consumed or relied on by users.
Learned about this from a security talk.
On Linux, we'd been using gettimeofday, which gives results with usec
resolution. Switch to clock_gettime, which has nsec results.
In practice, the round trip time of a pair of "get time" system calls
is about 30-50ns in either case. But the new call at least nominally
returns ns resolution values. Another way of saying that is: the
answers we get should be more accurate with this call, but the call
itself is the same expense as before, so we should still be just as
cautious about not inserting timers around events that are so brief
that the timers themselves are going to be a significant addition to
the cost.
This only effects Linux (and maybe BSD?). On MacOS, we use
mach_absolute_time(), and on Windows we use QueryPeformanceCounter(),
so neither of those platforms is affected by this patch.
Oops, calling it "m_max_tile_channels" (the name of the member
variable) instead of "max_tile_channels" (the name of the attribute)
actually prevented properly setting or retrieving the attribute by
name.
ustring is a unique string (that lives in an internal hash table) whose
representation is the pointer to the characters that live in the table.
ustring hash is its alter ego -- it also refers to a ustring in the
internal hash table, but the representation is just the hash itself.
It presents an interface that is a subset of that of ustring, and in
many ways they can be used interchangeably. For ustringhash,
accessing the characters requires using the hash to find the string in
the table -- which is not hard, but more expensive than accessing the
charcters with a ustring, where the representation is the pointer
directly to the characters.
For some uses where you don't need access to the characters in any
performance-critical paths, this may be a more convenient
representation. In particular, it's well suited to a GPU that doesn't
have access to the character memory. Another interesting difference is
that from run to run, a ustring may have a different literal value,
since there's no reason to expect that the pointer to a string like
"foo" will refer to the same memory location every time the program
executes, but in contrast, the hash is guaranteed to be identical
from run to run.
In the process of adding unit tests for ustringhash, I ended up
overhauling all of ustring_test.cpp, as I saw that the ustring tests
were extremely cursory. Did a tiny bit of (non-breaking) touch-ups
on a few ustring methods as well.
If the BMP image we're reading declares a certain sized palette, but
then has pixels that contain values outside the range of that palette
size, we would read past the end of arrays.
With a 1x1, 2bit image of a Targa 1.0 file, the entire image was only
22 bytes, so the seek to 26 bytes before the end to look for the
Targa 2.0 data... obviously failed.
Don't look for the Targa 2.0 block if the file itself is not long
enough to hold both the header (18 bytes) and the block (26 bytes).
This tiny change is a quality-of-life fix for me, addressing the fact
that there are two different simd.h files in our code base. I never
would have done that on purpose, but one of them is part of the
imported "squish" source code that we use for dds encoding.
For years I've been frustrated that reflexively typing `CMD-P simd.h`
in the editor has a 50/50 chance of pulling in this file, when in fact
there is literally a 100% chance of my wanting
src/include/OpenImageIO/simd.h.
So finally put my editing brain out of its misery by renaming the file
that I never want to edit.
When doing thread "wedge" tests, most of the time spent is for the
1-thread case. Who cares about 1 thread anyway? So we make two
improvements with new command line options:
* --minthreads sets the minimum number of threads that will be
used. For example, `--minthreads 2` will skip the single-thread case
and start the wedge with the 2 thread case.
* --lowtrials is an optional maximum number of trials just for the 1
or 2-thread case (overriding --trials just for those thread counts,
if it is a lower value).
By default, the old behavior is still the default. But these allow a
bit of customization in the tradeoff of how much time to invest in the
very low thread counts in a wedge of thread timings.
You'd think there would be no way that basic failure like this could
occur after 14 years, right?
But the TIFF reader code was a little mixed up about the fact that the
internal m_next_scanline field is relative to the starting scanline of
the image, not an absolute value. But in the case of the code that has
to deal with handling the stray scanlines at the end of a strip-read
file neglected to set it properly and the end of the stripped part.
Obviously, for TIFF files with a first scanline of 0 (the vast, vast
majority), failing to adjust by this offset doesn't matter. And also
as long as the reading continued to be strictly sequential, it also
was asymptomatic. But in the presence of certain multithreaded random
access patterns, it makes a difference. Oops.
The udim_lookup_mutex_pool is not necessary. Just using an atomic
pointer for udiminfo.icfile is enough.
I'm seeing about a 5-8% overall speed improvement in testtex-based
intense udim benchmarks under heavy threading, a ~2% overall
improvement in a real rendered scene that uses a bunch of udim
textures (I think there are bigger bottlenecks elsewhere that I'll
move on to next, but every little bit helps).
Some testtex enhancements coming along for the ride:
* UDIM-oriented tests (automatically decuded if the test textures
contain `<UDIM>` in their name.
* --maketests will construct random test textures and use them (handy
if you need lots of textures for a big test, but don't want to store
them). The tests are based on a file pattern using std::format-like
replacement with an index. Also, if the pattern contains `<UDIM>` in
the name, it's understood to be a udim pattern and many individual
udim piece files will be created.
An example of these two items in action is:
```
testtex --maketests 200 "tmp/foo{:04}_<UDIM>.exr" --threadtimes 8 -iters 100000000 -runstats -v --handle -maxfiles 1000 -trials 5 -wedge
```
fmt master seems to have just deprecated the ability for
fmt::sprintf("%s") to default to printing anything that allows stream
output. This breaks use of Strutil::sprintf formatting ustrings.
This isn't yet in any tagged release of fmtlib, but I'm trying to get
ahead of it.
The quick fix is to just change those handful of places to use
std::format style calls, which we are slowly converting to anyway.
Also throw in a fmt::formater for ustrings for good measure.
It's a lot of data, you don't need it if you don't want to run the
full testsuite, and a lot of places and distros have policies against
live downloads happening in the build step. Also, downstream projects
that need to build OIIO for their CI or other reasons don't want to
accidentally download a lot of test data they won't need.
So change the default of OIIO_DOWNLOAD_MISSING_TESTDATA to OFF.
Build with -DOIIO_DOWNLOAD_MISSING_TESTDATA=ON to turn on auto
download of the data images, or else run
src/build-scripts/install_test_images.
Now, this does have the following danger: That people who build OIIO
and do want to run the tests -- and don't know to turn this option
back on -- will end up with very few tests able to run.
We largely address this by observing that quite a lot of our tests
that depended on oiio-images actually depended only on one file
(grid.tif, which is not especially large). So by making a copy of that
file in testsuite/common, quite a few additional tests can run even
without oiio-images.
Though, to be clear, there are still many tests that want oiio-images,
particularly tests that validate support for various image file
formats, which often depend on large collections of images that are
stored in oiio-images. So remember, if you are developing OIIO, the
best policy is to run the whole testsuite. But on the other hand, if
you submit a PR, they'll all be run in CI as well.
For failed CI builds, save the cmake caches in a safe place, don't zip
them up in situ, because then when I unpack them in my build area
locally, they overwrite my working cache.
The Intel OneAPI LLVM-based C++ compiler, as we download it from Intel
for our CI, bumped from 2022.0.0 to 2022.1.0.
New warning popped up -- newest Intel LLVM-based compiler warns about
overriding certain floating point math flags.
Also had to adjust thresholds on a couple texture tests that have a
few pixels change slightly due to numerical differences.
Another round to hide Imath externally to OIIO public headers as well
as speed up our own builds.
* Remove the external reference to Imath::M44f from color.h, and instead
use M44fParam.
* The `<OpenImageIO/Imath.h>` includes the most commonly used Imath
headers including half.h. But this is unnecessarily expensive (and
presumptuous about wanting other Imath types as well as fmt.h) if
you just need to know about `half`. So we create a new
`<OpenImageIO/half.h>` that only includes what you need for `half`
(as before, with logic to handle the changing location of the half.h
header depending on whether you're using Imath 3.x or OpenEXR 2.x).
* Several files that included Imath.h, but in reality only needed half
all along, change to including OpenImageIO/half.h.
* simd.h leans more on vecparam.h and other template trickery to
reduce exposure of methods that only work in the presence of Imath
headers, and make fewer use cases that would care whether the Imath
headers are included before or after simd.h (which is less
controllable for unity builds).
* Vec3Param gains a `cast<V>()` templated method that allows easy
casting to foreign vector types even if Imath headers are not
included prior to vecparam.h being included (which is less
controllable for unity builds).
We have a template OIIO::bit_cast which has served us for a while, but
it turns out that C++20 has std::bit_cast which operates ALMOST the
same -- unfortunately, the tempalte arguments are reversed from the
way we did it!
So here we:
* Set up a new `bitcast<>` that works exactly like C++20 std::bit_cast.
* Implement our old bit_cast in terms of the new one, for now.
* Set up `#if` trick so that the confusing OIIO::bit_cast will be
marked as deprecated beginning with OIIO 2.5, and will be removed
entirely from OIIO 3.0. (Farther out than that, we may choose to
re-introduce bit_cast as truly being identical semantics to
std::bit_cast. But before that is possible, we need to completely
ween downstream users off the old one.)
* Took the opportunity to expand the set of compilers for which
certain bitcast overloads use Intel intrinsics. Previously, we only
did this for the Intel compiler, but it seems to work for modern
versions of clang and gcc as well. Do a little bitcast testing in
fmath_test.cpp to verify that the use of these intrinsics doesn't
break for any of the compilers we use for CI.
Over the span of several individual PRs, we've made the code base safe
for "unity" (a.k.a. "jumbo") builds -- a feature of CMake that
automatically combines multiple .cpp files into a single compilation
unit (basically making temporary .cpp that `#include`s each original
cpp) in order to, allegedly, speed up compilation by not re-parsing
headers multiple times, and other sources of per-module overhead.
We discovered early on that on a modern computer where modules are
built in parallel (another automatic feature of CMake), jumbo builds
are simply not an overall performance improvement for OIIO. Basically,
the efficiency gain from removing redundant parsing is mostly offset
by the loss of parallel efficiency -- fewer and larger modules means
there are periods where all cores are not able to find work to
do. (Reductive example to illustrate the basic problem: If you have 8
independent modules to compile, and each takes 1 second, 8 cores can
compile the entire collection in parallel in 1 second. But a unity
build that batches those files in groups of 4 and achieves a 2x
speedup, thus compiling each set of 4 modules in 2 seconds instead of
4, only can do two of these compilations in parallel, so it takes 2
seconds. We've cut the total work in half, and yet it takes twice as
long in real time to complete, and 6/8 of your cores are idle.
OK, but that's when you have several cores. If you had only one core
(or maybe two) so you weren't using much parallelism in the first
place, then the unity build would look like a win, since you'd really
see the improvement from the amortized overhead of the combined
modules.
Most of us don't have single core machines on our desks, or laps, or
even in our pockets (or on our wrists?).
But we do have single core machines (or close to it) in our CI runners!
We get 1.5-2 cores, roughly, in the slice of VMs we get to run CI on
GitHub Actions. So maybe this is the one place where a unity build would
be of benefit.
That's the theory anyway. Does it pan out? I tested it, and the answer
is basically yes. I can speed up the "build" step of our CI by
somewhere in the vacinity of 30% (varies a LOT from run to run). I
tried a whole bunch of different approaches, summarized in the table
below. I picked the best basic strategies revealed by those tests and
set up the CI to use them. (I didn't preserve all the other
permutations.)
| Name | try 1 build | try 2 | try 3 | try 4 |
|:----------------------- | ----------- | -------------------- | -------------- | -------- |
| **linux gcc9 BASELINE** | 14:19 | 11:46 | 11:56 | 13:32 |
| linux gcc9 group | 9:39 | 8:06 | 9:58 | **7:58** |
| linux gcc9 batch 8 | 8:06 | **7:17** | **8:02** | **7:59** |
| linux gcc9 batch 16 | 8:29 | 7:32 | **7:56** | 9:35 |
| linux gcc9 batch 4 | 8:39 | 8:38 | 8:52 | 10:31 |
| linux gcc9 batch 2 | 13:34 | 12:43 | 11:22 | 9:32 |
| linux gcc9 batch mix 4 | 7:22 | 9:14 | 9:01 | 9:39 |
| linux gcc9 batch mix 2 | 9:11 | 7:45 | 10:55 | 9:36 |
| **linux icx BASELINE** | | 14:39 | 15:07 | 15:06 |
| linux icx group | | 11:59 | **10:36** | 10:26 |
| linux icx batch 8 | | **9:40** | 11:31 | **9:14** |
| linux icx batch mix 4 | | 10:06 | 11:58 | 11:16 |
| **macos11 BASELINE** | 25:41 | 27:53 | 26:30 | 28:32 |
| macos11 group | 10:56 | 20:24 | 19:28 | 19:24 |
| macos11 batch 8 | 16:43 | 17:38 | **18:34** | **17:42** |
| macos11 batch mix 4 | | **9:22** | 19:12 | 18:26 |
| **windows BASELINE** | 20:17 | 23:27 | 28:43 | 19:13 |
| windows group | 23:05 | 15:57 (tests broken) | 21:57 (broken) | 15:33 |
| windows batch 8 | 21:16 | 15:57 | **17:38** | **15:08** |
| windows batch mix 4 | | **15:30** | 23:19 | 15:16 |
Notes:
* Timings vary wildly from run to run. You never know which VMs you'll
get or how overloaded they are. But over 3 trials, the trends were
clear.
* I rigged it to clear the ccache, so these are FULL uncached builds.
* This is only "build" time. The CI run also includes time to set up the
VM, download docker images, build or install dependencies, and of course
run the testsuite. So the overall speedup of the entire CI run will not
be as improved overall as these numbers indicate for the build step.
But still, saving several minutes per job means the CI tests turn
around faster, and that's what we want.
For C++17 and beyond (and maybe earlier in some compilers), we already
used std::gcd. Now for C++14 compilers without std::gcd, we roll our
own simple gcd implementation instead of using boost::gcd. Maybe it's
slightly slower, but this is nowhere near a performance-critical
path. It's worth it to take one more step at eliminating boost.
Reminder: When opening a file by name, first it tries to open assuming
that the file's extension indicates the format. If that doesn't work,
it tries all known format readers to see if one will work.
This patch includes several improvements to this process and surrounding
code in imageioplugin.cpp:
* When trying all formats because the extension was wrong, instead of
traversing the input_formats map in alphabetical order of the list
of all format names + extensions, try the three most common formats
we encounter (openexr, tiff, jpeg) FIRST, then try the other formats
in alphabetical order. We do this by traversing the
format_list_vector, which contains all format names in the order
they were declared, and ensure that those common formats are
declared first. Basically, this means that if the mislabeled file is
one of those most common formats, the correct reader is found after
only a couple tries, instead of having to try a dozen or more bad
guesses of rarely encountered formats.
* Debug mode messages that make it easier to understand which readers
are being tried for which files.
* Comments clarify that input_formats and output_formats hold not just
the map from format names to creators, but also map extensions to
creators. (That confused me while I was debugging this, so I figure
the clarification is needed.)
* Establish a new extension_to_format_map that lets you get the format
name if you know the extension.
* Get rid of input_extensions and output_extensions maps, which weren't
ever used.
* Some other minor refactoring.
If you know the hash of a ustring, this tells you the ustring.
(Caveat: in the very unlikely chance of two ustrings having the same
hash, this will return the first one it finds in the table.)
I realized that --attrib (and several related actions) only set the
attribute as directed for the first subimage if it's a multi-image
file (such as a multi-part exr), or all of them if the -a flag is
used. But there was no ability to set it for a particular (non-first)
subimage, like many other oiiotool actions have the ability to do.
So in this patch, I refactor the following commands to use the
OiiotoolOp and related macros, which automatically make it responsive
to the `:subimages=` optional modifiers:
--attrib
--sattrib
--eraseattrib
--caption
--orientation
--clear-keywords
--iscolorspace
As a bonus aside, one thing this lets you do is set a different
compression method per-subimage using, e.g.,
`--attrib compression:subimages=3 dwaa:100`.
But if you set compression with `--attrib compression`, do not also
use `--compression`, which will override it and also apply to all
subimages.
When this symbol is defined, this marks as deprecated the ErrorHandler
class methods that use sprintf formatting notation. It's not enabled
by default, but a downstream project can define the symbol in order to
help root out any uses of the old style formatting functions, to aid
identifying and changing those spots to the new style, long before
they are truly deprecated or removed from OIIO.
This change also sets up an OIIO_INTERNAL symbol that will be present
when OIIO itself is compiling, but not a downstream project using the
OIIO headers.
This is the first tiny step to fixing all the little things that break
in a unity build. This one is pretty small fry -- the oiiotool set of
modules only needed some minor rearrangement of header file include
order.
* Initial implementation of `ImageBufAlgo::st_warp`
* Simplify default filter logic, and share it with `ImageBufAlgo::warp`
* Add st_warp to oiiotool
* Add tests for bespoke `st_warp` validations that are done before IBAprep
* Add st_warp test to oiiotool-xform testsuite
* Address review notes
- Remove ST buffer datatype guard
- Simplify additional ROI prep
- Remove unnecessary buffer reallocation
* Use sample accumulator type for source/dest iterator USERT
* Add `--st_warp` section to `oiiotool.rst`
Cut/paste error inadvertently deleted a critical cmake line that I
wasn't intending to alter or remove, that resulted in the command line
utilities not being tagged for part of the install step. Oops!
Oh boy, never leave anything unbenchmarked.
Turns out the boost::algorithm functions we were relying on underneath
many Strutil "case-insensitive" comparisons were ridiculously slow.
We thought they were good enough because they allowed specification of
locale, so we could just pass the static classic locale, and so they
would be inexpensive because they didn't have to query the current
locale. But this is wrong, they were still ghastly.
So here I rewrite Strutil::iequals, iless, starts_with, istarts_with,
ends_with, iends_with in terms of a new (internal) Strutil::strcasecmp
and strncasecmp (which underneath handle differences in platform, and
use the locale-independent versions). The net result is that most of
those case-independent comparisons speed up by a factor of
50-100x. Wow.
I still need to tackle the family of 'ifind' related functions. They
are a bit trickier. But I'll leave them for another time, because I
need to roll this present fix out right away to fix a real production
bottleneck.
(Worth noting: iequals is instrumental when you're searching a
ParamValueList for a particular name, which is what happens when you
look up attributes from an ImageSpec, which is what happens when you
call get_texture_info(), which is what underlies OSL gettextureinfo()
calls in the cases that they cannot be constant-folded during runtime
optimization. So this came to my attention when analyzing a slow scene
whose shaders had a pathological explosion of gettextureinfo calls that
couldn't be optimized away.)
Reminder: isConstantColor() finds out if an ImageBuf is constant
color. It uses the thread pool to check several areas in
parallel. Once any task finds that its subregion is not constant, it
sets a flag so that as other subregions are pulled off the task queue,
they will take an early out if one of their siblings already knows the
image is not constant.
Turns out this isn't much of a big savings in practice, because lots
of regions may be in flight by the time one region realizes it's
non-constant, and once a region task starts, it doesn't check the flag
again.
This patch speeds up isConstantColor by a pretty big factor for
non-constant images (it can't really help for images that ARE
constant) in the following ways:
1. Before doing anything, it checks to see if the FIRST TWO pixels of
the image are unequal. If that's the case -- and it's a fairly
common case for non-constant images -- then it immediately knows
the whole image is not a constant and doesn't need to look at
anything else or launch any of the threads at all. (If it is a
constant image, or if the first two pixels are just coincidentally
identical, this quick check of only two pixels didn't really cost
us any masurable time.)
2. In addition to checking the result flag at the start of each
parallel job, now we also check if the flag has been changed as we
iterate to each pixel. So pretty much as soon as one thread finds
unequal pixels, the other threads can spot this and stop their
tasks early, no need to finish looking at their entire region. (To
make the more frequent checks of the flag inexpensive, I changed it
from an atomic to just a regular bool. It doesn't need to be
atomic, because the strict ordering is not important at all here
and it doesn't change the result if the threads' access to the flag
is unsynchronized.)
When outputting OpenEXR files, in a case where the "compression"
request was for "dwaa" (no compression specified in the main name,
like "dwaa:100") we accidentally forced a default quality level of 4,
which is the default we wanted for zip compression. But that's not a
good default for dwaa!
As I learned recently, a "unity" (aka "jumbo") build is where multiple
.cpp files are combined into one translation unit -- imagine a unity.cpp
that simply has
#include "file1.cpp"
#include "file2.cpp"
...
#include "fileN.cpp"
and so you compile unity.cpp instead of the separate file?.cpp files
individually.
Turns out that CMake understands this concept and can do it for you
automatically!
The benefit of a unity build is that file1...fileN probably include
most of the same headers, expand the same templates, etc., so a bunch
of the per-file work of the compiler can be done once rather than
redundantly for each file.
There are two potential downsides, however:
1. It may not be safe to concatenate your cpp files! For example, if
both file1.cpp and file2.cpp each contain a `static int foo;`, that
may have been safe when compiled separately, but is not allowed to
happen twice in one compilation unit. Similarly, if you have headers
that don't have proper guards against multiple includes, etc. So one
should expect a whole lot of little fix-ups are needed for this to
work properly. (We'll come back to that topic.)
2. Combining source file into these "jumbo" modules can make heavily
parallelized builds on many-core machine not be able to load balance
or keep all the cores busy. (Simplified examples: if you have 16 .cpp
files on a 16 core machine, each core can compile one cpp file in
parallel with the others. But if you mash the modules into just one
huge cpp file, give that to one core, and your other 15 cores are
idle, so the full build probably takes much longer.)
I tried this out, including the many fixups implied by (1) above, and
at first the unity builds were slower on both my laptop (8 cores) and
workstation at work (32 cores), because of downside (2) explained
earlier -- harder to take advantage of parallel builds when there are
fewer, bigger, compilation units. Some tweaking of strategy got me to
the point where I could always get the unity builds to go faster, but
not by a whole lot when many core were available. Slighty faster, but
not worth the trouble. A bit disappointing, nearly abandoned the whole
idea.
HOWEVER, in situation where you are limited to very few cores (like in our
CI, which allocates 2 cores, and it sure seems more like 1-1.5 for the
Windows and Mac CI runners), the unity builds are substantially faster --
there's already not much parallelism to exploit, so you come out ahead
with the savings of that redundant per-file work we talked about.
So my current thinking is to go ahead and make the changes that allow
unity builds. I don't particularly recommend them when highly parallel
builds are available to you, but it might help to cut our CI
turnaround time down on the GitHub runners. And maybe it will help in
other situations for other people.
Ok, then. The present patch introduces these concepts and makes the
CMake and other build system changes to allow unity builds. (N.B. It
won't WORK yet, so don't try it!) After we get that out of the way, in
subsequent PRs I'll show all the changs to the code that were
necessary to fix all the little things that went wrong when source
files got combined.
* Fix warning from duplicate define
* Fix warnings about bool/int expressions
* Fix discarded result warning in PSD input
* Fix codepage warning from unicode symbol in string constant
* Ensure the term object is initialized with the stream we intend to output to
* Fix pointer casts in paramlist test
Signed-off-by: Chris Kulla <ckulla@gmail.com>
Co-authored-by: Larry Gritz <lg@larrygritz.com>
In FFMpegInput::close(), some of the tear-down calls would happen
regardless of if their pointers were already null. Seemed safe, but
maybe not always? So check the pointers.
Fixes#3371
When trying to break filenames into the base name and any "RESTful"
arguments (such as "/foo/bar.exr?name=value&dog=cat"), we were
just looking for presence of the "?" to determine if it was a RESTful
thingie and using that as the split position.
But this is thrown off by an odd Windows notation where names that
start with (get this) "\\?\" somehow designate long filepaths. See
See: https://docs.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation
So the solution is to change Strutil::get_rest_arguments to explicitly
check for this pattern at the start of the string.
In OpenEXROutput::write_deep_scanlines and write_deep_tiles, don't
just pass the DeepData along -- handle the case of where the DD (the
buffer the caller is presenting) does not match the data types that
you said the file is going to hold. In this case, we need to make a
local temporary DD that holds a copy of the passed-in data, but with
the formats transcoded.
Some changes to DeepData were done to support this.
GHA is retiring the windows-2016 image, we won't be able to use it soon.
Also bump openexr versions in a couple places.
There's some in-progress work on a VS2022 test, but ignore that for now,
it's commnted out.
I'm a total convert to fmt (and/or std::format, std::print) and slowly
converting all prior use of sprintf and iostream to it.
If I print an Imath::V3f or a simd::vfloat3,
fmt::print ("v = {} {} {}\n", myvec[0], myvec[1], myvec[2]);
That's ok, I guess, but it's clunky and a big pain to put the format
specification for every element separately if it were a vfloat16,
say. I would like to be able to do this:
fmt::print ("v = {}\n", myvec);
and in fact, it does print all elements (thanks to some nice behavior
by fmt)! But this does not work as expected:
fmt::print ("v = {:.3f}\n", myvec);
fmt's tricks will print all elements, but only using the default format
spec, and the spec above applied so the "whole thing", not each element,
if you see the distinction.
So this patch sets up the custom formatter infrastructure to be able
to print array-like things (and in particular, the Imath and
OIIO::simd vector and matrix classes) by specifying the format spec
used for the elements.
The important elements to note:
* detail/fmt.h adds custom formatter helpers for (a) any class with
operator[] and size() ("index_formatter"); (b) any class that lacks
these but nonetheless has an array-like memory layout and a known
length ("array_formatter"). I also added a nonstandard extension to
the format spec for these, which optionally lets you use a comma
separator rather than just the space between elements.
* These building blocks are used to specify custom formatters for
the commonly-used Imath types (in our Imath.h) and our internal
SIMD vector types (in simd.h).
* simd.h also adds a few other goodies, including an `is_simd<>`
template, and `size()` constexpr methods to each vector type. These
make certain template metaprogramming tasks easier for these types.
* A new "type_traits.h" header for collecting a bunch of little utility
templates, such as `has_subscript<>` and `has_size_method<>`.
JPEG expresses pixel aspect ratio implicitly as header fields
"xdensity" and "ydensity" (which we call "XResolution" and
"YResolution" in our metadata after the TIFF names). We also have
"PixelAspectRatio", so the three of them, if they are all used, form
an over-constrained system. Plus, they may come both in the JPEG
header and also (not necessarily having the same values) in any Exif
block.
And on top of that, the way Nuke and PhotoShop infer the aspect ratio
is BACKWARDS! (If pixelaspect = width/height of a displayed pixel,
then it's also = ydensity/xdensity. But Nuke is the other way
around. Only for JPEG, it gets it right for TIFF and OpenEXR.)
Anyhoo, there were some edge cases in which we were still getting
confused (specifically, that we were making some files that Nuke was
not properly recognizing as anamorphic). This patch tightens up
these cases.
Create a small shim class template called
Vec3Param (and its specialization, V3fParam) that does nothing but
implicitly convert from and to things that looks like 3-vectors
(thanks to some template magic I developed a while back for the Imath
project). And similar for template MatrixParam (and specializations
M33fParam and M44fParam) to pass a square matrix.
Basically the idea is that we can replace the public API function
parameters where we take an Imath::V3f and instead use a
V3fParam. User code that passes an Imath::V3f still works because
V3fParam accepts and converts it. Implementation code can implicitly
convert the V3fParam to an Imath::V3f, so it doesn't need to be
changed, either. Only the function declaration itself. (Same basic
situation for MatrixParam and Imath::M33f/M44f.)
So in the process, now we have laundered our public API so that it
doesn't reference the Imath types and headers any longer. This confers
two big advantages:
1. It allows OIIO itself and any downstream client code calling OIIO
to use *different* versions of Imath if they want.
2. It allows client code to not need Imath at all, for example if they
have their own proprietary vector type that they prefer to use
internally.
I'm not necessarily 100% sold on this approach, but I think it's got
promise and am seeking feedback. It seems to work, and I was able
to compile OSL against this altered OIIO with *no changes*.
The use of ends_with instead of iends_with made it so that
capitizalized files (especially on case-independent filesystems like
Windows or some Mac filesystems) were rejected.
(And yes, I did grep the whole code base for ends_with to be sure that
this was the only place we made this mistake.)
Fixes#3362
icx is the *new* Intel LLVM-based compiler, and it behaves a lot like
clang (versus icc, which is the "classic" Intel compiler).
* An additional CI test in ci.yml for icx.
* Tweaks in compiler.cmake and platform.h to detect and set options for
this compiler.
* A variety of warning fixes encountered for this compiler.
* Add a new cmake build-time option `ENABLE_FAST_MATH` (default: OFF)
for enabling fast-math optimizations that trade IEEE compliance for
speed. The Intel compiler enables it by default (!), so this build
option makes icx behave like the other compilers (not fast math
unless explicitly asked for), which makes the CI tests and other
behavior more consisent across compilers and platforms.
* Note: The bug that prevents some versions of fmtlib from working
with icc does not appear to be a problem for icx.
I finally learned how the "strategy" stuff works and refactored our
CI workflow based on it.
In short, we previously treated each CI test as its own universe, with
a huge amount of repeated build logic. These changes cut down on
redundancy by, for each set of related tests, collapsing them into
just one recitation of the build script logic and a handful of
variables that create the different test variations.
Currently, the categories are (a) all tests based on ASWF containers and
corresponding to VFX Platform specs, (b) "off-year" combinations based
on the native Ubuntu installations, (c) Mac tests, (d) Windows tests,
(e) the clang-format checker.
This refactor cuts the size of the ci.yml file by more than 50%, and
makes it much easier to add or change individual test configurations
in the future.
Strutil::print wraps fmt::format to render the formatted output into a
std::string, then sends the string to Strutil::sync_output(), which
ensures that every string printed (or sent to a FILE or stream) is
done so atomically, using an internal mutex, and with a stream flush
after each string. But this makes it a lot more expensive than it
needs to be, when you don't need such care -- either because buffered
output is ok, or because you don't need the thread safety.
This patch:
* Adds an optional `flush` parameter to sync_output(), defaulting to
true, which controls whether sync_output flushes or not.
* Introduces Strutil::sync::print, which is the same fully
synchronous, thread-safe, non-buffering behavior that Strutil::print
has traditionally had. Use this when you need those features.
* Redefines Strutil::print to be an alias for fmt::print, which means
- There is one fewer layer of template expansion and generated code,
and possibly fewer internal string copies, so that should make for
both faster compilation and faster runtime performance.
- Results are buffered, no sync after every output, so much faster
when you don't need that.
- No internal mutex.
On the platforms we use -- and with the current implementation of
fmtlib -- fmtlib's underlying use of stream and FILE output appears to
be adequately thread-safe. But we can't 100% guarantee this on all
platforms or all future versions of fmtlib, so users who need the
stronger guarantees (and are willing to sacrifice performance) can
simply use sync::print.
* New CI test downloads the Intel icc compiler and builds with it.
* A variety of minor fixes to address warnings from icc that were
not apparent from the other compilers.
* Some minor changes to just a few tests and reference output where
the icc-built compiler produces slightly different (usually LSB)
results.
These are cases where we declared a template, then later gave its full
definition, and we marked the declaration of the OIIO_FORCEINLINE but
omitted it from the later definition.
This addresses some compiler warnings from icc.
Slight reworking of -Werror logic to let the default behavior be
changed by setting a cmake variable. In particular, this allows us to
make a CI build disable -Werror when we need to.
Previously, having the "OpenmageIO_CI" env variable set trumped all
else and forced -Werror. Now it just contributes to the default behavior,
but setting `-DSTOP_ON_WARNING=OFF` and cmake config time can still
disable it.
Seemed like a good idea when we wrote it, but subsequently,
std::string_view was accepted into C++17 without a str() method. So
for the sake of future full interchangeability with std::string_view,
stop using this method. We will eventually deprecate it.
Special case some methods of ParamValue, ParamList, and
ImageSpec::attribute for ustring and string_view data, to reduce some
redundant copies and/or ustring conversions.
The internal method init_noclear() is augmented with a flag that allows
the caller to specify that any string data it might point to is already
ustrings, and thus does avoid the costly default behavior of converting
strings in the data block to ustrings. (This is not publicly exposed, so
users can't screw it up.)
So the new ParamValue constructor directly from a ustring can use this
flag to avoid passing the already-ustring through the string-to-ustring
conversion a needless second time. But also, the copy constructor, move
constructor, and assignment can use the flag and avoid the extra
conversion, since by definition they are copying from the contents of an
existing ParamValue which is guaranteed to be already ustring-ified.
While I was at it, I turned the optional arguments of ParamValue
constructors into "strongly typed" flags, deprecating the ones that
take a raw bool, which might be confusing.
c_str() was cute, but it's a little dicey and it's not present in
the C++20 std::string_view. We didn't use it in a ton of places,
so replace with other idioms so that in the future we are better
able to interchange with std::string_view or replace our own entirely
once we have flying cars and C++20 minimum.
* Allow ustring and OIIO::string_view to auto-cast to fmt::string_view
(helps some fmt formattting of types without custom formatters).
* When std::string_view (C++17) and/or std::experimental::string_view
(some compilers pre-C++17) are available, add OIIO::string_view
auto-construction from and auto-casting to those types.
...and stop using them internally as well.
For some of the IBA functions, we were unable to fully mark them as
OIIO_DEPRECATED (so instead that hint is commented out) because the
function signatures of the deprecated and new ones can be
ambiguous. We're just going to have to delete those when we are at a
compatibility break point, without going through a period where they
are present but marked as deprecated.
Just like we had done for the other libraries recently. And need
to make sure that the imath libs are added explicitly to the python
module's needs, since for hidden static libraries, they will not get
picked up merely by linking to libOpenImageIO.
Add an OIIO build-time option for forcing our exported
OpenImageIOConfig.cmake to skip the find_dependency of Imath and
OpenEXR.
This is helpful if you're at a site that is trying to build OIIO
against statically compiled Imath/OpenEXR, and want to be super sure
the downstream projects are definitely not going to encounter that
find_dependency(Imath) when the read the OIIO config files.
(They could do their own `-DOPENIMAGEIO_CONFIG_DO_NOT_FIND_IMATH`,
but I want even that to be unnecessary.)
There was that one pesky place where we were still including Imath
headers, from simd.h in the case where OIIO_SIMD==0 and we wanted to
fall back on some Imath::Matrix methods.
Ugh, but this backfired because when including this file from Cuda,
OIIO_SIMD==0 and it included the header. But what happens if a
downstream app (a) includes this file from Cuda, and (b) uses a
DIFFERENT Imath (so names clash) or doesn't want to use Imath at all?
So one more change:
* Remove the include of Imath.h from simd.h entirely.
* Most places where we fell back on the Imath::Matrix methods, just
replace it with scalar code, it's not complicated. (For some technical
reasons, this required some minor surgery on the matrix44 class, but
not anything that changes anything from a caller's point of view.)
* The one complicated case is matrix44::inverse(). Punt! Just let that
only be implemented if either (a) there is SIMD available, or (b)
there is Imath available (i.e., included by the downstream code
prior to this header). That's it. If you want to invert a 4x4 matrix
without SIMD, you need to arrange inclusion of the correct
ImathMatrix.h yourself before including this header.
For vector displacement maps or normal maps that are in tangent space,
it is heplful to be able to record whether the data are intended to use
a local left-handed or right-handed coordinate system, and for that to
be querably from the shader accessing the map.
The `maketx --handed` option, or `oiiotool -attrib -otex:handed=...`,
or adding "handed" metadata to the configuration ImageSpec being
passed to `IBA::make_texture()` does this. It's easy for texture
formats like openexr that allow arbitrary metadata. For TIFF textures,
we embed it in the ImageDescription tag (as we do for several other
texture-specific hints).
Skip 3 bytes between "software" and "background (key) colour" for the "software
version" (was simply missing). Avoids that m_alpha_type is overwritten with
wrong values.
The very latest fmt master changes a behavior: fmt functions no longer
automatically know how to format any type that can output to an
ostream, and instead need custom formtter objects. This breaks A LOT.
Fix by defining the preprocessor symbol `FMT_DEPRECATED_OSTREAM`,
which restores the old behavior (for now, at least).
While I was at it, I moved the fmt inclusion logic out of strutil.h
and into a new separate header, detail/fmt.h. This is in turn included
from strutil.h, so nothing in user code needs to change.
It turns out that this can crop up all the time, sometimes from
malformed metadata inside image files. We weren't noticing it because
in many cases, things ended up being turned into ustrings twice, and
the sequence string-with-nuls -> ustring() -> c_str() -> ustring()
will launder it. As I was tracking down and eliminating some of the
double ustring constructions, the metadata strings that were malformed
all along started becoming symptomatic.
Bad paradoxes follow from a ustring contain embedded NUL characters,
including ustring.c_str() and ustring.string() being different
character sequences, as well as ustring.hash() not being equal to
strhash(ustring.c_str()). One should definitely be able to count on
ustring(cstring) == ustring(ustring(cstring).c_str()).
So this patch checks for embedded NUL characters at the appropriate
time of ustring construction, and truncates the character sequence at
the point of any internal NUL character.
* `OPENIMAGEIO_IMATH_DEPENDENCY_VISIBILITY` is a new CMake cache
variable at OIIO build time that controls whether the Imath library
dependencies will be declared as PUBLIC (default) or PRIVATE target
dependencies of libOpenImageIO and libOpenImageIO_Util.
* For *downstream projects* that consume OIIO's exported cmake config
files, setting CMake variable `OPENIMAGEIO_CONFIG_DO_NOT_FIND_IMATH`
to true will skip the find_depencency() calls for Imath and OpenEXR.
To clarify, this is not a variable that has any effect when building
OIIO, it's something set in the downstream project itself.
* Add a `OIIO_IMATH_H_INCLUDED` header guard in OpenImageIO/Imath.h,
which allows something downstream to pre-define it to prevent OIIO
headers it needs to read from using the contents of that header.
Most builders of OIIO won't care and shouldn't change the defaults or
use any of these. But here's the specific case it addresses: If you
are building OIIO against static libraries of Imath & OpenEXR, but the
libOpenImageIO will be consumed by a downstream project that wants to
use a different Imath version, this can help you avoid some build-time
conflicts in the downstream project.
PPM by specification is Rec709! Make it so.
PNG files were getting marked as sRGB or Gamma if certain header
attributes were found in the file, but if they were absent, we didn't
designate a color space at all. Change so that the default assumes
sRGB, in the absence of any metadata telling us for sure.
Fixes#3084
One new method that should have been inline and non-virtual (so that
this whole set of changes does not change ABI by shifting the vtable
around) was accidentally marked virtual -- cut and paste error. This
patch fixes it to restore ABI compatibility.
IPTC data blocks seems buggy, and I don't know why. Our code?
Something in libtiff? Since nobody is depending on it and none of our
testsuite appears to fail when disabled, only write the IPTC block if
the user explicitly asks for it. Also, when reading them, don't
overwrite other metadata of the same name.
We wrote our string_view before C++17 std::string_view was finalized,
and the std one doesn't have a c_str() method. For conformance with the
standard, I want to remove this over time from ours.
Part 1: There were some places we did not need to call c_str() at all,
and some places where it was much more appropriate to just use a
std::string.
As Doug Rogers points out in Issue 3304, it is customary for Windows
applications to use wstring extensively for Unicode support. But it's
especially critical for unicode filenames, because unlike Unix-like
OS's where fopen() and similar functions support UTF-8, on Windows
those functions are strictly ASCII, and Windows users are used to
calling _wfopen() and other nonstandard functions that expect
filenames to be std::wstring if they want to support Unicode.
This patch:
* More clearly documents through all of our classes when parameters that
represent filenames or searchpaths are assumed to support UTF-8.
* Augments ImageInput, ImageOutput, and ImageBuf so that their methods
that take file names or path lists have additional flavors that take
wstring (always implemented as an inline shim that translates the
UTF-16 wstring to a std::string with UTF-8 encoding to pass on to the
usual method).
* Augments ImageCache and TextureSystem in just a couple places, the
functions where a filename is turned into a handle. All the many
other functions that take ustring for function names were left alone,
because they are called very frequently and are performance-sensitive,
so we don't want any translation from wstring to string happening on
every call.
We do not attempt to add wstring-based versions of functions that take
strings for other purposes -- that is, that are not intended to
represent filenames.
C isspace() takes an int, but on Windows when building in debug mode,
the implementation of isspace() can hit an assert if the value is < 0,
which of course it can be if you're looking at a character in a string
whose high bit is set (because char is signed).
Easiest solution is to make our own Strutil::isspace that does exactly
what we want and is safe.
Fixes#3300
This makes sure that two new preprocessor symbols are defined in
oiioversion.h: `OIIO_USING_IMATH_VERSION_MAJOR` and
`OIIO_USING_IMATH_VERSION_MINOR`, revealing the version of Imath being
used by the build of OIIO (which may not be the same as the version
found or being used by the app that's calling OIIO). They also get
defined as CMake variables in our exported cmake configs, allowng
downstream cmake build systems to be able to easily test whether the
Imath they found is compatible with the version that we use in our
APIs.
This patch sets out to minimize the extent to which Imath is a
necessary source dependency for using OIIO C++ header files and
APIs. The major changes are:
* Remove the unconditional `#include <OpenImageIO/Imath.h>` from
fmath.h and simd.h. Those forced Imath headers on nearly all all
code that uses OIIO's public APIs, whether or not they needed the
very few calls where Imath types are used as parameters.
* Make sure all public OIIO non-virtual methods and freestanding
functions that have Imath types as parameters are enclosed in header
guards like `#ifdef INCLUDE_IMATHVECTOR_H`. Downstream client code
needs to include Imath headers before including OIIO headers, if
they want to call those few particular functions.
This means that client applications who don't need those particular
functions, but call other parts of the OIIO APIs, don't need to
`#include` any Imath headers (or pay the extra compilation time for
doing so), and don't need any knowledge of which specific version of
Imath is used by the OIIO they are using.
The small number of public API calls that will no longer be visible
downstream unless the client code also include Imath headers (before
the OIIO headers) are:
- color.h: ColorConfig::createMatrixTransform (needs ImathColor.h)
- fmath.h: type conversions involving `half` (needs half.h).
- imagebufalgo.h: warp(), colormatrixtransform() (needs ImathMatrix.h)
- simd.h: conversions between our SIMD types and Imath types (Vec3,
Matrix33, Matrix44) and vtransform() calls involving those types
(requires ImathVec.h and ImathMatrix.h), and SIMD load/store
involving `half` data (requires half.h).
These changes necessitated adding including of Imath.h in a number of
internal .cpp files, the ones where those types are actually used.
The following header files do still unconditionally reference Imath
types or headers:
- texture.h: Some virtual methods of TextureSystem use Imath types
as parameters.
- imagebufalgo_util.h: For `half` support. We use this a lot
internally, but presume most client code never uses this header.
- simd.h, ONLY IF USE_SIMD=0 (which is generally only used for
development debugging and if porting to obscure platforms that are
neither x86 nor ARM).
The bottom line is that the vast majority of downstream callers of OIIO
APIs who need ImageInput/ImageOutput, ImageBuf, ImageBufAlgo, and
generally everything in the OpenImageIO_Util library, do not need to
use Imath types, include Imath headers, or even be aware of which Imath
release OIIO is using.
Why would we do all this?
Because this paves the way for optionally building OIIO to link against
static libraries for OpenEXR and Imath, essentially embedding and
completely hiding an OpenEXR/Imath inside libOpenImageIO.
That allows libOpenImageIO to use the very latest Imath (3.1, say,
with the new faster core API), even though that libOpenImageIO is
being consumed by an application that for whatever reason is forced to
use an entirely different (older) OpenEXR or Imath library. In other
words, it lets OIIO use OpenEXR/Imath fully embedded without
transitively passing that OpenEXR version as a downstream dependency
(except for the few use cases that need to call the tiny number of
OIIO API methods that require Imath types.
There is a small chance that client code that use Imath types and
SHOULD have been includeing Imath headers themselves actually omitted
those includes and dependd on the fact that certain OIIO headers
included them. Now that they don't, it may require the client-side
code to add those includes. I'm going to claim that this is a "bug" in
the client code (they didn't include a header that defined a type they
used), and not a breaking incompatibility in OIIO's public calls
themselves. Thus, I feel comfortable with these changes being in OIIO
2.4 (today's master) and not necessitating a 3.0 release (in which
arbitrary loss of backward compatibility would be allowed). So watch
out for the occasional need for OIIO-calling code to need to add an
`#include` for Imath types the code was always using, but neglected
to explicitly include.
Our Strutil utilities for utf8 <-> utf16 conversion were based on
Windows specific calls that we had added even before C++11 was our
minimum. Replace the Windows-specific code with std calls, and
expose them in strutil.h for all platforms (why not?).
We were passing along, in the exported cmake config files, a demand
for downstream clients to use a C++ standard version equal to that
which we used to build OpenImageIO itself. But that's not
necessary. We may choose to build with C++17 (for example), but all of
our public APIs are safe to use for C++14.
Split the build-time options to CMAKE_CXX_STANDARD for the standard we
build this project with, and DOWNSTREAM_CXX_STANDARD for the standard
we are declaring is the minimum to *use* the library.
The main event:
* Assume that for a TGA 1.0 file (too old to have the header field
that says if alpha is associated, unassociated, or meaningless), if
the alpha is zero everywhere, it means that it's meaningless, not
unassociated, so in that case we should skip the
auto-premultiplication. (See issue 3277)
Public-ish stuff:
* Be more consistent with Targa-specific attributes all being called
"targa:foo" (most were, but one was "tga:alpha_type").
* Document all the targa-specific attributes (a few were not).
* Add "targa:version" to reveal whether the file was a TGA 1.0 or
2.0 version of the format.
Internal stuff, opportunistic:
* Some more minor conversions of string fomatting to fmt style.
* Convert some std::cerr debugging code (commented out) to the DBG macro
we use anywhere.
* The `std::unique_ptr<uint8_t> m_buf` should always have been
`unique_ptr<uint8_t[]>` because it's not a pointer to a single value.
I realized recently that if you said something like
oiiotool foo.exr --ch R,G,B,A -o out.exr
and there was no "A" channel... it would just silently fill the channel
with 0 in the output.
This patch adds a warning if --ch is told to copy a channel that doesn't
exist in the input.
Ugh, there's a situation where PhotoShop is misinterpreting an alpha
channel for certain OIIO-written TIFF files, but it's fine with some that
came from elsewhere, and the only thing that appears to be different is
that the non-OIIO one entirely lacks the EXTRASAMPLES TIFF tag.
So I'm adding an option (use with extreme caution) allowing you to
tell the TIFF output to not output this tag, even though clearly it
should be.
Some rogue app out there is writing 4-channel TIFF files without the
ExtraSamples tag in the header that explains that the data is a valid
alpha channel.
We were using TIFFGetFieldDefaulted to retrieve this tag, and guess what?
The defaulted behavior changed between libtiff 4.0 and 4.3!
Changing to TIFFGetField actually turns out to fall back to the right
logic on our side, which boils down to assuming a 4th channel is alpha
if there's no other information. That seams reasonable -- RGB + mystery
channel is almost certainly RGBA.
* Bugs fixed in build_opencolorio.bash, it botched some of the directory
choices when customizing where to place and build it. Also, switch to
the more standard BUILD_SHARED_LIBS cmake variable to control whether
to build dynamic or static libraries.
* Correctly handle the cmake variable VISIBILITY_MAP_FILE being set to
"" on the command line as a way to disable using the visibility map file
entirely.
* FindOpenColorIO.cmake changes:
- Prefer and use an exported cmake config for OCIO if found (is should
be present for OCIO 2.1+), unless caller set OPENCOLORIO_NO_CONFIG=ON.
- New hint OPENCOLORIO_INTERFACE_LINK_LIBRARIES lets caller force a
list of interface link libraries. This is helpful when purposely using
a static OCIO library, which itself has static dependencies, and you
need to name those specific library filepaths.
* Simplify some of the logic for setting extra link flag arguments for the
two main libraries.
* On Linux, also set link flags `--exclude-libs ALL` to hide symbols from
any embedded static library dependencies, to prevent clashes with other
versions of those libraries that may be compiled into a DCC app.
I realized that a call to make_texture() with the "maketx:ignore_unassoc"
attribute set did not properly set the configuration attribute up when
opening a new ImageBuf.
```include/OpenImageIO/typedesc.h:620:10: warning: 'const' type qualifier on return type has no effect [-Wignored-qualifiers]
auto parse(format_parse_context &ctx) -> decltype(ctx.begin()) const // c++11
```
Co-authored-by: Thiago Ize <thiago@solidangle.com>
The docs said that passing a tuple for min or max param of the clamp()
function gave per-channel values, but passing a single float gave one
value for all channels.
In fact, the implementation used the float to clamp only the first
channel. This patch fixes the behavior.
Fixes#3264
In addition to C++ code examples for each IBA function, work toward
also having equivalent Python code. Also, where applicable, show the
equivalent operation using oiiotool.
We had forgotten to mark --colorcount and --rangecheck as commands
that intentionally outputs to the console. This omission meant that an
oiiotool command consisting only of a those would warn about no
output. But we didn't really want it to.
How do you craft an oiiotool command that will examine a file and add
an alpha=1 channel if it's a 3-channel file, but leave the existing
alpha alone if it's already a 4 channel file?
This patch adds control flow, so you could express the above like this:
oiiotool in.exr --if "{TOP.nchannels < 4}" --ch R,G,B,A=1 --endif -o out.exr
The following slate of new fetures is added:
* `--set name value` sets a "user variable".
* Expression evaluation will expand identifiers that are the names of
user variables. For example, `{i}` is replaced by the value of user
variable `i`, and `{i+3}` returns 3 more than the current numerical
value of `i`.
* Expression evaluation has been extended to support the following
numerical comparison operators: `<` `>` `<=` `>=` `==` `!=` and
`<=>` (the last one being the 3-way comparison, like in C++20).
Also logical operators `&&` and `||`. All return 1 for true, 0 for
false. Note that the comprison and logical operators have the same
precedence as `+` and `-`, so use parentheses liberally to
disambiguate complex mixed expressions.
* New functions recognized within expressions:
- the value held by a named user variable: `var(name)`
- logical negation: `!`, `not()`
- string comparison: `eq()`, `neq()`
* `--if condition commands... [--else commands...] --endif`
* `--while condition commands... --endwhile`
* `--for variable range commands... --endfor` where the range may be,
like Python, any of `end` (implied begin=0, step=1), `begin,end`
(implied step=1), or `begin,end,step`.
Also needed to modify ArgParse so that it has the notion of whether
it's "running" -- actually calling the actions for each argument, or
if it's merely parsing but taking no action -- and to be able to set
and retrieve which command line argument index an action is responding
to.
This little change makes this work:
ImageBufAlgo::add (buf, { 0.2f, 0.2f, 0.2f });
instead of the more awkward
ImageBufAlgo::add (buf, cspan<float>({ 0.2f, 0.2f, 0.2f }));
and for anyplace tht has Image_or_Const parameter types.
New OpenColorIO and fmt versions are out.
* Bump "latest releases" test to fmt 8.1.0, and OCIO 2.1.1.
* Re-adjust CI test fmt versions across many tests to give a better
spread over the supported releases.
* Bump the default version to download and use if none is found to 8.0.0.
We had --line (wrapping IBA::render_line) and --box (wrapping
IBA::render_box).
This adds the obvious --point, wrapping the existing IBA::render_point.
The Python bindings for ParamValueList and ImageSpec allow a "dict"
style (associative array) interface:
imagespec["key"] = value
val = imagespec["key"]
This patch makes some minor improvements:
* Clarify in the docs that like a regular Python dict, the associative
retrieval will raise a KeyError if the key is not found.
* Add `__contains__` python method as a synonym for contains(), thus
implementing `key in dict`.
* Add `__delitem__` python method as a synonym for erase(), thus
implementing `del dict[key]`.
* Add an `ImageSpec.get()` method that is a synonym for getattribute
(which retrieves the item or returns None or a default if not
found), mirroring the behavior of Python's `dict.get()`.
Somehow I never implemented a copy() method for ROI, which of course is
super useful if you want to make an actual copy of an ROI and modify
the copy.
I also realized that ImageSpec.copy() unnecessarily/incorrectly set
return_value_policy to reference_internal, but actually it should be a
real copy, which it has to be anyway because the implementation
creates a separate copy.
* Docs: we previously only documented the "width=" optional modifier.
Add height, color1, color2.
* Change behavior ever so slightly so that if the "height=" modifier is
not supplied, the height defaults to the "width".
* Use the new pybind11 v2.9.0 for 2022 and "latest" tests.
* Bump the default version built by build_pybind11.bash to 2.8.1.
* Adjust the pybind11 version used in many of the other CI tests, to
give an even spread across versions, and to make each of the
"vfx platform" CI tests use the pybind11 version that was available
near the start of its repsective year.
For each ImageSpec ctr that takes a TypeDesc, add another one
that's the same but takes a string_view (which is just a simple
pass-through of the TypeDesc-from-string_view).
Also tagged Strutil and ustring::fmtformat as 'nodiscard', to produce
warnings if people call it and ignore the results. (As was the case in
my own bug that I tracked down in the course of this patch.)
When doing operations with filters that have negative lobes, you can
sometimes get visible ringing artifacts in very high contrast regions
of HDR images (you tend not to see these artifacts in images that have
range [0,1] because the ringing is very low amplitude unless the input
is very large).
You can do "highlight compensation" to reduce the artifacts when they
occur by doing --rangecompress, which does a log transform on the
input, then the filtered op, then --rangeexpand to transform back to a
linear space. But that makes for kind of verbose and clumsy command
lines, and makes it easy to err by doing the transforms in the wrong
sequence or omitting one.
The -otex command takes a highlightcomp=1 modifier, which does this
bracketing automatically, foolproof, with a compact notation.
This patch adds support for optional highlightcomp=1 modifier for all
of the oiiotool commands that let you specify a filter= option:
--rotate, --warp, --resize, --fit, and --pixelaspect.
* Clean up intra-document section references.
* New explanations about input and output configuration hints.
* More code examples in both C++ and Python.
* Fix some formatting flubs.
This was an obsolete experiment in adding a C binding. We abandoned
this effort (a replacement will happen eventually), but this one file
was left behind when we intended to remove it all.
Errors encountered while reading tiles were getting lost as we used
ImageBuf::Iterator to iterate over an image. Have the iterator keep a
flag for whether it encountered an error, set to true if the
ImageBuf::retile indicates an error occurred during the read.
Also, some ImageCache errors might get lost on the way to
TextureSystem calls, fix that as well.
TextureSystem is designed primarily for tiled images (typically 64x64
pixels each), but to handle untiled images, it just treats the whole
image as one big tile. That means for very large untiled images, you
might end up with one VERY large tile, and if the image is large
enough, the size and/or offset computations therein may overflow a
32 bit int.
This patch attempts to make all the offset computations in the texture
system be 64-bit safe.
After setting up the new IOProxy helper methods for ImageInput and
ImageOutput, here we go back to the readers/writers that had used
proxies for a long time, and switch them to using the new helpers.
Add two new global OIIO attributes: "limits:channels" and
"limits:imagesize_MB" which are imposed limits for validating a
reasonable number of channels and total uncompressed image size,
respectively. We presume that an image file header describing an image
that purports to exceed these is most likely corrupted or even
maliciously crafted. The limits, therefore, are designed to avoid
crashes, absurd memory allocations, or other cascading errors that
would result from taking such a claim at face value (for example,
a file that appears to have tens of thousands of color channels).
The limits default currently to a maximum of 1024 color channels, and
a maximum uncompressed data size of a single 2D image to 32GB. These
seem reasonable today, but if such large sizes become likely to be
seen in the wild in legit files in the future, we may change the
defaults. They may be adjusted now simply by setting the attributes,
of course, as should be done by applications that legitimately expect
to see input that exceeds them. (Applications also may make the
validation limits even smaller if they wish.)
I've modified the TIFF reader to check these limits as a first step.
We'll expand to doing such validation checks on other readers over time,
as we have opportunities to visit them.
This string utility computes the "edit distance" between two strings.
Currently it only uses the Levenshtein metric, but it has room for
future expansion by taking a parameter that can specify which metric
to use. Levenshtein is an algorithm for computing how many single
character insertions or deletions are necessary to turn one string
into the other.
I have a few ideas for where this can be useful, but as one
demonstration, I also modify the ArgParse class internals so that if a
command is not found, it will print a suggestion naming the valid
command option that has the closest edit distance (but only if one
choice is <= 2 edis away, and only if it's not a single-character
option, in order to reduce the number of useless suggestions). Here's
its behavior in action:
$ iinfo --stat foo.exr
iinfo error: Invalid option "--stat" (did you mean --stats?)
...
* The main event here is that we're now telling OpenJPEG to use
threading when compressing/decompressing JPEG-2000 files. This
improves the speed of JPEG-2000 reading on my laptop by 2-3x! The
threading feature was added to OpenJPEG 2.2 for decode, and 2.4 for
encode.
* CI: Build OpenJPEG 2.4 for the "latest releases" test, and build
their master for our "bleeding edge" test.
* Fixes to deal with the in-progress 2.5 deprecating a struct field.
* Generally clean up our detection of what OpenJPEG version we are
using.
* Some clean-up of the FindOpenJPEG.cmake file, use PATH_SUFFIXES.
Also fix longstanding bugs in PNM input:
* We weren't correctly emulating random scanline access, as is expected
(but in practice rarely used) of all ImageInput implementations.
* We weren't marking the color space in any way, now we do mark as
"Gamma2.2", which is the correct/preferred interpretation according
to the PNM spec.
* Add full IOProxy support to both input and output.
* This necessitated a major refactor, including sucking the I/O specific
code from rgbe.{cpp,h} into hdr{input,output}.cpp and getting rid of
the old files.
* Add a testsuite entry for hdr (shockingly, there was none before)
As well,
* Add a DDS test, apparently we never had one.
* Remove DDS setting of "texturetype", we don't ever use that, it's only
"textureformat" that we use. Not sure how that crept in there years ago.
Helper functions ioproxy_opened() and iotell() should have been const
all along, since they just return back information and should not
change state of the proxy, and also ioproxy() should have both const
and non-const versions.
* On Mac CI, use openvdb from Homebrew now that it's OpenVDB 9, which
properly supports OpenEXR/Imath 3.1.
* Update our FindOpenVDB.cmake target names to match the names used in
OpenVDB's own FindOpenVDB.cmake (even though we don't use their module,
may as well stick to their convention, especially if they eventually have
a properly exported config file).
For some reason, in the early days I got in the habit of having OIIO
input readers call their color space "GammaCorrected2.2" (for example)
when they appeared to be a straight-up gamma. (Generally only used for
certain fairly old formats that date from before sRGB existed.)
But there appears to be a widely used convention of naming OpenColorIO
spaces "GammaX.Y", so I'm switching OIIO to conform that
convention. It's more straightforward and increases the chances that
it will match the names of color spaces that already exist in people's
OCIO configs.
As we move toward fully equipping all image format implementations
with full IOProxy support, it's clear that a lot of repeated code is
error-prone. I first tried to collect this in a mixin class, but then
re-implemented after realizing that we can use the existing PIMPL
ImageInput::Impl and ImageInput::Impl classes, without changing any
ABI or making new headers.
This patch add a couple data members to the Impl data -- an
`IOProxy*`, (the current proxy, used for both passed-in and created
proxies) and also a `unique_ptr<IOProxy>` used when we have to create
our own and want to make sure it's destroyed with the ImageInput or
ImageOutput.
Then we add a number of protected member functions to ImageInput and
ImageOutput, not meant to be ever called by "users" of the OIIO APIs,
but used internally by II's and IO's to encapsulate several common
tasks related to managing IOProxy's. This coalesces redundant code
that would need to appear separately in most of the plugins that
support IOProxy, helps to reduce cut-and-paste errors from having to
make these changes separately in many places, and centralizes the
logic so that as our ideas of how to implement IOProxy evolve, we can
easily change it in just one place and have it apply to all plugins.
GIF revise with new IOProxy utility methods
JPEG simplify with new IOProxy utilities.
To try to combat long template instantiation times, did some refactoring
on the ImageBuf iterators:
* Pushed more functionality from the templated Iterator and
ConstIterator to the non-templated base class IteratorBase. This
should reduce the amount of code that is separately instantiated for
every data type, in every module that uses iterators.
* Pushed many implementations from inline defined in imagebuf.h to
separately compiled in imagebuf.cpp. The most important methods that
are called on every pixel are still inline, but we are switching to
separate compilation for all the methods that are only called upon
construction (once per image being iterated over) and most of the
code that is called only when we wrap to the next scanline.
* Some other miscellaneous refactor to remove redundant computation in
those classes.
Maybe somebody else is building this as a subproject or has a varible
called "CI"?
* For our CI runs, set an env variable "OpenImageIO_CI" (not just
generic "CI").
* Make sure STOP_ON_WARNING is true for our CI runs, even for release
branches -- but now check OpenImageIO_CI, not CI, in case somebody
else has set that.
This was deleted a few months back, but I think I did not realize
it still served a purpose. We don't use JasPer directly, but LibRaw
does, so if we are using static LibRaw, we still need it here.
Fixes#3209
When the make_texture option monochrome_detect is used to collapse 3
identical channels to 1, it was leaving the first channel name in
place (for example, "R"). That was confusing, because at this point,
we're not trying to represent a red image, we're trying to represent a
grayscale image. A strategic call to `default_channel_names()` clears
it up, imposing the standard "Y" we use for a single channel luminance
image.
Also amended the documentation of IBA::channels(). It turned out not
to be the way I called channels() that was the problem here, but by
the time I figured that out, I had already seen that its docs could be
improved.
Fixes#3204
* --onlyformat or env IMAGEINOUT_TEST_ONLYFORMAT to test only one format.
* --no-delete to preserve the temporary files for inspection
* other debugging checks
* Add a DDS test, apparently we never had one.
* Remove DDS setting of "texturetype", we don't ever use that, it's only
"textureformat" that we use. Not sure how that crept in there years ago.
The previously existing max and min take 2 images and compute the
pixel-by-pixel, channel-by-channel max or min of the two images.
The new maxchan/minchan take one multi-channel image and produce a
single-channel image where each pixel contains the max or min of the
*channels* of the corresponding pixel of the original image.
It is equivalent to this (for an RGB image):
oiiotool in.exr -ch R in.exr -ch G in.exr -ch B -max -max -o max.exr
Except, of course, that --maxchan is a single command, is much more
efficient, and doesn't require you to know the number or names of the
channels of the input.
This was a specific user request.
Note: in the course of making the tests, I noticed that we did not
have documentation or testsuite coverage of the python bindings for
IBA.min() an max(), although they were implemented. So I added that as
well.
* Make PROJECT_VERSION_RELEASE_TYPE be a cache string, so individual
build configs or sites can override it.
* Change OpenImageIO_IS_SUBPROJECT to PROJECT_IS_TOP_LEVEL (and with
the opposite sense), to match CMake 21+ behavior. And no need to set
it ourselves if using cmake 21+.
ImageBufAlgo has both premult and repremult (as well as unpremult).
But oiiotool only exposed unpremult and premult, but not repremult.
This little patch rectifies that oversight.
Just to remind everybody of the difference:
`premult` just multipies the color channels by alpha, like it says.
Note that for a pixel that has Alpha = 0 but nonzero color, this will
crush the color to black.
That's fine if you have an *original* image that's unpremultiplied and
you want to do a one-time conversion to premultiplied.
But there's another use case, which is when you start with a
premultiplied image, you need to temporarily unpremultiply it (like
for a color space conversion), and then re-premultiply it. In that
case, you may have those glowing pixels in the original image, the
unpremult will preserve them, and you DO NOT want to crush them to
black with the subsequent premult. So there is a "repremult"
operation that's like premult, but preserves the color values when
alpha == 0. This is the preferred operation when you are doing the
sequence of "unpremult, do something, re-premult" so that the glowing
pixels make it through the round trip.
When I implemented `oiiotool --invert`, I knew that the default should
be to only invert the first three channels (presumably R,G,B) and not
anything beyond, because we wouldn't want to invert the Alpha channel.
But I botched it, and instead what happened is that if given an RGBA
image, it would just leave the A black in the result image.
So this patch has two improvements:
1. If there are more channels in the source image than we are
inverting, make sure to copy the other channels unmodified, don't
just leave them blank.
2. Allow optional modifiers chbegin= and chend=, so users can change
the default if they want a different channel set to be inverted.
This new IBA/oiiotool command adjusts the saturation of a color image,
with a scale factor. Scale = 0 makes the image gray scale of
equivalent perceptual luminance, scale = 1 leaves color unchanged,
scale > 1 increases color saturation.
This adds a new cmake variable in our build system, TIME_COMMANDS,
that when set ON will adjust the compile and link rule launch to
include a time command. This is probably only useful when also setting
`CMAKE_BUILD_PARALLEL_LEVEL=1` so that build are happening serially.
The use case is to investigate how long each module takes to compile,
for optimizing the build.
* Add a test for MacOS 11
* Updated supported versions of dependencies
* Move linux clang test to clang13 on vfx2022 container
* Switch clang-format test to use clang12
In the early days of OIIO, it was 100MB. Ten years ago, we bumped it
to 256MB. I've heard a rumor that computer memories have grown in the
last 10 years, and so have the size of images some people deal
with. So it seems reasonable to raise it to 1GB.
`sscanf()` still has locale-dependent behavior that we wish to avoid
because it can lead to subtle bugs for people whose machines are
configured for certain locales.
This patch removes from the OIIO codebase almost every remaining
sscanf use (and definitely every one that could harbor locale bugs).
Instead, we use the rich set of parsing functions we've added to
Strutil over the years, which are locale-independent, and in most
cases are also faster than the standard C/C++ equivalent.
The official names we use are "box2" and "box3", but users of Imath
know these as box2f/box3f, so accept these as synonyms when
constructing a TypeDesc from a string.
* Typo in webp section -- said "gif:LoopCount" when we meant
"webp:LoopCount".
* In the code, did not actually set the "oiio:LoopCount" metadata
that the docs said would be set.
* Opportunistic fix: TIFF docs said we only suported IOProxy for input,
but we have supported it for output for a while now.
* `parse_values()` and `scan_values()` both can parse a series of
values in a string, with optional prefix, suffix, and separator. The
scan versions just figure out the values, the parse versions
optionally can update a string_view& like the other parsing
functions in Strutil.
* `scan_datetime` parses the int components of a string that looks like
"YYYY:MM:DD hh:mm:ss" (and a few minor variants).
* Fix stoui -- it was previously casting stoi results, which is not
correct.
* Add `from_string<>` specialties for `double`, `int64_t`, `uint64_t`.
When outputting to formats that don't support negative pixel origins
(such as JPEG), we crop away the negative areas. But we did not
consider the case where the entire data window was in the negative
region!
TIFF has an absolute limit of 4GB file size due to use of uint32
offsets within the file. If you try to write a big enough TIFF image,
it will fail. This is not a software problem, it's a limitation of the
original TIFF file specification itself.
The underlying libtiff supports a nonstandard "bigtiff" extension that
uses 64 bit offsets. This patch switches to this automatically for
images whose uncompressed size would get too close to (or exceed) the
4GB limit. Any such files can be read by OIIO-based software, as well
as anything else using a relatively modern version of libtiff, but may
not be readable by independent TIFF reading implementations.
This is a mixed bag, though. On one hand, this at least lets us write
files that would have been an error before. But note that some files
that would have worked as classic TIFF before -- because their
*compressed* size was still under the 4GB limit, even if their
uncompressed pixel size was not -- will now be written as bigtiff
because we have to commit at the time we open the file, before we've
seen the pixels and know the final compressed size. We just can't
know for sure, for an image with > 4GB of pixels, if the file will get
lucky and be below the limit by the time the scanlines or tiles are
compressed. So for those files, not all apps will be able to read
them (though anything using libtiff >= 4.0 or OIIO will be fine).
Fixes 3156
When ImageInput::create() or open() is called, it first uses the
extension of the filename to guess which reader plugin to try (for
example, "foo.jpg" will try the jpeg reader). If that fails, however,
the default behavior is to then try EVERY reader to see if any will
open the file, in case it guessed wrong, or the extension is used by
more than one format, or the file was simply given the wrong
name. Thus, ultimately, OIIO will never fail to open a valid image
file in a known format, even if the file is wildly misnamed. However,
it's a lot faster to open a correctly named file, because it will try
that one first, before trying a long succession of readers in the hope
that eventually one will work.
There are three cases where this is not the desired behavior:
1. You are confident that files are named with the correct extensions,
and you want a corrupt or invalid file to fail QUICKLY, without trying
every format reader.
2. If the file is corrupt, rather than see a generic "Could not find
any format reader that will open the file" because they all failed,
you want to see the more specific error message for why the (presumed)
correct reader could not succeed.
3. When doing regression tests to make sure that sample corrupted
files are correctly reporting the right errors, we really do want to
see error messages from that one reader we are testing.
This PR exposes a global OIIO attribute, "try_all_readers", which
defaults to 1, but when set to 0, changes the behavior so that ONLY
the one format implied by the file extension is tried, and OIIO does
not subsequently try every known reader.
This new oiiotool command lets you set global OIIO attributes, i.e.,
making calls to OIIO::attribute(). It allows oiiotool command lines to
set some additional OIIO-wide options for which no specific oiiotool
command flags could otherwise access.
When building against OpenEXR >= 3.1.3, exr output now allows zip/zips
compression to take a quality metric (such as compression
"zip:4"). The range is from 1-9, 4 is the default, and it's a control
over speed of compression vs compression ratio (higher number is more
expensive but may result in smaller files). This only affects
compressed output -- it's lossless and will be read to the same pixel
values upon input.
This is also incentive to upgrade to OpenEXR >= 3.1.3, because its new
default (and ours) is 4, versus the old (implied, not directly
controlled) 6. This change was made because compared to 6, it's
significantly faster to write exr files with zip compression 4 (tens
of % faster), while only increasing file size by 1-2%, so that seemed
like the right tradeoff. Lower numbers make much larger files for
little speed gain, and larger numbers than 6 make for much slower
writes of exr files for very tiny compression improvements. But YMMV
and you may want to experiment to find the right trade-off for your
use cases.
OIIO 2.3 marked it as deprecated. This new change is for master (2.4+)
only -- removing all support for Field3D.
F3d is effectively unmaintained, is known to be incompatible with
the latest versions of OpenEXR/Imath, and we haven't found anyone
prepared to speak up as a user of it via OpenImageIO support.
One more boost extraction: tokenizer.
The only place we used it is Filesystem::searchpath_split, which can
easily be rewritten in terms of the Strutil parsing functions.
While I'm at it, add a new version that returns the vector instead
of needing to pass in a ref.
Suggested by Luke Emrose, the idea here is that we introduce two new
MIP interpolation modes: StochasticTrilinear and StochasticAniso.
These work similarly to Trilinear and Aniso, respectively, except that
instead of sampling both of the bracketing MIP levels and blending
between them, in stochastic mode we use a random variate to select ONE
of the two MIP levels with probability proportional to the computed
blending weight.
If these mip modes are used, the caller needs to pass a
well-stratified value in the new "rnd" field of TextureOpt. Also note
that the method only really makes sense in the context of a renderer
that is using many samples per pixel. As a single texture lookup, it
looks horrible and noisy, as you can see by the testshade-based tests
we do of this, which are 1spp and looks pretty rough.
My benchmarks (in a hard stress test of the texture system using
`testtex --threadtimes 8`) show that this approach has speedups of
35-40% when using between 1-4 threads, dropping to more like 25-30%
speedup as threads are in the 12-32 range (that makes sense, it's
doing the same amount of total I/O).
In this first implementation, in order to preserve ABI compatibility
if we decide to backport this feature to the release branch,
TextureOpt::rnd is added as a union co-member with the unused "bias"
field. For TextureOptBatch, we had to add it outright, but it's
guarded by an `#if` and will not be backported to 2.3. (We may change
our minds and not backport it at all, I just wanted to leave the
option open with this initial patch. Also, there will probably be
additional further ABI-breaking changes coming later that will
definitely not be backported.)
In this first phase, we only use the stochastic component to turn the
2-MIP-level lookup into one level lookup. It has been further
suggested that we could apply stochastic techniques to anisotropic
filtering (instead of many samples spaced across the major axis of the
filter footprint, take one sample distributed across the axis), and
even for the ordinary bilinear or bicubic within each level (instead
of 4 or 16 weighted texels, just retrieve one texel chosen with the
weights as probabilities). But these deserve to be benchmarked and
have their quality evaluated separtely and deliberately, and so will
be relegated to a later investigation and implementation.
We will have a C API eventually, but it will be implemented
differently, and this cruft will probably get in the way. If the new
code needs any of this (particularly the test cases), it's easy enough
to revive it from the project history).
When requesting that raw image be returned as "ProPhoto" color space,
we were using the wrong gamma. Also, at user request, add
"ProPhoto-linear", which is the same but with linear response. (This
is analogous to the existing "sRGB" and "sRGB-linear".)
Fixes#3152
Thiago Ize points out that TextureSystemImpl::fill_gray_channels
assumes that the `float*` addresses it is passed are big enough for
a vfloat4, and also properly aligned as such. Not true.
This is an edge case anyway, only called in particular cases where
we are doing 3/4 channel lookups on 1/2 channel files with the option
turned on to expand single channel gray to RGB gray, and also doesn't
do enough SIMD to have a perf impact. So just recode as scalar and
have no more alignment mismatches to worry about.
Also make sure at least one of the Windows CI tests is explicitly
turning SSE on.
Fixes#3128
Field3D is essentially unmaintained, and is currently incompatible with
OpenEXR/Imath >= 3.
In this first stage (which will apply to OIIO 2.3):
* By default, Field3D support is disabled. To enable it, you must
invoke CMake with `-DENABLE_FIELD3D=1`. No warning or other messages
are printed by the build configure step when not enabled.
* If requested, but an incompatible OpenEXR is found, it will be disabled
with a clear error message.
* If requested and enabled, a warning will be printed indicating that
Field3D support will be removed from future versions (2.4 and beyond).
* Add a new test, texture-texture3d, that provides the same testing that
texture-field3d did, but with openvdb.
We have options that allow adding random dither to images before
output, which when converting float images to 8 bit or less, can
greatly reduce the appearance of banding artifacts. This PR makes a
number of improvements:
* When the option is used, it should apply dither any time you are
reducing an image with > 8 bits/channel to an output with <= 8
bits/channel. Not just from float/half, but also from uint16, for
example. (When outputting > 8 bits per channel, there are enough
gradations that you tend not to see banding artifacts, so dithering
is not necessary for 16 bit output, even if coming from a float data
source.)
* Change the actual dither we use to blue noise. The old dither just
used a hash to add uniformly distributed random numbers to each
pixel value before quantization, completely uncorrelated from pixel
to pixel. Blue noise is specially constructed to still have a
uniform distribution of the output range, but it ensures that
spatially, similar values are distributed as far away from each
other as possible, rather than having similar values sometimes
clumped together. The result is a MUCH NICER appearance.
* Of course, this means the we needed to store a blue noise table in
the library, and some access routines for it. We incorporate a
public domain blue noise texture as a table.
* ImageBufAlgo::noise() and `oiiotool --noise` now take "blue" as a
noise name to return blue noise. Also, make "white" the (preferred)
synonym for "uniform" (to disambiguate the fact that both white and
blue noise are uniformly distributed; the difference is that white
is independently random for each pixel, whereas blue shapes the
spectrum to favor high frequencies, making it better suited for
dithering and sampling purposes.
* ImageBufAlgo::bluenoise_image() returns a reference to a stored
periodic blue noise image.
* oiiotool -d would let you say "uint10" or "uint12" to indicate 10 or
12 bit output (for formats that support it), but had no way to ask
for less than 8 bit output (for example, TIFF supports that). So we
add "uint6", "uint4", "uint2", "uint1" as recognized names for the
`-d` option in those rare cases when you specifically want to
produce a low bit depth file. And furthermore, if you ask for
`--dither` when using lower bit depths, the dither amplitude is
adjusted to be right sized for the bit depth you are using.
* Many format output modules that support 8 bit output honor the
option to dither the output, but in the "built-in plugins" chapter
of the docs, most did not list this option in the section that
describes which special output options it knows. So now we've added
this to the docs for all the output plugins that did support it.
Turns out that heif files that contain alpha can have either
associated or unassociated alpha. We had been assuming
associated. With LibHeif 1.12 and beyond, you can ask, so now we
do. Thanks Frédéric Devernay for pointing this out.
Similar to other formats that optionally have unassociated alpha:
* The default behavior is to associate/premultiply automatically when
reading, but set metadata "heif:UnassociatedAlpha" to indicate that
it was unassociated in the file.
* Open configuration hint "oiio:UnassociatedAlpha", when nonzero,
means to please keep the data unassociated and not premultiply. In
that case, metadata "oiio:UnassociatedAlpha" will be set in the
ImageSpec to indicate that the returned image (not just the file)
consists of unassociated values.
I did a couple other minor docs touch-ups while I was at it.
Fixes#3129
New ":C=name" modifier for --dumpdata will cause the dumped image data
to be formatted with the syntax of a C array, that can be dropped right
into source code.
$ oiiotool --dumpdata:C=foo image.exr
// image.exr : 256 x 256, 4 channel, float openexr
float foo[256][256][4] =
{
{ /* (0, 0): */ { 0.517036676, 0.261921108, 0.017822538, 0.912108004 },
/* (1, 0): */ { 0.653315008, 0.527794302, 0.359594107, 0.277836263 },
...
};
Without this modifier, --dumpdata will keep the same formatting it
always had.
This works pretty well for ordinary boring flat, single subimage,
non-MIPmap images. The option should probably not be used (or at
least is very unpolished) for multi-subimage, MIP-mapped, deep, or
other oddball cases (but you probably don't want to shove those into
a single C array anyway).
We had long ago established that the one in fmath was a slightly
faster formulation, as well as having the property of squashing
NaNs. There were a few remaining places where we used a different
clamp so fix them.
The DSLR raw format reader has the ability to pass hints to the
underlying LibRaw requesting that the pixel data be returned to us in
particular color spaces.
* Add new supported color spaces "DCI-P3" and "Rec2020" when using a
LibRaw new enough to understand them -- which means the upcoming 0.21.
* A new name to request the data be returned using sRGB color
primaries but a linear response curve: "sRGB-linear".
* The old name we used for sRGB-linear, "linear", is now considered
deprecated since it's ambiguous -- since its first use, new color
spaces "ACES", "DCI-P3", and "Rec2020" have been added, all of which
are also linear response.
When reading tga files, make sure that every seek and every read
checks results and issues error messages. Also double check that
large allocations succeed.
Also improve error propagation in several spot of iinfo, oiiotool
Owing to our C++11 focus in previous years, we had a number of places
were constants were declared as `static constexpr`. When done in
headers, this potentially causes separate copies of those constants to
be instantiated in each module that includes the header. It's not
really harmful the way we're using it for small constand objects, but
it's wasteful to potentially have dozens of copies of each those
constants by having it repeated in each module.
C++17 has a new idiom, `inline constexpr` that is more like we want --
makes it a true compiler constant that doesn't get instantiated in the
memory of each module.
We still have to cater to C++14, and in platform.h we already had a
C++-standard-dependent definition of OIIO_INLINE_CONSTEXPR that was
almost correct. Fix it (it wasn't previously used, but I must have
seen this coming at some point) so that it expands to
`inline constexpr` for C++17 and beyond, but stays as
`static constexpr` for C++14, and then use it for these constants.
Fixes#3117
If think that our inventory of UDIM tile files won't find names with
wildly out-of-range tile indices, you haven't been working in a
production studio! When those indices imply tile numbers < 0, we were
running off the beginning of the vector when filling in the udim
table, to hilarious results.
Oiiotool::extract_options was kind of fishy and brittle before --
splitting the string at ':' characters, then split each piece (which
should form a `name=value` pattern) at the first '=' character. There
are probably many malformed option strings that would have had weird
results with the prior naive parsing method. And it has been noted
that it can't handle a value that is intended to be a Windows filename
(like "C:/foo"), nor a searchpath (like "/usr/bin:/usr/local/bin").
Rewrite to parse it more carefully as name=value pairs delimited by
':'. In the process, we now have the ability to have the value be
enclosed by a single or double quoted string, in which case the
contents of the quote may themselves contain a colon, for example, the
following command and modifier is now no longer a problem:
`--foo:filename='C:/blah':count=3:`
We were already passing this on to libtiff when it handles the
compression. But for the case where we were calling zlib ourselves
(to facilitate multi-thread compression, which libtiff doesn't do),
we were just using the default compression.
This is inspired by a great [blog post](https://aras-p.info/blog/2021/08/05/EXR-Zip-compression-levels/)
by Aras Pranckevičius in which he made a similar change to OpenEXR
and benchmarked the results. He reported approximately a slightly more
than 2x write speed increase, while getting files that were only around
2% larger.
I tried replicating his experiment for TIFF, and while I don't quite
get results as good as he did, it still is really interesting. I
tested with 100 images randomly selected from the [Animal Logic Lab
USD scene](https://animallogic.com/usd-alab/) (they are almost all
4k^2, a mix of 1- and 3-channel images), converted from half exr to
uint16 TIFF images. I ran the tests for both scanline and tiled
images.
Compared to the default zip level 6, using level 4 resulted in 3%
bigger files, and around 40% faster write times. That seems like it
is a potentially very helpful tradeoff.
I tested the same set of images also with zip levels 8 and 2. For
level 8, the files were only another 2% smaller than the default level
6, but the write speed was 45-50% slower, so there is little value to
asking it to work harder on compression -- it just burns up time. For
level 2, it went somewhat faster (maybe another 10% faster than level
4) but the files started getting bigger, level 2 was 12% bigger than
level 6, that's definitely getting into territory where people might
mind the larger files.
So I want to add the control now, though I'm deferring a decision on
changing the default value until I can do some more extensive
tests. In particular, I would like to assess how it behaves with 8 bit
files (I was testing uint16, specifically), how well it works with
rendered images and with natural/photographic images (I was testing a
set of textures, which may not have the same characteristics), and
perhaps also see if results are better or worse for lower resolution
images.
The old libOpenEXR/libIlmImf (default) implementation handled
multithreading on the exr side of things, inside the readPixels method.
The new (experimental, enabled only by option) OpenEXRCore
implementation leaves the threading to the caller, and we weren't
doing it, so the performance was unsurprisingly awful.
This patch turns the implementations of the exrcore-based
read_native_scanlines and read_native_tiles (and also the deep
versions thereof) to use parallel-for loops, restoring parallel
processing to the exr reader when using OpenEXRCore.
Reading of whole scanline and tiled images now is on par or slightly
faster (a few %) than when using the main OpenEXR library. Reading of whole
scanline and tiled deep images is substantially faster (2-3x!) than
before.
I did some extra refactoring while I was there, simplifying and
eliminating some redundant error handling and using a new
DecoderDestroyer utility class.
STILL TO DO:
* See the long comment at the end of read_native_scanlines -- the error
message strategy implemented here is temporary, I'd like to improve it,
will probably involve both some work here as well as on the openexr
side.
If set to ON, will include the testtex program in the install.
The default is OFF, which builds it for tests, but doesn't install it
with the real command line utilities.
In forward looking code (current master means OIIO 2.4, which won't
be released until mid-2022), get rid of some stray logic catering to
OpenEXR releases 2.2 and older.
Background: Old style CMake was to use FindFoo.cmake modules, which
set FOO_INCLUDES and FOO_LIBRARIES variables. Best new cmake practices
are for the upstream packages themselves to export a FooConfig.cmake
and friends, which set up cmake targets. Of course, many packages don't
provide exported configs, so we use the old Find modules for many of
our dependencies.
I've slowly been migrating our Find modules to set up targets in addition
to the variables, and favoring use of the targets over the variables.
But I was wrong about this. It's more than just a choice of style.
When targets are used, use of transitive linkage (especially for static
library versions of dependencies) make the dependency targets show up in
our own exported OpenImageIOConfig.cmake. But to a downstream package,
now our config talks about a target like Libsquish::Libsquish, but it
doesn't know what that is, and it doesn't have the FindLibsquish.cmake
module that we used (internally) when we found that dependency.
This was causing particular problems for Libsquish on Windows with
vcpkg, but I see now that this could be a problem elsewhere as well:
field3d, libheif, openvdb, and webp.
So this patch:
* Reverts from the ill-advised targets back to the old style
variables, for the dependencies where we use *custom* Find modules.
(We presume that a dependency's exported configs, as well as any
Find modules that are distributed as part of CMake itself and define
targets, are done correctly, and will be available to downstream
packages, and so we continue to use targets in those cases.)
* Beefs up our FindLibsquish.cmake module to correctly find debug
versions of the library, to use in debug builds.
* Alters build_webp.bash to allow WEBP_BUILD_TYPE to override the build
type, which had previously been hard-coded to Release.
* Removes search for HDF5, which as far as I can tell was unused
directly by OIIO.
Instead of a build-time option for enabling (and always using) the
experimental OpenEXR "core" library...
* Build support for it by default any time we're using a new enough
OpenEXR version (>= 3.1). You can still hard-disable it being compiled
in at all with `-DOIIO_USE_EXR_C_API=OFF`.
* Select whether it's actually used with the global OIIO attribute
"openexr:core" (default to 0, disabled, for now).
This easily lets an app select support at runtime, and also may be
selected by user with environment variable
`OPENIMAGEIO_OPTIONS="openexr:core=1".
This all is intended to make it easier to benchmark use of this new
feature, switching back and forth without a rebuild.
* Honor env $OPENIMAGEIO_FONTS as an additinal font searchpath.
* Global attribute "font_searchpath" as an additional font searchpath.
* Refactor and simplify the code that assembles the list of directories
to search.
When $OCIO was not set at all, the call to OCIO::GetCurrentConfig()
was checking it again and writing the info message that none was found
to stdout, which we really don't want. And as Patrick pointed out
recently, asking for the "current" config is suspect, since who knows
exatly what the app has done.
So we're making the following executive decision: At least from OIIO's
viewpoint, if you want an OCIO color config, you either have to name it
expliticly, or have it determined by the `$OCIO` env variable. That makes
cleaner logic, and also rids us of the info messages.
I also now have the exported cmake config set
`OpenImageIO_HAS_OpenColorIO`. This makes it easier for downstream
consumers of our exported OpenImageIOConfig.cmake to know if the OIIO
build they have found has build-in support of OpenColorIO.
When it comes time to read_image() into the ImageBuf's internal
memory, we should have said to convert to `m_spec.format`, which
describes the buffer we allocated. Instead, we asked for `convert`,
which is the user request, which at that time can still be UNKNOWN,
which indicates to preserve the file's data format, which in fact does
not necessarily match the internal buffer.
This becomes especially troublesome for images with mixed per-channel
data types, which will obviously never match the internal buffer since
ImageBuf only internally stores one data format for all channels.
This is a very low priority change, but every little bit helps when it
comes to limiting boost's footprint as thing we used to require it for
become available in the standard library.
Add tests to be sure we have reasonable error reporting for at least
two important cases: a specified OCIO config file is not found at all,
and an OCIO config is present but cannot be read (in particular, if
the config is newer than the library). Because the exact error spewage
is different for each OCIO version, and apparently also depending on
the exact version of the yaml library that OCIO used, we have a few
different possible output references.
* oiiotool --nostderr (new hidden command)
This hidden flag causes oiiotool errors to go to stdout instead of the
default stderr. It only does this for oiiotool's own error and warning
calls; if there are other places within library functions that
directly output to stderr, they will not be redirected by this
command.
The main purpose of this is for testsuite and debugging. I don't
expect ordinary users to have any use for this, and so it is currently
hidden from the oiiotool --help explanation. If it seems useful for
something other than testsuite, I'll expose it in oiiotool --help.
It seems that for the case of OCIOv1 reading a v2 config, OCIO itself spews a bunch of warnings to stderr. And apparently the order is nondeterministic, because I just could not get any set of reference error output to match stably from run to run in CI. But the problem is, if I don't capture stderr, I also don't get the oiiotool error messages, which are the very thing I'm trying to test!
So what I did is add a new flag to oiiotool, --nostderr, which causes oiiotool's error and warnings to get sent to stdout rather than stderr. This lets me capture the usual output and the error messages from oiiotool, for this test, while allowing OCIO's stray stderr output to go into the ether where we don't care about its order.
I'm not sure if there's any non-CI, non-debugging application for oiiotool --nostderr, but maybe it'll turn out to be generally useful.
When OIIO that was built against OCIO 1.x, but encounters an OCIO 2.x
config file, OCIO could throw an exception that was not caught by
OIIO. The problem was the call to OCIO::GetCurrentConfig(), it should
have been inside the try/catch block.
Calling close() in open will forget too much state. All we want to do
here is make sure the TIFF file itself is closed at that point (so it
can be re-opened in append mode to add a subsequent subimage).
* CI test for VP2022, fix revealed python binding bugs
* CI test against ASWF's latest VFXPlatform-2022 docker container,
including Python 3.9.
* Pesky crash resulted in testsuite/python-imagebufalgo, laboriously
traced to IBA_color_range_check releasing the GIL too liberally. The
scoped release cannot contain the tuple construction!
* Audited the rest of py_imagebufalgo.cpp and found the same mistake in
only one other place: IBA_histogram. Was asymptomatic so far, but fixed
now.
* Presumably we just got lucky and these bugs were never symptomatic
until we started building against Python 3.9.
* Noticed that the recent change in build tree layout where we now
copy the test images into the build/testsuite area, resulted in HUGE
GHA build artifacts for failed tests. Modify ci.yml use of
upload-artifact action to exclude those directories.
* more tweaks
oiiotool --runstats tries to report how much time was taken by each
operation. But it has always had a major flaw: Because we try to read
input files lazily and use ImageCache underneath, only in certain
circumstances do the true cost of reading the files show up in the
entry for "-i"; often the cost gets wrongly credited to the other
command (like --add or -o) whose operations triggered the I/O.
This patch addresses the problem, and I think correctly credits I/O
time where it belongs.
The workhorse here is a new helper class within oiiotool,
OTScopedTimer, which does some tricks to try to account for both
direct read time and ImageCache I/O time, subtracting those from the
op it was instructed to time and instead adding it to the counter for
"-i". Because the destructor apportions time to the right commands,
this eliminates a whole bunch of spots where we did that by hand.
Also fixed a few places where ops directly called ImageRec::read()
instead of Oiiotool::read(), which would also result in bypassing some
timers we relied on. I made the ImageRec one be private, so harder to
make that mistake in the future.
Use a cmake idiom so that you need an explicit `-DCMAKE_INSTALL_PREFIX=`
in order to write over /usr/local or any potentially sensitive area.
In the absence of this being set explicitly, it will configure the
cmake build so that an "install" ends up in ./dist (which is what the
Makefile wrapper did all along).
Before doing all the work of copying the image, first see if the
`--ch` is actually specifying any change versus the image as it
stands. If not, skip all the copying and just stick to the image we
have.
The main benefit here is if you have a --ch primarily for "conformance",
e.g. you want to reduce to a known subset of channels and remove any
extra that may be unanticipated in the input, but it's possible that
the input will come to you already with only the desired channels. As
an example,
oiiotool in.exr --ch R,G,B ...more ops... -o out.exr
Previously, if in.exr aleady had channels R,G,B (and no other), there
would be some unneceessary expense of creating (in memory) a new image
for the output of --ch copying data to fill it in. With this patch,
the --ch would very nearly turn into a no-op. In both cases, if
in.exr's channels were something other than R,G,B, channel copies
would need to be done identically in both old and new code.
I realized after this was merged that I often share and copy a lot of
these little dependency build scripts across multiple projects, so I
wanted to remove the OIIO-specific naming.
To help make testsuite pass even with nonstandard relative path to the
build area, slightly change how we reference the OCIO config that is
in the testsuite.
Fixes python error about OIIO_TESTSUITE_ROOT not defined in the case
where the testsuite dir wasn't found in the standard places. This can
happen if you put the build directory in an unusual location that's
not right in the main project area.
* Ensure that "CMAKE" built "OpenCV" libraries can be found.
* Fix incorrect url to "openexr-images" repository.
* Fix typo in "FindTBB.cmake" module.
Co-authored-by: Thomas Mansencal <tmansencal@wetafx.co.nz>
* New SETVARIABLES option sets named cmake variabes to true if the
package is found.
* DEBUG should print variables even if not found. Duh, failing to find
the package is the most important debugging situation.
For historical reasons, our wrapper makefiles assume that we build in
`build/ARCH[.variant]`, so maybe build/linux64, build/linux64.debug, etc.
I think this is a throwback to literally decades ago when I would
build multiple architectures from different machines that viewed a
common shared directory of code, this allowing them to build
simultaneously and/or erase one without needing to rebuild the others,
etc.
But now I find that:
* It's contrary to modern cmake practice, and there are many benefits
to having our build directory structure identical to the vast majority
of other open source projects.
* I never want or need those simultaneous builds in practice, haven't
for a long, long time.
* ccache means that there is no longer an onerous cost to, for example,
having to blow away a release build and do a debug build and then
blow that away and do release again. It's just a few seconds to do a
ccache-based full rebuild of a recently-built tree.
So this patch switches all our assumptions to having a single level
hierarchy to the build and dist areas.
Also, I'm changing the assumption that the oiio-images (and other data
sets used by the testsuite) live "up a level", but instead added code
that either copies it from there (or another area settable with the
cmake variable OIIO_LOCAL_TESTDATA_ROOT) or downloads it, into the
build/testsuite directory itself. This simplifies a great number of things
in the test scripts as well as the test reference output (a lot of which
changes slightly to accommodate the new paths to test files).
* raw: Fixed output when raw:Demosaic == "none"
* raw: Include metadata regarding filter pattern
* raw: Added missing call to set unpacked status
* raw: Implement "balance_clamped" mode to help HDRI processing
* raw: Implement raw:apply_scene_linear_scale feature
* Fix up formatting with clang-format
* Update testsuite reference output
* raw: Fixed error message format handling
* raw: Updated documentation to include new raw features
* Ensure that all the `build_*.bash` scripts don't download anything if the packages appear to already be there.
* Env variable `OIIO_DEP_DOWNLOAD_ONLY` can be used to do the download only (if not present) and skip the actual build.
This combination allow the scripts to be run twice: once to do all the network I/O to download the packages we'll need, and a second time to trigger the builds.
The existing --info and --stats seem a little confusing and
inflexible. Unlike the majority of oiiotool arguments which are
evaluated in order, these merely set flags which apply to the entire
oiiotool command (specifically, they cause all image inputs to print
the metadata or image statistics at the time that the input are read
from disk). There isn't a way to make it apply to a single image, nor
to print the metadata or stats of any *computed* image, leading some
people to awkwardly save a computed image and re-read it merely to
print information about it.
So in this patch, we add new commands
* `--printinfo` which is like `-info -v` but is done immediately for
the "current" image on the top of the stack (even if it's a computed
image and does not correspond to any disk file).
* `--printstats` which is like `--stats` but is done immediately for
the current image on top of the stack.
Furthermore, `--printstats` takes an optional modifier `:window=`
which can be given a rectangular region in either of the notations
we use ("WxH+X+Y" or "xmin,xmax,ymin,ymax"), for example:
oiiotool blah.exr --printstats:window=30x30+1000+500
prints the statistics of the 30x30 region whose upper left corner
is pixel (1000,500).
The code to implement this is actually pretty short and straightforward,
but this patch turns out to touch a lot more. I needed to plumb
some of the info-printing internals for this support and did some
refactoring along the way.
Also, upon adding these to the docs, I realized that it would be
better to reorganize a bit to separate descriptions of the "flags"
from the "actions", and so many of the oiiotool arguments got shuffled
between different subsections, and in a few cases the descriptions
themselves got rewritten for clarity.
To help applications that need to deal with UDIM textures, the
TextureSystem now exposes the following public API calls:
* is_udim() : is a filename (or a `TextureHandle*`) referring to a
UDIM pattern / virtual file?
* resolve_udim() : given a filename (or handle) and s,t coordinates,
return the handle for the concrete file corresponding to those
coordinates.
* inventory_udim(): retrieve the full list of concrete filenames, as a
vector indexed by `(utile + vtile * nvtiles)`, and the nutiles,
nvtiles counts. Empty filenames in the vector indicate UDIM tiles
that are not populated.
Some extra functionality falls out naturally from these and existing
ingredients:
* inventory_udim + get_texture_handle together can get you the list of
handles corresponding to each found udim tile.
* resolve_udim + get_texture_info together can retrieve the attributes
of the udim tile corresponding to a particular texture coordinate.
Additionally, I have done some refactoring and performance
improvements of the internals:
* The internal UdimInfo that's stored inside the ImageCacheFile record
is now a vector, not a flat_map, so conversion of texture
coordinates to a handle is now O(1) rather than O(log n).
* Per Luke Emrose's suggestion, use `rcontains()` when searching for
UDIM pattern markers, since they will almost always occur close to the
end of the full pathname.
* Added TextureSystem and ImageCache methods that lets you find out
the filename that corresponds to a handle. (That's bound to be handy
for somebody.)
* Beefed up the udim tests to be sure to test every udim notation that
we support.
* At the suggestion of Luke Emrose, add rcontains() and ircontains(),
which are like contains/icontains, but start the search at the end
of the string rather than the beginning. This is measurably faster
than contains() for the case when you are reasonably certain that if
the search pattern is found at all, it is highly likely to be close
to the end of the string. (They're approximately the same speed when
the pattern is not found.)
* To verify this, I added a bunch of benchmarking to strutil_test.
* The benchmarks also revealed that contains was about 2x slower than
calling find! It should be the same. I don't know what the underlying
boost::algorithm::contains is doing differently than find, but whatever.
Changing our implementation of Strutil::contains to simply call find
doubles its speed.
* The new implementation of contains has an extremely minor behavior
change for the degenerate case of `contains("", "")`, which used to
return true and now returns false. Since the right behavior here is
debatable at best (is it false because the first empty string
contains nothing? or is it true because containts(a,b) should return
true if a==b?), I'm going to accept this behavior change as well
worth the 2x speed improvement (besides, "contains(a,b) should have
the same result as find(a,b)!=npos" is also a reasonable
expectation).
* The benchmarks also reveal that the case-insensitive functions (like
ifind) are not just a little bit slower than their case-sensitive
siblings (like find), they are 20x or more slower! I don't think
there is anything to do about that, but I did note it in the
documentation of those functions so that we are disuaded from using
them except when a case-insensitive test is really necessary.
The primary issue being addressed here was that get_texture_info() on a
udim pattern filename (like "foo_<UDIM>.exr") was unable to retrieve
info, except for an extremely narrow range of queries, because of course,
different UDIM "tiles" might have different attributes or be missing.
So in consultation with Lee Kerley, we decided that get_texture_info should
work as follows:
* If the attribute exists and has the same value for all of the
matching UDIM file files, then succeed and return the value.
* If any of the tile files does not contain the attribute, or if the
attribute doesn't have the same value in all found files, then it's
a failure and the value cannot be retrieved.
This necessitated a more rebust accounting of all the tile files
comprising a UDIM pattern, by doing a full inventory the first time
the UDIM pattern is encountered, rather than (ick) iterating through
all possible tile files every time we retrieve an attribute, as we
were apparently doing before. And so the following improvements were
also made:
* get_texture_info now works for more queries -- in fact, any query
that would be possible to ask about on an individual file, without
restriction. (Though, as I said, the query will fail if the value
isn't identical across all the found tile files.)
* The info queries (even the few that were previously supported) will
be MUCH FASTER than before, because we are doing the tile file
inventory once per udim pattern, rather than on every query.
* That inventory will also pre-build the association between u,v tiles
and the concrete filename (and texture handle!), so that each
individual texture call does a simple map lookup to get all the way
to the texture handle, without any string manipulation to first
construct a concrete filename, then look that new filename up in the
cache to get a handle, etc. I'm expecting this to reduce some of the
overhead of using UDIM files with the TextureSystem.
Note: I added an ImageCacheImpl::find_file_no_add at some point in
this work, eventually not needing it by the time I got to the final
version of my implementation of this PR, but I'm going to keep it
because I can forsee its usefulness and I don't want to delete it only
to write it again some day.
We now think that this C API should be removed, and replaced by a
different one coming soon.
Merely commented out the option and other lines in the top-level
CMakeLists.txt, since almost the exact thing will be needed eventually.
Also kept libOpenImageIO-C/CMakeLists.txt since we'll eventually need
one much like it. But deleted the .cpp and .h files, since those will
be totally replaced.
For the upcoming OCIO 2.1, the library names will incorporate the
major/minor version numbers, so our Find module needs to take that into
consideration.
Work towards eventually using OCIO exported configs (though still not
100% working now, may need fixes on their end).
Change CI to use the OCIO's that come in the aswf containers instead
of building from scratch in many cases.
Sometimes we use a filename, or pieces of it, as a regex.
But filenames can have characters that are interpreted differently when
part of a regex pattern (like `.`, which in a filename is literal, but
in a regex means to match any character!).
So this handy utility inserts the right backslashes so the filename will
end up as a literal match when part of a regex.
Furthermore, there's an optional parameter `simple_glob`, which when
true, will also substitute `?` with `.?` and `*` with `.*`.
We then use this to fix a latent bug in scan_for_matching_filenames():
it could have incorrectly matched files during the search.
I noticed when working on an ImageCache thing that even the smallest
change to imagecache.h was causing a substantial amount of the
codebase to needlessly recompile even though very few places really
need to use the IC APIs directly.
It took a little bit of juggling of some declarations, but this patch
allows imagebuf.h to be able to treat ImageCache and ImageCacheTile as
opaque, forward-declared types, without the need for every file that
needs imagebuf.h to also have to parse imagecache.h and everything it
might pull in.
This speeds up my development cycle substantially in many cases, and
it also speeds up compilation of user code (by a small amount) by
avoiding parsing of imagecache.h in circumstances where the user code
only needed to know about ImageBuf but not ImageCache.
There's a downside to this -- it's possible that user code that truly
needs both imagebuf.h and imagecache.h only includes the former
(inadvertently relying on the fact that the latter was transitively
included). These apps may need `#include <OpenImageIO/imagecache.h>`
to be added after upgrading to OIIO 2.3. Is that really really bad?
Added this handy function that figures out if the first string contains
any character present in the second string.
Also rewrite the comments describing parse_until and parse_while for
better clarity and correctness.
Some of the OpenEXR sample files had an attribute "version".
All along, I assumed this was telling me the version of the OpenEXR file
itself, so I was passing it along and also renaming "openexr:version".
Turns out I was wrong! This is just arbitrary metadata. Who knows why
it's in some of these files? Maybe nobody. But it's not meant to indicate
the file format version, and indeed it does not correctly reflect it.
In fact, there is no simple API call in the OpenEXR API that tells you
the file format version, you have to infer it from the features
present (multi-part or deep or long attribute name indicate changes
that came along with OpenEXR 2.x, but there is no version number in
the file per se).
So I'm just going to stop doing this rename, let "version" be "version",
it's just some user's metadata, I guess.
A few image formats (but not most) support "thumbnails" -- low-res preview
images.
Traditionally, we have shoved them in the ImageSpec metadata as a byte
array (and ints for the width, height, nchannels). This is less than
ideal: It makes the metadata very bloated, taking more memory and
being more expensive to copy metadata, and potentially wastes time
doing I/O on this data that most applications don't want or
need. (Remember that any alterations to the image will make the
thumbnail "wrong", also. So it's rarely useful except for applications
that specifically want a preview image.)
Discussion in Issue 2902 led us to think that thumbnails should be
treated explicitly by the API, rather than shoved into the metadata in
this way. This patch is the attempt to overhaul thumbnail handling:
* ImageInput has a new `get_thumbnail()` method that can retrieve the
thumbnail (if it exists) into an ImageBuf. The base class / default
implementation does nothing; the image formats that support
thumbnails should have their readers override this method. They are
encouraged to read the thumbnails "lazily" if possible -- only
if/when this method is called, and avoid the cost, memory, and I/O
if not requested.
* ImageOutput likewise has a new `set_thumbnail` method.
* These are implemented for Targa and PSD, the only two formats for
which we previously had implemented thumbnail functionality. (There
are probably other formats that support thumbnails, but for which we
never added OIIO support. Leave that for another day.)
* ImageBuf has new `has_thumbnail`, `get_thumbnail`, `set_thumbnail`,
and `clear_thumbnail` methods.
* ImageCache has a new `get_thumbnail` method.
* oiiotool and iconvert attempt to preserve a thumbnail from input to
output (but of course, only if both file types support it).
* ImageBufAlgo::IBAPrep clears the thumbnail in the destination image,
on the theory that pretty much everything IBA would ever do will
likely invalidate any preview image that it started with. This gets
us to the point with oiiotool where a straight copy (like a format
conversion) `oiiotool in.tga -o out.tga` will preserve the thumbnail,
but just about anything else `oiiotool in.tga --stuff -o out.tga`
will not transmit the original thumbnail to the output (which would
almost certainly be wrong).
There is more work to do in the future. Some file formats that support
thumbnails don't yet have OIIO support for it. I'm not sure we ever
really want (except in the case of a truly straight copy) for oiiotool
to keep any input thumbnails; maybe if a thumbnail is wanted for
output, it should always be created anew (by downscaling) so that it
definitely matches the main image? Maybe also oiiotool needs a command
to explicitly extract and save the thumbnail (is that needed for any
legit use? or only as a debugging tool?).
Imagecache repeatedly has to determine whether a filename is a type
of file that's "procedural" and does it expensively -- every time, it
creates the type of ImageInput in question just to ask it if it
supports("procedural").
We can ask this of each plugin type once, when we first register it.
Then just look it up when we need it.
This is strictly internal, has nothing to do with public APIs.
OpenCue-launched apps are encouraged to honor the setting of
CUE_THREADS to limit the number of threads they use. This change makes
oiiotool (and other OIIO-based tools and scripts) better citizens when run
as part of an OpenCue job.
CUE_THREADS has lower precedence than the more OIIO-specific
`OPENIMAGEIO_THREADS`.
For ordinary command line usage, this should not change behavior, since
you wouldn't expect CUE_THREADS to be set.
(Also fix docs which incorrectly named `OPENIMAGEIO_TEXTURE_OPTIONS`.)
And make sure to switch all of our internal uses of those to the
preferred new equivalents!
Also, hide some deprecated things from the documentation system so that
new people won't be tempted to use them.
These are all things that have been deprecated starting with OIIO 2.0
(or in some cases, even earlier). Multiple releases (2.0, 2.1, 2.2)
have gone by with them commented or documented as deprecated, but
without our adding the warnings. It's about time. We're still not
taking them away entirely, I usually don't do that until we've had at
least one release where it's been marked as OIIO_DEPRECATED, and also
only do true removals of API at major releases. But we're laying the
groundwork for being able to totally remove these when we eventually
have a 3.0 release.
This patch modifies the build system to make it possible to control
whether OIIO should link Boost in the form of static or dynamic
libraries, independently from the LINKSTATIC setting, without
having to use the fully custom route (BOOST_CUSTOM).
This is done by preserving the value of the Boost_USE_STATIC_LIBS
variable, if it has been defined by the user. If the value
of Boost_USE_STATIC_LIBS has NOT been explicitly defined, it is
set to match the value of the LINKSTATIC variable, which should
result in Boost being linked in the same way as other dependencies.
Slightly amends PR 3012
* Add docs for this recently-added feature.
* Change the make_texture configuration hint to "maketx:uvslopes_scale"
for symmetry with all the other texture conversion configuration names.
Everything works as before, including setting "uvslopes_scale"
metadata in the file. (But still accept the name without the
prefix as a synonym.)
* Change it to a float for a bit more flexibility.
Ordinarily, --mosaic takes the N images you have, arranges them into a
rectangular array of "cells" where the cell size is simply the biggest
image of the collection.
This adds the optional modifier `:fit=WxH` that lets you specify a
cell size expliticly. Images will be sized to fit the dimensions in
the same way as the `--fit` works (maximally filling the dimensions
while preserving aspect ratio of the original image).
This is particularly helpful for creating a contact sheet from full-size
frames, without having to do any resizing as a separate step.
The existing semantics of `oiiotool -info`, `-info -v` and `-stats`
are that these options set flags so that each time a file is opened
for the first time, it will print the metadata and/or stats. But these
do not allow you to create new images and print their metadata or stats
(at least, not without writing to disk and reading those files).
This patch extends the expression substition syntax to recognize the
following new metadata notations:
* `METABRIEF` : a string containing the brief one-line description that
would be printed with `oiiotool -info`.
* `META` : a multi-line string containing the full metadata that would
be printed with `oiiotool -info -v`.
* `STATS` : a multi-line string containing the image statistics that would
be printed with `oiiotool -stats`.
Some interesting use cases are illustrated with these examples:
# Roughly the same behavior as `oiiotool -info img.exr`:
oiiotool img.exr -echo "{TOP.filename} : {TOP.METABRIEF}"
# Perform computations on an image and print statistics about
# the computed image (not the original):
oiiotool img.exr -blur 3x3 -echo "blurred stats: {TOP.STATS}"
# Print statistics on intermediate results... and keep going with
# more operations:
oiiotool fg.exr bg.exr -over -echo "{TOP.STATS}" -o comp.exr
# Write metdata as "burn-in" on an image:
oiiotool in.exr -text:x=10:y=20 "{TOP.META}" -o burned.exr
The implementation in the business logic of oiiotool.cpp was very
simple. The more extensive change behind the scenes to make this
possible was a massive refactoring of printinfo.cpp, which previously
just wrote everything to stdout in a totally undisciplined fashion,
and transforming it to very carefully write to output streams instead,
which could be passed a stringstream to render the metadata listing or
stats output as a string. Also needed some refactoring to allow it to
take either the name of a file to read anew (as it did before), or an
existing ImageBuf/ImageRec (to support the new feature).
Internally, ImageBuf has for some time now supported strided or
non-contiguous buffers (for IBs that reside in memory, rather than in
an ImageCache). It was originally anticipated to accommodate cases
where we would want pixels or scanlines to have particular alignment
or spacing, such as to be even multiples of the SIMD width.
But we never allowed the kind of IB that "wraps" an application's
memory buffer to specify the strides, in case the app buffer was in
some layout that was not contiguous. This patch adds that ability, by
amending that constructor with stride values (which are optional and
default to AutoStride, which indicates that contiguity should be
assumed). This is 100% back compatible for source code.
This lets you do some cool things with in-memory buffers:
* More flexible about the kinds and layouts of buffers that you can wrap
to look like an IB.
* Can easily make an IB that wraps an interior rectangular subset of
another buffer (including another ImageBuf).
* To a limited extent, makes it possible to have an IB that is a "view"
of another IB (if the second IB is in memory), including strided pixels
or scanlines -- and even if the strides are negative (this lets you make
an IB view of a buffer where copying the buffer can reverse left/right
or top/bottom, however you can't use it to change the channel order or
layout within each pixel).
* I think this also lets you make a 2D IB that wraps a 2D slice along any
axis of a 3D volumetric IB.
Some other changes that came along for the ride:
* Added `ImageBuf::contiguous()`, which tells you if the memory layout is
contiguous (will return false if it's not an in-memory buffer of some
kind, so ImageCache-backed IBs are NOT considered contiguous).
* Renamed some internal types and fields, in particular, changes like
`size_t ImageBufImpl::m_pixel_bytes` to `stride_t m_xstride` to make it
clear that the values are strides and can be negative.
There are several alpha choices. We were assuming that color values
were unassociated (need to be premultiplied) unless it was marked as
premultiplied alpha. But it seems that, actually, we should only
premultiply if it's "useful alpha". [Indeed, their terminology is
the worst.]
Rename the m_alpha member to m_alpha_type to clarifywhat it holds.
Also add the alpha type to the metadata as "tga:alpha_type" to make it
easier to inspect the value in tga files.
I don't think issues have been reported, so maybe this wasn't a
problem in practice for reasons I've long forgotten about, but doing
some other work in ImageBuf, I noticed what I think was a thread
safety problem.
ImageBuf reads are done lazily, a lot of operations call
validate_spec() or validate_pixels() which can trigger a call to
init_spec() or read() if they have not yet been done, and those had a
spin mutex protecting them from being called at the same time. But the
underlying read() and init_spec() themselves did not have any lock,
which made me worried about what would happen if one user thread was
explicitly calling read() while another was calling it implicitly
through validate.
Also, spin_lock is probably inappropriate for something that could
trigger an entire image read from disk -- it's meant specifically for
situtions where the lock only will be held for a few instructions.
So, this patch:
* Renames ImageBufImpl::m_valid_mutex to m_mutex, reflecting that it's
not merely guarding access to `m_spec_valid` and `m_pixels_valid`,
but to potentially anything internal to the ImageBufImpl that needs
thread safety.
* Changes that mutex from a spin_mutex to a std::recursive_mutex, to
reflect that it *may* block longer than a few cycles, so blocked
threads should yield rather than spin. And recursive to simplify the
hand-off logic between validate_spec, validate_pixels, init_spec,
and read, the former two may call the latter two.
* Add locks to read() and init_spec(), which previously lacked them (I
think in error), in addition to the existing ones in the validate
methods.
* For some locking ImageBufImpl methods, add a param that lets the
caller (which will always be internal code) specify that the lock is
already held.
* Add a timing test of recursive_mutex to spinlock_test, just to verify
that I'm not doing anything to totally botch performance. On my Intel
Mac laptop, an uncontested lock acquisition/release is 6.4ns for a
spin_mutex, 16ns for a std::mutex, and 27ns for std::recursive_mutex.
So this is definitely a more expensive lock, but 27ns is still very
reasonable, and *most* operations don't need to acquire the lock at
all (first it checks the m_spec_valid or m_pixels_valid, once they are
set there is no more locking).
* Improve ImageBuf docs, adding more explanation including some guideines
about which methods are considered thread-safe.
* strutil.h already had a symbol called OIIO_FORMAT_IS_FMT defined to
0 or 1 that reveals (to clients of OIIO) whether Strutil::format()
behaves like printf (the old and current way) or like std::format
(the future way). Now we adjust so that the caller may pre-set the
value to force one implementation or the other, at least for the
"inline" functions of the headers.
* As an aid to downstream packages trying to ensure that they aren't
using the function slated for eventual deprecation or behavior
change, you can now define OIIO_HIDE_FORMAT to cause a deprecation
warning now on those functions that will eventually change or be
deprecated. This can help to identify any places where you call the
future-deprecated functions and change them to a "safe" variety.
ImageInput and ImageOutput have ways for the caller to say that there
should be no "fan-out" in the internals of what the reader or writer
needs to do. But TIFF wasn't honoring this.
This patch provides an additional but optional parameter for --bumpslopes option to mormalize the derivatives in UV space and scale them by a user defined constant. The scale factor is used to prevent half floating point overflow at time of the UV normalization. A value of 256 is recommended to prevent floating point overflow with texture resolutions up to 32k. The constant value is stored as an extra attribute in the output file for reverting the scaling when using the texture. Note that this options is only meaningful with height maps.
Co-authored-by: Pascal Lecocq <plecocq@imageworks.com>
* Add TypeDesc support for metadata that is a 2D or 3D bounding box,
represented as an array of 2 VEC2 aggregates (for 2d) or VEC3
aggregates (for 3d), having the BOX semantic. The basetype
determines whether it's an integer-based (generally for pixel
ranges) or float-based (generally for 3D spatial) bounds. The new
text representations are "box2" and "box3" (for float-based), and
"box2i" and "box3i" (for int-based).
* Somehow we never had a C++ test thoroughly exercising all the corners
of TypeDesc. Now we do: typedesc_test.cpp
* The test revealed a number of edge cases we don't handle well, now
shored up.
* New fmt 8.0.0 is out -- make sure we test against both that and their
current master.
* Starting with fmt 8.0, and in accordance with C++20 std::format's
final incarnation, the format string needs to be constant, for the
sake of compile time checks. To use a nonconstant format string, as
we do for these wrapper functions, we need to lean on vformat.
* For Strutil::sprintf, just `using ::fmt::sprintf` and eliminate the
wrapper entirely. (Though we express the wrapper inside
`#ifdef OIIO_OXYGEN_ONLY` so that it fully appears in any docs we
generate.)
Add custom UDIM support and Houdini UDIM convention support
For our (Animal Logic) in-house renderer, we need support for an additional UDIM format of `/path/to/file/textureName_u##v##.ext`
Added second line to ImageCacheImpl::resolve_udim to make this work.
Additionally add support for Houdini UDIM specification
* New Ptex (since 2.3.1) has exported cmake configs. Use those if
available and seem to work.
* Rename FindPTex.cmake -> FindPtex.cmake to match the capitalization
in the Ptex project itself, thus avoiding warnings from CMake.
* Fix our FindPtex.cmake to set up Ptex::Ptex_dynamic target. Use
that instead of the old style variables. That makes a seamless future
transition between the old Find module and future exported configs.
* Our FindPtex.cmake got the version number all wrong. Fix.
* gh-installdeps.bash tried to install ptex-base. That's not Ptex! So
stop doing that.
* New build_Ptex.bash does a simple clone and build of Ptex. Do so for
a few of the CI test cases so that we are more thoroughly testing Ptex
in CI.
LibRaw 0.19 and earlier use auto_ptr in public headers, which was
deprecated in C++11 and removed in C++17.
When this combo is detected, issue a warning message from CMake.
When using filesystem::directory_iterator, I was using operator++,
which can throw exceptions for OS errors. I should have used
increment(ec) instead. Throw in a surrounding try/catch just in case,
since even increment is not marked as noexcept.
* Use g++11 on the bleeding edge CI test
* Remove OIIO_CONSTEXPR20 decls from string_view -- gcc 11, even in
C++20 mode, doesn't have std::string fully constexpr plumbed, it's
not ready for this.
* Suppress gcc11 warnings about arithmetic with enums that occur in the
OpenCV headers.
* Suppress gcc11 warning (correct) in the spot where we were trying to
force a crash for testing purposes.
We were double-printing error messages. If maketx/oiiotool is itself
printing errors returned by IBA::make_texture, it is not necessary to
pass std::cout to make_texture so that it can echo the message to the
console itself.
It's meaningless to try to use make_texture on input images that are
"deep". We never even thought to check for this, which led to a
cascade of problems resulting in a segfault.
There's something wrong with the behind the scenes tricks we're using
to doctor the internal string representation. Is it because of big
endian? The "alternate" string rep? Something else different on
Apple's libc++ on M1?
Fix for now by just disabling that clause. The only harm is that
ustring internals will take more memory on that platform. That's
better than crashing!
When I have more hands-on access to an M1 (or when GitHub Actions
supports running CI tests on M1), I'll return to this for a proper
fix.
A few varieties of encoded videos with bit depth 12 and having alpha
were not marked as such, and falling to the default case that reported
them as 8 bit RGB.
The docs were ambiguous about what would happen for many-channel
image, and the behavor in fact was to copy and color transform only up
to the first 4 channels, leaving additional channels in the
destination undefined (though in practice they turned out black).
Clarify now that for colorconvert, colormatrixtransform, ociolook,
ociodisplay, and ociofiletransform, the first 4 channels are assumed
to be RGBA to undergo the transformation, and any additional channels
will be copied unaltered from source to destination (not set to black).
In ImageInput::read_scanlines and read_image, if somehow the image has
a spec with 0 size, it will lead to a divide by zero error. Check this
and fail safely.
For BMP in particular, do some sanity checks to catch corrupted headers.
(This is in response to Issue 2982)
For TBB 2021, headers moved around, and our FindTBB.cmake was failing.
But actually, starting with TBB 2021, TBB exports a proper cmake config!
So take advantage of that by preferring to load TBB from a config file.
Fixes 2963
Both global and per-package controls have been added to our
checked_find_package cmake function, to cause it to first check for
an exported config for the dependency, and only if not found, revert
to potentially using a FindPackage.cmake module.
This reflects our general leaning toward assuming that a package that
exports a cmake config probably does a better job of it than we would
do writing a Find module for it.
Rename FindOpenJpeg -> FindOpenJPEG to match the conventions used by
that project, in particular for its exported CMake config files.
(We don't yet rely on their exported configs, I need to do some
homework to see if they exist and are reliable as far back as the
oldest OpenJPEG release that we support.)
Add helper script to build OpenJPEG
We still had a few stray localtime() calls (which is not reentrant,
because it depends on a global buffer), even though all along we had a
"safe" wrapper in Sysutil::get_local_time, which internally calls safe
versions localtime_r or localtime_s (depending on the OS).
* Writing: Write single channel BMP as 8 bit palette images (not 24/32
bit images). This fixes a problem I discovered related to the recent
change that reads 8 bit palette images as if they were single channel.
The lack of symmetry there made problems for round-tripping an image
and having it end with the same apparent number of channels as it
started with.
* Reading: respect a configuration hint "bmp:monochrome_detect" that if
set to 0, will disable the detection of monochrome palettes and spoofing
to look like a single-channel image. The default is 1.
* Read RLE-compressed images correcty now! Both 4 and 8 bpp varieties.
* Beef up the testsuite/bmp tests, now test against a wider set of sample
BMP images from bmpsuite (including rle compressed images).
OpenImageIO exported config has a dependency on Imath/OpenEXR. For
Imath 3, I accidentally used OpenEXR_VERSION when I should have said
Imath_VERSION. Makes no differece if they're in sync. But should
somebody downstream have a separately compiled Imath and OpenEXR that
just might be different patch versions, it could break builds by
rejecting if the Imath version appears to be behind the OpenEXR
version. There's no guarantee that Imath and OpenEXR will always be
released exactly in sync.
Found some files that encountered the problem described here:
https://prepression.blogspot.com/2017/06/metadata-bloat-photoshopdocumentancestors.htmlhttps://feedback.photoshop.com/conversations/photoshop/photoshop-corrupt-ancestors-tag-in-xmp-causing-giant-file-sizes/5f5f45f74b561a3d426ba97f
The pathologically long lists of "photoshop:DocumentAncestors" tags (a
4MB PNG of which 3MB consisting of XMP) was tripping us up with an
O(n^2) handling of appending to lists for those attributes that could take
minutes per file.
It seems like this was a widely acknowledged problem that has been fixed
in Photoshop 20.0.2 (January 2019) and later.
Solution:
* Just skip the problem tag; I can discern no real use for preserving
it in any OIIO context.
* More generally, be on the lookout for pathological or corrupt XMP by
keeping track of the length of lists and just cutting them off from
further appending when they seem senselessly long. I can imagine XMP
with lots of attributes, but it's hard to imagine that any single
attribute having thousands of list entries is intentional and useful.
* Fix some places where we were needlessly generating and then tearing
down std::string's.
This makes several minor formatting changes, mostly related to how
clang-format 11 has improved ability to track the alignment of = in
successive statements, even when one of those statements is split
across two lines.
This started out as just trying to fix our CI on the "bleeding edge"
building against OCIO master and complaining about a recently committed
OCIO deprecation of Config::parseColorSpaceFromString.
But it turned out that the new OCIO v2 method, getColorSpaceFromFilepath,
didn't work quite the same way and broke our tests. So I ended up having
to implement a bit more of a complicated set of functionality:
* Config::parseColorSpaceFromString now wraps the OCIO version if it's
available, but implements the equivalent logic if OCIO is new enough
that the function is deprecated.
* As a bonus, our pCSFS now works (didn't before) in cases where lack of
OCIO support or missing config meant the OCIO functionality wasn't
available. That is, it works on our meager set of default color spaces
that are present even without OCIO.
* New Config::getColorSpaceFromFilepath wraps the OCIO version if OCIO
is new enough, but if it isn't, or if there's no OCIO, or if OCIO ends
up not figuring it out and returning the default rule, we fall back
to Config::parseColorSpaceFromString.
* Add OCIO to VFX Platform CI builds -- because it's not in the aswf-osl
docker images, we weren't testing OCIO in as many versions as we had
previously thought.
* Bump default OCIO version built by build-scripts/build_opencolorio.bash
to 2.0 (from 1.1.1).
the temporary image buffer is created with only 3 channels, so the get_pixels call at the end of the
read_unmipped method must account for that when creating the ROI passed to it.
This also bumps compiler minimums to gcc 6.1, clang 3.4, MSVS 2017,
and icc 17 (all because those are the first versions to fully support
C++14).
This has been coming and has been socialized for a long time, with
nobody voicing serious objection. So here it is.
This WILL NOT be backported to release branches. The new minimums only
affect OIIO 2.3 and later.
When last discussed on the oiio-dev mail list, I said I would aim to
introduce this change on May 1. I will not merge this PR earlier than
that, and I will tag a developer preview release before that happens.
The new `--missingfile` argument (taking a value of either `error`,
`black`, or `checker`) specifies the policy for what oiiotool does
for input files that are entirely missing (not present but broken).
In all cases, an error messages is issued, but for the cases of
black and checker, an image is substituted and computation continues,
rather than immediately ending the attempt to execute the command
sequence.
On a personal project a while back, I wasted a full day chasing down
inexplicable behavior that turned out to stem from TWO implementations
of ustring being in the same application, a result of inadvertently
linking against both libOpenImageIO and libOpenImageIO_Util. This led
to two ustrings that contain the same character sequence, but with
different pointers because they resided in different ustring tables.
There is no way that anybody without very deep understanding of ustring
and a lot of patience could have tracked down the underlying issue.
This patch eliminates the possibility of duplicated modules in
libOpenImageIO and libOpenImageIO_Util. Now libOpenImageIO depends on
and links to the latter. This should all be automatic for users -- as
before, link against libOpenImageIO if you want everything, link
against libOpenImageIO_Util if you just want the util helper classes
but don't need any of the image I/O functionality. But now, if you
accidentally specify both... well, that's just the same thing as
asking for libOpenImageIO.
Fully separating libOpenImageIO from libOpenImageIO_Util requires new
OIIO_UTIL_API
Make whether namespace includes the patch number be dependent on
whether it's a dev or release branch. We generally want the patch
in the namespace for master, but not release.
This teaches oiiotool's command line expression substitution (`{...}`)
syntax to understand that
getattribute(name)
should be replaced by the value that OIIO::getattribute(name,...) would
retrieve. The `name` may optionally be single or double quoted, but doesn't
have to be if it's a simple identifier.
Some examples:
# Which SIMD features does this OIIO use?
oiiotool --echo "{getattribute('oiio:simd')}"
# Which dependencies are used for each format?
oiiotool --echo "{getattribute(library_list)}"
# Which file extensions are associated with each format?
oiiotool --echo "{getattribute(extension_list)}"
The cv::Mat::create method does the same thing as the constructor but "Allocates new array data if needed." And when not needed, which is my use case when all the images I process are the same size, create does nothing. This is more efficient than the implementation where a new mat is constructed and then moved to the destination. This is a minor optimization, but it does avoid allocating an image size block of pixels when it is not necessary.
Sorry this has taken so much tinkering to get right for downstream
projects. It's tricky business.
First, we start by adding testsuite/cmake-consumer -- a test that
consumes OIIO and its exported cmake modules, and is independent of
the rest of OIIO's build system, more accurately simulating whether
things will work for a downstream client.
Adjust our exported module, it needs to break out the OpenEXR 3 vs 2
cases separately, as well as whether it found an OpenEXR/Imath config
file or not. I think I have all the cases covered now. I also needed
to specify find_dependency of IlmBase and Threads, for the sake of
openexr 2.4 with config, and add the INCLUDES directory lists in the
case of no config files found.
Some adjustment of our library target_link_libraries -- better PRIVATE
hiding of things not needed in the interface by downstream builds.
In particular, the OpenEXR components and OpenCV should be private, not
public.
Add m_allgray field and color_table_is_all_gray method to detect
that the color table entries have the same r, g, b values and indicate
a single channel gray scale image rather than a 3 channel rgb image.
Temporary allocations were made and freed for each scanline read or
written. Do it once per image and reuse. When reading, eliminate one
temp buffer by writing data into its final location, also eliminating
a needless extra copy of each scanline.
We broke things subtly for older OpenEXR when we added the 3.x
support. Without this fix, our export CMake config files weren't
properly exporting the OpenEXR include path as the interface includes
for OIIO libraries. This doesn't affect the OIIO build itself, but it
can affect downstream libraries that use OIIO as a dependency and
rely on our exported cmake config files.
We also remove the need for OpenEXR/OpenEXRConfig.h entirely through
header configuration.
Ever since adding OpenEXR 3.0 support, we had some cases where
installations that built the 2.x IlmBase and OpenEXR portions
separately and installing to separate areas.
Also make some minor changes to ci.yml to make sure we are documenting
the right openexr versions used for each test.
It seems that for CMake, --config should only be used in conjunction
with --build. Who knew? Things are breaking right and left now, especially
on GitHub Actions CI. But also on my laptop.
Hypothesis: maybe this was silently ignored in older cmake versions, but
now with CMake 3.20 is a hard error?
This does not (YET) change the minimum to C++14. For now, you can still
build for C++11 -- and with compilers that only support C++11 -- by using
CMAKE_CXX_STANDARD=11.
By changing the default, we hope to more thoroughly test everybody's
ability to compile C++14, ahead of the actual cut-off when we make that
the true miimum.
Also, the latest OpenVDB 8.0 does has C++14 as a minimum, so compiling
in C++11 mode will automatically disable OpenVDB support if OpenVDB >=
8 is what is found. We expect an increasing number of our dependencies
to drop support for C++11, so we want our default compilation mode to
be one that doesn't silently drop format support without
users/builders being aware of it.
In many cases, this informative debugging code was left over from when
parts of this code was strictly a command line utility. It's not
appropriate for a library call, especially on Windows.
Make sure every last output to std::cerr is guarded by some
preprocessor symbol that would only be enabled when one of the
developers is trying to do some "printf debugging". And similarly,
std::cout should only be used for standalone command line
utilities that are by design supposed to output to the console.
We were treating some recoverable warnings from jpeglib as fatal errors.
Removed testsuite/jpeg-corrupt-header since now we can read it just
fine despite the very minor corruption! Also, turns out the image was
inappropriate, sorry, I had never actually seen it before because
OIIO wouldn't read and display it and I assumed it was just garbage.
Apparently all the recent farmhash changes are fine with C++14 mode,
and fine with C++11 mode on gcc 4.8, but gets some errors when
usig in newer compilers (gcc6+) using C++11 mode.
Add a CI test with gcc10 but with C++11 compatibility to verify this fix.
We will remove this test once we retire C++11 compatibility.
For the case of applying a color transformation to just one color, we
were sloppy and set the strides to 0 (just one color means we are not
iterating from point to point using the strides).
OpenColorIO 1.x didn't care, but OpenColorIO 2.0 throws an exception if
the strides don't look valid.
Thanks to Thomas Metais for the fix suggestion.
Change imageio.cpp convert_from_float to use the convert_type function
from fmath.h, rather than defining redundant functionality in that
module.
This has the added benefit that the ones in fmath.h have some interesting
specializations to use SIMD ops and other optimizations that the one-offs
in imageio.cpp lacked.
Ordinarily, oiiotool will exit upon encountering an error (such as an
input file not being found). When operating on a frame range or with
frame number wildcards, that means that it will exit without attempting
the rest of the frame range.
The new `--skip-bad-frames` command line argument means that an error
will skip the rest of that iteration, but not exit, instead starting
fresh on the next frame in the sequence. (But if any frame in the
sequence encounters an error, the exit code of the oiiotool command
will still be EXIT_FAILURE.)
GitHub Actions in its wisdom has removed pre-installed Boost from the Windows
images, causing our CI to fail.
Instead, get boost from vcpkg, just the pieces we need.
Also needed some JPEG adjustments for Windows CI build.
Added utility function OIIO::get_extension_map() which returns a
`std::map<std::string, std::vector<std::string>>` listing all
image file formats and their presumed file extensions.
`oiiotool --list-formats` helps to test this and is useful in its own
right.
Fix reading tiled tiff files when vertical read size is less than tile size, just for the code path where we use TIFFReadRGBATile (used for certain rarely encountered types of TIFF files). TIFFReadRGBATile returns the image data in bottom-to-top order, and we had been correctly reversing it all along, but neglected to account for the issue of a partially-filled tile ending up with the padding at the beginning of the buffer rather than the end.
All of our prior tga tests had valid thumbnails. The code skipped the
thumbnail if the "offset to the thumbnail" was 0. But if the offset was
nonzero but the thumbnail record at that spot had a 0x0 size, we just
ran merrily along, and on Windows in debug mode even it an assert.
This patch makes it understand that a 0x0 thumbnail size means no
thumbnail.
I added an image from Doug Rogers, the reporter of the problem, to the
oiio-images repo in the targa directory, and updated the OIIO testsuite
to also check that image. (So testsuite will fail for people who don't
update their oiio-images repo.)
Also changed some stray refs to &vector[0] to vector.data().
Fixes#2900
The very old OS2_V1 variety of BMP files -- which dates from the OS/2
and Windows 2.0 circa 1987 -- could have undefined results by relying
on the cpalete header field that didn't exist in that early
version. Our failure to initialize it led to expecting color palettes
of undefined size (which could either cause a read failure or even a
crash if it vastly over-allocated).
Some other minor touch-ups to the bmp code while we're at it:
* Zero-initialize all the other fields that don't exist in the OS2_V1
version of the header.
* Don't set the Resolution metadata if the values in the file aren't
positive (including for the V1 header that lacks it entirely).
* Fix reading V1 header resolution on big endian architectures (I just
happened to spot that potential bug).
* Fix longstanding error (that I only just noticed) in reading 4 bpp
images with odd horizontal resolutions, wherein the very last column
was not filled in correctly (that's the change at lines 188 and
193).
* Beef up one of the error messages invoked for failures while reading
the color palette.
* Rig the BMP reader to know the "dib" is another extension often
associated with BMP files.
* Add a regression test case for this format variant.
* Make sure CI tests bmp -- the GHA CI did not download the bmpsuite and
so was not actually running the testsuite/bmp tests.
* Add "bmp:bitsperpixel" (when not 24 or 32) and "bmp:version" metadata.
This is primarily for debugging and understanding which encodings the
BMP file is using.
* Recognize nonsensical values for the header's field for number of entries
in the palette. Add a test for that.
This patch ensures that two ustrings will never have the same hash value.
The basic strategy is that the usual 64 bit hash of the characters provides
an initial guess. If no other string has already used that hash (as is
almost always the case), we use it. However, if a different string already
took that hash value, we rehash, and repeat until we find an unused hash.
We maintain a "reverse hash map" that relates hashes to ustrings to
help figure this out efficiently. (Note: This adds at least another 16
bytes per unique ustring to maintain the reverse map. Do we care?
Probably few applications are in the business of creating more than a
few million distinct strings.)
Because we are recording the collisions as they happen, this also
greatly simplifies (and reduces to trivially low cost) the
recently-added ustring::hash_collisions() function, because there is
no complicated search when it is called.
When making a texture out of float values, but writing ultimately to a
`half` file, it's not too hard to end up with valid but large float
values that are outside the range representable by a `half` (fp16)
value and therefore will get converted to an `Inf`, which nobody wants
in their texture.
This patch prevents Inf from showing up in the output of make_texture
by clamping output buffers to the range of `half` when the source data
are float and the destination file is half.
Fix minor warnings and deprecations of C++20.
Bump to more modern ubuntu images for recent-year tests and make some
related adjustments. Also make sure that the "latest" and "bleeding edge"
tests are using python3, they were inadvertently still using python2.
A recent commit in the OpenEXR master changed which headers are
transitively included by other OpenEXR headers, and that broke our
build until this patch adds the missing headers explicitly.
The change may or may not be reverted on the OpenEXR side, but this
patch to OIIO at least unbreaks our CI in the mean time, and has
no down-side.
Update old calls to Strutil::format to new fmt conventions, convert
some sprintf-like calls to std::format like calls.
This is not meant to be comprehensive; there is more work to do.
It's just an opportunistic bite at getting some of it done.
The oiiotool --colorconvert operator (and the other color conversion ops)
have an optional modifier ":unpremult=1" that will bracket the color
transformation with an unpremultiply/premltiply. But the implied
colorconvert that happens upon input and output with --autocc has no such
option, which is an oversight. This patch adds the ability to control
with autocc transformations happen in an unpremultiplied state (which I
believe they SHOULD, generally speaking).
Also beef up the docs and help messages in some areas where I
coincidentally realized that -i and -o explanations omitted some of
the possible options.
There were some flaws in the logic of IB::set_write_format that could
cause a crash when used in conjunction with a file format that doesn't
support per-channel data types (even if the set_write_format call
specified a single type rather than per-channel types!).
There are basically two problems I fixed here.
First, in ImageBuf::write, where we are dealing with the fact that
set_write_format previously registered a request, be extra careful
that the "default" data format is the "biggest" type out of the
per-channel choices, and also make sure that if all the channels are
identical, we just use the default type and clear the channelformats
vector.
Secondly -- and more importantly to the crash itself -- in
ImageOutput::to_native_rectangle, if the output file format does not
suport per-channel data types, just use the main format. Do NOT
consider this case a hard error, which previously took an early out by
returning NULL for the location where the native data was copied. It
was that NULL that, when subsequently copied from (not noticing that
it was NULL indicating an error), actually produced the segfault that
caused the crash. In other words, don't worry, be happy -- just do
your best by treating the unsatisfiable request as the next best
thing, as if it were the same type for all channels.
Homebrew just upgraded to Qt6, and that breaks us. Peg to Qt5 as a
temporary measure to let our CI pass, until somebody figures out what
needs to be fixed for full Qt6 compatibility.
I have submitted a PR to Imath that adds "interopability" constructors
and assignment, that use some templates to allow seamless
construction, assignment, and passing of "foreign" vector types to
Imath vectors. This makes it easier library A that uses Imath vector
types in its API to interact with app or library B that has a custom
internal vector type, without a ton of ugly casts and copies.
(See https://github.com/AcademySoftwareFoundation/Imath/pull/91)
These created a few minor conflicts with OIIO's internal SIMD classes
and their construction and casting from Imath types. This patch cleans
up the ambiguities:
* Make Imath::V4f -> simd::vfloat4 and Imath::M44f -> simd::matrix44
constructor `explicit`.
* Make the simd types `simd()` method (returning the underlying raw SIMD
type) also have a variety that returns a ref to a non-const vector,
to ensure proper working of the _MM_TRANSPOSE4_PS macro.
Now I am going to commit a sin. This MUST get backported to OIIO 2.2
in order for it to not break against the upcoming Imath 3.0. This
doesn't break ABI compatibility, nor change OIIO's primary advertised
public APIs, but it does change the technically/informally public APIs
of simd.h in ways that introduce the possibility that an app using
those simd vector classes *might* need minor source code editing if
they relied on those implicit constructors. Such software might need
certain `simd::vector4 foo = imath_bar` to become `foo = vector4(bar)`
and the same for certain matrix44 assignments. It is usually a big
no-no to backport any change that could require editing of source
code. But since the alternative is not being able to use the new
Imath, I think this is the lesser evil, and maybe nobody out there is
doing the thing that would need to be changed?
Need some include fixes to more gracefully handle the case where
Imath_ROOT points to a custom build (say, of Imath 3.x), but there is
a system installation of a different version (say, OpenEXR 2.x in
/usr/include, which could easily end up earlier in the path list).
Also update some reference output due to recent bug fixes in OpenEXR,
where error messages changed.
* Better handling of lists/sequences -- append into attribute arrays
instead of allowing each value to overwrite the previous one.
* Better inference of unknown attributes that look like int, float, or
rational, rather than cramming into strings.
* Fixed what I now think were bugs in handling rational values.
Turns out it doesn't -- we're still seeing timeouts on the Windows CI
tests, even with the OpenEXR that we thought fixed the threadpool
problem.
But let's keep this change anyway.
Background: In our GitHub Actions CI, we were doing environment setup,
dependency setup, OIIO build, and testsuite -- all in one action
"step" because each stage is its own shell process, so I couldn't
figure out how to communicate between them (for example, a dependency
setup might build and install a dep package, but then the later build
stage needs the CMake Pkg_ROOT variable set to that location).
Doing it all as one "step" was clumsy, not best GHA practice, and it
made it hard to understand which stages were taking different amounts
of time and how to optimize them.
The trick, it turns out, is that $GITHUB_ENV is the name of a file
that will be executed at the start of each step. So you put commands
into that location that restore whatever env variable you need to
restore between steps, or any other setup you need.
So in this patch, we:
* Elevate env setting to be for all steps.
* Split build-and-test.bash into separate build and test scripts.
* Separate setup, dependency installs, build, and test into separate
GHA steps.
* Each step's script, at its very end, calls new script save-env.bash,
which puts commands to restore the environment into the file named
by $GITHUB_ENV, which will be read at the beginning of the subsequent
step.
* Merge gh-installdeps and gh-centos-installdeps scripts -- there was a
lot of redundancy between them anyway.
The net result of this is that when we examine the test results, it's
much easier to see which stage a failed test had trouble on, and
especially how much time each stage took. That's really handy with the
recent changes to add caching, and helps us verify that the caching is
working as intended, since it is expected to take a big bite out of
the dependency and build stages, but not the test stage.
This speeds up CI builds by 1/2 - 2/3 of total time, for PRs that are
minor changes by reusing the compiled results from previous runs.
Most of the remaining time cannot be squeezed down -- it's the
instantiation of docker containers and running the testsuite
itself. But the build (including of dependencies we build from
scratch) is dramatically faster.
Currently, this depends on ccache, so is for Linux/Mac only. I'm also
working on doing something similar for Windows, but it's not ready for
review yet.
Do some adjustment/simplification of the macros that set up the right
decorators on the new inline farmhash.
Some additional constexpr's, including Strutil::strhash().
Add to hash_test verification of the actual results of each of the
hashes, so we know that we don't break them over time, that they give
the same result on all platforms, and most immediately, to 100% verify
that the new constexpr inline version of the hash precisely matches
the old procedure call one.
Augment the existing benchmarking of the hashes to include
farmhash::inline. It tries a variety of string sizes. I was worried
that the constexpr/inline version would be a performance hit because
it can't use any of the SSE intrinsics (which, ugh, aren't declared
constexpr in the headers). But it turns out that the inline (but
SSE-free0 version is a little faster on some tests, a little slower on
other tests, overall a wash and never differing by more than
10%. Maybe that means that the inlining/constexpr makes up for the
lack of SSE? Maybe it means that the compiler optimization is adding
SSE back in well enough that losing the intrinsics isn't as bad as we
would have thought? In any case, the upshot is that this doesn't
represent a performance it, so I may consider a future change to just
go strictly with the inline constexpr versions and eliminating the
separately compiled version altogether. Will mull that over.
* cmake: Make LibRaw Jasper and LCMS2 detection optional, add LIBRAW_NODLL
* cmake: Fix OpenColorIO defines on Windows to use correct cmake mechanism
* cmake: Add missing LibRaw defines (needed when linking statically)
Some fields have disappeared from certain structures, so we need to make
adjustments to the various conditional compilation guards.
This patch just makes OIIO build against the new libraw, and is intended
to be backported to the release branch.
This patch does not incorporate any new fields added to libraw
structs, representing newly retrieved metadata. A separate PR patch
will address the newly added capabilities (TBD whether or not that
ends up safe to backport and when).
Bump CI "current releases" test to OCIO v2, it's released!
Oops, we used the wrong name for the env variable, and that CI test
wasn't getting the OCIOv2 RC all along!
Add OIIO::ColorConfig::OpenColorIO_version_hex().
Have oiiotool --help print the OCIO version.
Also Bump the pybind11 version for the "latest" test, and update
INSTALL.md a bit to document the latest versions known to work of
several dependencies.
When we wrote our `span<>`, we based it off the draft for std::span,
which at that time specified span::size() as ptrdiff_t, which is
signed. By the time std::span got finalized in C++20, it was changed
to size_t, which is unsigned. Ick. Switching to size_t will be an API
break, so we can't do that until we're at a version where it's ok to
make breaking changes.
But we can stage a few changes that will make that future transition
easier.
* Add std::size() and std::ssize() wrappers for both span and
string_view. This gives downstream apps the opportunity to avoid
span::size(), instead using std::size(the_span), which is guaranteed
to be unsigned, and std::ssize(the_span), which is guaranteed to be
signed, and both of these will work now as well as after we make the
other breaking changes.
* Doing this properly required adding a couple 'constexpr' in places
where they should have been all along.
* Define macros OIIO_SPAN_HAS_STD_SIZE and OIIO_STRING_VIEW_HAS_STD_SIZE
making it easy to know if std::size and std::ssize are defined for
span and string_view
* OIIO_SPAN_SIZE_IS_UNSIGNED will be defined when OIIO::span switches
to having an unsigned size like std::span (making it easy for client
software to know if span::size is signed or unsigned).
Turn this on automatically for OIIO >= 3.0 so we don't forget.
If client code predefines OIIO_SPAN_SIZE_IS_UNSIGNED before
including span.h, that forces span::size to unsigned now (good for
experimentation and compatibility testing, but I don't recommend
anybody ship software in that state unless they really know what
they are doing and don't mind potential incompatibility with the
out-of-the-box OIIO).
These SHOULD NOT break any existing code or ABIs that uses our
existing span and string_view classes or their size() methods. It only
adds options that will help bridge compatibility.
A variety of small problems were fixed:
* Symbols in libOpenImageIO-C that were not marked as OIIOC_API.
* MSVS not being happy with array length determined by a variable
* Issues with Windows paths
This is a weird thing we're doing in this very artificial test,
removing a file when the IB reading it is still alive. That's safe on
Unix, where a file with an open file handle still be safe to read even
if it's removed. But it's not kosher on Windows. Fix by simply making
sure that in this test, we aren't doing the unadvised thing of having
the IB still active on the file we're deleting. This should fix the
unit_imagebuf test that is failing our Windows CI.
* Try to address the OpenEXR thread pool hanging when we call exit().
* Strive to remove all exit() calls after any file reads happen,
in oiiotool, iconvert, and idiff.
* This DOES NOT stop all hanging. It's still a bit of a mystery whether
there is anything we can do to fully prevent it without a fix on the
OpenEXR side.
* Temp workaround: for the Windows CI tests, use ctest commands to
allow each test to timeout 3 times before giving up. To be clear,
this does not solve the problem, it only masks it. But it helps
us "pass" those tests, exposing any other problems occurring with
them, and we will return later to find the source of any residual
timeouts.
* Update ref output for several tests where the differ slightly under Windows.
* Make runtest.py convert text output line endings cr-lf -> lf before
comparing to reference output. (This eliminates unnecessary diffs.)
* Update many tests' run.py file to use double quotes instead of single
quotes for arguments containing some characters -- looks like the Windows
shell invoked by Python system() does not handle the single quotes quite
the same way as Unix shells.
* oiiotool: when printing the command line after errors, genericize
identifiable filenames so they render the same on Windows and POSIX,
and strip directory name from the oiiotool filepath. This makes
error output comparison more equal between Windows and POSIX, making
several more tests pass.
In total, this seems to get us down to failing only 7 tests (not counting
spurious timeouts that cause failures).
Similar to C++17 std::filesystem::path::generic_string(), but for a
filepath sent as string_view.
While I was there, I noticed that a few Filesystem functions that I
declared as `noexcept` actually call things that can throw, so I put a
few try/catch in stratgic spots. Nobody had reported a problem with
this, and maybe it would be fine, but it's the right thing to do.
* Change to finding a consistent Python version (3.7)
* Install numpy
* On Windows, convert crlf to lf before diffing reference output, so we
don't get mismatches just because of LF vs CRLF.
* Update ref output on testsuite/python-imageinput
* Fix windows detection in runtest.py oiio_relpath
* Fix some semicolon problems that were making bash command lines conused
Along for the ride:
* Have ci-build-and-test.bash send OIIO_CTEST_FLAGS as extra args to ctest
* Remove some needless verbosity from the dependency building scripts.
With some recent heic related packages, we lost the ability to build
against some older libheif versions that we purport to support. We may
eventually want to raise the minimum supported libheif, but in the
mean time, this restores the prior build abilities.
Use pybind11 functions to set up python module, it's got superior logic
to what we were doing.
Bump pybind11 version default in build_pybind11.bash
Suppress clang warnings about python register use
* Re-enable vcpkg installation of freetype on windows
* Add TESTSUITE-README.md file
We'll use this file to document how the testsuite works and how users
should update it when necessary.
* Re-enable windows tests
Fix a few issues parsing rational values with large denominator in OpenEXR header.
Incorrect array index can cause divide by zero crash (observed consistently on Windows with repro image).
Conditional comparison was assigning 1 or 0 to int f declaration instead of gcd value.
Changed gcd type parameter to long long to avoid unsigned int->long int overflow on windows (long int is 4-bytes on windows).
Co-authored-by: Chris Crosetto <ccrosetto@gmail.com>
* fixes to make vs2019 CI run. Not passing yet as some tests fail
* added explicit ctest timeout (180sec) + misc fixes to run CI local
* disable Windows VS2019 tests again as they are not passing yet
Co-authored-by: Dieter De Baets <dieter.baets@unity3d.com>
OSL has some concerns about how ustring hash collisions could impact
its strategy for strings on GPU.
This patch adds some instrumentation to ustring to detect hash
collisions. Added static ustring methods total_ustrings() and
hash_collisions(). The latter tells you the number of collision pairs,
including optionally enumerating them into a vector of ustrings. Also,
`getstats(verbose=true)`, in Debug builds only, will print all of the
colliding pairs. These are both somewhat expensive if there are lots
of strings (my laptop takes about 4 seconds to find and print all
collisions for 10M ustrings). So if you are concerned about
performance in debug builds, call getstats(false) instead if you have
a truly enormous ustring table.
This addresses slight ringing at image edges when using a combination of
black wrap mode and a filter with negative lobes. We need to be sure that
we really clamp the filter at the source image edges.
But we only want to do the clamping selectively. Works great for --fit,
not so much for more general warp and rotate. So add it as an internal
parameter to the underlying warp we're doing.
Needed to update a bunch of the fit-related output reference.
There are several ways to make an image fit a new size while preserving
aspect ratio. Originally, fit() would do what we now call "letterbox"
mode, which tries to maximally fill the new size, retain the original
aspect ratio of the visible image, exactly fill one of the dimensions,
but not lose any visible image data -- this means that a dimension that
cannot be filled will be padded with black pixels. This is still the
default behavior.
But additional modes are not possible: "width" means to exactly fill
the horizontal dimension (the vertical may be letterboxed, or may be
cropped, depending on if it's too big or too small), "height" means to
exactly fill the vertical dimension (the horizontal may be letterboxed
or cropped).
* Add `fillmode` parameter to IBA::fit() in both C++ and Python.
* Old fit() without this parameter are still present for
back-compatibility, but are deprecated and will eventually (when
we're at an ABI-breaking release in the future) be removed.
* oiiotool: Add optional `:fillmode=` modifier to the `--fit` command.
* More comprehensive tests in testsuite/oiiotool-xform, testing every
combination of portrait, landscape, shrinking, growing, each fill mode.
* Update docs
The WebPData::size is a size_t, on 32 bit systems it will balk at our
putting a uint64_t in it. Downcast explicitly so the warning doesn't
flag us.
Also some changes to ensure adequate testing of webp in CI. (Even though
we only CI test 64 bit systems, so this was not related to the specific
build break being fixed... but an excuse to make sure we are hitting it
a reasonable amount in CI.)
Filesystem::IOFile constructor now upon an error opening the file will
set the error message appropriately.
Move IOProxy::error() implementation from inline to be in
filesystem.cpp to hide its details, use a static mutex to make error
set and retrieval be thread-safe (we presume that the rare chance that
multiple threads will be setting or getting IOProxy errors
simultaneously are rare enough that it's not a thread performance
issue, and if the user is getting file I/O errors, a bit of minor
locking is the least of their worries).
Better checking of these errors in OpenEXR input and output open
operations. Hopefully this means that exr file I/O errors will
percolate up more helpful error messages.
TL;DR: This could crash during openexr output, turned out to be a subtle
race condition.
Long explanation:
Metadata gets prefixed with `<formatname>:` in cases where it's only
meant to be understood by the reader or writer of one format. Formats
that support arbitrary metadata filter out any items that are prefixed
by the name of a different format (so you don't, for example, when
converting file formats, incorrectly copy a TIFF-specific metadata
hint into an OpenEXR file as literal metadata).
The specific code we used in the OpenEXR writer tried to do this but
got the threading locks wrong, and this could lead to very rare, subtle
crashes. It could have been fixed by a 3-line very local fix, but...
I also noticed that the same code was replicated in Field3D output (and
quite incorrectly -- in the cut and paste from the openexr code, it was
still looking for "openexr:" prefix instead of the correcd field3d one!).
Since this idiom was happening (wrong) in two different places, and
seemed likely to crop up again some time, instead of fixing locally I
added a new API call, `is_imageio_format_name()`, to do this centrally
and correctly. So most of this patch is implementing that and then
using it instead of the incorrect code elsewhere.
Also add python bindings and docs for the new API function
Upon reading cmake docs mre carefully, it's clear that when you pass a
version to find_package (which does much of the heavy lifting inside
checked_find_package), it DOESN'T MEAN you want that version or
later. It means you want to find a version that is "compatible" with
the named version. The meaning of "compatible" is up to the package.
Some packages define it as being >= the requested version. Others
define it as being the same major release, or same major AND minor
release, or exact.
But generally speaking, we only care about the minimum version (e.g.,
"I want FooBar 8.0 or later"). And it's not guaranteed that is what
the package's compatibility will do. In fact, FooBar may be using
compatibility mode "SameMajorVersion", which will make our find of
FooBar succeed for 8.0, 8.1, 8.2, but if a user encounters FooBar 9.0,
our build will reject it, even though it's probably fine.
So I augmented our checked_find_package wrapper to take optional
VERSION_MIN to give the real minimum version, when that's the semantic
we really want (in which case we would omit the ordinary version
number used for the compatibility check and let VERSION_MIN do all the
work). There's also an optional (and usually omitted) VERSION_MAX that
you can use if you are sure that we must be less than a certain
version (the max is exclusive, you must be <, not <=, whereas the min
is inclusive >=).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Some permissions on GH changed due to security concerns, and that broke
the setup-nuget action we were using... which turned out to be obsolete,
anyway, and switching to the new recommended replacement fixes everything.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Inspired by OpenColorIO's beautiful docs, I'm adding the "sphinx_tabs"
extension to our docs, and testing them out in one little spot in
ImageBufAlgo to see if we like the effect.
Please look at https://openimageio.readthedocs.io/en/rtd/imagebufalgo.html
look at the first couple screenfuls to see the example of how the
tabs look. Opinions are welcome. Should we do this kind of thing more
extensively?
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Detect if OIIO is being built as a subproject, and if so, set
`OpenImageIO_IS_SUBPROJECT` (a.k.a. `${PROJECT_NAME}_IS_SUBPROJECT`).
* Move all the testing logic out of the main CMakeLists.txt (and a few
testing-related macros from oiio_macros.cmake) and into a new
src/cmake/testing.cmake.
* Move all of the packaging logic out of the main CMakeLists.txt and
into a new src/cmake/packaging.cmake.
* Make all the testing and packaging inactive if this is being built
as a subproject. Also disable the clang-format target for subprojects.
* Move fancy_add_executable and add_oiio_plugin from oiio_macros.cmake
into their own files (leaving oiio_macros.cmake empty and
eliminating it).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
As OpenEXR works toward their 3.0 release, which includes splitting
Imath (the vector, matrix, math, and 'half' stuff) out into a separate
repo, their top of tree master finally breaks our "bleeding edge"
build.
This is a little tricky, because now our support of this dependency is
attempting to span 3 eras: 3.0 with the full split of openexr from
imath, recent openexr+ilmbase 2.x that supplies reliable exported
cmake configs, and older 2.x that predates cmake configs. Eventually
we'll phase out older versions and clean up.
Fixes in this patch generally include:
* Add a new CI test of VFX2021 + openexr/imath master. (This will go away
when 3.0 is released and that's what we use for the main 2021 case.)
* For new enough versions of OpenEXR, use the exported config file and
targets, rather than the old style OPENEXR_LIBRARIES style variables.
We have kind of a mixed idiom use because we're still supporting older
versions that predate use of the config files.
* Move imath includes to new OpenImageIO/Imath.h Deal with the fact
that the Imath headers now live inside `<Imath/ImathBlah.h>` rather
than `<OpenEXR/ImathBlah.h>`. But of course, you need to start
including things just to find out the version. It's tricky. The
logic is to look at `<OpenEXR/OpenEXRConfig.h>` to determine the
version we're dealing with, then include Imath stuff from one or the
other depending on what we found. I rearranged quite a few includes
to ensure that all this logic only happens in a couple spots and the
rest gets included transitively. As it turns out, quite a few of our
`#include`s of Imath headers were redundant.
* OpenImageIO-C depended on OpenImageIO publicly, but actually it
should depend on OpenImageIO privately, but export its include
needs.
* OIIO's exported config needs to look for OpenEXR.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Use __builtin_bswap when available. These seem to be 8-15x faster than
the std::swap-based implementation when multiple data values are
being swapped. (It's not dramatically better when swapping a single
value, either way it's pretty fast.)
Also add correctness unit test and benchmark for swap_endian.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The core ImageInput and ImageOutput (and auxiliary classes they use, such as ROI, ImageSpec, DeepData) now have C99 interfaces that wrap the native C++ classes. In addition to being good for using OIIO with C, this is an important step along the way to simplifying the implementation of Rust bindings (and other languages that have a simple ffi interface to C but are tricky to wrap C++).
Still to do: ImageBuf, ImageBufAlgo, etc.
* Bump breathe release to the latest (we'e been holding back because of
errors that we fix in the other elements of this patch, below)
* Changing declaration formatting, particularly in the ImageBuf and
ImageBufAlgo chapters, eliminates function ambiguity warnings and even
python exceptions we were getting in Breathe before.
* Document missing texture option fields and improve formatting.
* Instruct Doxygen to not catalog deprecated ImageBufAlgo functions.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We were running some CI tests only for PRs or for certain branches.
This may save some time, but I found that I was, more often than not,
naming my branches to trigger all tests to run. That's silly. Just run
all the tests every time, it's not that expensive.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
OCIO master recently renamed the symbol in its headers that we used to
detect its version, breaking OIIO's build against the current OCIO
top of tree master.
This change makes our FindOpenColorIO.cmake work with both old and new
OCIO.
Also, cut out other useless OCIO building (for our purposes) by having
our build_opencolorio.cmake disable buildig of GPU tests (which,
recently, became controlled by a different option than is used to
disable building all the other tests).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
ImageBuf methods subimage(), nsubimages(), miplevel(), nmiplevels(),
and file_format_name(), which are concerned with the file to which the
ImageBuf refers, only make sense for an ImageBuf that directly refers
to a file -- either by being constructed/reset from a filename, or in
some cases, being a direct unmodified copy of such an imageBuf. All
other cases, including pixel-copies of other ImageBufs (even those
that came from files), it is merely an in-memory image and doesn't
really refer to a file.
Clarify the docs (and fix the implementation) to ensure that for
any ImageBuf that is not a simple reference to a file, these methods
return a suitale default value.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Accommodate changes for 0.20.2 that Homebrew now has.
* Mistake in variable naming meant we weren't testing libraw master as
we previously thought.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
There is a big bug lurking in how we read TIFF files with "separate"
planarconfig (RRRGGGBBB, versus "contig" RGBRGBRGB). I knew that
libtiff when reading scanline files doesn't support true "random
access" to scanline for most compression methods -- because you can't
seek to the start of a scanline in the middle of a compressed "strip,"
you have to read whole strips, or read individual scanlines
consecutively. But what somehow I've misunderstood for years (if not
decades) is that for "separate" planarconfig, this means that you
can't jump from scanline i in plane 0 to scanline i in plane 1! You
have to read all the scanlines (at least within one strip) in a single
plane before moving to the next plane.
This means that we get all sorts of read failures when reading TIFF
files meeting all of these conditions:
* "separate" planarconfig (these are rare in general)
* Scanline-oriented (tiles are fine, because they're read whole)
* More than one channel (with one channel, separate and contig are
basically the same)
* Certain compression codecs (turns out the separate+scanline combo
tests in the libtiff example suite are uncompressed, which is how
our testsuite failed to catch this).
* rowsperstrip > 1 (if rps == 1, every scanline is a new strip, and
that worked)
* written by an app that doesn't use OpenImageIO (because OIIO TIFF
output forced rps=1 for separate planarconfig)
Things were very broken. There is a comment in tiffoutput.cpp that says
// I can only seem to make separate planarconfig work when
// rowsperstrip is 1.
and forces rps=1 for separate planarconfig output. So at some point
many years ago I realized that something odd was happening here, but I
apparently I thought it was a bug in libtiff when writing with rps > 1.
Probably we were writing correctly all along, and the real bug was in
the reading. This also explains why we had mostly not seen this
problem before, which is because all OIIO-written files avoided
producing tiff files that fell into the failures case.
This patch mostly fixes it and also makes a number of performance
enhancements:
* Fix misunderstanding in read_scanlines where it was loose with thread
safety in retrieving the rows per strip.
* In read_scanlines when deciding the "chunk size" to limit temp
buffer size, and in read_image when deciding how many scanlines to
read at a time via read_scanlines, these computations weren't the
same, leaving slivers to be read alone instead of staying aligned
with strips. Make those both compute the same way, so from read_image
on down, we end up reading chunks that are always aligned with
strips.
* TIFF read_scanlines used to try to read whole strips, except for
reading individual scanlines that didn't comprise a whole strip
length. Modify it to recognize that the one exception is the last
strip in the file, in which the strip is shorter than usual if the
total number of scanlines is not an even multiple of rps. This
ensures that the last strip is read as a strip, not as individual
scanlines.
* Imagecache autotile behavior is modified to force the pseudo-tile
size to be a multiple of the strip length. Again, that keeps the
resulting reads to always be in units of strips, which is much more
efficient and is the safe way to seek and read separate
planarconfig. Also, always turn on autotile for any tiff file with
rowsperstrip > 1, because it's already organized that way in the
file. This is not only more efficient generally, but also makes it
so that ImageCache will only do reads aligned safely to strip
boundaries.
This is not a complete solution, in the sense that an app making
individual read_scanline calls for solo scanlines, or read_scanlines()
that is not aligned to strips, can still have read failures for TIFF
files that are just the right combination of separate planarconfig +
certain compressions + scanline order.
However, with this patch, the following common cases will ALWAYS work
correctly now: (a) read_image or read_scanlines of the whole image
size; (b) any reads mediated by ImageCache or TextureSystem; (c) all
images read by ImageBuf (which always use either (a) or (b)); (d) all
operations of oiiotool (which use ImageBuf).
After this patch is merged, solving the common cases, I'll return to
the topic to tackle the remaining issue of broken reads for these
files when doing individual ImageInput::read_scanline reads or
read_scanlines with ranges not aligned to strips. To make those work,
I'll need some kind of ugly internal strip buffering.
I'm going to leave the "force rps = 1 when writing planar separate tiff"
as it is, because even after this set of fixes, software using old
versions of OIIO will not be able to correctly read rps>1 planar files,
so we wouldn't want to be inadvertently writing them, at least not
until all the pre-fix OIIO releases are unlikely to be used by
anyone. Not writing planar files with rps>1 is not much of a
sacrifice, considering how rarely planar files are used at all.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Now that we use Linux and the ASWF docker containers for the
"clang-format" test, there's no reason to have logic in this setup
script for Mac to choose a more restrictive set of package for that
case.
Same for clang-tidy -- we don't have a test like that anyway, and
if we add one, we will surely do it with one of the Linux containers.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
There was no formal error setting! Errors were echoed to the optional
output stream, if supplied. But that's not good enough.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Although we were careful to make sure that the new versions of
ImageInput `read_scanlines`/`read_tiles`/`read_image` methods were
thread-safe when called concurrently, one thing that slipped through
the cracks was error reporting. A failed read (or other) call would
return an error flag and use error() to set the error message, which
could be retrieved via geterror()/has_error().
Except the flaw was that the error itself was a single std::string in
the class. It was guarded by a mutex, so at least the threads wouldn't
clobber each other over access to the string. But it means that the
following call sequence would do the wrong thing:
thread 1: read_tile A --> fails
thread 2: read_tile B --> fails
thread 1: geterror() --> gets both errors appended, oops!
thread 2: geterror() --> gets an empty error message, oops!
Why hadn't we noticed until now? Because read failures of individual
scanlines or tiles are rare -- usually the failure or discovery of a
missing or corrupt file is in open(). So I guess it's not often that
two concurrent reads would both have just the right timing to have the
error retrieval problem described above.
There is no easy fix, so a bit of a bigger, ABI-breaking, change was
necessary to make this work properly once and for all:
* ImageInput and ImageOutput error messages are now thread-specific. That
is, geterror() retrieves errors generated by method calls made by the
same thread.
* ImageInput and ImgageOutput now use a PIMPL idiom to store some data
behind an opaque pointer. This is good -- it means that the details
are hidden from the public API, and can use classes we don't wish to
expose (which we will do here with boost::thread_specific_pointer) as
well as allow us to change what's behind the pointer without breaking
the ABI.
* The error message uses a thread_specific_pointer.
* We therefore removed the m_errormessage member that was unfortunately
in the public class. Also removed the mutex and the m_threads, which
can now be safely tucked behind the PIMPL.
* With the mutex hidden from view, we now re-jigger references to it
to use the lock/unlock methods we had already. In fact, instead of
using lock_guard<mutex>, we can use lock_guard<ImageInput> because
the presence of lock/unlock methods make ImageInput (and ImageOutput)
themselves compliant with the BasicLockable concept.
Note that this changes the data layout of ImageInput and ImageOutput,
and therefore is an ABI break. So this is a change meant for master
only (future OIIO 2.3+), and will not be backported to the 2.2 release
branch.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* WebP input overhaul (including animated webp and metadata)
Huge overhaul of webp reading. Since webp support was first added in
2011 (with no major changes since, other than a couple minor bug
fixes), we have been using the basic/simplified webp API, which had
many limitations. This patch rewrites it to use the more complete
API. This results in the following changes and new capabilities:
* RGB images are kept as 3-channel instead of forced to be RGBA upon
input.
* Skip alpha premultiplication for RGB files. And for RGBA, skip the
dubious color space transformation when doing premultiplication of
RGBA files. I think this has been wrong all along.
* We can now read animated webp images (akin to animated GIFs, they look
like multi-subimage files).
* Much more efficient to read just the header, the pixels are read and
decoded lazily.
* Switch to the WebPDecode*Into API calls, which lets us preallocate
and manage the decoded image buffer (as a unique_ptr). Also switch
the encoded buffer from a std::vector to std::unique_ptr, which
avoids an unnecessary zeroing out of the buffer.
* Close the file handle immediately after we're done reading, rather
than leaving the handle open until the client is done with the
ImageInput.
* Read Exif metadata properly from webp files.
* To use the more sophisticated API, we need to find and use the
libwebpdemux library.
* Since there is a "LoopCount" metadata here, similarly to with GIF, I
decided to make an OIIO metadata "oiio:LoopCount" that will work
with both. GIF saves with this, too, and "gif:LoopCount" is now
considered a deprecated synonym.
Fixes#2721
N.B. This only covers webp input, not output. A similar overhaul should be
done eventually on the output side.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Cuda 10.1 seems to have trouble because of the `__attribute__` used in
`OPENIMAGEIO_PRINTF_ARGS`. Declan Russell suggests modifying the `#if`
to avoid using it for `__CUDACC__`.
LG further observes that `OPENIMAGEIO_PRINTF_ARGS` is only used for
the declaration of two functions, it seems like a nice decluttering to
just directly use the short attribute exactly where it is needed
instead of having a lengthy explanation and possibly defining
`__attribute__` everywhere as a side effect.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Co-authored-by: Declan Russell <declan@solidangle.com>
For IC::add_tile() and IC::get_tile(), the documentation advertised that
passing chend < chbegin meant "all channels". But the implementation was
setting chend=nchannels but not setting chbegin to 0.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Different classes and modules had minor inconsistencies on how errors
were reported and retrieved. Let's fix these up.
* geterror() should always take a `clear` parameter that allows the
caller to control whether or not retrieving the error clears the
internal error message (default is true, clear it after retrieving).
* Any class with a geterror() should also have a has_error() that merely
reports whether there is an error message pending retrieval.
* geterror() will always *append* to an existing error message that
has not yet been retrieved, with a newline separating error
messages. Previously, some of our geterror implementations appended
and others replaced.
* geterror() will trim a single trailing newline from the message it
is passed, thus storing error messages that are not
newline-terminated and making it so that callers of geterror() don't
need to wonder whether or not they should terminate their error
messages with a newline or what happens if you make the wrong
choice.
* Usage guidance is therefore that individual errorf/errorfmt() calls
not include a trailing newline (though if one is present, it will
be harmlessly stripped), and that whoever retrieves with geterror()
should, after outputting it, add a newline (which they seem to
invariably do already).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
One of our CI matrix entries -- building with clang -- has
mysteriously broken with compile errors complaining about reallocarray
being inconsistently declared in stdlib.h and gif_lib.h. It just broke
a few days ago, unrelated to any changes on our part. I have no idea
if it was the compiler or gif library that got updated and then broke.
Whatever. Insulate ourselves from random changes on GHCI by converting
this test to use one of the ASWF containers that we can count on to
have stable dependencies installed.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Bring back the idea of a Findpugixml.cmake file because it turns out
that (for pugixml <= 1.10) the exported config files are not reliable.
* Fix CI scripts to build PugiXML (with a few different versions). We
previously were relying on the embedded version for all CI tests, which
masked problems associated with using an external PugiXML.
* Fix deprecation warnings about xml_document.load(), the new call is
xml_document::load_string().
Signed-off-by: Larry Gritz <lg@larrygritz.com>
It was setting it, but later, too late for the particular case where
constant_color_detect was used and found a constant color and also the
config spec came in without any tile preference (still 0's). So move
the default initialization up to the top.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This is good for optimization, allowing "de-virtualization" of function
calls in cases when the compiler can be sure that there can be no further
subclassing.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This used to work, was inadvertently broken in #2398, in Nov 2019, and
no release of OIIO 2.1 ever had it working! (I guess people don't
come across 16 bit iff files very often?)
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Boost date_time, though never explicitly called in OIIO code, can be transitively pulled in by boost threads. This seems to be a problem on Windows when you build a partial boost containing only the libraries you think you need.
The solution seems to be to set BOOST_ALL_NO_LIB on MSVC, even for the dynamic linking case.
What data format does oiiotool write for its outputs? The ones
specified by -d, if supplied, but otherwise it will make a guess based
on the types of inputs it has seen.
This patch makes several improvements:
* Multi-subimage files were getting very confused because different
subimages may want different data formats, and it was at best
remembering data formats overall and by channel name, but not taking
subimages into consideration. Make this much more specific by
remembering both raw channel names (for all subimages), as well as
specific channels of named subimages.
* The way the input data formats were remembered had repeated (and
non-matching) code depending on whether the input was read
immediately or lazily. That is now cleared up, with the logic
refactored and expressed only once.
* The -d directive can specify a type (for all channels), a type for a
particular channel name (`channel=type`), and -- this part is new --
a type for all channels of a named subimage (specified as
`subimage.*=type`) or just one channel of a specific subimage
(`subimage.channel=type`).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This can cause cache re-reads unnecessarily because it thinks you're
asking for potentially different configuration hints every time you
set the ImageBuf to a different subimage.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
A bug (issue 2713) revealed that an alloca in IBA::colorconvert could
cause a stack overflow on an image that had an unusually wide number of
columns per scanline.
This patch takes a number of measures to help root out these problems
in the future:
* The OIIO_ALLOCA macro is amended to assert (in DEBUG builds only)
upon any attempt to alloca more than 1MB, which is surely a sign that
we have used alloca in an inappropriate place.
* A new OIIO_ALLOCATE_STACK_OR_HEAP macro is used to encapsulate an
idiom I have used before, to alloca small things but new[] big things,
and ensure in both cases that the memory is freed when the function
exists scope. I choose arbitrarily for "big thing" to mean <= 64 KB,
which is big enough for one 64x64 tile or one scanline of a 4k image
if it has 4 float channels, that means most of the time we'll safely
allocate on the stack, but for anything larger, it will use heap.
* Use OIIO_ALLOCATE_STACK_OR_HEAP in those problematic spots of
colorconvert.
* Use OIIO_ALLOCATE_STACK_OR_HEAP in a few other places that seem
prudent.
* Grepped through the code for all instances of OIIO_ALLOCA and I think
that all the potentiallly "big" ones are now using the safe idiom.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
For "mip-mapped" TIFF files where we use the multiple images in the file
to emulate MIP-maps rather than what we ordinarily consider subimages,
a call to `ImageInput::spec(subimage,miplevel)` was not properly
recognizing invalid subimage values and returning an empty ImageSpec
as advertised. Basically, we assumed that they would only ask for a
valid subimage and miplevel and didn't properly handle out of range
values.
Same problem for spec_dimensions().
Signed-off-by: Larry Gritz <lg@larrygritz.com>
For both, allow an optional modifier ":n=" to specify the number of
images from the stack to be combined by having their channels or
subimages appended (versus the default of combining just the top 2
images).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Fix cause of a subtle bug: splitting an empty string returned one
string piece (also empty) instead of 0 pieces.
Beef up the Strutil unit tests in this area.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This was subtle. Empty subimage include/exclude specifications would
confuse compute_subimages(), which in turn would MUCH later do a
mis-allocation and lead to a situation in --resize where it would
iterate over subimages that had not need allocated.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Switch from old cmake macros that were marked obsolete in cmake 3.12 to
the new FindPython.cmake module.
Instead of defaulting to python 2.7, default to whatever is found.
But user can still set PYTHON_VERSION variable to request a specific
one in case there are multiple.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
ImageBuf uses ImageCache underneath. But this can lead to the following
unexpected behavior if you don't understand this:
1. Make an ImageBuf for an image, read it (even just header info).
2. Done with it, destroy the ImageBuf.
3. Some other part of the program, or even a different process, changes
that file on disk.
4. Make another ImageBuf with the same filename, thinking you're getting
the updated image, but in fact it's relying on the cached representation
which doesn't contain the changes made in step 3.
So these tiny changes try to make this much harder to shoot yourself in the
foot:
* New ImageBufs automatically call cache->invalidate on the filename
before they start reading, to ensure that if the file has changed on
disk, anything in the cache is invalidated.
* When you destroy or clear an ImageBuf, it calls cache->close() on the
file, to release the handle (especially helpful on Windows, where an
open file handle can lock the file to other processes).
There is a chance that some image operations will have degraded
performance, if you are mixing direct ImageCache use (including
TextureSystem) and ImageBuf reads of the same files -- since creating
and destroying the IB's will flush tiles from the cache and close cached
file handles. But I think that this should not be a problem in practice,
since most apps that are heavy users of ImageBuf (like oiiotool) aren't
heavey users of IC/TS, and vice versa. And eliminating this confusing
behavior is probably more important than occasional over-eager
invalidation of the cache.
Also clarify in the docs (and imagebuf_test) that if you use the
variety of ImageBuf ctr where you give it a custom ImageCache, it's
important that the ImageBuf outlives the ImageCache, as it is a
non-owning reference.
Fixes 1987
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This new command takes two images as arguments, and appends all the
metadata (only) from the first image onto the second image's pixels
and metadata, producing a combined image.
This can be useful for users who have some (either non-OIIO-based or
just badly implemented) application that operates on image files but
fails to propagate the metadata, by having a simple way to transfer
metadata from one file to another:
bad-app in.exr -o out.exr
# oops, out.exr no longer has the crucial metadata from in.exr
oiiotool in.exr out.exr --pastemeta -o out2.exr
# Now out2.exr has the pixels of out.exr, but with all metadata
# from in.exr added to or overwriting that found in out.exr.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We have many classes that have methods
.attribute(name, intval)
.attribute(name, floatval)
.attribute(name, stringval)
.attribute(name, TypeDesc, anyval)
For ImageSpec.attribute() and OIIO.attribute(), the anyval was
restricted to only accept tuples. Fix these to bring them in line with
many other classes' attribute calls in which the 'anyval' really can
be just about anything -- tuples, lists, numpy arrays or other buffer
objects, or scalar values.
Fixes#2674
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Seems that cmake's FindGIF.cmake isn't good at discerning minor
releases (prior to GIF 5.x including more info in its headers), so
saying "find_package(GIF 4.1)" doesn't work the way we'd like, and
seems to fail to find a legit 4.1 installation. Go back to
find_package(GIF 4).
We seem to have broken this in PR 2681, and I'm worried that people
with gif_lib 4.1 will find themselves mysteriously lacking GIF support
in their OIIO builds. I may do an emergency tweak release of 2.2 just
to address this.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
When transferring blocks of pixels (for example, by ImageInput.read_image
or ImageOutput.write_image), "half" pixels ended up mangled into uint16,
the only type of the right bit width.
But NumPy has a numpy.float16 type! A little bit of extra hinting to
pybind11 makes this work as users probably expect.
Fixes#2689
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Don't copy from the data pointer if it's null, that indicates an error.
Related to issue 2688, but doesn't fully solve it.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
For a few rare edge cases of photometric interpretation, we fall back
on libtiff's ability to auto-translate to an RGBA image for us. It
only does this for the whole image at once, so we do it for the first
scanline we read (storing the whole image in a buffer, skipping the
conversion for subsequent scanlines -- and it can tell it's the
"first" scanline because the buffer isn't yet allocated).
The bug was that for these cases (such as a photometric YCbCr), if the
file was *multi-subimage*, we were not clearing the rgba buffer, so
upon reading the second subimage, it thought its buffer was already
filled, so it wasn't actually doing the read and convert, just using
whatever was already living in the buffer. Oops.
Also, I noticed that we should clear m_use_rgba_interface for each
subimage -- it was previously just setting it when it found those edge
cases. We never saw a symptom from this, but I realized it would be
problematic for a file where the first subimage was YCbCr and the
second subimage was regular RGB, it wouldn't know to fall back to our
usual code path.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
It's been endless trouble, because the nature of the test is malformed
images that fail in inconsistent ways. I can never seem to make a
stable set of reference output for it, especially for OpenEXR
development master, which is always adding new fixes or tinkering with
exceptions and error messages.
So just don't run that test in the automated CI.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Plain old bug in all cases:
--croptofull botched the subimage count by reserving space for results
based on the subimages in the input image, but then only producing
results for the first one if -a (or :allsubimages=1) was not used.
Output specific bugs:
There are some operations that happen ephemerally upon output, mostly
to account for output file formats that don't support some feature of
the top image (like action_croptofull if the output format doesn't
support differing display and data windows, or action_channels if
there are more channels than are supported by the format, or
action_colorconvert when --autocc is used).
In these cases, we were forgetting that those operations may tend to
drop all but the first subimages, if no -a or :allsubimages=1 is not
used. But that's NOT the behavior we want in this case, because for
output specifically, we never want to drop subimages. This is because
we want a simple format conversion like
oiiotool in.xyz -o out.pdq
to faithfully copy, not "narrow" to one subimage, even if some other
operations on their own will tend to drop all but the first subimage if
not instructed to preserve them all.
So the solution is that we need to add ":allsubimages=1" modifier to
these ephemerally called adjustments.
Periodic compare of analogous build scripts in OIIO & OSL and synchronize
to ensure that my current most favored idioms are in both.
The biggest change is that I split the checked_find_package into a separate
cmake file, for easier copying between projects.
Also fix some minor typos.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
HEIF is a lossy format, and it seems libheif 1.8 changed internally
in such a way that the bit-for-bit comparsions of the write-read
cycle we do in imageinout_test started failing. (Bummer on our Mac
CI, where the homebrew version of libheif updated automatically and
then started failing.)
The easy solution is that just for this test, write zeros to the image,
not 1.0 values (which were coming out 254, not the expected 255).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
In the case of a dependency not being found, we look for
src/build-scripts/build_{package}.bash and if it exists, suggest using
it. But the scripts were all lower case, and the packages were not, so
it was often not giving the hint! Also, in some cases, the package
might be named 'TIFF' but the script was 'build_libtiff.bash'.
So now, look for a build script with the package name straight, and
also all-upper and all-lower-case versions, and "lib<lower>".
Rename build_ocio.bash to build_opencolorio.bash, because it's the one
case where the script didn't match any of those obvious
transformations.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This new file contains the hashes of commits whose only function was
to reformat large amounts of code.
Ordinarily, this would make for confusing 'git blame' output, by
making old code written by author A appear to have a new modification
date and authorship by B, when in fact we know that B just did some
reformatting. This can make it especially hard to discern code history
after after commits that establish the use of "clang-format".
A new git feature (requiring git >= 2.23) allows pointing to a file
containing hashes of commits to be ignored by git blame:
git blame --ignore-revs-file .git-blame-ignore-revs
To avoid having to remember this all the time, consider adding this to
your git config:
git config blame.ignoreRevsFile .git-blame-ignore-revs
Note that git supports this feature but does not dictate the name of
the file. I'm just following the convention I'm seeing pop up in other
projects of calling it ".git-blame-ignore-revs".
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Sometimes the true minimum version of a dependency that we are able to
build against is lower than the minimum *recommended* version (perhaps
because some important feature was added in a subsequent release that
we would like to expose if available).
This patch allows checked_find_package to take two new optional
arguments: RECOMMEND_MIN is the minimum *recommended* version (but not
the true rejection minimum) and RECOMMEND_MIN_REASON is some text to
add to the error message if there is a succinct explanation for why we
recommend a higher version than the true minimum.
As an example, we will (currently) build against any OpenEXR 2.x. But
if it's not at least 2.2, we can't support DWAA and DWAB compression.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
These were so out of date and unhelpful, it has to be better to
have no instructions at all. Tear it down and make room for somebody
to do it right.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Oops, ParamValue.type ought to be a TypeDesc, just like in C++. In
the Python binding overhaul of 2.0, it seems that I made a mistake and
unintentionally changed it to appear to Python as a string (e.g.,
'int' rather than TypeDesc('int')).
Fixing it to be the way it was originally intended would involve a
breaking change and not appropriate until such things are allowed by a
2.x -> 3.0 transition, so we'll just have to live with it.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The thing about IOProxy is, once closed, it can't be reopened. So it's
important that if an IOProxy-based file is in the IC, its ImageInput
should not be reclaimed by closing it (and that might not help the
file handle count anyway, since the main use of an IOProxy under those
circumstances is probably to read an image from memory).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
It's important not to mess with a passed-in IOProxy* in Input::close(),
because that proxy may not still exist.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Add support for reading DPX files from any IOProxy
* Replace memcpy_s function with memcpy
* Fix IOProxy bug (not just for DPX) seeking from end
Signed-off-by: ra-mt <radu.arjocu@masstech.com>
I just like this better than the prior habit of putting it one PAST
the column width, which made the backslashes go in column 81, ick.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Last week, Vcpkg apparently had some flaw that recently caused it to
fail when building/installing libpng (among other things). It's been
fixed in the vcpkg main branch, but that hasn't propagated to the
vcpkg that's installed on the GitHub Actions CI runners, and that's
causing our Windows CI tests to fail.
Rather than wait out the fix, I'm just taking it into my own hands by
building all our dependencies from source on Windows. This means adding
bash scripts to download and build zlib, libpng, and libjpeg-turbo (and
maybe those will be helpful to others as well).
**Late breaking news** They beat me to it, vcpkg has been updated and
the GitHub CI Windows tests are passing again.
But I had already done this work! So I still would like to switch to
this methodology because:
* Sometimes it's faster to build ourselves rather than have vcpkg
build, since vcpkg always builds both release and debug, but we only
need the release builds of the dependencies.
* This makes it easier to customize the specific version of a
dependency, or particular build options, which are hard or
impossible to change for a vcpkg build.
* These same dependency-building scripts could be useful for people who
are not on Windows or not interested in using vcpkg.
Also some minor changes to the build_libtiff and build_openexr scripts
to synchronize some of the idioms I use in these scripts.
Rename workflow.yml to ci.yml to adhere to convention and also make it
so we can have multiple workflows in the future.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Fixes a compilation error, that appears on alpine linux.
/builds/lmarz/aports/testing/openimageio/src/oiio-Release-2.1.18.0/src/libOpenImageIO/exif.cpp: In function 'void OpenImageIO_v2_1::read_exif_tag(OpenImageIO_v2_1::ImageSpec&, const TIFFDirEntry*, OpenImageIO_v2_1::cspan<unsigned char>, bool, int, std::set<unsigned int>&, const OpenImageIO_v2_1::pvt::TagMap&)':
/builds/lmarz/aports/testing/openimageio/src/oiio-Release-2.1.18.0/src/libOpenImageIO/exif.cpp:789:20: error: comparison of integer expressions of different signedness: 'unsigned int' and 'OpenImageIO_v2_1::span<const unsigned char, -1>::index_type' {aka 'int'} [-Werror=sign-compare]
789 | if (offset >= buf.size()) {
|
When the texture cache reads scanlines in random order, the same pixels could
be read many times over. Now the file offset for scanlines is cached for quick
random access.
* OpenImageIO/imagebufalgo.h: remove OIIO_API from enums
* OpenImageIO/strutil.h: remove OIIO_API from inlined lower and upper functions
* libutil/plugin.cpp: cast return value of GetProcAddress to void*
* libutil/sysutil.cpp: define _POSIX_C_SOURCE for localtime_r declaration
* libutil/thread.cpp: cast t->native_handle() to HANDLE type
* ptex.imageio/ptexinput.cpp: use perr.c_str() for %s string format
* Fix some incorrect error calls in ImageBuf internals.
* Simplify because we know that ImageBuf::error() and warning(), with
just one string argument, doesn't need a format command.
* Make sure that if no message bubbles up, we say "unknown error".
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We weren't setting up the proper handler callbacks for libpng internal
errors to bubble up to the caller.
Tell libpng to turn off sRGB color profile check. There are many
false positives where embedded ICC profiles that don't match its
expectations (but that other people and apps consider fine) trigger
errors in libpng.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
| master | :white_check_mark: :construction: ALL fixes immediately, but this is a branch under development with a frequently unstable ABI and occasionally unstable API. |
| 2.3.x | :white_check_mark: All fixes that can be backported without breaking ABI compatibility. New tagged releases monthly. |
| 2.2.x | :x: Only receives occasional critical fixes, upon request. |
| <= 2.1.x | :x: No longer receiving patches of any kind. |
## Reporting a Vulnerability
If you think you've found a potential vulnerability in OpenImageIO, please
report it by emailing security@openimageio.org. Only the project administrator
has access to these messages. Include detailed steps to reproduce the issue,
and any other information that could aid an investigation. Our policy is to
respond to vulnerability reports within 14 days.
Our policy is to address critical security vulnerabilities rapidly and post
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Larry Gritz