* For analysis, USE_QT=0 will disable osltoy from building (since we
don't/can't test it in the testsuite, it's unexecuted but really
shouldn't count against our code coverage metric).
* Exclude Imath duplicate headers from Sonar analysis.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Traditionally, OSL has supported building against the current and also
the previous supported version of OIIO, but not necessarily any older.
With OIIO 2.4 released (any day now), and OSL "main" is actually going
to be next year's supported release, it seems reasonable to change the
forward-looking development of OSL to have a minimum of OIIO 2.3,
which means that OSL-next will support OIIO 2.3, 2.4, and the
currently-in-progress 2.5 (or whatever it's called by that point).
The main reason to do this bump is because we'd like to depend on the
ustringhash class, and possibly other things we will probably be able
to easily backport to 2.3 but not to 2.2.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* A variety of changes to get a clean build and passing tests when
using LLVM 15.0.
* I've noticed problems when using LLVM 15 but building with earlier
clang, so the cmake scripts now print a warning in that case, so if
users encounter trouble they have a hint about what to do to fix it.
* For our CI tests on Mac, force the MacOS-11 test to use llvm 14,
and the MacOS-12 test to use llvm 15.
IMPORTANT TO-DO / CAVEATS:
1. When doing JIT at optlevel 12 or 13, I had to disable a number of
passes that don't seem to exist anymore in LLVM 15. This is enough
to get it working, and to be honest, I don't know if anybody uses
these opt levels. But we need to revisit this, because I don't
know if there these are cases where the names of the passes merely
changed or that new passes take their place (in which case we
should add the new passes, not stop after merely disabling the
deprecated ones). For that matter, optlevel modes 11, 12, 13 are
supposed to match what clang does for -O1, -O2, -O3, and that
changes from one release to the next, so we should probably revisit
this list and make sure it's matching clang's current choices
(which I assume are crafted and periodically revised by clang's
authors).
2. LLVM 15 switches to "opaque pointers". It still supports typed
pointers... for now. But as you can see in the diffs, I had to
disable a variety of deprecation warnings and take some other
actions to put LLVM 15 in the old "opaque ptr" mode to match our
use of LLVM <= 14. But this is only temporary, because the typed
pointer mode is expected to be removed entirely in LLVM 16. So at
some point in the next few months, we are going to need to support
opaque pointers in our use of LLVM. (Also note: for a while, we're
going to have a bunch of ugly `#if` guards or other logic to
support both opaque pointers for users with llvm >= 16 and also
typed pointers for users with llvm <= 14, until such time as our
LLVM minimum is at least 15, which I expect is not a reasonable
assumption for at least a couple years.)
Signed-off-by: Larry Gritz <lg@larrygritz.com>
- There are a handful of tests we disable entirely with because they
are irrelevant or just cannot be made to be clean (and that's ok).
- Tests with sanitizers run slow, so we increase timeouts, and also
set AA to 1 for render tests to speed them up. We rig it so a
non-matching output won't fail, and that's ok because every other CI
test is checking the results; here we have no reason to believe
results will differ from other tests, we just need to run with
sanitizers turned on.
- In testing.cmake, we rig texture tests to disable the leak
sanitizer, because we can't eliminate some completely benign leaks
from the texture system, and because they are called from JITed
code, we don't get a clean stack trace of things that happen in OIIO
and therefore we just can't enumerate the allowed exceptions. So we
trust that OIIO itself tests the texture system adequately and is
able to ignore the benign "leaks."
- We already submitted PRs and merged the little fixes that were
discovered by enabling address and leak sanitizers. This is just the
final bit to turn it on for CI runs from here on out.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We had hard-coded longer allowable timeouts for particular testsuite
tests. But for the static analysis runs, it takes extra long and we
need a way to tell it to increase the timeouts even more, and to
exclude the longest running tests from the analysis coverage run (it's
not necessary to run those tests, they don't increase coverage enough
to justify being an hours long run).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Exclude osltoy from coverage analysis, since we don't have a way to
run it from CI
* Run batch shading for analysis to get better code coverage
* llvm_gen_select: extra check that we can't overrun memory of cond[]
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Minor changes to fix a couple more warnings given by SonarCloud
static analysis.
* Get the code coverage enabled, it wasn't running properly.
* testrender: allow secret TESTRENDER_AA to override command line --aa
* runtest.py: allow secret OSL_TESTSUITE_SKIP_DIFF to force skipping
the reference comparison step.
* For analysis run, set TESTRENDER_AA=1 to keep the tests from being
prohibitively expensive, and OSL_TESTSUITE_SKIP_DIFF=1 to keep it
from failing because the aa=1 images are different.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Another round of clang-format
* Bump the standard clang-format we use for CI from clang11 to clang14.
This will make some minor additional changes.
* Due to a mismatched target name, we weren't actually checking the
formatting in CI on all the files we thought we were, and therefore
not assuring that certain code was correctly formatted. So there is
some fixup needed, which is included here.
* And a bunch of files we didn't try to do before, either.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Set up a SECURITY.md explaining how to confidentially report
vulnerabilities.
* Create scorecards.yml workflow to run OSSF scorecard to check supply
chain security issues.
* Use exact SHA's for actions versions to guard against hijacked
actions repos that might try to substutite something else at the
version tag we use.
* Make sure actions we use don't get write perms on our repo.
* Tried to use harden-runner action to ensure no actions on the runner
overwrite source code or have unauthorized outgoing net connections
during the GHA run, but currently leaving it disabled because of too
many false positives -- it flags every dependency download as a case
of altering source code. Which it technically is, but it doesn't
matter for any jobs that aren't producing any artifacts that will be
consumed or relied on by users.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
A few minor changes were needed to address new warnings.
clang14 warns if you use bitwise `|` or `&` in an `if`, but there are
some places where we do that purposely to avoid branches.
Disable clang14 warnings about bitwise ops for all of liboslexec.
Also some batch shading related vectorized loops fail to vectorize
with clang14 -- though they worked fine with clang12 -- and so we
needed to mark those as "complex" SIMD loops. Is this a regression
with clang14?
Revise simd loop pragma for wide_transformc due to llvm14 regression
Signed-off-by: Larry Gritz <lg@larrygritz.com>
checked_find_package:
* Optional SETVARIABLES gives a list of vars to set to TRUE if the package
is found.
* Optional DEBUG flag turns on extra debugging information about how the
package is found.
* Optional PREFER_CONFIG uses a dependency's exported config, if available,
rather than our Find module.
* Global ALWAYS_PREFER_CONFIG option can be set to ON to always prefer
config over Find modules for all dependencies.
fancy_add_executable:
* -DENABLE_INSTALL_<executable>=OFF lets you turn off the installation
of an executable.
other:
* Change the target name for doing the clang-format over the code to
"run-clang-format", because "clang-format" is an imported target name
of the libclang exported cmake (which we may wish to use soon).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
API changes we had to take into account:
* TargetRegistry.h location
* No more DisableTailCalls field in PassManagerBuilder.
Needed to update the ref image for render-microfacet test, some sparklies changed.
Looks like the new LLVM probably JITs to ever so slightly different math code, tickling
some LSB differences that at 1 sample per pixel, results in some different sampling
directions leading to fireflies. We decided to just commit a new ref image and move on.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This eliminates a huge amount of redundancy in our CI descriptions --
instead of the entire CI logic replicated for each test in its own
universe, we group tests into (a) all the ones based on ASWF VFX
Platform containers, (b) non-VFXP-year oddballs run on Ubuntu, (c) Mac
tests, (d) clang-format verification. Each has the build logic
expressed once, and a short list of variables that change for each
individual test of that category.
This organization is much easier to read, maintain, modify tests, or
add new test variants, and cuts the size of the ci.yml file by over 50%.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Add compiler support for Intel(r) oneAPI DPC++/C++ Compiler (based on LLVM technology).
Detect if CMAKE_CXX_COMPILER_ID matches "IntelLLVM" and set CMAKE_COMPILER_IS_INTELCLANG=1
NOTE: CMAKE_COMPILER_IS_INTEL will not be set, that is reserved for the classic Intel(r) C++ Compiler.
Define OSL_INTEL_LLVM_COMPILER to hold an encoded Intel(r) LLVM Compiler version
(e.g. 20220000), or 0 if not an Intel(r) LLVM Compiler.
Define OSL_INTEL_CLANG_VERSION to hold the encoded Clang version the
Intel(r) LLVM Compiler is based on (e.g. 140000),
or 0 if not an Intel(r) LLVM compiler.
Ensure OSL_CLANG_VERSION is when its a vanilla clang compiler, not apple or Intel(r) oneAPI DPC++/C++ Compiler.
Added OSL_ANY_CLANG which is set for any clang based compiler.
Removed OSL_NON_CLANG_COMPILER.
Updated uses of OSL_NON_CLANG_COMPILER with more specific and understandable combinations of OSL_ANY_CLANG, OSL_INTEL_CLANG_VERSION, and OSL_INTEL_COMPILER.
Make sure our math calculations are consistent,
especially division. Different compilers may have choose to use
reciprocal division loosing precision causing slightly different
results which can lead to aliasing differences when running the testsuite
Disabled reciprocal division for all projects.
if -ffast-math is enabled to to the underlying compiler, inf's and NaN's may not be handled properly (by design)
When building liboslexec, ensure any source files that require proper INF or NaN handling have additional command line flags passed if needed (as the default behavior maybe -ffast-math):
shadingsys.cpp
wide/wide_shadingsys
wide/wide_optest_float
Enable unitialized Matrix44, because all of its members get overwritten anyway, in impl_transform_normal_masked.
Added alternative test results where needed, wsing '.icx.' as shorthand for the Intel(r) oneAPI DPC++/C++ Compiler in alternative test result filenames.
Changed some tests to use the pretty(value) which clamps values close to 0 to be 0 to avoid lots of different alternative rest result files when using printf.
Added "--center" option to several regression tests to avoid aliasing in results between SIMD batched and scalar execution.
* Added icx-vp2022 to CI plan.
* To better clarify the classic vs. LLVM based Intel compilers and that they have version #'s not just 0 or 1,
renamed OSL_INTEL_COMPILER to OSL_INTEL_CLASSIC_COMPILER_VERSION
renamed OSL_INTEL_LLVM_COMPILER to OSL_INTEL_LLVM_COMPILER_VERSION
Removed unnecessary FMT_VERSION and OPENIMAGEIO_CMAKE_FLAGS from the icx-vp2022 worklow, added back USE_SIMD.
Fix spelling of losing.
Added OSL_OMP_COMPLEX_SIMD_LOOP(...) which is enabled or disabled in platform.h based on which compilers/versions can handle vectorizing complex nested control flow. This makes it cleaner to document which loops need to vectorization conditionally controlled and centralizes where it is managed.
* Removed extra fp-model=consistent for icx-vp2022 workflow
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
(many ported/replicated from various recent OIIO PRs)
* CI: bump to latest dependency releases
* Bugs fixed in build_opencolorio.bash, it botched some of the
directory choices when customizing where to place and build
it. Also, switch to the more standard BUILD_SHARED_LIBS cmake
variable to control whether to build dynamic or static libraries.
* Correctly handle the cmake variable VISIBILITY_MAP_FILE being set to
"" on the command line as a way to disable using the visibility map
file entirely.
* On Linux, also set link flags `--exclude-libs ALL` to hide symbols
from any embedded static library dependencies, to prevent clashes
with other versions of those libraries that may be compiled into a
DCC app.
* Some other minor touch-ups in compiler.cmake.
* Refresh build_openexr.bash with the latest changes from OIIO.
* CI workflow: revise failed build artifact storage to capture cmake
files.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We used to build MaterialX shaders on our end. Now it's all done in
the MaterialX project itself, and along the way, we failed to keep in
sync with them. (And even if we had "kept up", what does that even mean?
We want OSL to be independent of any particular MX release, but the MX
shaders may change from release to release. Better to be entirely on
their side of the fence.)
We turned off building these shaders by default a long
time ago, now let's finally remove the last remnants and urge everyone
to rely on the MaterialX project for this material.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* pybind11 2.9.0 release -- bump the vfx2022 and "latest" CI tests to
use it.
* Adjust other CI test pybind11 versions to get good coverage across all
the pybind11 releases, and to synchronize the vfx platfom tests with
the pybind11 release that was current for its respective year.
* Sync build_scripts/build_pybind11.bash with the latest in OIIO, including
bumping the default pybind11 version we download to 2.8.1.
* Opportunistic touch-up of warnings about llvm version to bring
into sync with our actual current requirements.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Among other cruft that existed to support versions 7 and 8, this lets
us completely get rid of PrePromoteLogicalOpsOnBitMasks custom pass,
which is only needed for LLVM < 8.
Also get rid of some cmake checks for LLVM versions that require
C++14 or higher -- not necessary now that C++14 is our minimum.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Increase testing timeout, some of the batch shading regression tests
seem to take longer than the prior 240s limit.
* Turn off OpenVDB when using the osl:2021 in combination with OpenEXR
3.x -- the version of OpenVDB in the 2021 container is too old to
understand OpenEXR 3.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Use LLVM 13 / clang 13 for the VFX Platform 2022 test.
* Add MacOS-11 test
* Poke around how to use arbitrary LLVM versions via Conan. (Works! But
I have not yet enabled it.)
Signed-off-by: Larry Gritz <lg@larrygritz.com>
OIIO master recently removed support for Field3D, which is rarely used
and the project is effectively unmaintained and isn't able to build
against the newer OpenEXR releases that we want to encourage people to
use.
So the primary purpose of this patch is to switch our unit test for
texture3d from using a field3d file (and thus would not run when using
newer OIIO) to a roughly equivalent test using openvdb (which is
supported, assuming OIIO is built in the presence of OpenVDB).
But, ha, I discovered two things, either one of which means that we
had actually not been unit testing texture3d at all lately: (1) To
discover whether the installed OIIO supported openvdb (and therefore
whether to bother running the test), we were relying on a totally
nonexistant OPENIMAGEIO_BIN cmake varaible. I don't know what that was
supposed to be (did I mean OIIOTOOL_BIN? but that also is not a
thing), but anyway, it caused it to always skip the test. (2) Oh boy,
the OIIO that we were relying on for may tests, provided by the ASWF
docker containers, was an OIIO that was not built in the presence of
OpenVDB (even though our OSL testing containers DO have OpenVDB), so
again it skipped the test because it needs that OIIO-side support for
texture3d() to know how to read openvdb files. Fix this by building
OIIO from scratch with openvdb support when doing the "VFX Platform
2022" test, rather than relying on the one in the container.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Most important: make sure we have a proper VFXPlatfom-2022 like test,
with Python 3.9 and OpenEXR/Imath 3.1.
Fix a few of the "name" entries for tests to match the set of versions
actually present on those images.
Remove apt-get install of libboost-regex-dev, since we use only
std::regex now that we're no longer using gcc4.8.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Similar to a recent change in OIIO... For historical reasons, our
wrapper makefiles assume that we build in `build/ARCH[.variant]`, so
maybe build/linux64, build/linux64.debug, etc.
Get rid of this extra bit of hierarchy, it no longer serves us and
it's contrary to modern cmake practice.
Also used this as an excuse to remove some SPI build script
customization that we no longer use.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Update the "bleeding edge" CI test to gcc-11, and also LLVM 12,
which we hadn't tested before.
* A "latest releases" CI test uses gcc10 and LLVM 11.
* Turns out that nothing else on the OSL side needed to be fixed to
accommodate gcc 11 (though you need a very up-to-date OIIO if you
want to compile it with gcc11 unless you turn warnings off).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
As discussed at the TSC meeting and on the mail list, I won't merge this
change until June 1. It will only be in master (future OSL 1.12), not
backported to any release branch.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Larry Gritz