On the OIIO side, there's a PR in review that changes the ImageCache
and TextureSystem APIs to return shared_ptr instead of raw.
(https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/4377)
This is destined for the upcoming OIIO 3.0 release.
This patch on the OSL side ensures that we build cleanly against both
OIIO 2.x and 3.0.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Some explanation is needed:
OIIO master recently added the ability to auto-build OCIO when a new
enough one is not found at its build time, and bumped the minimum OCIO
to 2.2.
Sometimes this is triggered by OSL's CI, when OCIO < 2.2, so building
OIIO also triggers building of OCIO.
All fine, except for the case when we are testing Intel's icx
compiler, which OCIO's build system does not recognize and fails to
build. So we make a little modification here that lets us, just for
that case, ensure that OCIO is pre-built with gcc even though we're
building OIIO and OSL with icx.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The convention in the single-point RendererServices is that the texture call returns
derivatives in st space, and they are transformed to xy space before returning
from the wrapper to RenderServices. This change makes BatchedRendererServices
follow the same convention.
---------
Signed-off-by: Stephen Friedman <sfriedma@pixar.com>
This was breaking CI for us. Hard break for icc/icx tests, since those
REQUIRED a yum install of the intel compilers. Softer undetected break
for all the ASWF <= 2022 containers based on CentOS 7, which were not
failing outright but were failing to install certain optional
packages.
The solution (for now) is to configure yum to exclude the missing repo.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Various minor changes to eliminate reference to some long-ago
deprecated OIIO items (mostly which simply alias to std versions)
before then disappear for real.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The backfacing shadeop now directly accesses the backfacing field
on the shader globals rather than calculating it from N and I.
This updates the shader-globals-needed query to reflect that
change.
Signed-off-by: Stephen Friedman <sfriedma@pixar.com>
Fixes to documentation of new oren_nayar_diffuse_bsdf parameters
* The new parameter was explained, but not added to the function
prototype in the docs.
* The semi-deprecated languagespec.tex was fixed, but it slipped our minds
that we've shifted the docs to markdown for ReadTheDocs (the real place
we should have changed was stdlib.md).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Implement new EON mode for Oren-Nayar closures (via an opt-in keyword arg for backwards compatibility)
* Add test for energy conserving Oren-Nayar (on half the sphere so we keep checking the old mode as well)
Signed-off-by: Chris Kulla <ckulla@gmail.com>
* Fix references to old OIIO home
* Fix embarrassing release notes generating script botch that
accidentally used OpenImageIO URLs where OSL URLs were needed.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We've been C++14 minimum for a long time now, we can just say constexpr.
Change a couple of stray OIIO_CONSTEXPR14 occurrences as well.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
fix: Fix typo
This is the only way that makes any sense -- define OIIO_HIDE_FORMAT
if it ISN'T already defined.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Some methods and enums got renamed from LLVM 17 -> 18, etc.
Also some adjustment to warning messages about version mixing
between clang and llvm.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Especially for our "bleeding edge" test that builds from the current
master of several dependencies, sometimes our tests fail not because
of anything we've done, but because of a change in one of the
dependencies. To help track down exactly which change broke the
dependency, make sure our CI build logs print the specific commit hash
of those packages that we built.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Background: Intel's icc is essentially discontinued and we think all
the renderers using OSL have moved on to the newer icx, but Alex Wells
still likes to sometime use icc for its better facilities for some
SIMD code generation sleuthing that he does. So we keep it in our CI
matrix to esure we don't break it for OSL until it is truly
unavoidable or he doesn't feel like he needs it anymore.
Some recent changes to OIIO's simd.h, from PR 4187 in that project,
seem to tweak what appears to be a compiler bug in icc -- bugs were
also noted in OIIO in the course of this PR and some workarounds were
put there just for icc to make OIIO's own tests pass. Well, once that
went in, OSL's CI tests for the icc case also started failing. It's
only texture derivatives that are broken, and in a weird wy that
smells like a compiler icc bug with SIMD generation similar to what we
discovered on the OIIO side.
Like I said, this is a discontinued compiler that no major renderers
are still using, so it doesn't seem worth tracking it down any further
to find a true fix. So let's just lock that one test case down to OIIO
2.5 (the current release) to avoid the changes that will eventually be
in OIIO 2.6/3.0 and hope that we fully retire icc before OIIO 2.5 is
no longer supported by OSL.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
OIIO recently pushed its master to be C++17 minimum. When using a C++17-minimum OIIO, you need to build OSL with C++17 as well. This broke a couple CI cases for OSL.
There were a few spots where we were hard-coded to C++14 on the OSL side rather than even using the C++ version specified for the OSL build, so these are some fixes here to address this.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The isconnected() returns 2 when used on an output that is connected to another downstream shader node -- this can be used to limit computation to only the outputs used. However, that misses the important case of where the renderer has declared a shader node output as a renderer output, which happens often when using OSL purely for pattern generation purposes. This change makes that case also return 2.
The isconnected test has been updated to include testing this functionality.
Signed-off-by: Stephen Friedman <sfriedma@pixar.com>
Prevent complications on the GPU by printing error messages at compile time instead of generating code to print them at runtime when a closure doesn't exist.
---------
Signed-off-by: Stephen Friedman <sfriedma@pixar.com>
* Have OSL indicate when it needs fliphandedness in shader globals.
* Add testsuite test to check that calculatenormal() triggers a globals_needed flipHandedness value.
---------
Signed-off-by: Stephen Friedman <sfriedma@pixar.com>
Fix bug in batched codegen of compref when the index is varying.
The loadvalue was left to defaulted to uniform load, which we can't store into a varying/wide result.
Fixed similar bug in getmessage, and added check for spline knotcount to be uniform (although varying support could be added).
This bug exposed a missed optimization in BatchedAnalyser where a early out (return) needs to change the conditional to be varying to support masking. However no check was being done to see if the loop control existed higher in the callstack. So now the execution scope tracks loop nesting depth to see if the current function is really in a loop or not. This allows the loop control flow to remain uniform (avoiding masking and more complex control flow).
Added nestloop-reg to testsuite to reproduce bug and verify its fixed.
---------
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
Tweak results for ARM tests that have minor numerical differences and
update references.
Set parallel build level to nprocs+2 instead of hard-coding to be
better for the new runners having more cores than they used to.
Speed up macosx build of deps by using unity build of OIIO.
Fix runtest, was clobbering parameters to oiiodiff
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We had been using clang-format 14. Bump to 17 as our official
reference standard for formatting.
Moved the clang-format CI test from an aswf container to bare Ubuntu
to make it run faster -- it doesn't need to do a real build, so
there's no point wasting 2 minutes or so downloading a container holding
library dependencies it won't need.
Also fix a duplicate line in .clang-format that newer clang-format
complained about.
Document that it's specifically clang-format 17 that is our formatting
standard
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Delete the NVPTX target machine in the LLVM_Util destructor.
* Initalize m_nvptx_target_machine to nullptr.
* Workaround clang/CUDA12 texref issue.
* Remove the unnecessary null check before deleting the NVPTX TargetMachine.
---------
Signed-off-by: Tim Grant <tgrant@nvidia.com>
A recent change in OIIO master made some of the ustring methods
"trivial", and somehow that makes these variables warn about being
unused, which it did not do before.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Bump the "bleeding edge" test to LLVM 17, the "stable latest releases"
test to 16.
Cover clang 17 compilation with MacOS-13 build
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This can come up if, for example, you build an OIIO using a static
Imath library that's more recent than what you're forced to use for
your OSL build. (Don't ask, you don't want to know.) So we need
to see half.h before we include any OIIO headers. But it's the OIIO
headers that adjudicated where we get half.h from, so we need to
be a little more explicit in, of all places, rend_lib.h of testrender.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Fixes#1755: Build error on MSVC with Partio
Explicitly mark unsupported/unused PointCould copy/move constructors and assignment operators as deleted.
Change SortedPointRecord index type from int to uint32_t and add static_cast during construction to avoid a warning.
Change SortedPointRecord index type from uint32_t to Partio::ParticleIndex to handle 64bit indices (where we can)
clang-format changes
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
Added OSL_INTEL_CLASSIC_PRAGMA(...) which only emits for Intel(r) Classic C++ Compiler, previously OSL_INTEL_PRAGMA did that.
As many Intel compiler specicif pragmas are now supported by icx and icc,
redefined OSL_INTEL_PRAGMA to emit for the classic and llvm based Intel compilers (icc and icx).
Updated some uses of OSL_INTEL_PRAGMA to OSL_INTEL_CLASSIC_PRAGMA as appropriate.
Fixed long icx 2023.1.0, 2023.2.0, 2023.2.1 build times for certain files by removing the "-mllvm -inline-threshold=100000" flag for all but 1 file avoiding the overhead of optimization passes being applied to inlined code that wasn't actually part of the SIMD loop that we truly needed inlined. icx now supports "#pragma forceinline recursive" which allows us to target just the SIMD loops to be inlined (as we did for icc).
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
This patch adds a new test to the CI matrix: verifying that we do not
break the ABI during releases. This is based on similar work done a
couple months ago in OpenImageIO.
Basically, in addition to building the current checkout, it also
builds a checkout of the designated "ABI standard" (generally, the
first non-beta release of a major version), then generates ABI dumps
of both versions and runs the ABI compliance checker using the tools
from https://github.com/lvc
Making two builds sounds expensive, but both builds are rigged to be
minimal, since only the libraries are needed for ABI checks (we can
turn off the binary tools, unit tests, python bindings, etc.), and of
course the test suite need not run. So actually, it's less expensive
than most of the test matrix entries, and it helps ensure that we
don't accidentally break ABI when we don't mean to.
I'm not sure how many false positives we'll have. We might need to
adjust the methodology slightly as we gain experience with this.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
When we bumped to cmake 3.15 minimum recently, we changed lots of
constructs like this:
if (VERBOSE)
message (STATUS "blah")
endif ()
to this:
message (VERBOSE "blah")
but I didn't realize that this was not enough, and to do it right
requires CMAKE_MESSAGE_LOG_LEVEL. So even when VERBOSE=ON, we weren't
actually getting all the verbose messages. Here, we fix it.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Fixes#1750
Batched shading crashes in point_cloud search if "index" is not passed as an attribute.
Fixed by allocating space for indices on stack before calling into underlying search algorithm which requires them (thus crashed if not provided).
Added 2 new tests to exercise it with no indices provided and with indices array provided but not long enough.
testsuite/pointcloud/rdcloud_varying_mismatch.osl
testsuite/pointcloud/rdcloud_varying_no_index.osl
Added alternative test results for
testsuite/pointcloud/ref/out_rdcloud_varying_mismatch.macos.tif
testsuite/geomath/ref/out.icx.alt.txt
Format change for wide_oppointcloud.cpp
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
This is probably the last time we'll regen the PDF docs, since we
are mid-transition to ReadTheDocs. We will probably not directly
distribute a PDF version with 1.14 or subsequent releases.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The newly released OIIO 2.5 (and changes in master) adjusts a few of
its minimum dependencies (cmake, openexr, fmt) in ways that break some
of our CI test cases for OSL 1.12, and also deprecated a helper method
called ustringHash that OSL referenced in a few places.
These have already been addressed in the OSL main branch (what will
soon be OSL 1.13), and in some cases OSL-next has also bumped some of
the very same dependencies.
But this PR is dealing with the 1.12 branch, in which we cannot raise
minimum dependencies that OSL requires. So here we are only adjusting
which versions of those dependencies we are using or building for the
specific tests in our CI matrix that use the sufficiently new OIIO that
it needs the updated changes.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Stephen Friedman alerted me to the fact that on Windows, bit fields
within a struct will only be combined if they are the same type. We
were a bit sloppy in the defintion of SymOverrideInfo, and that was
causing the struct to end up a different size in Windows. I think the
solution is that all those bit fields should be `unsigned int`.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The MacOS 11 CI seems to be broken on GitHub's runners. I'm not sure
why, but since it's old and getting various deprecation warnings from
Homebrew and GH, I'm inclined to just disable it rather than work hard
to keep this old version of the OS running.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Convert the LaTeX docs to Sphinx + Myst Markdown extension
I think this is in decent shape. It's going to need a lot of thorough
proofreading to ensure that thing got lost or mangled on the way from
the original docs. I suspect we will have several follow-up PRs to fix
thing.
After we're reasonably happy with it, we'll eventually remove the
.tex and the pdf.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
It gave a VERY old link to the SPI CLA, which we no longer use. In
fact, because we have CLA enforcement through the CLA bot, there is
not really a need to have it in the checklist... nobody can forget.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Add a GHA workflow that is triggered up on drafting a release, which
will echo a notice to the ASWF slack `#release-announcements` channel.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Based on Brecht's tests, we realize that OSL doesn't work at all with
Cuda 8.0 (which didn't support the C++14 that we use), so we're
raising the minimum to 9.0. And there are some flaky tests with 9 and
10, so we're making the recommended minimum be 11.0.
It's not our policy to change the minimum versions of dependencies on
release branches, but I think we should backport this to the release
branch. I believe what this change is doing is not removing actual
support for old versions, but rather, correctly documenting what has
been working (or not) for quite some time.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Co-authored-by: Brecht Van Lommel <brecht@blender.org>
* Bump minimum supported LLVM version to 17
* Rename `makeArrayRef` to `toArrayRef` to avoid `llvm::makeArrayRef`
deprecation warnings.
* Minor required changes related to `llvm::Align`.
* Disable `initializeInstrumentation` and `createPruneEHPass` which
were remove and appear to serve no purpose now.
* Add assert when using default optimization levels with legacy pass
manager, which is no longer supported.
This depends on both opaque pointers and new pass manager support.
Signed-off-by: Brecht Van Lommel <brecht@blender.org>
* Keep CMAKE_CXX_STANDARD warning for LLVM 16
* Enable LLVM 17 too, adjust for changes in new pass manager PR
* Fix OptiX data layout mismatch with LLVM 16+
i128:128 was already in the NVPTX backend since LLVM 6, the docs
this was taken from seem outdated. Unclear it started failing now,
but the data layout seems clearly incomplete without this.
* Modify CI to run the bleeding edge test on LLVM 16
* Manually make global variable names valid for PTX
* Fix build with older LLVM
---------
Signed-off-by: Brecht Van Lommel <brecht@blender.org>
Co-authored-by: Larry Gritz <lg@larrygritz.com>
Co-authored-by: Tim Grant <tgrant@nvidia.com>
Based on Brecht's tests, we realize that OSL doesn't work at all with
Cuda 8.0 (which didn't support the C++14 that we use), so we're
raising the minimum to 9.0. And there are some flaky tests with 9 and
10, so we're making the recommended minimum be 11.0.
It's not our policy to change the minimum versions of dependencies on
release branches, but I think we should backport this to the release
branch. I believe what this change is doing is not removing actual
support for old versions, but rather, correctly documenting what has
been working (or not) for quite some time.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Co-authored-by: Brecht Van Lommel <brecht@blender.org>
Use the absolute path to the Python executable to avoid picking up
a different executable with the same version.
Signed-off-by: Brecht Van Lommel <brecht@blender.org>
In #1668 USE_OPTIX changed to OSL_USE_OPTIX, but CI was not updated.
Also fix use of CPU SIMD flags for compiling CUDA bitcode
---------
Signed-off-by: Brecht Van Lommel <brecht@blender.org>
PR #1563 accidentally changed tests to accept failing commands by
default. (This seems to have been a mistake -- a temporary change
added when debugging that LG neglected to remove before submitting
the PR.)
This can cause tests to pass even though the program crashed, when
a previous run created the correct output file.
---------
Signed-off-by: Brecht Van Lommel <brecht@blender.org>
LLVM 16 removes support for default optimizations levels (O1, O2, O3)
from the legacy pass manager, which are used by OSL. This adds supports
for the new pass manager while keeping the legacy pass manager working
for older LLVM versions.
* Enable new pass manager for LLVM 15+
* Both default and custom optimization levels have been implemented
* But the performance and correctness beyond passing tests has not been
verified
* `ptx_compile_group` still uses the legacy pass manager
* Fix unecessary alias analysis and wrong #endif placement
* Fix LLVM 16 compatibility
* Ensure running prevent bitmasks pass is reentrant, and initialize once
* Make compiling with the new and legacy pass manager mutually exclusive
LLVM 17 removes even more legacy pass manager functions. With these
changes LLVM 17 works as well.
* OptiX fix, avoid using legacy pass manager for opt command since it does
not support default -O levels anymore
* Only enable new pass manager for LLVM 16+
---------
Signed-off-by: Brecht Van Lommel <brecht@blender.org>
* LLVM opaque pointers support
New LLVM versions require opaque pointers, that is you can no longer tell
from a pointer value which data type it points to.
* Remove deprecated GEP and op_load methods without type, and refactor
code to always pass the type.
* Add element type argument to op_scatter and op_gather.
* Eliminate pointer type comparison which no longer work with
opaque pointers.
* Keep interactive params marked as uniform in batched analysis, so
we can deduce uniform/varying correctly from just the symbol
* Enable opaque pointers on LLVM 15, except when using OptiX
OptiX opaque pointers were not tested yet and left disabled. It is
likely broken due to the sneaky conversion in
`BackendLLVM::llvm_store_value` between `int64` and `char*`, but we
can no longer compare the pointer type. This function is called from
many places and it's unclear to me when this happens.
According to a4e779c4 this code is only temporary, so this may resolve
itself as the ustring refactoring gets completed.
Signed-off-by: Brecht Van Lommel <brecht@blender.org>
* Fix debug_uninit and pointcloud batched testsß
Signed-off-by: Brecht Van Lommel <brecht@blender.org>
* Variable renaming and comment for clarity
Signed-off-by: Brecht Van Lommel <brecht@blender.org>
* Add/restore debug names for groupdata access
Make batched backend more consistent with the regular backend.
Signed-off-by: Brecht Van Lommel <brecht@blender.org>
* Rename scalar_type to component_type
Signed-off-by: Brecht Van Lommel <brecht@blender.org>
* Alternative interactive param fix based on feedback from Alex
Signed-off-by: Brecht Van Lommel <brecht@blender.org>
* OptiX fixes
Signed-off-by: Brecht Van Lommel <brecht@blender.org>
* Use opaque pointers only for LLVM 16+
To avoid potentially breaking LLVM 15 with OptiX.
Signed-off-by: Brecht Van Lommel <brecht@blender.org>
---------
Signed-off-by: Brecht Van Lommel <brecht@blender.org>
The unconditional dereferencing of the sg pointer is only necessary if
texture_thread_info is not supplied. And for that matter, the
texture_thread_info is only necessary if the texture_handle is not
supplied.
So rearranging the nesting skips a bunch of the work, and the need
for the caller to supply the sg if they have the texture_thread_info,
nor for them to need texture_thread_info if they already have the
handle.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Fixes#1716: Crash assigning boolean expression to int in batched mode
Modifying BatchedAnalysis::establish_symbols_forced_llvm_bool to excluded renderer outputs to fix the issue.
Added new test compassign-bool to recreate the issue and verify it is fixed.
Comprehensive Fix for Issue 1716: Crash assigning boolean expression to int in batched mode
Rearchitected BatchedAnalysis::establish_symbols_forced_llvm_bool to track a BoolStatus per Symbol evaluating all operations that write to the symbol. Only if all ops that write to it always produce a logically boolean result will the BoolStatus be set to Yes.
There is another set of ops that could produce a logically boolean result if their inputs were also boolean. In that case the BoolStatus is set to Maybe and its input symbols tracked (as they may not have a known BoolStatus yet) and deferred for a follow on pass.
After the initial pass is made, if there were any deferred Symbols, we can check their dependencies to see if they have a known BoolStatus and we can set the Symbol's BoolStatus to Yes or No depending on those input dependencies. If the BoolStatus of any dependencies is still Unknown, we just defer it again. We continue making passes over deferred Symbols until we have established a known BoolStatus for them all. Afterwards we use forced_llvm_bool(true) on the Symbols where BoolStatus is Yes.
Improved support for storing forced_llvm_bool Symbol values by automatically converting from int->bool and then a vector of bools (mask) to the platforms native mask type (might be vector of integers on platforms without native mask support).
Fixed bug in BatchedAnalysis where arrays of closures were not being marked as varying, only closures where.
Allow GroupData to hold actual boolean or native mask data type vs. integers when a Symbol is forced_llvm_bool.
Improved batched GroupData layout calculation to use llvm offset and alignment calculations based on the actual DataLayout (might consider upgrading the scalar code gen to do the same).
Fixed bug where results of partial results from RendererServices::get_userdata allowed initops to execute unmasked effectively overwriting the results from RendererServices::get_userdata. Issue stemmed from op_useparam not being modelled properly by BatchedAnalysis.
Added new test compassign-bool to recreate the issue and verify it is fixed.
Added new test closure-layered to recreate the issue and verify it is fixed.
Added new test userdata-partial recreate the issue and verify it is fixed.
Fixed bug in analysis of forced_llvm_bool for getattribute, whose return value is always a bool but result parameter when it is an integer should NOT be forced_llvm_bool.
Added test getattribute-shading to recreate the issue and verify its fixed.
Added testsuite/getattribute-shading/ref/out-scalar.txt to support builds that exclude batched.
---------
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
This bump brings access to the following interesting features (not a
comprehensive list):
- 3.13 Nov 21, 2018
- -B, -S let you specify source and build directories without the "cd" dance
- target_link_options()
- ctest --progress / CTEST_PROGRESS_OUTPUT
- 3.14 Mar 14, 2019
- cmake --build now supports -v/--verbose
- file(CREATE_LINK )
- 3.15 Jul 17, 2019
- env CMAKE_GENERATOR can be used instead of -G
- when using --build, the --target can now list multiple targets (and also can be `-t`)
- --install can be used separately without running through the whole build
- message() now supports NOTICE, VERBOSE, DEBUG and TRACE.
- CMAKE_FIND_PACKAGE_PREFER_CONFIG variable
- Enable C++20 for Apple Clang
Make sure all CI cases have a new enough cmake
Use message(VERBOSE ...) feature to simplify lots of `if` blocks.
Clean up code for older cmake versions
Freshen some of our CI build_<dependency>.bash scripts.
Note that this will NOT be backported to the current release branch,
but it will be part of the upcoming 1.13 family of releases.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We were overzealous in switching everything to ustringhash. The
filename argument to get_texture_handle should allow taking a ustring.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
On SPI Arnold, we are seeing GroupData as a bottleneck for GPU performance. Regardless of whether the buffer is backed by local or global memory, it's off-chip and slow to access.
This change moves qualifying parameters out of the GroupData struct and onto the stack. On GPU this makes those parameters more likely to be stored in registers, leading to better performance.
Qualifying parameter details
The qualifying parameters are all the output parameters minus three exceptions.
A typical output parameter's lifetime is scoped to the execution of the instance it is associated with. The parameter is initialized at the beginning of instance execution, accessed during the body of the instance, and after the instance body its final state is broadcast to all its downstream connections. This lifetime makes output parameters a good fit for stack storage.
The three exceptions:
- output parameters can be connected upstream and written to before instance execution
- output closures are initialized at the beginning of shader execution
- output parameters can be renderer outputs and read after instance execution
These are all identifiable at code generation time and can be flagged as disqualifying.
Improvements
On internal scenes, this change improves performance by about 15% on GPU. On CPU there aren't any performance regressions, but we don't see significant gains either. It also improves ptx generation and compilation by about 10%, and slightly lowers our memory footprint due to the GroupData size reduction.
Tests
Exceptions 2 & 3 are covered by existing tests. A new test is added for exception 1, and for the case of successful optimization. This patch was also run through our internal shading testsuite.
---------
Signed-off-by: Chris Hellmuth <chellmuth@gmail.com>
## Description
This PR is an overhaul of PTX compilation in OSL. The goal is to minimize the size of the generated PTX to help reduce time spent in subsequent stages (e.g., module creation, pipeline linking, etc.).
### rend_lib
The main functional change is in how the functions defined in rend_lib.cu are handled. In the current setup, rend_lib.cu is compiled to a standalone PTX file using nvcc. The functions are considered `extern` in generated shaders, and the dependencies are satisfied when the final pipeline is linked together. Since the functions are `extern`, they are not available for optimization when each shader is generated. This prevents the inlining of small functions, which incurs a lot of function call overhead in terms of PTX size and run-time cost.
So instead of compiling rend_lib.cu separately, it is compiled to LLVM bitcode along with the rest of the "shadeops" sources. This bitcode is used to seed the LLVM Module for each shader. This makes the definitions available when each shader is being generated and optimized, which allows the optimizer to make better decisions. It also allows for better control of inlining decisions which affect the size and quality of the generated PTX.
### shadeops
Another significant functional change is in how the shadeops functions (including those defined in rend_lib.cu) are treated. In the current pipeline, each shader must carry along with it definitions for all non-inlined shadeops functions that are used. This bloats the size of the PTX for each shader, and results in many duplicate copies of those functions existing in the final linked pipeline.
In the new pipeline, the unified shadeops+rend_lib bitcode is compiled to PTX using the offline `llc` tool. This PTX is used by the renderer to create a single "shadeops" module that provides the definitions for those functions. This makes it possible to drop the definitions for those functions from the PTX for each shader, which can significantly reduce the size of the generated PTX.
### quirks
This PR contains some interesting "quirks" which are largely fallout from compiling rend_lib.cu using clang instead of nvcc:
- The clang-generated PTX is processed by a Python script to transform some function & symbol declarations to better match what was generated by nvcc. It might be possible to add the right decorators to the C++/CUDA code to get the correct visibility for these symbols, but I was not successful in doing so.
- It was necessary to change the function signatures for some of the rend_lib functions to use `void*` instead of the actual OSL types (e.g., `OSL::Color3*`). Trying to use the actual types produces an LLVM assertion failure, for reasons that I don't understand.
- I have enabled flush-to-zero (FTZ) for clang-compiled code, to better match the behavior of nvcc. I added a CMake variable to enable or disable FTZ in the event that somebody wants to disable it.
- I have added some alignment hints on pointer accesses in rend_lib.cu. clang does not appear to be able to deduce the alignment of pointers in many cases, which turns things like `memcpy` and `memset` into long series of byte operations. Functions like `osl_get_matrix` are bloated substantially if the hints aren't provided.
---------
Signed-off-by: Tim Grant <tgrant@nvidia.com>
This is just for future-looking OSL 1.13 releases, it won't be
backported to the 1.12 release branch.
Remember that by the time we release OSL 1.13 as a supported release
family, OIIO will also have done its annual release, of 2.5. So our
minimum of OIIO 2.4 means that we will be supporting the current
release, as well as the prior year's release (and, of course, we
always stay compatible with OIIO's trunk as it evolves). So we're
really keeping compatibility with 3 major releases, which seems like
plenty to juggle.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Adds a new RendererServices API build_attribute_getter which allows for building custom free functions that provide values for attributes, effectively replacing both get_attribute and get_array_attribute.
This new API is invoked at material compilation time to allow render developers to provide specialized functions capable of taking advantage of known compile time information. This will allow for many optimization opportunities, such as replacing run-time map lookups with direct memory reads from known locations, remove brancing required for checking if derivatives are needed, branching on type-conversions if required, etc.
This PR currently implements this approach for attributes, a future PR could do the same for user-data.
Tests
TestShade provides an example implementation showing how this compile time information can be used to select an appropriate function to use as the attribute provider, and how to configure the function signature.
---------
Signed-off-by: Curtis Black <curtis.w.black@gmail.com>
* Add the missing thread_index and shade_index members to the copy of ShaderGlobals in testshade/cuda/rend_lib.h.
* Reorder the ShaderGlobals members in rend_lib.h to match the definition in shaderglobals.h.
---------
Signed-off-by: Tim Grant <tgrant@nvidia.com>
The string situation is still a bit tricky on the OptiX side. There's
a full overhaul of that coming separately as we continue to shift to
an all-ustringhash world.
But in the mean time, we were getting some testsuite failures that
were because of a mismatch between times we think of a gpu-side
ustring as a char* and when it's a uint64. IN FACT, they are always
the hashes, but we backed ourselves into a corner where sometimes that
64 bit pattern is passed around as a pointer, ugh.
This small patch attempts to catch this case and just do the
appropriate cast to compensate. It's a band-aid, a more comprehensive
fix is coming, as I said.
This gets us down to just ONE filing optix test case, which I will
tackle separately.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Certain fmt library versions don't automatically know how to format
atomics. Explicitly load them to turn into regular ints to avoid new
build errors introduced by the recent journaling changes.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Move from cryptically named ASWF/images to more descriptive ASWF/logos.
Add an icon only version of the logo.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Intel's servers have again updated the default icx that you get when
you install from their yum repository, and the new one seems
incompatible with our libstdc++ toolchain. So lock down on 2023.1.0
for now so that our CI can pass.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Move from cryptically named ASWF/images to more descriptive ASWF/logos.
Add an icon only version of the logo.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Remove leftover logic from when we used TravisCI.
* Pick a default for LLVM_DISTRO_NAME so it works when the script is run
standalone when not on GitHub Actions CI.
* Switch the logic about file location to enumerate only the couple of
old versions that uses the old download location, rather than trying
to enumerate an ever-growing set of modern releases that use the current
download location.
* Comment o say that when LLVM 10+ is the floor, the old case can go away
entirely.
Also bump the CI "bleeding edge" test to use LLVM 15.0.6.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The recently merged PR 1702 fails all the OptiX tests. I tracked down
the problem and it's that the patch was a bit too aggressive about
removing things no longer needed. The intent was that even if you
don't use the new journaling, nothing should break. But it does break
for OptiX, and this patch restores some of the functions 1702 removed
(like the implementation of osl_printf) and some other necessary
stubs.
This takes us from failing all ~150 OptiX oriented tests to failing
only 4, which I believe are unrelated to 1702 and are merely revealed
by them. I will fix those in a separate PR.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Journaling Algorithm for error, warning, fprintf
Removes dependence on ShadingContext and OIIO for logging error, warning, printf, and fprintf calls
for use on CPU and device, and tracks logging calls per thread index and shade index.
All calls to shader execution track thread index; ShaderGlobals contains shade_index (for processing
fmt specification calls) and thread_index for allocating memory per thread.
* Add new customization points in RendererServices for reporting errors, warnings, printf, and fprintf.
These customization points accept a fmtlib style format specifier (vs. printf style) whose argument types are represented as EncodedTypes (encodedtypes.h) and can be subsequently decoded via a utility function.
Removes dependence on ShadingContext and OSL::ErrorHandler for logging error, warning, printf, and fprintf calls
for use on CPU and device during Shader execution. ShadingContext and OSL::ErrorHandler remain utilized during shader compilation/JIT.
Journal Buffer approach: each thread gets its own chain of pages within the shared journal buffer for recording errors, warnings, etc. By maintaining a separate chain per thread we can record errors (etc.) in a non-blocking fashion. Shade index is recorded for use by error reporter to sort messages but it is not used at present.
Using a Journal buffer is up to a Renderer. The Renderer use the new customization points (rs_errorfmt, rs_filefmt, rs_printfmt, rs_warningfmt or their virtual function equivalents) to interact with a journal::Writer and journal buffer owned by its RenderState. The Renderer would of have to allocate and initialized the journal buffer with the requested page size and number of threads before executing a shader. After a shader is done executing, the contents of the journal buffer can be transfered back to the host and processed with a journal::Reader, which reports them through a journal::Reporter's virtual interface. Intent is for Renderers's to use the Journal buffer and provide their own overriden version of the journal::Reporter. Testshade provides example usage. For legacy purposes, the default virtual RendererServices will route errors to ShadingSystem's OSL::ErrorHandler, but intent is for Renderer's to switch over to using a Journal buffer or their own custom logging.
NOTE: OSL library functions as well use Renderer Service free functions can directly call OSL::filefmt, OSL::printfmt, OSL::errorfmt, OSL::warningfmt which wrappers to the underlying free functions (who may record the messages into a journal buffer).
Added a new hashcode datatype in typespec.cpp; represented as "h" in function arguments in builtindecl.
To print closures added a new OSL function osl_closure_to_ustringhash to return a ustringhash_pod value directly.
Shader execute functions now require a zero-based thread_index to identify the calling thread; used for journalling.
RendererServices::get_texture_info parameter ShadingContext * changed to ShaderGlobals * (aka ExecutionContext).
ShaderGlobals now tracks both shade_index and thread_index.
As we look to hide/remove ShaderGlobals in the future, we still need a context object to replace the role the ShaderGlobals struct is currently serving (just minus the actual shader global variables). As a first step towards this goal, we introduce the concept of an ExecutionContext and OpaqueExecutionContext. These just alias to ShaderGlobals currently, but start to establish the idea of hiding its contents. Instead of direct memory access to the ExecutionContext/ShaderGlobals new getter functions have been added that accept an OpaqueExecutionContext and provide the requested data. When inlined there should be no performance penalty, and reliance on the ShaderGlobals implementation detail will be reduced. IE:
inline const Vec3 & get_P(OpaqueExecContextPtr oec);
template<typename RenderStateT> RenderStateT* get_rs(OpaqueExecContextPtr oec);
inline int get_raytype(OpaqueExecContextPtr oec);
Updated free function renderer services and opcolor to use the ExecContext, but left the rest alone to minimize size of this PR.
We introduce opfmt.cpp that provides shim function for llvm_gen to call, where the shim adapts datatypes from llvm to c++ forwarding calls to the free function renderer service error, warning, print, fileprint functions.
Split llvm_gen_printf into llvm_gen_printf_legacy, for string format calls or when using optix, and the new llvm_gen_print_fmt which will convert the printf style specifier to the fmtlib format that the Renderer Service customization points are expecting.
fprintf by default no longer writes out to a file, is routed to journal::Reporter::report_file_print which could be overriden (example in TestShade to maintain existing behavior). However for security reasons, we do not recommend allowing OSL shader's direct file system but instead just an annotated log entry.
To maintain existing behavior of disallowing output of repeated errors/warnings over a history window, we have provided 2 classes that could be used during reporting: TrackRecentlyReported and Report2ErrorHandler. TrackRecentlyReported can turn on/off limitting errors and warnings and control the history capacity. Report2ErrorHandler simply reports errors to an existing OSL::ErrorHandler to allow for easy integration. However we expect Renderers to not use Report2ErrorHandler and provide their own to directly interact with their logging systems.
---------
Signed-off-by: Steena Monteiro <steena.monteiro@intel.com>
Our latest license scan from LF reveailed that the usual copyright and
license declaration was missing on many shader source files in
testsuite. This was never intentional, it was simply overlooked for
certain test files. Just add the standard notice to those files.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Like other oslcomp source files that are duplicated in oslexec, this
duplication should only be done for shared library builds where the
symbols are hidden from each other.
Signed-off-by: Brecht Van Lommel <brecht@blender.org>
Sometimes an interactive param is used in a way that gets
flagged as needing derivs. But we were neither allocating space
for that nor zeroing them. This resulted on memory corruption.
This fixes incorrect renders in a production context where params are
often tagged as needing derivs.
Signed-off-by: Alejandro Conty <aconty@imageworks.com>
OIIO's 2.3 family is obsolete and no longer updating, so we removed
the branch marker. (Branch markers should always indicate places where
development may happen.)
But we still reference that named branch in a couple places for OSL's CI,
so just replace it with the immutable tag of the last release in the 2.3
family.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
On gpu, the renderer currently owns a shader's groupdata params buffer so that it can pass its pointer to both the init and entry functions.
The downside of this approach is that in order to avoid dynamic memory allocation, the renderer must commit to a buffer size at build time. Using a conservative size to accommodate potentially large shaders can lead to gigabytes of unnecessary memory footprint.
This patch adds a new OptiX direct callable alternative to the existing init and entry callables. This new function allocates a perfectly-sized groupdata params buffer, so that the gpu only pays for the memory it needs, and then it calls init and entry.
Particularly large shaders can require larger buffers than there is space on the cuda stack. To handle this case, there is a new option, "max_optix_groupdata_alloc". Any shader requiring a groupdata buffer larger than this value will not allocate its own buffer, and instead use the pointer passed in by the renderer (presumably coming from a global memory pool).
Signed-off-by: Chris Hellmuth <chellmuth@gmail.com>
For GPU in particular, this prevents needless transfers to the device
when no values actually change.
Also, keep track of some statistics about how much ReParameter is
being used. (Which also help to verify that this change works as
intended.)
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Sometimes, after optimization, symbols tagged as interactive
get removed because they are unused. But ReParameter is failing
with errors because it goes looking for the removed symbol to
master, which is, of course, not interactive. This patch avoids
that so the symbol is not found and we just return false.
Signed-off-by: Alejandro Conty <aconty@imageworks.com>
Oops, I noticed that OSL was still using deprecated OIIO::simd type
aliases (float4, int4, bool4) instead of the modern vfloat4, vint4,
vbool4.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Tighten up some of the explanations and update obsolete parts of the
CONTRIBUTING document.
This doesn't change any of our policies or procedures. But some advice
was out of date or no longer applicable (for example, now that we use
clang-format, we do not need to detail all the formatting rules, they
are enforced automatically).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The Intel yum repo renamed this package a couple weeks ago, and it
broke our CI. Update to the new name.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The Intel yum repo renamed this package a couple weeks ago, and it
broke our CI. Update to the new name.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This change updates testrender's optix implementation of reparameter,
and adds a new test.
Previously we were grabbing the pointers from incorrect SBT records, so
this moves them into a global lookup table.
Signed-off-by: Chris Hellmuth <chellmuth@gmail.com>
BREAKING CHANGE: to RendererServices ABI (including for CPU) and to
the renderer-side setup when using OptiX.
This overhauls the implementation of how interactively-editable
parameters work, where they live in memory, and get it all working on
GPU/OptiX so that renderers can support interactive adjustment of
those params without recompiling shaders.
The basic gist is as follows:
* We continue work to finish making a clean separation between
"interpolated" parameters and "interactive" (editable) parameters.
* Interpolated params are collected and put into a separate memory
area -- a separate per-group allocation on both the CPU and GPU
(where applicable). It needs to remember the offset into this arena
where each of the interpolated parameters resides. These allocations
and eventual release are taken care of by the OSL shading system,
they live in the ShaderGroup. When the group is set up, this block
of memory is initialized with the correct initial values of the
params and are ready to go.
* The implementation of ReParameter writes to this special memory
area also, that's how it works now (both CPU and GPU).
* How does the OSL library know how to allocate, free, and copy to the
device memory? It doesn't! Instead, we add new RendererServices
methods `device_alloc()`, `device_free()`, and
`copy_to_device()`. It's up to the renderer to provide those, so
that the OSL library doesn't itself need to know about the Cuda
runtime. These are trivial, there's really only one implementation
that makes sense, and you can copy it from the ones in testshade and
testrender.
* Interactive parameters are NOT constant folded during runtime
optimization.
* The shader entry points themselves now take an extra parameter in
the main call -- this will be the pointer to the beginning of the
shader group's interactive parameter arena.
* When JITing, references to interactive parameters know to retrieve
them from their designated offset into the interactive parameter
area.
* This means that the renderer-side OptiX/Cuda code is responsible for
adding this extra pointer parameter to the call to the shader entry
points. You can see how this is done in the testshade and testrender
Cuda code.
* It's up to the renderer to figure out how to make the OptiX hit program
aware of the interactive parameter pointer for that particular material,
in order to pass it to the osl shader entry point. The way I did it in
testshade and testrender is using a field in the struct that's given
to each entry of the shader binding table and can be retrieved on the
OptiX side via optixGetSbtDataPointer(). In testshade/testrender, a
data pointer already existed which wasn't used. In a real renderer, you
may need to add a field or come up with whatever other way you want
to somehow get this pointer, which can be retrieved via
shadingsys->getattribute(shadergroupptr, "device_interactive_params",
TypeDesc::PTR, &myptr);
you can see how I do that in optixraytracer.cpp (testrender) and in
optixgridrender.cpp (testshade).
A number of other things you will see that's worth calling out:
I added a device_ptr utility template that is just a wrapper around a
device side pointer that makes it hard to accidentally dereference it
on the host side.
Since I was changing RendererServices anyway, I also remove unused
register_global, fetch_global, global_map which were unused. They were
leftovers from the way we handled strings in OptiX 6.x.
Encapsulate cuda global symbol name mangling into
BackendLLVM::global_unique_symname(). I did this early on, turns out
it wasn't necessary, but I still like that encapsulation, so I'm
keeping it.
I bumped the 3rd set of digits in the version to reflect that the
changes in RendererServices break ABI. This is only in main, it
obviously cannot be backported to a release branch.
All tests pass for scalar and batch and optix.
I added a new simple reparam test, and renamed the old reparem to
reparam-array. Oddly, the reparam-array test doesn't work properly on
optix (it had never been tried before), but it also failed in optix at
main -- so it's not related to this patch! Putting that on my list of
other oddities to investigate later. It may just be a quirk of
testshade, I'm not really sure yet.
Added to BackendLLVM (and batched) a llvm_ptr_type(TypeSpec) method
that returns the LLVM type of a pointer to the specified type.
Note: This patch doesn't account for the face that a parameter marked
"interactive" could prevent a shader from correctly building for the GPU
because it used the kind of construct that is fine in shader source code but
only will work on GPU if it can be resolved to be a constant by the time
we get done with the runtime optimization (as pointed out by Stephen
Friedman. We'll come back to the problem later with a more robust and
automatic fix -- and if we are lucky, Stephen will have the opportunity to
upstream the approach he already has.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Following #1657, it turns out we forgot to zero initialize the userdata derivatives for the optix path.
This leads to some rendering glitches caused by the memory garbage left in those derivatives.
Signed-off-by: Pascal Lecocq <pascal.lecocq@gmail.com>
Co-authored-by: Pascal Lecocq <plecocq@imageworks.com>
Oops, I noticed that OSL was still using deprecated OIIO::simd type
aliases (float4, int4, bool4) instead of the modern vfloat4, vint4,
vbool4.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
As Fabrice Macagno pointed out on the oiio-dev mail list, we weren't
setting the correct variable name CMAKE_CXX_VISIBILITY_PRESET.
And, oops, this also revealed that there were some things that needed
to be declared public that were not (because, specifically, they
needed to be called across library boundaries by the per-ISA dynamic
libraries used by batch shading!).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This was apparently something I put in to track down a possible
llvm-side problem many years ago. I haven't seen the error message in
recent memory (and it was only enabled at SPI anway), so I guess it
was a red herring or has long since been fixed. Get this cruft out of
here.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This issue was caused by 8c52d48 . The problem is that the constant float values are being turned into ints. The PR restores the previous behavior and preserves the float format.
Note from LG explaining:
When outputting the oso and using fmt library to write the value, {} (equivalent to {:g}) prints the "simplest" text representation that, when re-parsed as a float, will give you the SAME binary float value. So the "true" value of -9.99999996e+11 is printed as -1000000000000, which when reparsed will give you the closest float to that, which is the original value. (Remember that the value of -1e12 is not exactly representable as a float.)
But if both will parse to the same float, why does the shader print -2147483648.000000 instead of -999999995904.000000? That's the interesting part.
The answer, and the real core of the problem, is that when we parse the oso file it sees -1000000000000 and thinks it's an int, lexically. Now, once it gets to figuring out what to do with that, if the initial value of a float parameter looks like an int, it's still no problem, it will convert the int to the equivalent float.
Except that in this case, because the text that looked lexically like an int is out of range for a 32 bit int, it gets INT_MIN, so by the time it's converted back to float, it's -2147483648.000000
Your fix works (at least in this instance) because it forces the value to print in a way that lexically can only be a float and not an int.
Signed-off-by: Declan Russell <declan.russell@autodesk.com>
This issue was caused by 5efe15a. Essentially, we now use Python_EXECUTABLE to execute the serialize-bc.py script. Therefore, we have to run find_package(python) so that the Python_EXECUTABLE variable gets setup.
Signed-off-by: Declan Russell <declan.russell@autodesk.com>
This avoids some possible confusion with other projects, or if this
is built as a subproject.
Also, change from the C++ side having an OSL_USE_OPTIX defined -- only
for building OSL itself -- to having it always defined in oslconfig.h
and set to 0 or 1 depending on whether OptiX support was turned on at
build time. This allows app-side use of OSL headers to know if it's
dealing with an OptiX-capable OSL build.
Clean up OSL_USE_OPTIX and OSL_USE_BATCHED symbol defs
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This issue was caused by 8c52d48 . The problem is that the constant float values are being turned into ints. The PR restores the previous behavior and preserves the float format.
Note from LG explaining:
When outputting the oso and using fmt library to write the value, {} (equivalent to {:g}) prints the "simplest" text representation that, when re-parsed as a float, will give you the SAME binary float value. So the "true" value of -9.99999996e+11 is printed as -1000000000000, which when reparsed will give you the closest float to that, which is the original value. (Remember that the value of -1e12 is not exactly representable as a float.)
But if both will parse to the same float, why does the shader print -2147483648.000000 instead of -999999995904.000000? That's the interesting part.
The answer, and the real core of the problem, is that when we parse the oso file it sees -1000000000000 and thinks it's an int, lexically. Now, once it gets to figuring out what to do with that, if the initial value of a float parameter looks like an int, it's still no problem, it will convert the int to the equivalent float.
Except that in this case, because the text that looked lexically like an int is out of range for a 32 bit int, it gets INT_MIN, so by the time it's converted back to float, it's -2147483648.000000
Your fix works (at least in this instance) because it forces the value to print in a way that lexically can only be a float and not an int.
Signed-off-by: Declan Russell <declan.russell@autodesk.com>
This issue was caused by 5efe15a. Essentially, we now use Python_EXECUTABLE to execute the serialize-bc.py script. Therefore, we have to run find_package(python) so that the Python_EXECUTABLE variable gets setup.
Signed-off-by: Declan Russell <declan.russell@autodesk.com>
This includes two fixes:
1) When registering symbols that need user data, sort the entries in
the set so the layer number is ignored. A needed udata iteam shouldn't
depend on the layer and separating them makes find_userdata_index()
sometimes find an index with different derivs status.
2) osl_bind_interpolated_param() is memcpy'ing derivs that might not
be there, yielding corrupted derivs and possibly a crash.
Signed-off-by: Alejandro Conty <aconty@imageworks.com>
This includes two fixes:
1) When registering symbols that need user data, sort the entries in
the set so the layer number is ignored. A needed udata iteam shouldn't
depend on the layer and separating them makes find_userdata_index()
sometimes find an index with different derivs status.
2) osl_bind_interpolated_param() is memcpy'ing derivs that might not
be there, yielding corrupted derivs and possibly a crash.
Signed-off-by: Alejandro Conty <aconty@imageworks.com>
My initial implementation of the run layer check optimizations didn’t account for the fact that init ops for symbols have their llvm code generated before we build basic blocks from the OSL IR.
This caused the code to be wrong in two ways: 1) the set of known run layer calls was being reset between instances too late in the pipeline, so a useparam inside an init op would actually be checking the previously compiled layer’s set, and 2) even when reset earlier, it assumed valid basic block ids existed for the init ops when they did not.
So, if a layer had a useparam inside one of its init ops that generally matched the same code location as a useparam in the previously compiled layer, the run layer check could be incorrectly omitted.
I’ve updated the code so that the default optimization behavior is reverted to the way it was before #1665, and added a new option opt_useparam that can be turned on to enable a fixed, restricted version of the optimization that only applies to code in the main section.
I'm working on a test that properly triggers the issue.
Signed-off-by: Chris Hellmuth <chellmuth@gmail.com>
Take into account whether a parameter can change due to interactivity
when comparing parameters for "mergeability".
This proposed patch fixes a corner case we were seeing during live rendering with ReParameterization.
We were experiencing a problem when ss->ReParameterize() wasn't successfully applying parameter changes for us.
In our situation, a shader's parameter, "A", was marked with lockgeom=0 (ie. "live" or "editable") and was set via ss->Parameter(). This parameter was connected (through a series of copies and network links) to another shader's parameter, "B", downstream whose initialized value (ie. the one specified in the .osl/.oso file) matched A's set value. When this happened, A's shading layer was getting marked as "mergeable" and deleted, and we wouldn't see any updates to A via "reparameterization" take effect.
Signed-off-by: Clifford Stein <cstein@imageworks.com>
We can currently query which attribute names and scopes are requested in a shader. This PR extends that idea to also allow querying the attribute types. This now more closely matches the equivalent available queries for user data.
For example, this shader:
```
shader Shader
{
color c;
getattribute("user_color", c);
string s;
getattribute("user_string", s);
}
```
allows us to do this:
```
int nattr = 0;
if (shadingSys->getattribute(shaderRef.get(), "num_attributes_needed", nattr) && nattr)
{
// this already works
OIIO::ustring* names = nullptr;
shadingSys->getattribute(shaderRef.get(), "attributes_needed", OSL::TypeDesc::PTR, &names);
// this is the added feature
OIIO::TypeDesc* types = nullptr;
shadingSys->getattribute(shaderRef.get(), "attribute_types", OSL::TypeDesc::PTR, &types);
// nattr: 2
// names: ["user_color", "user_string"]
// types: [OIIO::TypeColor, OIIO::TypeString]
}
```
The motivation behind this feature is to perform additional type validation at material compile time when determining what primvars can be provided to the shader.
One side-effect of this change to be aware of is that, as mentioned in the code comments, if the same name is requested multiple times with different types, it will now be reported multiple times. Hopefully this is not too much of an issue as the same behaviour also occurs when an attribute is requested in multiple scopes, or when user data is requested with multiple types.
Signed-off-by: Curtis Black <curtis.w.black@gmail.com>
We can currently query which attribute names and scopes are requested in a shader. This PR extends that idea to also allow querying the attribute types. This now more closely matches the equivalent available queries for user data.
For example, this shader:
```
shader Shader
{
color c;
getattribute("user_color", c);
string s;
getattribute("user_string", s);
}
```
allows us to do this:
```
int nattr = 0;
if (shadingSys->getattribute(shaderRef.get(), "num_attributes_needed", nattr) && nattr)
{
// this already works
OIIO::ustring* names = nullptr;
shadingSys->getattribute(shaderRef.get(), "attributes_needed", OSL::TypeDesc::PTR, &names);
// this is the added feature
OIIO::TypeDesc* types = nullptr;
shadingSys->getattribute(shaderRef.get(), "attribute_types", OSL::TypeDesc::PTR, &types);
// nattr: 2
// names: ["user_color", "user_string"]
// types: [OIIO::TypeColor, OIIO::TypeString]
}
```
The motivation behind this feature is to perform additional type validation at material compile time when determining what primvars can be provided to the shader.
One side-effect of this change to be aware of is that, as mentioned in the code comments, if the same name is requested multiple times with different types, it will now be reported multiple times. Hopefully this is not too much of an issue as the same behaviour also occurs when an attribute is requested in multiple scopes, or when user data is requested with multiple types.
Signed-off-by: Curtis Black <curtis.w.black@gmail.com>
* Switch lockgeom to interpolated and interactive
Establish two new metadata hints about shader parameters:
- "interpolated" means that the parameter comes from the geometry
somehow and may be bound with a different value at different
points on the surfaces, or different values for different
primitives that share the material. This corresponds to concepts
known variously as "geometric primitive data/variables", "user
data", or "vertex variables", depending on the renderer.
- "interactive" means that the user desires the parameter to be
allowed to change when the scene is interactively rerendered,
without incurring a full recompile of the shader. (Variables not
so marked might still be capable of being changed, but not
necessarily without a re-JIT of the shaders.)
Both of these require that the parameter not be fully constant-folded
or optimized away during runtime optimizeation or JIT, and thus
supplant the old "lockgeom=0" hint. But their semantics may differ in
other ways, so we thought it was worth introducing a separation, and
at the same time, finally fixing the mis-design of "lockgeom" (which
in retrospect is both a confusing name and also has the 0|1 sense
backwards, thwarting even my own intuition every time I use it).
We introduce new variants of `ShadingSystem::Parameter()` that changes
the optional `bool lockgeom` parameter for a new enum `ParamHints`
that allows for future expansion with more hinted properties. For
now, the old calls still work (lockgeom=false is the same as passing
ParamHints::interpolated) and this does not introduce any break to
backwards ABI compatibility at this time.
In this PR, "interpolated" is basically the replacement for the old
lockgeom (though the sense is reversed). The "interactive" hint is put
in place but doesn't do anything especially useful at the moment that
is different from interpolated. A few more things are coming down the
pike in subsequent PRs: (1) Make interactive work and fully hook it up
to ReParameter(), including on GPU. (2) Make a new global SS option
that says if you're building shaders for an interactive render or an
offline render. (If offline, the even "interactive" hinted params
should be constant-folded.)
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This patch is a very simple part one of work to optimize the number of conditional run layer calls that OSL makes, which take the form:
if (!groupdata->run[parentlayer])
parent_layer([...]);
After examining our slowest JIT’ing shaders on gpu, @tgrant-nv showed us that these run layer calls were significantly bloating our ptx and slowing down their compilation. Many of them are unnecessary because static analysis can show we are guaranteed to have already run the layer in question.
For example, reading different fields from a connected struct like below will often generate multiple conditional run layer calls:
```
struct Pair {
float x;
float y;
};
shader example_shader(Pair p = {})
{
if ([…]) {
float xx = p.x;
float yy = p.y;
[…]
}
}
```
Since this is not really a performance issue on cpu (runtime overhead is just checking the conditional bit), OSL doesn’t make a huge effort to optimize away these redundant cases. Each useparam op will be careful not to insert more than one per layer, and code that is called unconditionally in the main section of the shader will also not have redundant checks, but there are plenty of examples like above that are not currently removed.
This patch takes a basic first step, adding metrics to track run layer call counts and removing duplicate run layer calls inside a basic block, covering cases like the example above.
On my set of production test scenes, this alone removes anywhere from 20-50% of total run layer calls, with our biggest shaders seeing the most improvement. Total ptx line reduction varies between 3-20%, and optix module compilation (jit time) is reduced between 1-15%.
Signed-off-by: Chris Hellmuth <chellmuth@gmail.com>
But note: OSL does not yet work with LLVM 16.
Because homebrew recently upgraded its default llvm to 16.0, there may
be users for whom builds now break. The solution is:
brew install llvm@14
cmake ... -DLLVM_ROOT=/usr/local/opt/llvm@14
Signed-off-by: Larry Gritz <lg@larrygritz.com>
But note: OSL does not yet work with LLVM 16.
Because homebrew recently upgraded its default llvm to 16.0, there may
be users for whom builds now break. The solution is:
brew install llvm@14
cmake ... -DLLVM_ROOT=/usr/local/opt/llvm@14
Signed-off-by: Larry Gritz <lg@larrygritz.com>
LLVM 16 was just released. I'm pretty sure we won't build properly
against it because of the phase-out of typed pointers in LLVM IR, so
warn prominently so that anybody building against it who has problems
will know why.
Also, LLVM itself has bumped their toolchain requirements with the
release of 16. Technically, I'm not sure if that extends to projects
using the LLVM APIs, but warn anyway and hope for the best. If it
fails, at least the warning will supply an explanation.
At a later time, we'll need to upgrade the code on our side to have the
option of using opaque pointers so we can run with LLVM 16+.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
For LLVM 15+, we need to disable opaque pointers. We did this for
regular bitcode generation in llvm_macros.cmake, but neglected to make
the corresponding change for copmiling to ptx in cuda_macros.cmake.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
For LLVM 15+, we need to disable opaque pointers. We did this for
regular bitcode generation in llvm_macros.cmake, but neglected to make
the corresponding change for copmiling to ptx in cuda_macros.cmake.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
LLVM 16 was just released. I'm pretty sure we won't build properly
against it because of the phase-out of typed pointers in LLVM IR, so
warn prominently so that anybody building against it who has problems
will know why.
Also, LLVM itself has bumped their toolchain requirements with the
release of 16. Technically, I'm not sure if that extends to projects
using the LLVM APIs, but warn anyway and hope for the best. If it
fails, at least the warning will supply an explanation.
At a later time, we'll need to upgrade the code on our side to have the
option of using opaque pointers so we can run with LLVM 16+.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
When generating calls to osl_bind_interpolated_param in llvm_instance.cpp, the userdata_has_derivs field is set to true if the parameter's derivatives are needed, but the code does not take into account that on gpu there is no space allocated for these derivatives.
This leads to an incorrect derivatives pointer getting passed into the renderer, which the renderer will use to write into unrelated groupdata fields.
For our renders, getting the correct image was worth the 1-2% inflation of groupdata struct size that we observed by simply re-enabling gpu userdata derivatives. If desired, adding an option to turn them on/off is straightforward, but I didn't want to preemptively bloat the code if it's not necessary.
Signed-off-by: Chris Hellmuth <chellmuth@gmail.com>
In particular, `${LLVM_ROOT}/bin` will end up being (incorrectly!)
`/bin` if LLVM_ROOT is empty. That is bad if some other llvm-config
happens to be in /bin.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
In particular, `${LLVM_ROOT}/bin` will end up being (incorrectly!)
`/bin` if LLVM_ROOT is empty. That is bad if some other llvm-config
happens to be in /bin.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Stop using the deprecated, soon to be removed, ubuntu-18.04 runners
and switch to 20.04.
* Bump the "latest" test openexr to 3.1.6
* Update INSTALL.md to document some things we're already testing and
are known to work.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We had turned it off in one function to preserve formatting, but
failed to turn it on at the end of the function. This led to several
small functions not being clang-formatted, now fixed.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The runtime optimization does a great job of constant folding, and
we'd pushed it there because, with instance parameter values and
shader network connections known, it can find so much more opportunity
to optimize than we could in oslc.
However, there is one pesky problem, which is that shader parameters
that are initialized to even simple expressions such as 3+1 end up
with "init ops", which although later constant folded by the time the
shader is JITed, make it impossible to know the value via OSLQuery.
So this patch just takes the simplest cases -- certain `int OP int`
and `float OP float` expressions involving literal constants, and
performs the operation as it's parsing the code. So
int val = 3 + 1;
actually just immediately is turned into `val = 4` instead of making
an "add".
To reiterate, the add would never have happened while executing the
shader -- at runtime in the renderer, when it's time to optimize and
JIT the shader, it would know it's a constant 4 value. This is
strictly about making oslc directly output an .oso file that knows
that parameter `val` has default value 4 instead of throwing up its
hands and saying "it's math code that will be evaluated later."
Signed-off-by: Larry Gritz <lg@larrygritz.com>
In particular, 1 prints a short version of the compile report (just
the fact that it happened and the time), >1 prints more verbose info.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Stop using the deprecated, soon to be removed, ubuntu-18.04 runners
and switch to 20.04.
* Bump the "latest" test openexr to 3.1.6
* Update INSTALL.md to document some things we're already testing and
are known to work.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We had turned it off in one function to preserve formatting, but
failed to turn it on at the end of the function. This led to several
small functions not being clang-formatted, now fixed.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* testrender: Don't use the cached background map on the first bounce (camera ray)
* testrender: Change the default resolution of the background so that importance sampling is enabled by default
Signed-off-by: Chris Kulla <ckulla@gmail.com>
* testrender: Don't use the cached background map on the first bounce (camera ray)
* testrender: Change the default resolution of the background so that importance sampling is enabled by default
Signed-off-by: Chris Kulla <ckulla@gmail.com>
Apparently some varieties of fmod(triple,float) -- which were not
documented in the spec but were intended to work and did allow that
combination in oslc -- were not fully implemented in the runtime.
* Add all these tests to the arithmetic-reg test, to fully test all
possible argument type and unif/vary/const combinations once and for
all for fmod. Also add some more rudimentary testing to miscmath.
* Implement the missing bits and/or avoid the code paths that rely
on the unimplemented parts.
The implementation of fmod for funny -- both for batch and non-batch,
it's got one code path that just directly generates code and doesn't
need any functional implementation in llvm_ops at all, and a second
code path that uses the functional implementations to work around
(different, in each case!) what appears to be bugs or limitations in
long-ago LLVM versions we don't need any more. I'm trying to assess
whether those are still needed at all in the oldest versions of LLVM
we are using, and if not, I may amend even more simplification to this
whole thing.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Try to set ray types correctly for rays -- a little naive now, only
using "camera", "shadow", and "diffuse" for all rays (leave
distinguishing between reflection, refraction, diffuse, glossy for
another day, and currently testrender doesn't support subsurface or
displacement, but maybe someday).
Also, for testshade, improve efficency -- don't decode raytype on
every point. For each individual shade, testshade was converting the
name of the raytype from a string to a ustring, then decoding to a
bitfield. Pull it all out of the loops by computing the bitfield once.
Adjust glass shaders to not change behavior based on raytype:
The code path that was supposed to suppress noisy caustic rays never
worked properly before because we never set the ray type in
testrender. Now that testrender sets it -- but does not distinguish
between the different types of secondary rays -- the code path is
active, but incorrect, and causes tests to fail. Just remove the
offending clause for now and return to the old behavior of the shader
(no attempt to eliminate caustic rays). Later, we should revamp the
glass shader entirely by switching to the newer MaterialX-inspired
closures.
Note: when you specify a resolution in the Background command in our
scene xml, it will run the shader out of any ray context at all (therefore
having no raytype) in order to populate an environment map that will be
used as needed. If you omit the resolution, no importance sampling is
used at all, so it's much noisier. I think the right approach is to do both --
populate the map for importance sampling, but run the shader on the rays
themselves to get higher quality visible results.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Apparently some varieties of fmod(triple,float) -- which were not
documented in the spec but were intended to work and did allow that
combination in oslc -- were not fully implemented in the runtime.
* Add all these tests to the arithmetic-reg test, to fully test all
possible argument type and unif/vary/const combinations once and for
all for fmod. Also add some more rudimentary testing to miscmath.
* Implement the missing bits and/or avoid the code paths that rely
on the unimplemented parts.
The implementation of fmod for funny -- both for batch and non-batch,
it's got one code path that just directly generates code and doesn't
need any functional implementation in llvm_ops at all, and a second
code path that uses the functional implementations to work around
(different, in each case!) what appears to be bugs or limitations in
long-ago LLVM versions we don't need any more. I'm trying to assess
whether those are still needed at all in the oldest versions of LLVM
we are using, and if not, I may amend even more simplification to this
whole thing.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Try to set ray types correctly for rays -- a little naive now, only
using "camera", "shadow", and "diffuse" for all rays (leave
distinguishing between reflection, refraction, diffuse, glossy for
another day, and currently testrender doesn't support subsurface or
displacement, but maybe someday).
Also, for testshade, improve efficency -- don't decode raytype on
every point. For each individual shade, testshade was converting the
name of the raytype from a string to a ustring, then decoding to a
bitfield. Pull it all out of the loops by computing the bitfield once.
Adjust glass shaders to not change behavior based on raytype:
The code path that was supposed to suppress noisy caustic rays never
worked properly before because we never set the ray type in
testrender. Now that testrender sets it -- but does not distinguish
between the different types of secondary rays -- the code path is
active, but incorrect, and causes tests to fail. Just remove the
offending clause for now and return to the old behavior of the shader
(no attempt to eliminate caustic rays). Later, we should revamp the
glass shader entirely by switching to the newer MaterialX-inspired
closures.
Note: when you specify a resolution in the Background command in our
scene xml, it will run the shader out of any ray context at all (therefore
having no raytype) in order to populate an environment map that will be
used as needed. If you omit the resolution, no importance sampling is
used at all, so it's much noisier. I think the right approach is to do both --
populate the map for importance sampling, but run the shader on the rays
themselves to get higher quality visible results.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We were getting an exception thrown on the OCIO side, because we were
passing 0 for the y stride (which, us humans know, should have been fine
since there was only one row, so the stride shouldn't have mattered).
Now pass the right stride even though we don't need to step from row
to row.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Make sure LLVM_ROOT/bin (either as a cmake var or as an env var) end
up in the hints list when searching for llvm-config.
Without this, one of our Mac CI jobs was ending up with clang 14 and
llvm 15, and that was causing problems. The particular test was
supposed to be using 14 for both.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We were getting an exception thrown on the OCIO side, because we were
passing 0 for the y stride (which, us humans know, should have been fine
since there was only one row, so the stride shouldn't have mattered).
Now pass the right stride even though we don't need to step from row
to row.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Make sure LLVM_ROOT/bin (either as a cmake var or as an env var) end
up in the hints list when searching for llvm-config.
Without this, one of our Mac CI jobs was ending up with clang 14 and
llvm 15, and that was causing problems. The particular test was
supposed to be using 14 for both.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This makes some minor changes to the RS APIs to add an additional
optional parameter to get_texture_handle so it takes a `TextureOpt*`
to accommodate a similar change on the OIIO TextureSystem side
from https://github.com/OpenImageIO/oiio/pull/3761
The purpose here is to add color management options to texture calls.
However, please note that this PR only reserves a public API change we
know we'll want, but it does not fully implement the color management.
Further work will be needed, as follows:
* Current calls to RS::get_texture_handle should be updated to pass
the TextureOpt*. (This is a formality for future-proofing in case
future TS or alternate TS implementations want to bake the color
space choice into the texture handle itself. But the OIIO change
doesn't currently rely on that at the moment.)
* When the texture.h symbol `OIIO_TEXTURESYSTEM_SUPPORTS_COLORSPACE`
exists, we are building against an OIIO that has the new int field
`colortransformid` in the TextureOpt and TextureOptBatch structures.
We should fill this in with a nonzero value when we want the texture
call to be doing a color transformation as it reads texture tiles.
* Use the new TextureSystem:get_colortransform_id() to get that color
transform ID given two named from/to color spaces. We agreed that
it's a reasonable restriction that the color space names should be
known by the end of runtime optimization, so that should be constant
foldable.
* Recognize new optional argument(s) to the texture() call family that
lets the caller specify the presumed color space of the texture,
and use that value (with constant folded lookup of the ID, above)
to set the value in the TextureOpt.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Ported from OIIO. This lets us not have to depend on the specific
definition in OIIO's fmath.h, and in particular not worry about
whether it's a new enough version of OIIO that the old bit_cast is now
marked as deprecated.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Fixed batched support for closures with array types.
Enabled testsuite/userdata-custom/BATCHED (follow up to PR1620/PR1621).
Added modified version of PR1625 (Fix Issues with Derivative of getattribute) from danieldresser-ie
changing default behavior of MaskedData<DataT>::assign_all_from_scalar(const void *) to zero out Dx Dy when DataT has derviatives.
Added testsuite/userdata-custom
Fixed bug in testshades handling of userdata with derivatives.
Added bool LLVM_Util::is_type_array(llvm::Type* type)
llvm::Type* LLVM_Util::element_type_of(llvm::Type* array_type)
Fixed icx compiler warning with src/liboslexec/osogram.y
Fixed unreferenced function warning in src/testrender/shading.cpp
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
This is another fix I'm attempting without really understanding all this LLVM stuff.
I started by adding a simple test case that attempts to add two closure parameters, where one has not been set. This crashes or returns random values from uninitialized memory, matching a problem I ran into while trying to get batched mode working in Gaffer.
I investigated some, and found that add_closure_closure is assuming closure pointers are stored with the width of the current batch, but build_llvm_init is creating uniform closures. I checked with @sfriedmapixar, who confirmed that all the batched closure code is assuming wide closures, so it seems build_llvm_init is what needs fixing.
Adding a `sym.make_varying();` to build_llvm_init appears to fix the problem ... I really don't know enough about the overall structure to know if this is the right place to make this fix, or if it could have other consequences, but it's enough to get this test passing, so hopefully someone who does know the background can review it.
Signed-off-by: Daniel Dresser <danield@image-engine.com>
When evaluating varying closure keyword parameters in batched execution,
the value of the first lane in the batch would be used for the whole
batch.
Signed-off-by: Daniel Dresser <danield@image-engine.com>
It seems something in the latest builds of the aswftesting/ci-osl:2022
containers has changed, and explicitly running /usr/bin/yum (instead of
the broken /usr/local/bin/yum) fixes it for now. This was completely
breaking the icc & icx tests, which require yum to install the Intel
compilers.
Also, bump up to the -clang14 containers for those tests, just to
get to a more recent LLVM.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
To save build time, we had OSL's Mac CI tests use the homebrew
installation of openimageio. But... Homebrew recently switched its
default python to 3.11, which didn't match the versions we're testing
OSL against here and broke the python bindings, so need to make a few
adjustments.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
It seems something in the latest builds of the aswftesting/ci-osl:2022
containers has changed, and explicitly running /usr/bin/yum (instead of
the broken /usr/local/bin/yum) fixes it for now. This was completely
breaking the icc & icx tests, which require yum to install the Intel
compilers.
Also, bump up to the -clang14 containers for those tests, just to
get to a more recent LLVM.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Fixed batched support for closures with array types.
Enabled testsuite/userdata-custom/BATCHED (follow up to PR1620/PR1621).
Added modified version of PR1625 (Fix Issues with Derivative of getattribute) from danieldresser-ie
changing default behavior of MaskedData<DataT>::assign_all_from_scalar(const void *) to zero out Dx Dy when DataT has derviatives.
Added testsuite/userdata-custom
Fixed bug in testshades handling of userdata with derivatives.
Added bool LLVM_Util::is_type_array(llvm::Type* type)
llvm::Type* LLVM_Util::element_type_of(llvm::Type* array_type)
Fixed icx compiler warning with src/liboslexec/osogram.y
Fixed unreferenced function warning in src/testrender/shading.cpp
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
To save build time, we had OSL's Mac CI tests use the homebrew
installation of openimageio. But... Homebrew recently switched its
default python to 3.11, which didn't match the versions we're testing
OSL against here and broke the python bindings, so need to make a few
adjustments.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Related to https://github.com/OpenImageIO/oiio/pull/3653
Kai Pastor (dg0yt) points out that the recent change I made both in
OIIO and OSL to make the version a cache variable has some unexpected
side effects that can lead to unexpeted behavior by inadvertently
retaining a version number upon recompiles if the cache itself was not
cleared, and this can happen unexpectedly even when the user isn't
trying to override the version.
This patch changes the behavior as follows:
You must use the special `-DOSL_VERSION_OVERRIDE=...` to override the
version.
* if you do so, you are playing with fire and get what you deserve
if you misuse it or change branches and don't remember to clear
the cache.
If you do a normal build (without trying to override the version),
* you will get the real version number because it IS NOT cached
* if you switch branches, you'll always get the right version number
* ...but switching branches without clearing the cache can still, as
always, have other subtle behavior differences versus doing a
truly fresh build from scratch.
So in short, it still allows a way to override the version if you
really want to, but if you are not trying to do that, you can no
longer get yourself inadvertently into unexpected trouble.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Related to https://github.com/OpenImageIO/oiio/pull/3653
Kai Pastor (dg0yt) points out that the recent change I made both in
OIIO and OSL to make the version a cache variable has some unexpected
side effects that can lead to unexpeted behavior by inadvertently
retaining a version number upon recompiles if the cache itself was not
cleared, and this can happen unexpectedly even when the user isn't
trying to override the version.
This patch changes the behavior as follows:
You must use the special `-DOSL_VERSION_OVERRIDE=...` to override the
version.
* if you do so, you are playing with fire and get what you deserve
if you misuse it or change branches and don't remember to clear
the cache.
If you do a normal build (without trying to override the version),
* you will get the real version number because it IS NOT cached
* if you switch branches, you'll always get the right version number
* ...but switching branches without clearing the cache can still, as
always, have other subtle behavior differences versus doing a
truly fresh build from scratch.
So in short, it still allows a way to override the version if you
really want to, but if you are not trying to do that, you can no
longer get yourself inadvertently into unexpected trouble.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Lately, it has been experiencing intermittent availability problems
that causes our CI to fail while setting up dependencies.
What happens if we don't use the problematic ubuntu-toolchain-r at all?
Nothing, it seems. So eliminate it.
I think that once upon a time, it was necessary to add this repo to
get certain new packages? But maybe they are mainstream enough now
that it's not necessary to add this repo? In any case, it doesn't seem
to hurt anything, it's one less moving part that can't fail if it's
not there.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This is another fix I'm attempting without really understanding all this LLVM stuff.
I started by adding a simple test case that attempts to add two closure parameters, where one has not been set. This crashes or returns random values from uninitialized memory, matching a problem I ran into while trying to get batched mode working in Gaffer.
I investigated some, and found that add_closure_closure is assuming closure pointers are stored with the width of the current batch, but build_llvm_init is creating uniform closures. I checked with @sfriedmapixar, who confirmed that all the batched closure code is assuming wide closures, so it seems build_llvm_init is what needs fixing.
Adding a `sym.make_varying();` to build_llvm_init appears to fix the problem ... I really don't know enough about the overall structure to know if this is the right place to make this fix, or if it could have other consequences, but it's enough to get this test passing, so hopefully someone who does know the background can review it.
Signed-off-by: Daniel Dresser <danield@image-engine.com>
Lately, it has been experiencing intermittent availability problems
that causes our CI to fail while setting up dependencies.
What happens if we don't use the problematic ubuntu-toolchain-r at all?
Nothing, it seems. So eliminate it.
I think that once upon a time, it was necessary to add this repo to
get certain new packages? But maybe they are mainstream enough now
that it's not necessary to add this repo? In any case, it doesn't seem
to hurt anything, it's one less moving part that can't fail if it's
not there.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
When evaluating varying closure keyword parameters in batched execution,
the value of the first lane in the batch would be used for the whole
batch.
Signed-off-by: Daniel Dresser <danield@image-engine.com>
Changes:
* Scorecards and analysis -- allow new to cancel running old
* Scorecard only runs on main repo, no need to run on forks
* Nightly CI only runs on main repo, no need to run on forks
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Reminder: phase 1 changed the RendererServices APIs to be based
completely on ustringhash for public method calls, not ustring.
This PR starts along the path of phase 2.
In oslconfig.h, I define a new type `ustringrep` which is the
in-memory/in-shader CPU-side representation of a ustring. Currently,
it's still equivalent to a ustring.
I also define a type called `ustring_pod`, which is the "plain old
data" representation inside a ustringrep (i.e., a const char* when
it's a ustring, and a uint64_t when it's a ustringhash). For reasons
not important, it's exceedingly hard to pass a true
ustring/ustringhash as parameters to or from LLVM-jitted code without
running afoul of LLVM's type checking, so it's just easier to pass the
underlying data type.
So the main work of phase 2 is to change everything related to shader
data from a ustring into a ustringrep, and things passed as const
char* into passing a ustring_pod. In some sense, it's just renaming
and type conversion plumbing. When that's done, phase 3 is to throw
the switch and change the definition of ustringrep to ustringhash (and
of course fix everything that's broken).
This PR does not complete phase 2, but takes a big bite out of it.
It's an arbitrary bite, not much rhyme or reason to which specific
parts I did so far. Just working on it piece by piece and felt that
this was a place where much has been done, everything still works, and
it's a reviewable chunk without getting too much out of control.
Areas I worked on are related to range_check, raytype, getattribute,
bind_interpolated_param, uninit_check, range_check, naninf_check,
getmessage/setmessage, some texture setup, pointcloud.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Changes:
* Scorecards and analysis -- allow new to cancel running old
* Scorecard only runs on main repo, no need to run on forks
* Nightly CI only runs on main repo, no need to run on forks
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This seems to have shifted just enough things around to make
clang-format want to re-jigger a few things.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Wow, this has been lurking for a while. Our Findpartio.cmake tries
looking for a partio exported config, but if not found, falls back
to a simple code to find it manually.
All was fine with the fallback code, which if found would set both
"partio_FOUND" and "PARTIO_FOUND". But the config file, if found --
only for the newer partio versions -- only set partio_FOUND. And in
that circumstance, we were not seting up the pointcloud tests!
Although the partio tests were running in CI (thankfully), I was not
running them locally on my machine with a new-ish partio.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Wow, this has been lurking for a while. Our Findpartio.cmake tries
looking for a partio exported config, but if not found, falls back
to a simple code to find it manually.
All was fine with the fallback code, which if found would set both
"partio_FOUND" and "PARTIO_FOUND". But the config file, if found --
only for the newer partio versions -- only set partio_FOUND. And in
that circumstance, we were not seting up the pointcloud tests!
Although the partio tests were running in CI (thankfully), I was not
running them locally on my machine with a new-ish partio.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Address -Winconsistent-missing-override clang warning
* Fix ptx definitions of OSL_CONSTANT_DATA arrays
LLVM 14 introduced two ptx regressions to our file-scoped static variables:
1) They gained the .visible directive
2) They stopped being optimized out when unused
This lead to duplicate symbol errors when building an OptiX pipeline with
multiple shaders.
We attempted a fix using anonymous namespaces in #1561, but kept the static
keyword, resulting in a partial solution.
Removing static generates ptx that matches LLVM 12 for the affected variable
definitions.
Signed-off-by: Chris Hellmuth <chellmuth@gmail.com>
Looking at the code coverage report and trying to get a higher
coverage amount, employing two strategies: (a) remove unused code
that is not covered because it's... unused; and (b) test things
that are used but aren't in the test run. Details follow.
Disable contents of dump_struct_data_layout when OSL_DEV is not
defined, it will never be called in any other circumstances -- it's
strictly a debugging tool, not part of the shipping library.
When OSL_USE_OPTIX is not enabled, exclude compiling contents of
ptx_compile_group.
get_alignment can also be OSL_DEV only as well as simplified
Arbitrarily pick a couple of render tests to use llvm_opt levels 12
and 13 to verify that they work correctly. (That's a couple hundred
lines of pass setup code that wouldn't otherwise be included in the
coverage analysis, nor tested in the testsuite.) This also required us
to add the --llvm_opt command to testrender, as it is in testshade.
Make the analysis run build for avx512 as well for more code coverage.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The first step of the great ustringhash conversion: This changes all
RendererServices methods (i.e., callbacks to the renderer from the
shader JITed code) that used to take ustring parameters, switched to
take ustringhash instead.
The second step -- which I will tackle after this part is merged --
will be to fully switch the in-memory representation of strings during
shading execution to ustringhash on the CPU (thus matching how we've
been doing it on GPU and making a number of data stuctures and
representations identical/shared for CPU and GPU).
But in the mean time, this lets renderers get started changing their
RendererServices specializations. Just like with OSL's internal
implementation, it may be easier for renderers to break the job into
phase I (rendererservices) and phase II (in-memory rep).
Some notes and guideposts:
* In oslconfig.h, define ustringrep to alias to either ustring or
ustringhash, depending on a new (but temporary) CMake variable
`OSL_USTRINGREP_IS_HASH`, which is OFF for now, meaning that
ustringrep is still ustring.
* Also in oslconfig.h, several helper conversion functions among
ustring, ustringhash, ustringrip.
* RendererServices and BatchedRendererServices (and their various
subclasses in testshade and testrender): change the method
signatures from ustring to ustringhash for all methods that are
reachable from shader code.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Looking at the code coverage report and trying to get a higher
coverage amount, employing two strategies: (a) remove unused code
that is not covered because it's... unused; and (b) test things
that are used but aren't in the test run. Details follow.
Remove deprecated version of LLVM_Util::make_function.
Disable contents of dump_struct_data_layout when OSL_DEV is not
defined, it will never be called in any other circumstances -- it's
strictly a debugging tool, not part of the shipping library.
When OSL_USE_OPTIX is not enabled, exclude compiling contents of
ptx_compile_group.
get_alignment can also be OSL_DEV only as well as simplified
Arbitrarily pick a couple of render tests to use llvm_opt levels 12
and 13 to verify that they work correctly. (That's a couple hundred
lines of pass setup code that wouldn't otherwise be included in the
coverage analysis, nor tested in the testsuite.) This also required us
to add the --llvm_opt command to testrender, as it is in testshade.
Make the analysis run build for avx512 as well for more code coverage.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The immintrin.h header is required for the implementation of the
popcount and coutr_zero function only when the Intel compiler is used.
Move the inclusion accordingly. The current state breaks build on
platforms that don't provide the header, but provide the alternative
means (gcc/clang built-ins, etc).
Signed-off-by: Dan Horák <dan@danny.cz>
The immintrin.h header is required for the implementation of the
popcount and coutr_zero function only when the Intel compiler is used.
Move the inclusion accordingly. The current state breaks build on
platforms that don't provide the header, but provide the alternative
means (gcc/clang built-ins, etc).
Signed-off-by: Dan Horák <dan@danny.cz>
We compile testshade.cpp twice -- one for a special "does it compile
as a DSO" check but that isn't run from the testsuite. This was
causing the coverage analysis to think nothing in testshade.cpp (at
least one copy of it) was getting run, which is technically true, and
counting all of its lines as untested for the coverage analysis. Just
exclude that funny dso build from the coverage case.
Also exclude anything in src/build-scripts from coverage analysis,
that's not code that ships, so to speak.
We're trying to chip away at untested code bit by bit to get our
coverage to over 80% of lines. We're getting close.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Define std::hash for ustring and ustringhash, when not using an OIIO
new enough to do so.
Define std::less for ustringhash, when not using an OIIO new enough
to have operator< defined for ustringhash (will be fixed separately).
No need now for ustringHash, which is also unwisely similar to
ustringhash, just use std::hash instead.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Reduce dependency on ShadingContext and ShadingSystem objects
for accessing the ShadingSystem and consequently the ColorSystem. In its current form
it is not device-friendly.
We introduce a self-contained pvt::ShadingStateUniform that maintains a proper
state and is independent of the ShadingContext and ShadingSystem. We add an
opaque pointer to ShadingStateUniform in the ShaderGlobals structure.
Update uses of ColorSystem in opcolor.cpp to go through
ShaderGlobals::shadingStateUniform. Also update uses of m_commonspace_synonym
and m_unknown_coordsys_error in opmatrix.cpp to go through
ShaderGlobals::shadingStateUniform.
NOTE: The ShadingStateUniform is private and an OSL implementation detail and
does not change the public interface other than adding an opaque pointer to
ShaderGlobals.
* Fixed LLVM representation to accommodate ShaderGlobals::shadingStateUniform.
* Added missing Copyright Contributors license to shading_state_uniform.h
* Removed m_commonspace_synonym data member
Moved ShadingStateUniform closer to its use in osl_prepend_matrix_from
* Fixed clang-format diffs. Added void* shadingStateUniform to ShaderGlobals in rend_lib.h. Needed for proper working.
* Added shadingStateUniform to wide_opmatrix and batched_shader_globals.h
Adjusted batched struct data layout check in codegen.
* Reordered headers to include shading_state_uniform.h with the other headers
from liboslexec
Signed-off-by: Steena Monteiro <steena.monteiro@intel.com>
Several places in leparse.cpp were found by static analysis to have
memory leaks -- results of calling _parse() that, in the case of
early exists in error conditions, never got deleted.
I also did some other modernization: changed NULL to nullptr, changed
some loops to more compact range-for, utilize some Strutil helpers
that will be more efficient than the alternative std::string
manipulation.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This was meant to work all along, which is why this patch is so small!
But a tiny detail -- not explicitly passing info about the `--help`
argument -- made ArgParse substitute its own default, which exits
after printing the usage info. But, as I apparently meant to do all
along, if you specify it explicitly, you get to handle it, and the way
we handled it alsp printed the extra information.
Anyway, with this patch, `testshade --help` not only prints all the
testshade command options, but also it prints some information at the
end about some of the build settings of OSL, versions of key
dependencies, hardware info, and a run-down of the ShadingSystem
options. This can be helpful for debugging.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
- There are a handful of tests we disable entirely with because they
are irrelevant or just cannot be made to be clean (and that's ok).
- Tests with sanitizers run slow, so we increase timeouts, and also
set AA to 1 for render tests to speed them up. We rig it so a
non-matching output won't fail, and that's ok because every other CI
test is checking the results; here we have no reason to believe
results will differ from other tests, we just need to run with
sanitizers turned on.
- In testing.cmake, we rig texture tests to disable the leak
sanitizer, because we can't eliminate some completely benign leaks
from the texture system, and because they are called from JITed
code, we don't get a clean stack trace of things that happen in OIIO
and therefore we just can't enumerate the allowed exceptions. So we
trust that OIIO itself tests the texture system adequately and is
able to ignore the benign "leaks."
- We already submitted PRs and merged the little fixes that were
discovered by enabling address and leak sanitizers. This is just the
final bit to turn it on for CI runs from here on out.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Some explicit SIMD loops were technically illegal because each lane was not truly independent, fixed by adding "reduction" clause. Some Mask objects had to be replaced with just "int" to allow it to be specified in the reduction clause.
icx v2022.2.0 does apppear to have some regressions for large amounts of Dual code in a couple use cases, added alternative(improved) Single Statement Assignment (SSA) version of OIIO::fast_sincos which allowed unit tests to pass. Consider updating OIIO fmath.h to use SSA coding style (which would allow this workaround to be removed).
Another regression with transform_normals was avoiding by introducing an OSL_UNLIKELY(condition) to generate a branch vs. masked execution, this change is a general improvement and should be left in.
Avoid flex_bison produced warning in osogram.cpp "variable 'osonerrs' set but not used".
Due to a number of regressions that appear to be fixed in next revision, choose to skip icx 2022.2 and continue CI for the time being with icx 2022.1 until the next release.
Because multiple (possibly newer) versions of oneapi may be installed, use a config file to specific a specific compiler and tbb version NOTE: Individual oneapi components may independent version numbering.
For centos images, use a config file to specificy a specific oneapi compiler and tbb version
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
Some explicit SIMD loops were technically illegal because each lane was not truly independent, fixed by adding "reduction" clause. Some Mask objects had to be replaced with just "int" to allow it to be specified in the reduction clause.
icx v2022.2.0 does apppear to have some regressions for large amounts of Dual code in a couple use cases, added alternative(improved) Single Statement Assignment (SSA) version of OIIO::fast_sincos which allowed unit tests to pass. Consider updating OIIO fmath.h to use SSA coding style (which would allow this workaround to be removed).
Another regression with transform_normals was avoiding by introducing an OSL_UNLIKELY(condition) to generate a branch vs. masked execution, this change is a general improvement and should be left in.
Avoid flex_bison produced warning in osogram.cpp "variable 'osonerrs' set but not used".
Due to a number of regressions that appear to be fixed in next revision, choose to skip icx 2022.2 and continue CI for the time being with icx 2022.1 until the next release.
Because multiple (possibly newer) versions of oneapi may be installed, use a config file to specific a specific compiler and tbb version NOTE: Individual oneapi components may independent version numbering.
For centos images, use a config file to specificy a specific oneapi compiler and tbb version
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* For analysis, USE_QT=0 will disable osltoy from building (since we
don't/can't test it in the testsuite, it's unexecuted but really
shouldn't count against our code coverage metric).
* Exclude Imath duplicate headers from Sonar analysis.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Traditionally, OSL has supported building against the current and also
the previous supported version of OIIO, but not necessarily any older.
With OIIO 2.4 released (any day now), and OSL "main" is actually going
to be next year's supported release, it seems reasonable to change the
forward-looking development of OSL to have a minimum of OIIO 2.3,
which means that OSL-next will support OIIO 2.3, 2.4, and the
currently-in-progress 2.5 (or whatever it's called by that point).
The main reason to do this bump is because we'd like to depend on the
ustringhash class, and possibly other things we will probably be able
to easily backport to 2.3 but not to 2.2.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The little ReadEvent class had two members that are never set when
OSL_DEV is not defined, nor are they accessible (the accessor
functions also are disabled based on OSL_DEV). It looked to the static
analyzer like the data members were uninitialized, which they
were. Add OSL_DEV guards around the declarations of those members as
well, so they aren't even present in the struct when not needed for
this debugging mode.
Food for thought: another alternative is to keep the members but just
remove the OSL_DEV guards entirely -- just let the two int data
members be used all the time, not just for OSL_DEV. It's too small to
have any perf penalty, and it unclutters the declaration by removing
all those `#ifdef` lines.
For that matter, I also wonder if there's any real point to having
private members and accessor functions for this tiny internal-only
class, and maybe it's better to just let it be compact declaration
of a simple `struct` with three directly accessible data members.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
In testshade, there was a "loop" that wasn't really a loop, and it
was getting flagged by static analysis as being fishy. This patch
rewrites it as an "if".
Also take the opportunity to change from stream output to OSL::print
formatted output.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We had hard-coded longer allowable timeouts for particular testsuite
tests. But for the static analysis runs, it takes extra long and we
need a way to tell it to increase the timeouts even more, and to
exclude the longest running tests from the analysis coverage run (it's
not necessary to run those tests, they don't increase coverage enough
to justify being an hours long run).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Signed-off-by: Larry Gritz <lg@larrygritz.com>
In the course of our runtime optimization of shader networks, we
generate lots of new constant symbols (for example, as we optimize an
"a = b + c" into "a = newconst" if b and c can be determined to be
constants). The new constants are named with a numerical suffix doled
out in a sequental manner.
I've always thought that maybe it would be helpful for me when
debugging optimizer output if the name of the symbol also revealed the
value of the constant it held (when possible). Never did anything
about it.
Now it turns out that this has an application for JIT to PTX, where
the PTX cache can save compilation-to-PTX time if it encounters LLVM
IR that is exactly the same as it has seen before and is cache. If
the constants are generated in a different order, it generates
non-functional, purely text differences in the code we generate, and
this means the PTX cache doesn't function as efficiently as possible.
So this change alters the compiler-generated constant naming for two
cases: int constants, and string constants. The int constants append
the value of the integer (using '_' instead of '-' for negatives), and
the string constants append the hexidecimal value of the ustring hash
for that string. In both cases it does this without using or
incrementing the `m_next_newconst` counter that it uses to
sequentially name the other types that can't easily encode the value
to generate a unique name.
It's really to help the PTX cache, but as a side benefit, it will
be slightly easier for me to make sense of the optimized code when
I need to read it with my own eyes while debugging.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* A variety of changes to get a clean build and passing tests when
using LLVM 15.0.
* I've noticed problems when using LLVM 15 but building with earlier
clang, so the cmake scripts now print a warning in that case, so if
users encounter trouble they have a hint about what to do to fix it.
* For our CI tests on Mac, force the MacOS-11 test to use llvm 14,
and the MacOS-12 test to use llvm 15.
IMPORTANT TO-DO / CAVEATS:
1. When doing JIT at optlevel 12 or 13, I had to disable a number of
passes that don't seem to exist anymore in LLVM 15. This is enough
to get it working, and to be honest, I don't know if anybody uses
these opt levels. But we need to revisit this, because I don't
know if there these are cases where the names of the passes merely
changed or that new passes take their place (in which case we
should add the new passes, not stop after merely disabling the
deprecated ones). For that matter, optlevel modes 11, 12, 13 are
supposed to match what clang does for -O1, -O2, -O3, and that
changes from one release to the next, so we should probably revisit
this list and make sure it's matching clang's current choices
(which I assume are crafted and periodically revised by clang's
authors).
2. LLVM 15 switches to "opaque pointers". It still supports typed
pointers... for now. But as you can see in the diffs, I had to
disable a variety of deprecation warnings and take some other
actions to put LLVM 15 in the old "opaque ptr" mode to match our
use of LLVM <= 14. But this is only temporary, because the typed
pointer mode is expected to be removed entirely in LLVM 16. So at
some point in the next few months, we are going to need to support
opaque pointers in our use of LLVM. (Also note: for a while, we're
going to have a bunch of ugly `#if` guards or other logic to
support both opaque pointers for users with llvm >= 16 and also
typed pointers for users with llvm <= 14, until such time as our
LLVM minimum is at least 15, which I expect is not a reasonable
assumption for at least a couple years.)
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* A variety of changes to get a clean build and passing tests when
using LLVM 15.0.
* I've noticed problems when using LLVM 15 but building with earlier
clang, so the cmake scripts now print a warning in that case, so if
users encounter trouble they have a hint about what to do to fix it.
* For our CI tests on Mac, force the MacOS-11 test to use llvm 14,
and the MacOS-12 test to use llvm 15.
IMPORTANT TO-DO / CAVEATS:
1. When doing JIT at optlevel 12 or 13, I had to disable a number of
passes that don't seem to exist anymore in LLVM 15. This is enough
to get it working, and to be honest, I don't know if anybody uses
these opt levels. But we need to revisit this, because I don't
know if there these are cases where the names of the passes merely
changed or that new passes take their place (in which case we
should add the new passes, not stop after merely disabling the
deprecated ones). For that matter, optlevel modes 11, 12, 13 are
supposed to match what clang does for -O1, -O2, -O3, and that
changes from one release to the next, so we should probably revisit
this list and make sure it's matching clang's current choices
(which I assume are crafted and periodically revised by clang's
authors).
2. LLVM 15 switches to "opaque pointers". It still supports typed
pointers... for now. But as you can see in the diffs, I had to
disable a variety of deprecation warnings and take some other
actions to put LLVM 15 in the old "opaque ptr" mode to match our
use of LLVM <= 14. But this is only temporary, because the typed
pointer mode is expected to be removed entirely in LLVM 16. So at
some point in the next few months, we are going to need to support
opaque pointers in our use of LLVM. (Also note: for a while, we're
going to have a bunch of ugly `#if` guards or other logic to
support both opaque pointers for users with llvm >= 16 and also
typed pointers for users with llvm <= 14, until such time as our
LLVM minimum is at least 15, which I expect is not a reasonable
assumption for at least a couple years.)
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Several places in leparse.cpp were found by static analysis to have
memory leaks -- results of calling _parse() that, in the case of
early exists in error conditions, never got deleted.
I also did some other modernization: changed NULL to nullptr, changed
some loops to more compact range-for, utilize some Strutil helpers
that will be more efficient than the alternative std::string
manipulation.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
direnv (https://github.com/direnv/direnv) can be used for local developer workflow enhancements. direnv uses .envrc file for configuration. Because .envrc is usually specific to a developer workflow, it is generally not committed.
Signed-off-by: Mark Sisson <5761292+marksisson@users.noreply.github.com>
This was meant to work all along, which is why this patch is so small!
But a tiny detail -- not explicitly passing info about the `--help`
argument -- made ArgParse substitute its own default, which exits
after printing the usage info. But, as I apparently meant to do all
along, if you specify it explicitly, you get to handle it, and the way
we handled it alsp printed the extra information.
Anyway, with this patch, `testshade --help` not only prints all the
testshade command options, but also it prints some information at the
end about some of the build settings of OSL, versions of key
dependencies, hardware info, and a run-down of the ShadingSystem
options. This can be helpful for debugging.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
direnv (https://github.com/direnv/direnv) can be used for local developer workflow enhancements. direnv uses .envrc file for configuration. Because .envrc is usually specific to a developer workflow, it is generally not committed.
Signed-off-by: Mark Sisson <5761292+marksisson@users.noreply.github.com>
- There are a handful of tests we disable entirely with because they
are irrelevant or just cannot be made to be clean (and that's ok).
- Tests with sanitizers run slow, so we increase timeouts, and also
set AA to 1 for render tests to speed them up. We rig it so a
non-matching output won't fail, and that's ok because every other CI
test is checking the results; here we have no reason to believe
results will differ from other tests, we just need to run with
sanitizers turned on.
- In testing.cmake, we rig texture tests to disable the leak
sanitizer, because we can't eliminate some completely benign leaks
from the texture system, and because they are called from JITed
code, we don't get a clean stack trace of things that happen in OIIO
and therefore we just can't enumerate the allowed exceptions. So we
trust that OIIO itself tests the texture system adequately and is
able to ignore the benign "leaks."
- We already submitted PRs and merged the little fixes that were
discovered by enabling address and leak sanitizers. This is just the
final bit to turn it on for CI runs from here on out.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Rather than specify the version in the usual project() statement,
project (OSL VERSION 1.13.0.0 ...)
this patch decouples it
set (OSL_VERSION "1.13.0.0" CACHE STRING "Version")
project (OSL VERSION ${OSLVERSION} ...)
thus allowing the possibility of overriding the version number at
build confuguration time (`cmake -DOSL_VERSION=3.1.4.1`).
This should be used with extreme caution. Probably never used by most
people. I don't need to describe all the ways that things can go
sideways if you make your build of x.y.z.w actually be arbitrarily
different from the x.y.z.w that everyone else uses and that is
documented.
That said, there are two scenarios I have in mind for which this
ability is very useful:
1. Sometimes, we add or deprecate API features "in advance" by having
an `#if` enable or disable code based on the version, essentially
putting in place changes that can't be enabled yet because they
would break compatibility rules, but we want to integrate the code
now and have it become active upon reaching a particular future
version in which that incompatibility will be allowed. The version
number override makes it easy to test this by quickly doing a build
"as if" we had already bumped the version number to see what will
happen when we actually do.
2. Every once in a while we need to do an internal build that, to
conform to version compatibility rules, needs to appear to still be
part of a past version family. This lets individual sites play some
games with version number overrides rather than have to create new
branches and tags for certain one-off internal releases.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Rather than specify the version in the usual project() statement,
project (OSL VERSION 1.13.0.0 ...)
this patch decouples it
set (OSL_VERSION "1.13.0.0" CACHE STRING "Version")
project (OSL VERSION ${OSLVERSION} ...)
thus allowing the possibility of overriding the version number at
build confuguration time (`cmake -DOSL_VERSION=3.1.4.1`).
This should be used with extreme caution. Probably never used by most
people. I don't need to describe all the ways that things can go
sideways if you make your build of x.y.z.w actually be arbitrarily
different from the x.y.z.w that everyone else uses and that is
documented.
That said, there are two scenarios I have in mind for which this
ability is very useful:
1. Sometimes, we add or deprecate API features "in advance" by having
an `#if` enable or disable code based on the version, essentially
putting in place changes that can't be enabled yet because they
would break compatibility rules, but we want to integrate the code
now and have it become active upon reaching a particular future
version in which that incompatibility will be allowed. The version
number override makes it easy to test this by quickly doing a build
"as if" we had already bumped the version number to see what will
happen when we actually do.
2. Every once in a while we need to do an internal build that, to
conform to version compatibility rules, needs to appear to still be
part of a past version family. This lets individual sites play some
games with version number overrides rather than have to create new
branches and tags for certain one-off internal releases.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This is a tricky situation to describe. Consider this shader:
shader test (float A = 0.5,
float B = A)
{
printf("A = %g, B = %g\n", A, B);
}
By default, A is 0.5 and B gets A's value, also 0.5. If you give A a
different instance value, like 1.0, then B will also get the same
value, they'll both be 1.0. If you give B an instance value, that
will be used for B rather than copying A's value, like this:
$ testshade -param B 1.0 -shader test test
A = 0.5, B = 1
But there is a corner case bug: if we assign B an instance value of
0.0, it will incorrectly fall into a special case where it thinks it
gets a default value, only coincidentally because 0.0 is in an unused
slot for the default value (unused because it has no default -- its
default comes from "init ops", that assignment).
$ testshade -param B 0.0 -shader test test
A = 0.5, B = 0.5
!!!
Reading the patch should be self-explanatory for what the bug is and
how we fix it.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Signed-off-by: Larry Gritz <lg@larrygritz.com>
I've tried to write a complete set of step-by-step instructions for
everything about how to coordinate and deploy a release. This is a
living document, we will amend over time. I'm sure there must be
things I've forgotten.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This is a tricky situation to describe. Consider this shader:
shader test (float A = 0.5,
float B = A)
{
printf("A = %g, B = %g\n", A, B);
}
By default, A is 0.5 and B gets A's value, also 0.5. If you give A a
different instance value, like 1.0, then B will also get the same
value, they'll both be 1.0. If you give B an instance value, that
will be used for B rather than copying A's value, like this:
$ testshade -param B 1.0 -shader test test
A = 0.5, B = 1
But there is a corner case bug: if we assign B an instance value of
0.0, it will incorrectly fall into a special case where it thinks it
gets a default value, only coincidentally because 0.0 is in an unused
slot for the default value (unused because it has no default -- its
default comes from "init ops", that assignment).
$ testshade -param B 0.0 -shader test test
A = 0.5, B = 0.5
!!!
Reading the patch should be self-explanatory for what the bug is and
how we fix it.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Signed-off-by: Larry Gritz <lg@larrygritz.com>
I've tried to write a complete set of step-by-step instructions for
everything about how to coordinate and deploy a release. This is a
living document, we will amend over time. I'm sure there must be
things I've forgotten.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The little ReadEvent class had two members that are never set when
OSL_DEV is not defined, nor are they accessible (the accessor
functions also are disabled based on OSL_DEV). It looked to the static
analyzer like the data members were uninitialized, which they
were. Add OSL_DEV guards around the declarations of those members as
well, so they aren't even present in the struct when not needed for
this debugging mode.
Food for thought: another alternative is to keep the members but just
remove the OSL_DEV guards entirely -- just let the two int data
members be used all the time, not just for OSL_DEV. It's too small to
have any perf penalty, and it unclutters the declaration by removing
all those `#ifdef` lines.
For that matter, I also wonder if there's any real point to having
private members and accessor functions for this tiny internal-only
class, and maybe it's better to just let it be compact declaration
of a simple `struct` with three directly accessible data members.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
In testshade, there was a "loop" that wasn't really a loop, and it
was getting flagged by static analysis as being fishy. This patch
rewrites it as an "if".
Also take the opportunity to change from stream output to OSL::print
formatted output.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We had hard-coded longer allowable timeouts for particular testsuite
tests. But for the static analysis runs, it takes extra long and we
need a way to tell it to increase the timeouts even more, and to
exclude the longest running tests from the analysis coverage run (it's
not necessary to run those tests, they don't increase coverage enough
to justify being an hours long run).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Signed-off-by: Larry Gritz <lg@larrygritz.com>
In the course of our runtime optimization of shader networks, we
generate lots of new constant symbols (for example, as we optimize an
"a = b + c" into "a = newconst" if b and c can be determined to be
constants). The new constants are named with a numerical suffix doled
out in a sequental manner.
I've always thought that maybe it would be helpful for me when
debugging optimizer output if the name of the symbol also revealed the
value of the constant it held (when possible). Never did anything
about it.
Now it turns out that this has an application for JIT to PTX, where
the PTX cache can save compilation-to-PTX time if it encounters LLVM
IR that is exactly the same as it has seen before and is cache. If
the constants are generated in a different order, it generates
non-functional, purely text differences in the code we generate, and
this means the PTX cache doesn't function as efficiently as possible.
So this change alters the compiler-generated constant naming for two
cases: int constants, and string constants. The int constants append
the value of the integer (using '_' instead of '-' for negatives), and
the string constants append the hexidecimal value of the ustring hash
for that string. In both cases it does this without using or
incrementing the `m_next_newconst` counter that it uses to
sequentially name the other types that can't easily encode the value
to generate a unique name.
It's really to help the PTX cache, but as a side benefit, it will
be slightly easier for me to make sense of the optimized code when
I need to read it with my own eyes while debugging.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Today's branching of OIIO 2.4 and subsequent changing of its master
branch to think it's "2.5" seems to have triggered an automatic fuse
that annotates a particular function in OIIO as deprecated for version
2.5 and beyond. OSL was still calling the old version, so some
immediate changes are necessary to call the correct new name when
using a new OIIO.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Today's branching of OIIO 2.4 and subsequent changing of its master
branch to think it's "2.5" seems to have triggered an automatic fuse
that annotates a particular function in OIIO as deprecated for version
2.5 and beyond. OSL was still calling the old version, so some
immediate changes are necessary to call the correct new name when
using a new OIIO.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Exclude osltoy from coverage analysis, since we don't have a way to
run it from CI
* Run batch shading for analysis to get better code coverage
* llvm_gen_select: extra check that we can't overrun memory of cond[]
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Change the languagespec to document the new MaterialX-synchronized
closures.
* Document the old ones as "deprecated" -- still supported for 1.x but
assumed to be on the slate for removal in OSL 2.x.
* Change stdosl.h so that the new closures, instead of being seen only
if MATERIALX_CLOSURES is defined, to being enabled by default, but
(for now) can be disabled if NO_MATERIALX_CLOSURES is defined.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Change the languagespec to document the new MaterialX-synchronized
closures.
* Document the old ones as "deprecated" -- still supported for 1.x but
assumed to be on the slate for removal in OSL 2.x.
* Change stdosl.h so that the new closures, instead of being seen only
if MATERIALX_CLOSURES is defined, to being enabled by default, but
(for now) can be disabled if NO_MATERIALX_CLOSURES is defined.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Removing the suffix ids and callseq comments generated by global counters
leads to deterministic ptx in a multithreaded environment.
Signed-off-by: Chris Hellmuth <chellmuth@gmail.com>
* Minor changes to fix a couple more warnings given by SonarCloud
static analysis.
* Get the code coverage enabled, it wasn't running properly.
* testrender: allow secret TESTRENDER_AA to override command line --aa
* runtest.py: allow secret OSL_TESTSUITE_SKIP_DIFF to force skipping
the reference comparison step.
* For analysis run, set TESTRENDER_AA=1 to keep the tests from being
prohibitively expensive, and OSL_TESTSUITE_SKIP_DIFF=1 to keep it
from failing because the aa=1 images are different.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We had been using clang/llvm 12 and when trying out 14 I started
getting test failures with OptiX complaining about duplicate
symbols. They all seemed to be the things we declared as
OSL_CONSTANT_DATA. It looks like there's a difference in llvm 12 vs 14
in how the generated PTX marks its symbol visibility.
The solution (thanks to a clever suggestion by Alex Conty) was just to
enclose those declarations in anonymous namespaces. Works like a
charm!
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Minor changes to fix a couple more warnings given by SonarCloud
static analysis.
* Get the code coverage enabled, it wasn't running properly.
* testrender: allow secret TESTRENDER_AA to override command line --aa
* runtest.py: allow secret OSL_TESTSUITE_SKIP_DIFF to force skipping
the reference comparison step.
* For analysis run, set TESTRENDER_AA=1 to keep the tests from being
prohibitively expensive, and OSL_TESTSUITE_SKIP_DIFF=1 to keep it
from failing because the aa=1 images are different.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We had been using clang/llvm 12 and when trying out 14 I started
getting test failures with OptiX complaining about duplicate
symbols. They all seemed to be the things we declared as
OSL_CONSTANT_DATA. It looks like there's a difference in llvm 12 vs 14
in how the generated PTX marks its symbol visibility.
The solution (thanks to a clever suggestion by Alex Conty) was just to
enclose those declarations in anonymous namespaces. Works like a
charm!
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This allows the refraction index to be detected even if a medium closure
is not present. Additionally, the medium detection now traverses layer()
closures.
Signed-off-by: Chris Kulla <ckulla@gmail.com>
There seems to be just a teensy bit of nondeterminism or
platform-variance in this one test that we haven't tracked down the
source of yet, that moves some specular fireflies around. Smells like
maybe an uninitialized variable somewhere in testrender? Because it
doesn't happen on any other test and we haven't detected it in our
real renderer, we don't currently suspect that it's an actual error
in oslexec.
Big hack: Re-run failed "render" tests up to 5 additional times before
giving up.
This makes me feel dirty. But it makes tests pass for now. Eventually we
will find the embarrassing testrender bug that is responsible for the
occasionally different specular sampling noise.
Also slightly bump up the error tolerance for this test.
Also add one alternative that seems to frequently pop up.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This allows the refraction index to be detected even if a medium closure
is not present. Additionally, the medium detection now traverses layer()
closures.
Signed-off-by: Chris Kulla <ckulla@gmail.com>
There seems to be just a teensy bit of nondeterminism or
platform-variance in this one test that we haven't tracked down the
source of yet, that moves some specular fireflies around. Smells like
maybe an uninitialized variable somewhere in testrender? Because it
doesn't happen on any other test and we haven't detected it in our
real renderer, we don't currently suspect that it's an actual error
in oslexec.
Big hack: Re-run failed "render" tests up to 5 additional times before
giving up.
This makes me feel dirty. But it makes tests pass for now. Eventually we
will find the embarrassing testrender bug that is responsible for the
occasionally different specular sampling noise.
Also slightly bump up the error tolerance for this test.
Also add one alternative that seems to frequently pop up.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Didn't properly detect the disallowed use of arrays for the false
clause because we checked `t` twice.
Yay for static analysis!
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Didn't properly detect the disallowed use of arrays for the false
clause because we checked `t` twice.
Yay for static analysis!
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Another round of clang-format
* Bump the standard clang-format we use for CI from clang11 to clang14.
This will make some minor additional changes.
* Due to a mismatched target name, we weren't actually checking the
formatting in CI on all the files we thought we were, and therefore
not assuring that certain code was correctly formatted. So there is
some fixup needed, which is included here.
* And a bunch of files we didn't try to do before, either.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The idea is to stop propagating derivs through BSDFs, which we
never did properly. Instead we trace rays with radius + spread,
which we turn into OSL derivs in SG when we hit a surface. Then
when bouncing off a surface we use the BSDF roughness to set the
spread. In addition we also:
* Return BSDF samples/eval as a small struct Sample(). This
avoids output parameters in the methods.
* Stop passing a full ShaderGlobals to the BSDF methods. Pass
a viewing direction wo instead so they could run in a simpler
context.
Signed-off-by: Alejandro Conty <aconty@imageworks.com>
* Another round of clang-format
* Bump the standard clang-format we use for CI from clang11 to clang14.
This will make some minor additional changes.
* Due to a mismatched target name, we weren't actually checking the
formatting in CI on all the files we thought we were, and therefore
not assuring that certain code was correctly formatted. So there is
some fixup needed, which is included here.
* And a bunch of files we didn't try to do before, either.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The idea is to stop propagating derivs through BSDFs, which we
never did properly. Instead we trace rays with radius + spread,
which we turn into OSL derivs in SG when we hit a surface. Then
when bouncing off a surface we use the BSDF roughness to set the
spread. In addition we also:
* Return BSDF samples/eval as a small struct Sample(). This
avoids output parameters in the methods.
* Stop passing a full ShaderGlobals to the BSDF methods. Pass
a viewing direction wo instead so they could run in a simpler
context.
Signed-off-by: Alejandro Conty <aconty@imageworks.com>
This simplifies the logic around the ior which drives reflection/transmission (fresnel effect)
from the ior which drives refraction (and total internal reflection). Setting both iors equal to each other
gives the expected physical behavior.
Also tweak the bsdf constructor to support variadic arguments which simplifies a bunch of code.
Add a new test to validate the refractive version of generalized_schlick_bsdf.
Signed-off-by: Chris Kulla <ckulla@gmail.com>
Implement get_albedo for the sheen closure, based on fit derived by the MaterialX team
Add optional mode to testrender to visualize the result of the get_albedo method. This can be useful for validation of the get_albedo method.
Signed-off-by: Chris Kulla <ckulla@gmail.com>
This simplifies the logic around the ior which drives reflection/transmission (fresnel effect)
from the ior which drives refraction (and total internal reflection). Setting both iors equal to each other
gives the expected physical behavior.
Also tweak the bsdf constructor to support variadic arguments which simplifies a bunch of code.
Add a new test to validate the refractive version of generalized_schlick_bsdf.
Signed-off-by: Chris Kulla <ckulla@gmail.com>
Implement get_albedo for the sheen closure, based on fit derived by the MaterialX team
Add optional mode to testrender to visualize the result of the get_albedo method. This can be useful for validation of the get_albedo method.
Signed-off-by: Chris Kulla <ckulla@gmail.com>
* Implement reflective microfacet based MaterialX closures
Add test cases for dielectric, conductor and generalized schlick closures
Signed-off-by: Chris Kulla <ckulla@gmail.com>
* Implement refraction case in microfacet closures
Signed-off-by: Chris Kulla <ckulla@gmail.com>
* Update reference images
Signed-off-by: Chris Kulla <ckulla@gmail.com>
* Relax diff threshold on render tests
Signed-off-by: Chris Kulla <ckulla@gmail.com>
* Implement reflective microfacet based MaterialX closures
Add test cases for dielectric, conductor and generalized schlick closures
Signed-off-by: Chris Kulla <ckulla@gmail.com>
* Implement refraction case in microfacet closures
Signed-off-by: Chris Kulla <ckulla@gmail.com>
* Update reference images
Signed-off-by: Chris Kulla <ckulla@gmail.com>
* Relax diff threshold on render tests
Signed-off-by: Chris Kulla <ckulla@gmail.com>
It printed the u,v coordinates where the error was first discovered.
This is not necessarily deterministic, of course!
Signed-off-by: Larry Gritz <lg@larrygritz.com>
It printed the u,v coordinates where the error was first discovered.
This is not necessarily deterministic, of course!
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Add support for layer closure with dielectric reflections as top layers, and a test with nested layer closures
* Add support for sheen in the layer closure
Signed-off-by: Adrien Herubel <adrien.herubel@autodesk.com>
* Add support for layer closure with dielectric reflections as top layers, and a test with nested layer closures
* Add support for sheen in the layer closure
Signed-off-by: Adrien Herubel <adrien.herubel@autodesk.com>
* Extend the bsdf interface in testrender to use Color3 instead of float in preparation for MaterialX closures which are almost all tinted
* Add stub parameters and IDs for all MaterialX closures
* Register all MaterialX closure types
* Implement most MaterialX closures, reusing the existing implementations for now
This commit is mainly intended to do the busy work of adding the new closures, no new BSDFs are actually implemented yet. This means that some of
the closures are handled in a very rudimentary way, ignoring some key parameters. The intent is to improve support for these over time.
Signed-off-by: Chris Kulla <ckulla@gmail.com>
The sampler implementation in testrender was very close to the 2020 article from Brent Burley. This patch
catches up to best practices, including a faster owen scrambler from Nathan Vegdahl and recognizing that
the index scrambling can be achived by the same owen scrambler method (which saves a bunch of logic and runs
faster). Finally, avoid the redundant bit reversals and use compiler intrinsics when available to benefit
from hardware instructions (for ARM or GPUs). Finally the integer hash constants were updated to the latest
from the hash prospector project.
While more optimizations are certainly possible, this is a reasonable balance between quality and speed.
Update reference images for new sampler
Signed-off-by: Chris Kulla <ckulla@gmail.com>
* Extend the bsdf interface in testrender to use Color3 instead of float in preparation for MaterialX closures which are almost all tinted
* Add stub parameters and IDs for all MaterialX closures
* Register all MaterialX closure types
* Implement most MaterialX closures, reusing the existing implementations for now
This commit is mainly intended to do the busy work of adding the new closures, no new BSDFs are actually implemented yet. This means that some of
the closures are handled in a very rudimentary way, ignoring some key parameters. The intent is to improve support for these over time.
Signed-off-by: Chris Kulla <ckulla@gmail.com>
The sampler implementation in testrender was very close to the 2020 article from Brent Burley. This patch
catches up to best practices, including a faster owen scrambler from Nathan Vegdahl and recognizing that
the index scrambling can be achived by the same owen scrambler method (which saves a bunch of logic and runs
faster). Finally, avoid the redundant bit reversals and use compiler intrinsics when available to benefit
from hardware instructions (for ARM or GPUs). Finally the integer hash constants were updated to the latest
from the hash prospector project.
While more optimizations are certainly possible, this is a reasonable balance between quality and speed.
Update reference images for new sampler
Signed-off-by: Chris Kulla <ckulla@gmail.com>
* GPU string overhaul
Big string cleanup, yet again, for GPU. We still had this tricky
double indirection happening with strings for OptiX -- the string was
a pointer to a "global", which in turn held the hash. No, throw all
that out. Just directly store strings as the ustring hash on
OptiX. No extra indirection. Many fewer differences between how things
are handled on OptiX versus CPU (except for the fact itself that the
"string" holds the hash rather than the char*).
Some details worth noting:
* LLVM_Util loses "type_string()" and calls it "type_ustring()" to
emphasize it's the type we're using to represent a ustring, and
definitely not a C or C++std string. Also, LLVM_Util itself now lets
the caller tell it whether it prefers that the representation of a
ustring be the charptr or the hash, and then handles most of the
details internally. The BackendLLVM, then, makes that decision early
on, charptr for CPU and hash for GPU.
* Several more spots in LLVM_Util and BackendLLVM have been plumbed to
pass around llvm IR SSA variable names to try to make the resulting
LLVM IR more readable when I'm trying to debug it. (This was
strictly debugging scaffalding aid for me, but comes along for the
ride because it's useful to keep.) I also added a
BackendLLVM::llnamefmt to format strings, which only does the
formatting when the shading system knows it's going to save the IR
for inspection, and otherwise (that is, in production) it does not
do the string formatting or pass the names along, so it's not doing
all that string manipulation when no human is going to be looking at
the IR dumps.
* A lot of clunky OptiX-only code disappears.
* Some side effects of my reducing complexity of the IR as I debugged:
there were times we were generating redundant pointer casting in the
IR, such as grabbing a pointer, then casting it to a void*, then
casting back to some other kind of pointer. In some cases I reduced
that quite a bit. Again, this was mostly just a scramble to make the
IR more readable while I debugged, but it probably has a minor
performance gain as well since it reduced somewhat the sheer amount
of IR we generate and pass to the optimizer and JIT.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Remove register_string entirely from RendererServices
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Remove more register_string remnants
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* comment typo
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Set up a SECURITY.md explaining how to confidentially report
vulnerabilities.
* Create scorecards.yml workflow to run OSSF scorecard to check supply
chain security issues.
* Use exact SHA's for actions versions to guard against hijacked
actions repos that might try to substutite something else at the
version tag we use.
* Make sure actions we use don't get write perms on our repo.
* Tried to use harden-runner action to ensure no actions on the runner
overwrite source code or have unauthorized outgoing net connections
during the GHA run, but currently leaving it disabled because of too
many false positives -- it flags every dependency download as a case
of altering source code. Which it technically is, but it doesn't
matter for any jobs that aren't producing any artifacts that will be
consumed or relied on by users.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Set up a SECURITY.md explaining how to confidentially report
vulnerabilities.
* Create scorecards.yml workflow to run OSSF scorecard to check supply
chain security issues.
* Use exact SHA's for actions versions to guard against hijacked
actions repos that might try to substutite something else at the
version tag we use.
* Make sure actions we use don't get write perms on our repo.
* Tried to use harden-runner action to ensure no actions on the runner
overwrite source code or have unauthorized outgoing net connections
during the GHA run, but currently leaving it disabled because of too
many false positives -- it flags every dependency download as a case
of altering source code. Which it technically is, but it doesn't
matter for any jobs that aren't producing any artifacts that will be
consumed or relied on by users.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Most LLVM API IR calls that generate ops allow an optional name of the
llvm symbol that will hold the result of the operation. By default, it
will make one up, such as "%37", "%38", etc. Needless to say, these
are difficult to understand when reading volumes of IR dumped for
debugging purposes.
So in this patch, I'm starting to expose the ability for our own code
to pass names through our wrapper functions. I needed this for some
deep debugging.
I haven't instrumented the entire set, just trying it on for some of
the specific areas I needed to debug. But this is a useful first step.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Most LLVM API IR calls that generate ops allow an optional name of the
llvm symbol that will hold the result of the operation. By default, it
will make one up, such as "%37", "%38", etc. Needless to say, these
are difficult to understand when reading volumes of IR dumped for
debugging purposes.
So in this patch, I'm starting to expose the ability for our own code
to pass names through our wrapper functions. I needed this for some
deep debugging.
I haven't instrumented the entire set, just trying it on for some of
the specific areas I needed to debug. But this is a useful first step.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This makes OptiX 7.0 the minimum version we accept, and greatly
simplifies a lot of code we were separately maintaining for 6 and 7.
This will go into main AFTER the release branch for 1.12 is split off.
The dropping of OptiX 6 will be a 1.13 (or whatever we call it) change.
Don't build the optix modules at all when not USE_OPTIX.
Totally remove stringtable.
Other simplifications stemming from these.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The implementation of simplex noise we've been using is public domain,
but looks like when we pasted the license notice into our repo, there
was one stray sentence (probably left over from an earlier license
scheme Stefan Gustavson used?) that makes reference to GPL. But the
software and its repo are NOT -- it's very clear that it's public
domain: https://github.com/stegu/perlin-noise/blob/master/LICENSE.md
So remove this confusing sentence.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Aiming to declare a stable beta soon and release circa SIGGRAPH.
Primarily, this branch is to allow us to continue changing main in
ways that would be destabilizing before the release.
New things that go in main that are risk or break compatibility
will not be backported to this new branch.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The implementation of simplex noise we've been using is public domain,
but looks like when we pasted the license notice into our repo, there
was one stray sentence (probably left over from an earlier license
scheme Stefan Gustavson used?) that makes reference to GPL. But the
software and its repo are NOT -- it's very clear that it's public
domain: https://github.com/stegu/perlin-noise/blob/master/LICENSE.md
So remove this confusing sentence.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We'd been waiting until the minimum OIIO requirement was high enough
to include support for the newer syntax.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
In a few of my big clang-format revisions, I was over-zealous, and did
not realize that I was making the command line option parsing unreadable.
Revert those secions.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Recent OIIO and fmt changes slightly changed this one place's output
with some version/platform combinations. Also make it match exactly
between the example-deformer and example-batch-deformer tests.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
fmt master seems to have just deprecated the default ability of
fmt::sprintf("%s") to printing anything that allows stream
output. This breaks use of Strutil::sprintf formatting of ustrings.
This isn't yet in any tagged release of fmtlib, but I'm trying to get
ahead of it.
The quick fix is to just change those handful of places that still use
sprintf convention with nonstandard types to use std::format style
calls, which we are slowly converting to anyway. (And maybe a couple
other spots while I was at it.)
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Banish the last of the sprintf calls (except the handful that need
to be there, because such formatting is still supported in shaders
themselves, so something needs to understand them).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Another semi-random collection of conversions. Bit by bit.
These are all internals, none of it involves public interfaces.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Use unity build to speed up OIIO build (for sufficiently new OIIO).
* Build OIIO with STOP_ON_WARNING=OFF, so that if a compiler upgrades
and makes new warnings for OIIO, that doesn't break OSL's CI build.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Add support for closures to the batched interface.
The renderer interface for closures, including registering
and processing them remains unchanged.
The Ci variable in BatchedShaderGlobals has moved from
the uniform portion to the varying portion, which may
require changes to any renderer code that was utilizing
the BatchedShaderGlobals.
To utilize closures with the batched interface, you register
the closures per usual, then once shading has finished,
the Ci in BatchedShaderGlobals will contain an Block of pointers
to the roots of a closure trees that can be processed
independently just like in the normal single-point interface.
This initial implementation is largely functional, however,
there may be more optimization gains available in the code
that copies the run-time SIMD values and scatters them
to the separate closure-data locations.
One remaining un-implemented feature is passing
_closures_ via get/setmessage() when using the batched
shading interface, and the locations where that would
need to be implemented are marked in the source.
Signed-off-by: Stephen Friedman <sfriedma@pixar.com>
* Address batched closures review comments:
* Remove unneded Block specializations for pointers.
* Rename null variable to null_value.
* remove a 0 default param to rop.llvm_void_ptr in batched_llvm_gen.cpp:llvm_gen_mul
* Optimizations to masking and loading in wide_opclosure.cpp
* Fix LLVM11+ builds by using llvm_vecor_type instead of llvm::VectorType::get
Signed-off-by: Stephen Friedman <sfriedma@pixar.com>
* Remove unneeded oslclosure.h in wide.h now that we aren't explicitly instantiating the templates.
Signed-off-by: Stephen Friedman <sfriedma@pixar.com>
The Intel OneAPI LLVM-based C++ compiler, as we download it from Intel
for our CI, bumped from 2022.0.0 to 2022.1.0.
New warning popped up -- newest Intel LLVM-based compiler warns about
overriding certain floating point math flags.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Fix warnings from stdint on windows by pre-including the standard header
* Make sure register_JIT_Global is a public symbol since it needs to be called by client code
* Fix how llvm-as, llvm-link and python are invoked by cmake so they can be properly found on Windows platforms
Signed-off-by: Chris Kulla <ckulla@gmail.com>
There's no rhyme or reason to this particular set of changes... I just
picked a few files and did the obvious conversion. Will do more in
subsequent PRs.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
A few minor changes were needed to address new warnings.
clang14 warns if you use bitwise `|` or `&` in an `if`, but there are
some places where we do that purposely to avoid branches.
Disable clang14 warnings about bitwise ops for all of liboslexec.
Also some batch shading related vectorized loops fail to vectorize
with clang14 -- though they worked fine with clang12 -- and so we
needed to mark those as "complex" SIMD loops. Is this a regression
with clang14?
Revise simd loop pragma for wide_transformc due to llvm14 regression
Signed-off-by: Larry Gritz <lg@larrygritz.com>
In order to move towards shader execution on devices, we need to replace
host side interactions with new concepts that can exist on the device (or host).
The current virtual function based RendererServices exemplifies this dependency
on host side objects.
To break the dependency we introduce free function based renderer services (rs_free_function.h)
To simplify its introduction, we only added the renderer service free functions required by opmatrix.cpp.
Goal is to have a single opmatrix that can be compiled for host or device (eventually).
Renderers can provide llvm bitcode representation of their renderer service
free functions to the ShadingSystem. During shader compilation/JIT the
renderer's bitcode is combined with opmatrix.cpp bitcode along with the JIT'd shader network.
The idea is the combined LLVM IR representation can then be lowered to the target
(be it on the host or device[SPIRV|PTX]). Thus a renderer's free function implementation can be shared.
opmatrix.cpp now only calls the free functions (not virtual RendererServices).
However liboslexec on the host contains an rs_fallback.cpp which forwards
calls through the existing virtual RendererServices for seamless compatibility
when a renderer does not provide bitcode for the renderer services.
Testshade has been modified to optionally to provide rs_bitcode,
utilizing a new Cmake macro, EMBED_LLVM_BITCODE_IN_CPP, to turn rs_simplerend.cpp (the free function implementation)
into bitcode.
Free functions can be enabled via --use-rs-bitcode flag in testshade.
Free functions need access to RenderState values because they lack a this
pointer like the RendererServices virtual methods have. Added a RenderState
containing minimal set of values required by free functions.
Testshade/SimpleRenderer is responsible for setting RenderState values.
Added ShadingSystem attribute rs_bitcode to hold renderer services free function bitcode (provided by the renderer)
Renderers using global variables (like ustring's) not in strdecls.h must register their global variables' addresses
using OSL::register_JIT_Global (const char *global_var_name, void *global_var_addr).
This is necessary for the JIT process to resolve global variables that are embedded in RendererServices bitcode.
A validation step occurs that emits an error with a list of unmapped global variables.
RendererServices can reference strings with the STRING_PARAM(varname) macro.
By default, STRING_PARAM(varname) works for any string in strdecls.h.
For any string NOT in strdecls.h, it will map to a global variable with a prefix of "RS_".
For example, STRING_PARAM(raster) maps to a global ustring RS_raster.
NOTE: C linkage with a "RS_" prefix is used to allow for unmangled
non-colliding global symbol names, so its easier to pass them to
OSL::register_JIT_Global(name, addr) for host execution
NOTE: the STRING_PARAMS macro adapts to OSL_HOST_RS_BITCODE macro
to utilize the RS_ prefix. RS_ prefixed versions of all OSL::Strings
intances have been created by rs_free_function.h, so the same STRING_PARAMS
macro can be used for renderer service or OSL strings.
NOTE: We expect the handling of strings to change in the future possibly being replaced by Handles entirely
It is the renderer's responsibility to instantiate any NEW ustring global variables with the proper RS_ prefix;
It is the renderer's free function implementation to import/extern any of these NEW global variables with the proper RS_ prefix;
For testshade, a rs_strdecls.h has been added and is utilized to instance the RS_ global variables and register their addresses with OSL in simplerend.cpp.
rs_strdecls.h is also used by rs_simplerend.cpp (free function implementation) to create the externs with the RS_ prefix.
NOTE: DO NOT instantiate global variables inside the free function renderer services, because we strip global constructors as part of the pruning process.
Added LLVM_Util::validate_global_mapping to identify unresolved globals.
Added utility macro __OSL_STRINGIFY(string_value) in oslversion.h.in so the
proper naming prefix is used and we don't have multiple implementations around.
Added liboslexec build step to produce bitcode containing OSL library functions that depend on free functions (here, opmatrix.cpp)
Added a new Cmake EMBED_LLVM_BITCODE_IN_CPP function in llvm_macros.cmake that replaces the LLVM_COMPILE macro
and provides for both linking and serializing multiple OSL cpp files.
Fixed cuda.macro to use new location of serialize-bc.py in src/build-scripts.
Fixed formatting issues.
Disabled erasing llvm.global_ctors from LLVM module because of issues on MacOS Apple Clang.
Left as future work.
Added better error checking for module_from_bitcode().
Signed-off-by: Steena Monteiro <steena.monteiro@intel.com>
New classes of warnings triggered when enabling option "opt_warnings":
* Creation of new strings mid-shader.
* Access to the characters of strings mid-shader.
We already warned about texture access by name that could not be
turned into a handle.
In all of these cases, we only warn *after* runtime optimization, so that shader
idioms in the source code that appear to do these forbidden things will not warn
as long as they can be reduced to safe operations in the course of the runtime
optimization step.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
checked_find_package:
* Optional SETVARIABLES gives a list of vars to set to TRUE if the package
is found.
* Optional DEBUG flag turns on extra debugging information about how the
package is found.
* Optional PREFER_CONFIG uses a dependency's exported config, if available,
rather than our Find module.
* Global ALWAYS_PREFER_CONFIG option can be set to ON to always prefer
config over Find modules for all dependencies.
fancy_add_executable:
* -DENABLE_INSTALL_<executable>=OFF lets you turn off the installation
of an executable.
other:
* Change the target name for doing the clang-format over the code to
"run-clang-format", because "clang-format" is an imported target name
of the libclang exported cmake (which we may wish to use soon).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Pass some extra headers to NVCC_COMPILE to make sure that the PTX files are regenerated whenever those headers are changed.
Signed-off-by: Tim Grant <tgrant@nvidia.com>
* Don't generate PTX for rend_lib.cu when using OptiX 6.
Signed-off-by: Tim Grant <tgrant@nvidia.com>
* Fix the callable program signature for the init and group functions in the OptiX wrappers.
Signed-off-by: Tim Grant <tgrant@nvidia.com>
* Fix issues with the LLVM arguments for osl_printf when targeting CUDA.
Signed-off-by: Tim Grant <tgrant@nvidia.com>
* Fix llvm_gen_construct_triple for OptiX. The 'to' argument was not handled correctly.
Signed-off-by: Tim Grant <tgrant@nvidia.com>
* Improve support for CUDA osl_get_matrix.
Signed-off-by: Tim Grant <tgrant@nvidia.com>
* Add basic support for named transforms to testshade/testrender.
Signed-off-by: Tim Grant <tgrant@nvidia.com>
API changes we had to take into account:
* TargetRegistry.h location
* No more DisableTailCalls field in PassManagerBuilder.
Needed to update the ref image for render-microfacet test, some sparklies changed.
Looks like the new LLVM probably JITs to ever so slightly different math code, tickling
some LSB differences that at 1 sample per pixel, results in some different sampling
directions leading to fireflies. We decided to just commit a new ref image and move on.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
In our generated OSLConfig.cmake, we inserted a find_dependency(Imath)
that accidentally demanded the OpenEXR_VERSION (should have been
Imath_VERSION). That works fine as long as Imath version is the same
(or higher) than the OpenEXR version. But if the installed OpenEXR
version is somehow ahead of the Imath version, it will fail.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
A small instalment on the long-term shift from printf-style formatting
to std::format style formatting.
Define preprocessor OSL_INTERNAL, which is defined when compiling OSL
itself, but will not be defined for downstream projects using the OSL
headers.
When building OSL, enable the symbol OIIO_ERRORHANDLER_HIDE_PRINTF,
which causes ErrorHandler methods using the old conventions to be
marked as deprecated and generate warnings.
Clean up all the calls revealed by those warnings. (And
opportunistically clean up some other things that happened to be in
the same files.)
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Also change the one exception mentioned to a std one, matching
Imath 3.x and insulating us from any of the old Iex custom exceptions
from Imath 2.x.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Changed a function signature in OIIO master that threw us off here
just a tad, as there was no direct type conversion the compiler could
figure out on its own. Easy adjustment gives it the right hint.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This eliminates a huge amount of redundancy in our CI descriptions --
instead of the entire CI logic replicated for each test in its own
universe, we group tests into (a) all the ones based on ASWF VFX
Platform containers, (b) non-VFXP-year oddballs run on Ubuntu, (c) Mac
tests, (d) clang-format verification. Each has the build logic
expressed once, and a short list of variables that change for each
individual test of that category.
This organization is much easier to read, maintain, modify tests, or
add new test variants, and cuts the size of the ci.yml file by over 50%.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Avoid many clang vectorization workarounds for OSL_INTEL_LLVM_COMPILER_VERSION
Added Makefile option VEC_REPORT to turn on generation of optimization reports
Reduce impact of REQUIRE_INF_NAN for icx by explicitly honoring inf, nan, and signed zeros vs. disabling fast-math
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
Add compiler support for Intel(r) oneAPI DPC++/C++ Compiler (based on LLVM technology).
Detect if CMAKE_CXX_COMPILER_ID matches "IntelLLVM" and set CMAKE_COMPILER_IS_INTELCLANG=1
NOTE: CMAKE_COMPILER_IS_INTEL will not be set, that is reserved for the classic Intel(r) C++ Compiler.
Define OSL_INTEL_LLVM_COMPILER to hold an encoded Intel(r) LLVM Compiler version
(e.g. 20220000), or 0 if not an Intel(r) LLVM Compiler.
Define OSL_INTEL_CLANG_VERSION to hold the encoded Clang version the
Intel(r) LLVM Compiler is based on (e.g. 140000),
or 0 if not an Intel(r) LLVM compiler.
Ensure OSL_CLANG_VERSION is when its a vanilla clang compiler, not apple or Intel(r) oneAPI DPC++/C++ Compiler.
Added OSL_ANY_CLANG which is set for any clang based compiler.
Removed OSL_NON_CLANG_COMPILER.
Updated uses of OSL_NON_CLANG_COMPILER with more specific and understandable combinations of OSL_ANY_CLANG, OSL_INTEL_CLANG_VERSION, and OSL_INTEL_COMPILER.
Make sure our math calculations are consistent,
especially division. Different compilers may have choose to use
reciprocal division loosing precision causing slightly different
results which can lead to aliasing differences when running the testsuite
Disabled reciprocal division for all projects.
if -ffast-math is enabled to to the underlying compiler, inf's and NaN's may not be handled properly (by design)
When building liboslexec, ensure any source files that require proper INF or NaN handling have additional command line flags passed if needed (as the default behavior maybe -ffast-math):
shadingsys.cpp
wide/wide_shadingsys
wide/wide_optest_float
Enable unitialized Matrix44, because all of its members get overwritten anyway, in impl_transform_normal_masked.
Added alternative test results where needed, wsing '.icx.' as shorthand for the Intel(r) oneAPI DPC++/C++ Compiler in alternative test result filenames.
Changed some tests to use the pretty(value) which clamps values close to 0 to be 0 to avoid lots of different alternative rest result files when using printf.
Added "--center" option to several regression tests to avoid aliasing in results between SIMD batched and scalar execution.
* Added icx-vp2022 to CI plan.
* To better clarify the classic vs. LLVM based Intel compilers and that they have version #'s not just 0 or 1,
renamed OSL_INTEL_COMPILER to OSL_INTEL_CLASSIC_COMPILER_VERSION
renamed OSL_INTEL_LLVM_COMPILER to OSL_INTEL_LLVM_COMPILER_VERSION
Removed unnecessary FMT_VERSION and OPENIMAGEIO_CMAKE_FLAGS from the icx-vp2022 worklow, added back USE_SIMD.
Fix spelling of losing.
Added OSL_OMP_COMPLEX_SIMD_LOOP(...) which is enabled or disabled in platform.h based on which compilers/versions can handle vectorizing complex nested control flow. This makes it cleaner to document which loops need to vectorization conditionally controlled and centralizes where it is managed.
* Removed extra fp-model=consistent for icx-vp2022 workflow
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
Slight reworking of -Werror logic to let the default behavior be
changed by setting a cmake variable. In particular, this allows us to
make a CI build disable -Werror when we need to.
Previously, having the "OSL_CI" env variable set trumped all else and
forced -Werror. Now it just contributes to the default behavior, but
setting `-DSTOP_ON_WARNING=OFF` and cmake config time can still
disable it.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Additionally, allow custom command line arguments to run after each
image is loaded, but before the comparison.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Turn on warnings about OIIO::Strutil::format and ustring::format
We should be using Strutil::sprintf / ustring::sprintf, or upgrading
entirely to Strutil::fmt::format / ustring::fmtformat.
Some day, the plain "format" will be deprecated, then disappear, then
reappear as compatible with std::format (as are fmt::format/fmtformat now).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The str() method was not part of the final C++17 std::string_view.
To ensure future interchangeability with std::string_view, change the
few places where we call str() in the OSL code base.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Track that the backfacing() shadeop requires N and I shader globals.
* Test ensuring that N and I are reported as needed shader globals when a backfacing() shadeop is used.
Signed-off-by: Stephen Friedman <sfriedma@pixar.com>
* Implement Batched pointcloud_search, pointcloud_get, and pointcloud_write.
Updated behavior of scalar pointcloud_search and pointcloud_get to not code gen branch with code gen runtime error for maxpoints/num_points larger than array size. Instead now it codgens a clamp of the maxpoints/num_points to the arraysize. This allows code to continue executing, and if accessed outside of the range, existing codgen range checks will still emit a runtime error.
Extracted class PointCloud defnition and some helper functions to pointcloud.h so it can be shared between batched and scalar implementation.
Moved PointCloud out of anonymous namespace to OSL::pvt so a single implementation (including static maps) could be shared between batched and scalar implementation.
Refactored implementation of
template<typename DataT, int WidthT> struct Masked;
template<typename DataT> struct Ref;
to flatten class hierarchy and fix bug where derivatives of unbounded arrays could not be constructed, IE:
MaskedDx<int[]> myDeriv(ptr, arraylength);
Added MaskedData::assign_val_lane_from_scalar
Renamed MaskedData::assign_val_from to MaskedData::assign_val_from_wide
Renamed MaskedData::assign_all_from to MaskedData::assign_all_from_scalar
Updated BatchedAnalysis to indicate that results of pointcloud_search, pointcloud_get, and pointcloud_write are always varying regardless of the symbols passed in.
Updated debugging in BatchedAnalysis to emit unmangled symbol names.
Properly implement (was broken previously) BatchedBackendLLVM::llvm_zero_derivs(const Symbol& sym, llvm::Value* count) including using code gen loop to reduce code bloat.
* Fixed warnings about BatchedRendererServices<WidthT>::pointcloud_get and BatchedRendererServices<WidthT>::pointcloud_write not returning a value.
* Fixed unused variable "ok" in batched pointcloud_write.
Fixed undefined type when building with USE_PARTIO=0
* Update pointcloud tests to avoid non-deterministic results for unsorted results and out of range max_points/num_points.
Added support for NOOPTIMIZE file tag for batched regression tests.
Made array-length-reg NOOPTIMIZE so its code gen would actually execute vs. constant fold.
Fixed bug for storing arraylength result in varying result symbol.
* Fix missing virtual function overrides of MyBatchedRendererServices in example-batched-deformer
* Change varying point cloud tests to use pixel centers in order to avoid situations where point cloud could provice multiple data points at equal distances causing non-determinism.
* More changes to avoid non-determinism in pointcloud test results
* Add alternate reference images for point cloud results that were coming back slightly different for essentially identical distances.
* Fixed bug in llvm_gen for raytype where the wrong type was passed as a parameter for a string.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
(many ported/replicated from various recent OIIO PRs)
* CI: bump to latest dependency releases
* Bugs fixed in build_opencolorio.bash, it botched some of the
directory choices when customizing where to place and build
it. Also, switch to the more standard BUILD_SHARED_LIBS cmake
variable to control whether to build dynamic or static libraries.
* Correctly handle the cmake variable VISIBILITY_MAP_FILE being set to
"" on the command line as a way to disable using the visibility map
file entirely.
* On Linux, also set link flags `--exclude-libs ALL` to hide symbols
from any embedded static library dependencies, to prevent clashes
with other versions of those libraries that may be compiled into a
DCC app.
* Some other minor touch-ups in compiler.cmake.
* Refresh build_openexr.bash with the latest changes from OIIO.
* CI workflow: revise failed build artifact storage to capture cmake
files.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
New OIIO guards the contents of its Imath.h with the
`OIIO_IMATH_H_INCLUDED` preprocessor symbol. This lets us in OSL
pre-define it to absolutely ensure that OSL headers are including
exactly the Imath headers they want, not accidentally getting the ones
that OIIO includes.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
OIIO's string_view::c_str() was cute, but it's a little dicey and it's
not present in the C++20 std::string_view. We didn't use it in a ton
of places, so replace with other idioms so that in the future we are
better able to interchange with or switch to std::string_view (when
our minimum supported C++ standard is high enough).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Here was the construct it was confused by:
struct Data {
float height;
};
Data foo(Data a, float b)
{
return Data; // <----- what's this madness?
}
Curiously, this revealed a source code error in one of our other tests,
oslc-err-struct-print, which had not been detected previously!
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Added Wide<const int, WidthT> wide_shadeindex, void* userdata_base_ptr, void* output_base_ptr to BatchedExecutor interface (and underlying implementation) that allows each data lane
of a batch to utilize a different shade index in combination with a uniform userdata and output base pointer.
Fixed bug in batched version of llvm_gen_compassign where index was being loaded as uniform value even when it was varying (really just caused assert/extra output in Release).
For scalar and batched, avoid generating code for init ops or default values when a sym_loc is found because unlike get_userdata, when supplied sym_loc can't fail.
Improved LLVM_Util::op_scatter to handle scalar destination data types, previously only wide destination were required but to handle scatter results from SOA wide data layout to AOS
output placement layout scalar destinations need to be supported.
Updated BatchedRendererServices to be less abstract replacing several pure virtual functions with default noop versions to better match behavior of RendererServices.
Changed testshade to utilize output placement by default in batched mode.
Added NOSCALAR option to testing framework to allow for batched only tests.
Added testsuite/example-batched-deformer to exercise userdata placement and simple example of utilizing batched interface.
Added alternate output for example-batched-deformer to account for some rounding differences.
Added comment to note that the batched implementation assumes that SymLocationDesc::offset is readable at point 0 (because we clamp masked off lanes to 0 vs. adding conditionals).
Renamed local variable wide_index_to_output to wide_index_for_userdata to more accurately describe how it is used.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
Implement batched versions of isnan, isinf, and isfinite.
Added batched regression testsuite ieee-fp-reg.
NOTE: its really difficult to test isnan(value) == true,
as its unclear to actually generate a NAN in OSL because
most functions are "safe" and check/avoid NAN's
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
We were passing along, in the exported cmake config files, a demand
for downstream clients to use a C++ standard version equal to that
which we used to build OSL itself. But that's not necessary. We may
choose to build with C++17 (for example), but all of our public APIs
are safe to use for C++14.
Split the build-time options to CMAKE_CXX_STANDARD for the standard we
build this project with, and DOWNSTREAM_CXX_STANDARD for the standard
we are declaring is the minimum to *use* the library.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Enable BATCHED execution of most testsuite entries that don't use closures or point cloud.
Added batched implementation of hash function.
Updated batched implementation to use the symbol.get_int(), symbol.get_float(), and symbol.get_string() methods vs. *(int*)symbol.data() approach.
Fixed bug in batched code gen where mangled symbol names were output as part of debug uninit warnings and range checking.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Fixed reparameter in batched execution.
Updated batched behavior for unlocked geom shader output parameters to function like scalar version.
Fixed batched bug when non-entry layers are executed and execution masks are compared producing a <i1x16> instead of i1 for use in conditional operation.
Added more reparameter test cases to cover arrays and arrays of aggregate types.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Enabled command line assignable user data for batched execution in testshade, which fixes testsuite/userdata-passthrough
Changed print_outputs for batched execution to not use << operator which includes parenthesis, but instead output each componenet with spaces to match the non-batched print_outputs.
renamed MaskedData<...>::assign_from(void* ptr_wide_data) to assign_val_from(const void* ptr_wide_data) to better reflect that only the value is assigned, and Dx and Dy are NOT assigned.
Added MaskedData<...>::assign_all_from(const void* ptr_data) to take a scalar DataT and broadcast it to all lanes of the MaskedData, masked off lanes are NOT overwritten, and value, Dx, Dy are assigned.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Apply clang-format to wide.h
Add some paranetheses to conditional to avoid warning.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Wrapped all printf arguments with pretty() in testsuite/geomath/test.osl to avoid small floating point differences from affecting printed results in test.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Add testsuite/geomath/ref/out-alt.txt to allow for minute floating point differences with some compiler/platform combinations.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
We used to build MaterialX shaders on our end. Now it's all done in
the MaterialX project itself, and along the way, we failed to keep in
sync with them. (And even if we had "kept up", what does that even mean?
We want OSL to be independent of any particular MX release, but the MX
shaders may change from release to release. Better to be entirely on
their side of the fence.)
We turned off building these shaders by default a long
time ago, now let's finally remove the last remnants and urge everyone
to rely on the MaterialX project for this material.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Added MaskedData::assign_from(void *ptr_wide_data) to perform a masked copy of wide data at some address.
Added enum class RenderServices::TraceOpt::LLVMMemberIndex to correlate C++ data members with LLVM type generated by BatchedBackendLLVM::llvm_type_batched_trace_options()
Fixed bug in BatchedBackendLLVM::llvm_load_arg where array types where only properly broadcasting uniform to varying arguments for the 0th element of the array.
For batched implemented trace options as a LLVM type with direct code gen to alloca, init default values, and set optional values vs. callbacks. Same code could be resused for non-batched impl.
Added BatchedMessageBuffer to context and BatchedMessageList to wide_opmessage.cpp to allow a shared buffer to be cleared by the ShadingSystem while allowing target specific compilation of SIMD import/export of message values into the list to compile in wide_opmessage.cpp.
Extended testshade's SimpleRenderer and BatchedSimpleRenderer to implement trace and getmessage to allow new tests to exercise some basic functionality.
Added new batched regression tests:
message-reg
trace-reg
Added new test: message-no-closure which is a simplied version of message with no closures
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Temporarily disable simd compilation of message import and export data to test if that is cause of crash in bleeding edge CI workflow.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Temporarily disable more simd compilation of batched message test if that is cause of crash in bleeding edge CI workflow.
Temporarily run only message-reg test for bleeding edge CI workflow
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Temporarily run only message-reg.regress.batched.opt for bleeding edge CI workflow
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Fix alignment of MessageBlock allocation so that reads/writes can be assumed aligned to Block<> data members
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Re-enable disabled simd statements from trouble shooting bleeding edge CI.
Re-neable all tests for bleeding edge workflow.
Applied clang-format to wide_opmessage.cpp
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* pybind11 2.9.0 release -- bump the vfx2022 and "latest" CI tests to
use it.
* Adjust other CI test pybind11 versions to get good coverage across all
the pybind11 releases, and to synchronize the vfx platfom tests with
the pybind11 release that was current for its respective year.
* Sync build_scripts/build_pybind11.bash with the latest in OIIO, including
bumping the default pybind11 version we download to 2.8.1.
* Opportunistic touch-up of warnings about llvm version to bring
into sync with our actual current requirements.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Among other cruft that existed to support versions 7 and 8, this lets
us completely get rid of PrePromoteLogicalOpsOnBitMasks custom pass,
which is only needed for LLVM < 8.
Also get rid of some cmake checks for LLVM versions that require
C++14 or higher -- not necessary now that C++14 is our minimum.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Add batched support for dict find, dict next, dict value.
Implementation takes advantage of uniform values and only does lookup once and broadcasts results if necessary.
When nodeid is varying, it falls back to simple loop over active data lanes performing lookups.
* Added new regression test xml-reg.
Enabled BATCHED execution of existing xml test
* Add new wide_opdictionary.cpp missed in original commit
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
Maybe somebody else is building this as a subproject or has a variable
called "CI"?
* For our CI runs, set an env variable "OSL_CI" (not just generic "CI").
* Default STOP_ON_WARNING to be off for supported releases, only on by
default for main branch where we're actively developing. Because you
want downstream users who are building supported releases to not get
tripped up when their compiler upgrades and there's a new warning.
* Make sure STOP_ON_WARNING is true for our CI runs, even for release
branches -- but now check OSL_CI, not CI, in case somebody else has
set that.
Fixes#1428
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Make PROJECT_VERSION_RELEASE_TYPE be a cache string, so individual
build configs or sites can override it.
* Make OSL_SUPPORTED_RELEASE be an overrideable option, and make
OSL_DEV_RELEASE be driven by it, rather than (possibly incorrectly)
independently set.
* Change OSL_IS_SUBPROJECT to PROJECT_IS_TOP_LEVEL (and with the opposite
sense), to match CMake 21+ behavior. And no need to set it ourselves if
using cmake 21+.
* All of the clang-format stuff should only apply if PROJECT_IS_TOP_LEVEL.
* Add a TIME_COMMANDS option to help debug performance of the build.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Add batched support for 2d texture calls, gettextureinfo.
To enable, added:
bool RenderServices::is_udim (TextureHandle *texture_handle)
TextureSystem::TextureHandle* BatchedRendererServices::resolve_udim_uniform(...,TextureSystem::TextureHandle*, float S, float T)
void BatchedRendererServices::resolve_udim(...,TextureSystem::TextureHandle*,Wide<const float> wS, Wide<const float> wT, Masked<TextureSystem::TextureHandle*> wresult)
which allows batched code gen to resolve_udim's for varying S & T and bin the resulting TextureHandle* and make calls to get_texture_info_uniform with a single resolved udim texture handle, then broadcast the results to data lanes with matching TextureHandle *'s.
Added extensive new regression tests:
gettextureinfo-reg
gettextureinfo-udim-reg
texture-opts-reg
Enabled BATCHED execution of existing tests:
gettextureinfo-udim
gettextureinfo
texture-alpha-derivs
texture-alpha
texture-blur
texture-connected-options
texture-derivs
texture-errormsg
texture-firstchannel
texture-interp
texture-missingalpha
texture-missingcolor
texture-simple
texture-smallderivs
texture-swirl
texture-udim
texture-width
texture-withderivs
texture-wrap
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Removed some commented out includes
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Add conditional compilation dependent on OIIO version to enable RendererServices::is_udim
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Fix clang-format issues
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Fix bug in testing.cmake where non-existant OPENIMAGEIO_VERSION was being checked, when it should have been OpenImageIO_VERSION.
Added quiet option to oiio-tool calls in regression tests to allow produced out.txt to match between baseline and batched runs.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Added optional silent=False argument to oiiotool in test suite, allowing us to avoid generating warnings for file already exists that was causing differences when comparing baseline/out.txt and out.txt in regression tests
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Added missing batched texture option rnd for the llvm code gen representation
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Fixed compilations issues with OIIO 2.4 by enabling "rnd" batched texture option.
Fixed some other compilation issues when OSL_DEV is defined.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Cleared up some comments around populating rnd.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Add support for batched texture3d.
Discovered mismatch between documentation, llvm_gen implementation, and oslc when passing user derivatives to texture3d.
oslc and documentation expect optional parameters "vector dpdx, vector dpdy, vector dpdz",
however llvm_gen and optexture implementation only accept "vector dpdx, vector dpdy"
and use 0 for dpdz when calling into the texture subsystem.
This commit includes changes to oslc to make the compiler match the implementation of accepting only "vector dpdx, vector dpdy".
Enable BATCHED execution of testsuite/texture3d
Add new regression test textur3d-opts-reg
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Revert oslc to make the compiler match the documentation of accepting only user inputs that include dpdz: "vector dpdx, vector dpdy, vector dpdz".
Fixed llvm_gen and optexture to properly extract user dpdz from texture3d options or initialize a Vec3(0) and pass through to texture subsystem.
Modified batched_llvm_gen and wide_optexture to properly extract user dpdz from texture3d options or initialize a Wide<Vec3>(0) and pass through to texture subsystem.
Updated texture3d-opts-reg test to pass dpdz for calls taking user derivatives.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Added batched support for environment.
Added BatchedRendererServices::environment virtual method
Added new test environment
Added new regression test environment-opt-reg
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* clang-format of batched_rendererservices.h
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Rename testsuite/environment-opts-reg to testsuite/texture-environment-opts-reg to line up with a different PR adding testsuite/texture-environment.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Removed some environment option test for "wrap" as that is an unsupported option for environment.
Added test and handling for "time" option to batched texture3d. Current choice is to just silently ignore "time" as currently supported 3d formats do not support time.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Removed redundant environment test as texture-environment covers it.
Enabled BATCHED execution of texture-environment
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Extended test timouts further for arithmetic-reg.regress.batched.opt and transform-reg.regress.batched.opt which were causing spurious CI timeouts.
Reduced some multipliers of user supplied derivs to environment tests to reduce differences in results.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
Co-authored-by: Larry Gritz <lg@larrygritz.com>
It was documnted for texture(), but not for the others, for which it
also has worked for many years.
Thanks for Alex Wells for pointing this out.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
I recently bumped the ctest per-test timeout to 300s because one of the
batched shading regression tests was occasionally taking longer than the
prior limit of 240s.
But there is a better way: Individual tests can have their timeouts
set individually via set_tests_properties(TIMEOUT), so I raised the
timeout to 300s for all tests whose names contain "-reg", also kept it
rather long (the old 240s) for the long render tests that are all named
"render-". Then for everything else, we can actually lower the default
to 60s, since no other tests should be long-running.
(For the curious, the reason these batched regression tests are so
expensive is that they test a bunch of combinations, and they first
run the test in the non-batch scalar mode, then again in batched mode,
and compare the results to ensure that the batch code never deviates
from scalar. This makes them take at least twice as long as they
ordinarily would. But also remember that these are only run for the
few CI test matrix entries where we are specifically testing batch
shading.)
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Increase testing timeout, some of the batch shading regression tests
seem to take longer than the prior 240s limit.
* Turn off OpenVDB when using the osl:2021 in combination with OpenEXR
3.x -- the version of OpenVDB in the 2021 container is too old to
understand OpenEXR 3.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Use LLVM 13 / clang 13 for the VFX Platform 2022 test.
* Add MacOS-11 test
* Poke around how to use arbitrary LLVM versions via Conan. (Works! But
I have not yet enabled it.)
Signed-off-by: Larry Gritz <lg@larrygritz.com>
OIIO master recently removed support for Field3D, which is rarely used
and the project is effectively unmaintained and isn't able to build
against the newer OpenEXR releases that we want to encourage people to
use.
So the primary purpose of this patch is to switch our unit test for
texture3d from using a field3d file (and thus would not run when using
newer OIIO) to a roughly equivalent test using openvdb (which is
supported, assuming OIIO is built in the presence of OpenVDB).
But, ha, I discovered two things, either one of which means that we
had actually not been unit testing texture3d at all lately: (1) To
discover whether the installed OIIO supported openvdb (and therefore
whether to bother running the test), we were relying on a totally
nonexistant OPENIMAGEIO_BIN cmake varaible. I don't know what that was
supposed to be (did I mean OIIOTOOL_BIN? but that also is not a
thing), but anyway, it caused it to always skip the test. (2) Oh boy,
the OIIO that we were relying on for may tests, provided by the ASWF
docker containers, was an OIIO that was not built in the presence of
OpenVDB (even though our OSL testing containers DO have OpenVDB), so
again it skipped the test because it needs that OIIO-side support for
texture3d() to know how to read openvdb files. Fix this by building
OIIO from scratch with openvdb support when doing the "VFX Platform
2022" test, rather than relying on the one in the container.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Because they can fail with some locales. Replace with OIIO::Strutil
parsing functions that are locale-independent.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Renaming the master branch to main is imminent. Here are the few
places where we refer directly to branch names that will no longer
exist.
While searching for "master/", I also noticed a couple files that
had the wrong project's copyright boilerplate on it (a lot of the
cmake scripts I write and submit to both projects simultaneously).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This is primarily for the sake of mingw, but also is the proper kind
of stream to pass to OIIO::Filesystem::open.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Use a cmake idiom so that you need an explicit -DCMAKE_INSTALL_PREFIX=
in order to write over /usr/local or any potentially sensitive area.
In the absence of this being set explicitly, it will configure the
cmake build so that an "install" ends up in ./dist (which is what the
Makefile wrapper did all along).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Added
template <int M00, int M01, int M02, int M03,
int M10, int M11, int M12, int M13,
int M20, int M21, int M22, int M23,
int M30, int M31, int M32, int M33, int DivisorT>
struct sfm::StaticMatrix44
to model a matrix whose elements are known at compile time. Internally it utilizes proxys instead of floats in order to optimize for special cases of 0, 1, -1 elements and not even generate math when possible. The StaticMatrix44 is used by wide_opspline.cpp to create specialized functions for each of the spline basis types. A dispatch function routes the compile time optimized spline or spline inverse routine.
Altered splineimpl.h to provide "int basis_type_of(StringParam basis_name)" to allow the batched implementation to map a string to spline enum value used as an index in the dispatch function to functions optimized for each type of spline basis (instead of generic 4x4 matrix implementation)
Added "T sfm::select_val(bool cond, const T left, const T right)" as alternative to the ternary operator which appeared to possibly take address of its arguments which prevented privatization of those variables when compiled inside a SIMD loop.
Added note to BatchedAnalysis implementation identifying that local variables scope is lost between .osl and .oso and when a function is inlined multiple times, its local variables could be treated as varying when they could have been uniform if somehow identified as a local variable by the Analyzer.
Enable BATCHED testing of:
spline-boundarybug
spline-derivbug
spline-nonuniformknots
spline
splineinverse-ident
splineinverse
Added new BATCHED_REGRESSION tests:
spline-reg
splineinverse-knots-ascend-reg
splineinverse-knots-descend-reg
Added sfm::Inverter to remove nested loop behavior of batched splineinverse as OIIO::invert has a loop which was called from inside the search interval loop, created a nested loop. Under SIMD/SIMT the nested loop could execute incoherently. Instead we choose to use the class sfm::Inverter to manage state between is_bracketed_by(min,max), meant to be called from the search interval loop, and bracketed_invert(min,max) meant to be called after exiting the search interval loop, avoiding a nested loop that could be executed at different interval's for each SIMD/SIMT lane/thread.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
Patrick Hodoul pointed out that we were forcing OpenColorIO to
initialize and read configs as soon as the ShadingSystem was
created. In this patch, we're changing to a lazy creation approach.
We remove the `OCIOColorSystem m_ocio_system` member and the
ocio_transform() method from ShadingSystem. Put them in the
ShadingContext instead. This allows lock-free access because each
context is used by only one thread at a time.
Within the OCIOColorSystem, make the `OIIO::ColorConfig` be a
shared_ptr rather than a member, so it can be created lazily. If the
Context's OCIOColorSystem needs a config and doesn't have its own yet,
it asks the ShadingSystem, which allocates one the first time (and
only that one time).
So, in short, SS allocates *one* shared ColorConfig, the first time
any context asks for one. The Context only asks the SS for this once
per context (i.e., once per thread). Individual shades just use the
ColorConfig of their context, not needing to lock for it (except for
that brief lock the first time any context asks for the very first
ColorConfig that's needed).
Some things were guarded by `OIIO_HAS_COLORPROCESSOR`, which is silly
because we no longer support OIIO versions so old that it's not
defined. But then OSL `undef`ed it for Cuda. I replaced these guards
by a more straightforward direct test of whether it was Cuda code.
I added a sample OCIO config to testsuite so we can use it for tests.
In particular, the new testsuite/ocio -- test using OCIO for color
transforms.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Fixes 1416
In OSL, function parameters must be marked as `output` or they are not
supposed to be writable. In issuing errors when a write is performed on
something not writable, we missed an interesting case:
```
void g(output float b) {
b = 1; // ok, since b is marked as output
}
void f(float a) {
g(a); // <--- should be disallowed, g is writing to f's non-output param
}
float x;
f(x);
```
This patch catches cases like these. The new error looks like this:
`test.osl:22: error: cannot write to non-output parameter "a", as passed to function g() as arg 1 (output parameter "b")`
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Add batched SIMD implementation for all color operations.
Added batched llvm code gen for: llvm_gen_blackbody, llvm_gen_luminance, llvm_gen_transformc (including code gen of loop to identify unique "from" and "to" space combinations to call library function with).
Made ColorSystem const correct.
Pulled much of ColorSystem's implementation into src/liboslexec/opcolor_impl.h so it could be inlined in both opcolor.cpp and wide_opcolor.cpp.
To enable SIMD fast path for ColorSystem::blackbody_rgb added new methods to expose underlying lookup table optimization.
bool can_lookup_blackbody(float T /*Kelvin*/) const;
Color3 lookup_blackbody_rgb (float T /*Kelvin*/) const;
Color3 compute_blackbody_rgb (float T /*Kelvin*/) const;
When can_lookup_blackbody() returns true
then lookup_blackbody_rgb can be safely called,
otherwise the expensive compute_blackbody_rgb is required.
Updated implementation of ColorSystem to succesfully vectorize on multiple compilers, optimized lookup table data types and indexed dereferences to generate prefered 32bit indexed gathers vs 64bit gathers which require multiple instructions and registers.
To improve code generation and less chances of aliasing issues, changed nested M[i][j] operator access to Matrix in ImathMatrix.h to directly access underlying 2d array M.x[i][j] also changed Vector operator V[i] to directly access V.x, V.y, V.z in OSL/IMathx/IMath*.h
Moved helper testIfAnyLaneIsNonZero from wide_opmatrix.cpp to be testIfAnyLaneIsNonZero in OSL/wide.h
Added sfm::min_val to provide an implementation of min which returns values not references to the original through a ternary (which can cause issues with vectorization). Added Clang specific sfm::min_val and sfm::max_val implementations to assist in vectorization.
Enabled BATCHED execution of existing testsuites: blackbody, color, transformc, wavelength_color
Added new testsuite BATCHED regression tests: blackbody-reg, color-reg, luminance-reg, transformc-reg, wavelength_color-reg
Added sfm::min_val to provide an implementation of min which returns values not references to the original through a ternary (which can cause issues with vectorization). Added Clang specific sfm::min_val and sfm::max_val implementations to assist in vectorization.
* Adopt OIIO calls to fast_cbrt and fast_expm1 for blackbody computation.
Use single precision for bb_spectrum object/operator and manually implement std::pow(wlm,-5.0f) to better enable vectorization.
Reduced masked code generation for rgb_to_hsl.
Removed duplicated code by implementing ColorSystem::blackbody_rgb(T) in terms of ColorSystem::can_lookup_blackbody(T), ColorSystem::lookup_blackbody_rgb(T), and ColorSystem::compute_blackbody_rgb(T). Removed some casting to uint16_t that was unnecessary.
Removed some test shaders that were not intended to be promoted.
* Implement batched llvm code generation for ops: andor, bitwise_binary_op, clamp, get_simple_SG_field, isconstant, select, unary_op, mix
Fixed bug in BatchedAnalysis where the complement operator was being treated as always having a boolean result. In reality the result of complement or other bitwise operation would only be boolean if its input parameters were forced to be boolean (left TODO note for future improvement).
Fixed bug in printf where integer who is forced_llvm_bool() is formatted as a float was not converted to an integer first.
Enabled BATCHED for execution for tests: and-or-not-synonyms, isconstant, logic, select.
Expanded testsuite/shaderglobals to exercise/access all shader global data members.
Add regression tests: andor-reg, bitwise-and-reg bitwise-or-reg bitwise-shl-reg
bitwise-shr-reg bitwise-xor-reg, complement-reg, mix-reg, select-reg
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
Remember that our current 'master' is really aiming for a release in
late 2021 at the earliest, and thus will be a version primarily
intended to be used widely in 2022. Release branches never drop
supported dependencies, so the 1.11 branch will continue to support
OpenEXR all the way back to 2.0.
VFX Platform guidance is to support the current year, plus up to 3 years
prior, so that means that for 2022-ish software, we should support back
to 2019 dependency ecosystem.
For OpenEXR, 2019 state of the art was 2.3 -- that's still dreadfully
old, and is no longer getting even critical or security bug fixes. But
it still means that we can comfortably upgrade our advertised minimum
OpenEXR from 2.0+ to 2.3+ and eliminate some build and dependency
logic that was dedicated to dealing with 2.0..2.2.
We're also raising the OpenEXR/Imath "recommended minimum" (build-time
warning, but not rejection) to 2.4. Even that is shaky, but the fact
that it's still in wide use and was officially (though IMHO unwisely)
the VFX Platform 2021 recommendation, means we can't yell at people
too much about it.
It is possible that by the time we make a supported OSL 1.12 release,
we will bump even further to OpenEXR 2.4 as the minimum. We'll make
that decision later. It would depend on our being confident that even
though 2.3 is technically within the 3 year window we try to support,
we cannot discern that anybody is actually using it (who would also
need the most recent OSL in the same app).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Added batched llvm code gen for: llvm_gen_blackbody, llvm_gen_luminance, llvm_gen_transformc (including code gen of loop to identify unique "from" and "to" space combinations to call library function with).
Made ColorSystem const correct.
Pulled much of ColorSystem's implementation into src/liboslexec/opcolor_impl.h so it could be inlined in both opcolor.cpp and wide_opcolor.cpp.
To enable SIMD fast path for ColorSystem::blackbody_rgb added new methods to expose underlying lookup table optimization.
bool can_lookup_blackbody(float T /*Kelvin*/) const;
Color3 lookup_blackbody_rgb (float T /*Kelvin*/) const;
Color3 compute_blackbody_rgb (float T /*Kelvin*/) const;
When can_lookup_blackbody() returns true
then lookup_blackbody_rgb can be safely called,
otherwise the expensive compute_blackbody_rgb is required.
Updated implementation of ColorSystem to succesfully vectorize on multiple compilers, optimized lookup table data types and indexed dereferences to generate prefered 32bit indexed gathers vs 64bit gathers which require multiple instructions and registers.
To improve code generation and less chances of aliasing issues, changed nested M[i][j] operator access to Matrix in ImathMatrix.h to directly access underlying 2d array M.x[i][j] also changed Vector operator V[i] to directly access V.x, V.y, V.z in OSL/IMathx/IMath*.h
Moved helper testIfAnyLaneIsNonZero from wide_opmatrix.cpp to be testIfAnyLaneIsNonZero in OSL/wide.h
Added sfm::min_val to provide an implementation of min which returns values not references to the original through a ternary (which can cause issues with vectorization). Added Clang specific sfm::min_val and sfm::max_val implementations to assist in vectorization.
Enabled BATCHED execution of existing testsuites: blackbody, color, transformc, wavelength_color
Added new testsuite BATCHED regression tests: blackbody-reg, color-reg, luminance-reg, transformc-reg, wavelength_color-reg
Added sfm::min_val to provide an implementation of min which returns values not references to the original through a ternary (which can cause issues with vectorization). Added Clang specific sfm::min_val and sfm::max_val implementations to assist in vectorization.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Adopt OIIO calls to fast_cbrt and fast_expm1 for blackbody computation.
Use single precision for bb_spectrum object/operator and manually implement std::pow(wlm,-5.0f) to better enable vectorization.
Reduced masked code generation for rgb_to_hsl.
Removed duplicated code by implementing ColorSystem::blackbody_rgb(T) in terms of ColorSystem::can_lookup_blackbody(T), ColorSystem::lookup_blackbody_rgb(T), and ColorSystem::compute_blackbody_rgb(T). Removed some casting to uint16_t that was unnecessary.
Removed some test shaders that were not intended to be promoted.
* Added additional test result for black body because the change from double to single precision caused a few results to differ slightly (max delta 0.0000534058)
* Incorporated changes from PR to "fix color conversion from RGB to xyY"
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
This change now requires renderers to manually link in the compiled rend_lib.cu
PTX code when using OptiX 7.
Signed-off-by: Clifford Stein <cstein@imageworks.com>
Changed the prototype of osl_get_attribute callback in liboslexec in
order it takes a copy of the attribute TypeDesc casted into a long long
instead of taking a pointer to it. This allows to make it work properly
on GPU else we have an invalid pointer.
Signed-off-by: nicolas <nicolas@isotropix.com>
Fix problem with unresolved symbols when multiple shaders call certain (#1401)functions on the GPU (ie. osl_printf()).
This fixes a problem I was seeingon the GPU: if I had two shaders that contained a printf(), then the second shader group would report osl_printf as being an unresolved function (the first shader group was fine). If I looked at the PTX for the second shader group, osl_printf was tagged as a "weak" function. Removing the "weak" attribute resolves the problem.
I'm not sure why osl_printf and rend_get_userdata should be different from the other functions in rend_lib.cu.
This is easily reproduced in testsuite/testoptix by adding a printf() to checkerboard.osl and microfacet_test.osl.
Signed-off-by: Clifford Stein <cstein@imageworks.com>
Added clang pragma and gcc attribute to disable Fused Multiply Add for nonAffineInverse to allow offline compilers to produce results closer to those of LLVM IR compilation (which has FMA disabled by default) which produced the reference images of the testsuite.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
Newly added functionality in TextureSystem (only in OIIO >= 2.3.7)
allows us to add a new flavor of gettextureinfo that takes texture
coordinates, and in the case of UDIM textures, will retrieve the info
for the particular UDIM texture that will be used for the given
position:
int gettextureinfo (string texturename, string paramname,
float s, float t, output TYPE destination)
NOTE: Use with caution, this will be more expensive than ordinary
calls to gettextureinfo that will tend to constant-fold effectively,
since the s,t coordinates are runtime-determined and won't be known
for runtime optimization.
If built against older versions of TextureSystem that lack support for
the new is_udim() and udim_resolve() methods, this new gettextureinfo
will simply use the old gettextureinfo() without coordinates, and thus
will not be able to retrieve position-specific texture info.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Most important: make sure we have a proper VFXPlatfom-2022 like test,
with Python 3.9 and OpenEXR/Imath 3.1.
Fix a few of the "name" entries for tests to match the set of versions
actually present on those images.
Remove apt-get install of libboost-regex-dev, since we use only
std::regex now that we're no longer using gcc4.8.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Similar to a recent change in OIIO... For historical reasons, our
wrapper makefiles assume that we build in `build/ARCH[.variant]`, so
maybe build/linux64, build/linux64.debug, etc.
Get rid of this extra bit of hierarchy, it no longer serves us and
it's contrary to modern cmake practice.
Also used this as an excuse to remove some SPI build script
customization that we no longer use.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This is analogous to the "output placement" that we added recently, but
for input of userdata.
Reminder: "userdata" means shader parameters marked as lockgeom=0 that
are expected to be supplied on a per-shade basis, such as interpolated
vertex data on geometric primitives. The primitive data binds by name to
matching shader parameters.
This patch concerns how the shader gets this per-point data.
Old school: in the middle of the shader, when userdata is needed, the
shader calls osl_bind_interpolated_param(), which makes a callback to
RendererServices::get_userdata(), which... does something magical and
renderer-specific to retrieve the data by name. This is expensive and
also challenging to implement on the GPU (which is why it remains
unimplemented).
This patch changes the methodology to the following, which is somewhat
analogous to how we did output placement:
0. It is assumed that there is an arena of memory where the userdata
is going to be, and the pointer to its start is passed down to the
shader via execute() (just like how the shaderglobals and output
arena pointers are passed), along with an index of the current
shade point.
1. The renderer pre-declares SymLocationDesc records for the userdata,
which specifies the offsets and strides within the arena where the
data can be found.
2. Prior to launching the shader execute(), the renderer is expected
fill in those values in the userdata arena for the points to be
shaded.
3. When JITing the part of the shader that initializes those userdata
parameter values, we simply JIT a simple data copy directly from
the computed position within the arena (offsetting the right amount
for this data, at this shade index) -- just a couple of
instructions per shade, no function calls, no RendererServices
callbacks, no name lookups!
4. There is no 4. That's it.
See the changes to osldeformer.cpp for an example of how this looks from
the renderer side. It's really straightforward.
Currently, if there is no SymLocationDesc for a lockgeom=0 parameter
that needs initialization, it will fall back to the old callback. We
may fully deprecate the old way, but not for a while, so this is back
compatible for now.
This seems to work, passes all tests, doesn't break anything, and works
for the osldeformer example!
There is follow-up work to do: I think we can pretty easily do the
same thing for ShaderGlobals ("make your own SG struct"), named marix
retrieval, and other attributes. Will do those in subsequent PRs if
this meets everybody's approval.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Well intentioned attempt to ensure that Cuda runtime library was in
the library path when tests were run, but it clobbered the existing
LD_LIBRARY_PATH.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Add batched code gen to generate a loop to calls to OSL library noise functions with binning of varying noisetypes and options. For example if half of a SIMD batch uses gabor option "bandwidth" of 10 and the other half default "bandwidth" of 1, then gabornoise will be called twice (once with bandwith option set to 10 and a mask with the 1st half of SIMD lanes enabled, then a 2nd time with bandwith option set to 1 and a mask with the 2nd half of SIMD lanes enabled.
Added batched OSL noise library functions which compile with outer loop explicit vectorization. NOTE: Perlin and some other noises have SIMD intrinsics that work on multiple data elements (block vectorization). When the the active number of lanes in a batch is below a tunable threshold, the legacy block vectorization version is called. This avoids the worst case scenarios of 1 active SIMD lane executing, and allows the 3-4 wide x,y,z data element version to run instead.
Add previosly omitted /Qopenmp-simd and -fopenmp compiler flags along with -DOSL_OPENMP_SIMD to cause OpenMP outer loop explicit SIMD vectorization of loops in batched OSL libraries.
Fix Mask::invoke_foreach to be OSL_NOINLINE, it was mistakenly set to OSL_FORCEINLINE
Update oslnoise.h to access target specific library sfm namespace
Improve compile time for 4d perlin noise with Dual arguments by using a 2 iteration loops to avoid repeating a large amount of code.
Changed ShadingSystemImpl::count_noise(void) to count_noise (int number=1) to allow batched version to update the call count with the number of active lanes in the current mask.
Fixed sfm::scalar_gabor3 and sfm::scalar_pgabor3 to not use the loop array index when storing partial results, when vectorizing an aliasing issue on the stack caused incorrect results.
Enabled BATCHED execution of all noise tests.
Added new tests: noise-generic, pnoise-generic
Added new BATCHED_REGRESSION tests: noise-gabor-reg, noise-reg, pnoise-reg
Avoid issue with constant folding in hashnoise with different compilers causing slightly different results from sin & cos to cause massively different results due to aliasing in hashnoise.
* Add openmp-simd and target compiler flags when CMAKE_COMPILER_IS_GNUCC for batched target specific liboslexec libraries
Move filter_gabor_kernel_2d from gabornoise.h to .cpp to avoid unused function warning.
* Revert to reference 4d perlin noise for Clang vectorization.
* Utilize reference filter_gabor_kernel_2d which uses its own gabor_mul_m22_v2(const Matrix22& m, const Vec2& v) instead of relying on Imath version which differs in naming between OSL/matrix22.h and <Imath/ImathMatrix.h>
Move filter_gabor_kernel_2d back from gabornoise.cpp to .h to share it between gabornoise.cpp and sfm_gabornoise.h.
Disable Clang vectorization of strlen because resulting binary causes crashes in testsuite (TODO: investigate)
* To avoid "may be used uninitialized in this function [-Werror=maybe-uninitialized]" messages add extra (unnecessary) initialization, as its a unlikely fallback case shouldn't affect performance much.
* Disabled GCC optimizations for nonAffineInverse to allow results to match LLVM-IR version for Matrix44 inversion.
Implemented better optimization for transform normal when matrix is uniform.
* Mark nonAffineInverse definition as OSL_HOSTDEVICE to match its forward declaration.
* Added environment variable TESTSHADE_LLVM_JIT_FMA to force FMA generation for JIT'd code, needed to produce reference i mages for tests.
Fixed warning message that was accessing a null pointer when testshade requests FMA without specifiying a target.
Added out_LLVM_JIT_FMA.tif image for hashnoise tests whose results vastly differ for small changes in input values, so we need a 2nd test image to compare against when FMA is enabled for LLVM JIT.
* Rearranged local stack variables inside of sfm::simplexnoise4 to instantiate Vec4 locals closer where they are used. This was necessary to get correct results with gcc (even with vectorization disabled), suspect issue with register overflow or stack stomping.
* Fix issue with impl_transform_normal_masked failing to vectorize when only the result is varying.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
OIIO 2.3 will rearrange some header files slightly. If you need to use
ImageCache, you must include imagecache.h (duh!). We were accidentally
relying on some leaked definitions in other header files, which will
soon be removed.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Implement batched versions algebraic, trigonometric, and transcendental OSL library functions.
Add extensive regression tests to exercise binary and ternary operations with combinations of uniform, varying, and constant arguments.
Added some reusable xmacros for shaders in the testsuite/common/shader's directory to create unary, binary, and ternary test shaders.
Increase timeout of CI tests from 180 seconds to 240 seconds to allow comprehensive regression tests to complete.
Alternative is to break regression tests into separate test directories so the individually finish faster.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
This feature (used, very possibly, only by me, for debugging) lets me
force OSL to write an archive that contains the oso files and a
serialized version of the shader network.
If no explicit name is given (by the "archive_filename" attribute), it
picks a name based on the shader group name. But that may not be a
safe string for a valid filename. We already eliminated anything
before the last slash, but it turns out that a colon (':') and pipe ('|')
can also really mess things up. So this small change protects against that.
For the actual commands sent to system(), escape special characters
in the strings and enclose any arguments that come from the user in
double quotes. That prevents shenanigans like asking for an archive
filename called "; rm -r *". By enclosing in double quotes, we'll
end up with a badly named filename produced by tar, rather than
possibly an arbitrary command being executed.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Background: OIIO now has errorf that are explicitly printf-like, and
errorfmt that are explicitly std::format-like, and also an old style
error() that is ambiguous. The ambiguous calls are (at this moment)
aliases to the printf ones, but will be deprecated at some point (and
may, later when it seems safe, get reborn as the std::format compliant
behavior).
This roots out and removes the last few calls to the old ambiguous
ones from the OSL codebase.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Update the "bleeding edge" CI test to gcc-11, and also LLVM 12,
which we hadn't tested before.
* A "latest releases" CI test uses gcc10 and LLVM 11.
* Turns out that nothing else on the OSL side needed to be fixed to
accommodate gcc 11 (though you need a very up-to-date OIIO if you
want to compile it with gcc11 unless you turn warnings off).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Implement batched matrix ops, including transform, determinant, transpose, and construction and getmatrix with spaces, multiplication, division, negation.
Moved helpers det2x2, det3x3, and det4x4 from opmatrix.cpp to Imathx.h so they could be shared with the wide_opmatrix.cpp.
Enabled BATCHED testing for matrix and transform testsuites
Added new BATCHED_REGRESSION testsuites array-reg, matrix-arithmetic-reg, matrix-compref-reg, matrix-reg, transform-reg to more fully exercise combinations of uniform and varying data types and indexing variables.
* Removed #if 0 code block of determinant match that was moved to Imathx.h
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
As discussed at the TSC meeting and on the mail list, I won't merge this
change until June 1. It will only be in master (future OSL 1.12), not
backported to any release branch.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Implement batched loop control flow (including, break, continue) along with early outs (return, exit). Also add batched support for printf and other string operations. Implement batched support for shadingsystem ops like bind_interpolated_param, get_userdata, rangecheck, uninitcheck_values, and nancheck. Implement batched support for negate, min/max, arraycopy, compassign, mxcompref, mxcompassign, .Added BATCHED_REGRESSION tests to the suite to exercise control flow and string ops and compare scaler and batched results. Enabled various tests to also execute in BATCHED mode. Enable unoptimized BATCHED mode testing, as constfolding was avoiding any work in optimized mode for some tests.
Fixed bug in testshaded batched mode that inverted population of x and y shader global values, but only noticed in order of side effects like printf in shaders.
Added buffered file support to ShadingSystem to allow batched fprint to write out to files in the correct order.
Fixed bug in BatchedAnalysis which was not handling unlocked geom correctly by requiring unlocked input parameters be varying inorder to accept Wide results from BatchedRendererServices::get_userdata.
LLVM_Util had previously added support managing masks, and its strategy should
be reviewed before trying to understand this commit's changes:
// Overview of masking strategy
//
// For if/then based on a conditional, the result of the conditional is
// conditionals written to a symbol that is a wide boolean. To handle
// nesting we will keep a stack of conditional masks. When a new
// conditional mask is pushed, it is combined with the previous mask on
// the top of the stack. It is the top of the mask stack that is used
// as the mask for any instructions requiring masking. When a
// conditional scope is exited, we just pop the top of the conditional
// mask stack.
//
// Loops are handled by pushing their conditional result onto the
// conditional mask stack, and popping it back off at the end of the
// loop.
//
// Handling of early outs (exit, return, break, continue) is done by
// storing a mask representing the lanes which are going to be
// deactivated (or continue executing) to an allocated location in the
// stack. We actually have to store these early out masks, so that we
// can apply it to the conditional mask stack. As that stack unwinds,
// any early out masks are applied to the top of the conditional mask
// stack. As there are different types of early outs that apply to
// different control flow scopes and multiple types can be present at
// the same time for different data lanes, we must store these early
// out masks at separate locations on the stack.
//
// To handle 'exit', we allocate a 'shader mask' that is initially
// populated with a startMaskValue representing the active lanes in a
// shader global batch. For example perhaps only 7 of 16 data lanes
// are populated, the startMaskValue would store that value and push it
// onto the conditional mask stack. We require a location on the stack
// for this 'shader mask' in order for an 'exit' statement to modify it
// and cause a subset of the lanes to be deactivated. If an exit was
// processed, the 'shader mask' is where it will be stored and pulled
// from when applying to the unwinding conditional mask stack.
//
// To handle 'return', we allocate a 'function mask' that is initially
// populated with a startMaskValue representing the active lanes when
// an inlined function was called. This startMaskValue should just be
// value at the top of the conditional mask stack. As function calls
// can be nested, we maintain a stack of function masks. We require a
// location on the stack for each 'function mask' in order for a
// 'return' statement to modify it and cause a subset of the lanes to
// be deactivated. If an return was processed, the 'return mask' is
// where it will be stored and pulled from when applying to the
// unwinding conditional mask stack. When a function scope is exited,
// the function mask is popped so any return's processed would apply to
// the outer function mask vs. the one just exited.
//
// To handle 'break', we use the condition symbol's location on the
// stack to directly disable data lanes for the loop execution. As we
// already had a location to store the conditional symbol's mask, no
// need to allocate or store a separate one. However code will need to
// reload the condition's value when a break 'could have been' called
// vs. relying on a conditional result already in a register.
//
// To handle 'continue', we allocate a 'continue mask' that is
// initially populated with a 0 representing the deactived lanes by
// 'continue' operations. We require a location on the stack for each
// 'continue mask' in order for a 'continue' statement to modify it and
// cause a subset of the lanes to be deactivated. If an continue was
// processed, the 'continue mask' is where it will be stored and pulled
// from when applying to the unwinding conditional mask stack. When a
// loop's scope is exited, the continue mask is popped so it would't be
// applied to an outer loop.
//
// The logic of when and how to actually apply these masks is handled
// by the caller, tracking/storing the different stacks and masks is
// handled here. And queries exist to see how many early outs
// operations have been processed, these counts can be used by the
// caller to determine which to generate code to apply each of them.
In our case BatchedBackendLLVM is the caller and during code gen will be calling:
push_shader_instance(llvm::Value* startMaskValue) and pop_shader_instance()
push_function_mask(llvm::Value* startMaskValue) and pop_function_mask()
push_masked_loop(llvm::Value* location_of_control_mask, llvm::Value* location_of_continue_mask) and pop_masked_loop()
And to handle if/then/else be calling
push_mask(llvm::Value *mask, bool negate, bool absolute) and pop_mask()
By checking masked_exit_count(), masked_return_count(), masked_break_count(), and masked_continue_count()
before and after code gen of a function call, then block, else block, loop body, etc.
we can detect if a exit, return, break or continue existed in the generated scope
and then can call apply_exit_to_mask_stack(), apply_return_to_mask_stack(),
apply_break_to_mask_stack(), or apply_continue_to_mask_stack() when needed.
Add utility function "foreach_unique" to provide scalar execution of a Functor
for each unique value in the a Wide data block along with a mask identify which
data lanes match the unique value.
Add utility function "unproxy(const T &elem)" to extract value type from a proxied
element. This is alternative to static_cast<ValueType>(elem), and better self documents.
Fixed compatibility bug where llvm 9+ asserted that bool was not a valid datatype for llvm::ConstantDataVector::getSplat.
Added LLVM_Util::test_mask_lane(llvm::Value *mask, llvm::Value * lane_index) to test a mask with a variable lane index.
* Updated OIIO::vformat calls to OIIO::sprintf.
Conditionally compile buffered_file_output_mutex to avoid unused variable warning when not building with USE_BATCHED.
Ran couple files through clang-format.
* Initialize local variable mask_slot to avoid maybe-uninitialized errors
* Fixed paths to additional oiiodiff that runs when a test fails to find a match to use the /baseline subdir when performing a BATCHED_REGRESSION test.
Improved control-flow-test to use 0.5 default value instead of 0 to tell difference between actual assignment and 0's on the heap.
* Changed some control-flow-reg tests to use non-zero default values to tell difference between uninitialized heap and setting of default values.
Changed order of parameters so that any initops that contain an "exit" are the final parameter so the exit doesn't skip assigning default values. Otherwise those parameters would be uninitialized data and cause intermitent CI failures.
* Deleted extraneious cmakefile settings for clang
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* "SOVERSION" option was set twice, oops!
* Stray "OIIO" in a name that should have been this project's name.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This PR adds closure declarations for the MaterialX PBS nodes. A discussion of their design has been ongoing in a thread at the MaterialX GitHub repo. The full history of this discussion can be found here: https://github.com/materialx/MaterialX/pull/614
Signed-off-by: Niklas Harrysson <niklas.harrysson@gmail.com>
This fixes CI in the Mac test. Something about the Homebrew upgrade to llvm 12.0.0_1 made this fix necessary to build.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Up until now, client apps declare which output variables are "renderer
outputs" before shader groups are compiled, and also after a shader is
executed, it is up to the app to extract the individual output values
and do with them what it needs to do. Those outputs live on the shader
heap (the temp storage used to run the shader), and information about
where it lives on the heap can be retrieved via find_symbol() and
symbol_address().
New paradigm:
Various inputs and outputs of shaders live in different memory arenas,
and for those inputs/outputs, the app declares (prior to shader group
compilation) which arena it lives in, its offset within the arena, and
the stride between entries (for different shades, such as different
rays in flight, different points on a mesh, pixels in an image, etc.).
The shader JITs into code that automatically pull inputs from their
sources and copy outputs to their destinations with no need for client
app code to call find_symbol or symbol_address or do any extra data
copies.
This symbol location data is communicated to the shading system through
new calls to add_symlocs() passing a new struct called a SymLocationDesc,
which has the aformentioned arena, offset, stride, etc.
Right now, the only supported arena is Outputs, and the actual call
signature to execute(), and to the shader JIT code itself, includes
extra parameters for the base pointer to the output arena (this could
conceivably change from shade to shade) and the integer shade index of
this shade.
I've tried to set this up in a general way, so that additional arenas
will be added later -- places where ShaderGlobals can be customized,
UserData, etc. Also there are placeholders for being able to talk about
absolute addreses and offsets into the heap, though I don't use them
at the moment. Some of the flexibility may be later removed as we learn
more about how we want to use this.
To show it in action, I have modified the "osldeformer" example to use
this new method to place the outputs directly where we want them.
It's just storing results into an array. The output arena base pointer
is the start of the array, the offset of the single output is 0, and
the stride is the size of each array element (one 3-float point).
As a much more complex example, testshade also uses the new technique
(unless you use --no-output-placement, but I'm not sure how long I'll
keep that). This is a good example of storing to images. Testshade thinks
of itself as shading pixels in an image, so there is an ImageBuf for every
renderer output (think of it like an AOV). The output arena base pointer
is the address of the (0,0) pixel of the ImageBuf for the first AOV; the
"offset" of each output variable is the address difference between that
AOV's first ImageBuf pixel and the arena base pointer, the stride is the
pixel data size for each output, and the shadeindex is the pixel index of
that shade.
It all works. Crazy!
OK, not quite all. Here's what hasn't been done yet:
* testrender does not yet use the new API (only testshade)
* GPU (have only tried CPU so far)
* Batch shading -- I don't want to mess with this while Alex is
still integrating all his work. We'll go back and catch the batch
shading up after he's all done.
These will come in subsequent PRs.
All the old calls should still work, so this ought not to be a source
compatibility break for renderers using OSL. Not at this point, anyway.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Left out some imath targets in liboslnoise
* Conform a couple libraries to conventions of the ${local_lib} notation
we use in the othe rlibraries.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This leaves the minimum C++ at 11, but bumps the default (if not set by
the user via CMAKE_CXX_STANDARD) from 11 to 14.
This is meant to start identifying any problems people have switching
to 14. We anticipate a hard switch to 14 minimum before the next OSL
major release. This change is in master only (i.e. future OSL), we would
never change minimum or default C++ standard in the supported release
branches (currently OSL 1.11.x).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Introduce cmake option USE_BATCHED to enabling building batched SIMD shader execution for a comma separated list of batch sizes, ISA's, and noFMA: (0, b8_AVX, b8_AVX2, b8_AVX2_noFMA, b8_AVX512, b8_AVX512_noFMA, b16_AVX512, b16_AVX512_noFMA)
When USE_BATCHED is set, -DOSL_USE_BATCHED=1 will be passed allowing for source files to conditionally compile in batched execution support.
When USE_BATCHED is set, cmake variable BUILD_BATCHED=True and projects can conditionally add batched sources to be built.
Updated OSL source with #ifdef OSL_USE_BATCHED/#endif were needed.
Improved/streamlined liboslexec/CMakeLists.txt to iterate over targets in USE_BATCHED vs. hardcoding all variants.
Updated _OSL_SUPPORTS_* defines to match USE_BATCHED values simplying creating compiler definitions in Cmake.
Added new BATCHED_REGRESSION testing mode, when a file named BATCHED_REGRESSION exists in a testsuite folder, the run.py will be executed and its output files copied to "./baseline". The same run.py is then run again but with environment variable TESTSHADE_BATCHED=1, which cause testshade to execute in batched mode. Then the results/image comparison happens between the batched output files and those in "./baseline". The "./ref" dir is never used in a regression test.
Presence of BATCHED_REGRESSION prevents non-regression tests from being created for a folder.
Converted arithmetic-cov to be a BATCHED_REGRESSION test.
Removed testsuite/arithmetic-cov/ref image files.
Renamed ShadingSystem::supports_batch_execution_at(int width) to ShadingSystem::configure_batch_execution_at(int width) to better identify its behavior: based on currently set attributes for llvm_jit_target and llvm_jit_fma, test if current machine is capable of supporting batched execution at the specified width. If no specific llvm_jit_target was requested, sets llvm_jit_target and llvm_jit_fma to the supported configuration for the requested width. Returns true if supported, false otherwise
Fix bug in batched Matrix*Matrix code generation where the order of arguments were being reversed (swap was intended to transform float*Matrix to a Matrix*float function call, but accidentally happened for Matrix*Matrix)
* Moved define of OSL_USE_BATCHED to oslconfig.h.in
Fixed order of includes in batched*.cpp so oslconfig.h is higher
* Renamed testsuite/arithmetic-cov to testsuite/arithmetic-reg to better represent it is a regression test.
Changed #ifdef OSL_USE_BATCHED to #if OSL_USE_BATCHED and rely on it being set to 0 or 1 by <OSL/oslconfig.h>
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
Generating code for prepare_closure and gen_closure was crashing.
The code now matches the implementation of the removed LLVM function.
Signed-off-by: Brecht Van Lommel <brecht@blender.org>
Ever since adding OpenEXR 3.0 support, we had some cases where
installations that built the 2.x IlmBase and OpenEXR portions
separately and installing to separate areas, had trouble being found
properly.
Remove the need to include OpenEXR/OpenEXRConfig.h
Also make some minor changes to ci.yml to make sure we are documenting
the right openexr versions used for each test.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
It seems that our liberal use of cmake argument `--config` should only
be used in conjunction with the `--build` flag, not for the initial
configuration step. Apparently this was just silently ignored for a
long time, but for CMake 3.20, it is an error and breaks a lot of our
CI cases and our build_blah.bash scripts.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Added building of ISA specific shared libraries lib_b16_AVX512_oslexec, lib_b8_AVX2_oslexec, lib_b8_AVX512_oslexec, lib_b8_AVX_oslexec to house precompiled OSL library functions that execute over batches of 8 or 16 in SIMD for the ISA. Compiler flags for OpenMP simd code gen and ISA targets has been added for Intel(r) C++ Compiler (ICC) and CLang (newer versions of GCC 6+ might be possible, but untested).
Implement batched llvm code gen for: generic function calls, useparam, compare ops, addition, subtraction, multiplication, division, modulus, assignment, component reference, construct triple, construct color, derivative extraction. Stubbed out all other code gen functions with TBD asserts.
Populate OpDescriptiors with valid wide version of llvm-generating routine
Added wide_opalgebraic.cpp which uses X-macros (instead of #define like llvm_ops.cpp) to define wide(batched) versions of OSL library functions: sqrt, inversesqrt, floor, ceil, trunc, round, sign, abs, fabs, fmod, and step.
The X-macro wrappers follow a pattern of manufacturing a target specific library function name with enough parameter types embeded in its name to uniquely identify it (vs. other versions). Then it declares local Wide<T> or Masked<T> wrappers that convert any void */char * parameters to references to Block<T,WidthT> data blocks of wide SOA data. Then an explicit OpenMP simd loop iterates over the data lanes and extract a local scalar values from the Wide|Masked wrappers, then the scalar implementation of the library function is then inlined using the local scalar values. Finally the result is writen back out to the data lane inside the Wide|Masked wrapper. This paradigm allows scalar implementations to be resused inside simd loops and avoid having to use intrinsics or assembly. It also allows the same implementation to be recompiled for different target ISA's and varius Widths (8|16). The build system will create copy of each wide_*.cpp to a target and batch size specific named b(8|16)_(AVX512|AVX2|AVX)_wide*.cpp and build it with different -D__OSL_TARGET_ISA and -D__OSL_WIDTH values which inturn will manufacture uniquie function names. Sometimes scalar algorithms/functions can be refactored to provide better performance when executing inside a SIMD loop. sfmath.h (SIMD friendly math) houses these alternative math functions, although many improvements have already been moved into OIIO as they benefit (or do no harm) to scalar code gen.
Made ShadingContext remember the ShaderGroup it just optimized. This allows symbol queries without actually jitting or executing a shader.
Improved TestShade to not actually execute the shader during setup_output_images, but to instead explicitly JIT scalar or batched version of the ShaderGroup (primarily to make sure JIT happens during the "setup" stage vs. lazily later).
Fix TestShade to explicitly set the number of OIIO worker threads to avoid overhead (and debugging confusion) of OIIO thread pools being created even when "-t 1" was requested.
Modified ShadingSystem to only perform group_post_jit_cleanup (delete operations of shader group) only if both scalar and wide JIT's have occured or if RendererServices doesn't support batching. Without this changed the operations were being deleted before a batched JIT could occur.
Extended testsuite framework to look for file named "BATCHED" which causes another run of the test with TESTSHADE_BATCHED=1
Added testsuite new tests with BATCHED enabled for passing tgh shaderglobal values, and increased coverage of arithmetic tests with reference images for float, color, point, vector, normal data types along with Dx Dy results.
Added utility macros __OSL_CONCAT, __OSL_CONCAT3, ..., __OSL_CONCAT10 to be able to easily manufacture function names.
Added macro __OSL_WIDE_PVT to give each target specific library its own namespace avoiding collisions should multiple libraries be loaded.
Added sfm::negate(const T &x) with optimized implementation.
Disabled some unreferenced functions warnings for ICC and removed some unused functions from batched_analysis.cpp
Updated BatchedBackendLLVM to match behavior of BackendLLVM by configuring its LLVM_Util based on ShadingSystem attributes.
Disable clang format for X macro based building of initializer arrays to prevent clang format from reordering the #include files.
Fix control flow in factory function TargetLibraryHelper::build to not trigger assert unnecessarily.
Limit list of OSL library functions in builtindecl_wide_xmacro to just those we have implemented so far because all functions listed must exist in the target specific library for it to successfully be loaded and resolved.
Added LLVM_Util::op_zero_if(llvm::Value *cond, llvm::Value *v) which allows its implementation to work around an LLVM issue where expensive instructions to produce the value (div, sqrt, etc) are duplicated (once with a mask, once without).
Fix bug in ShadingSystem::supports_batch_execution_at where jit_fma was being accidentally negated causing rest of logic to fail.
Implement ShadingSystem::BatchedExecutor<WidthT>::jit_group
* In Cmake, changed ISA specific target libraries from SHARED to MODULE so that on OSX they will be named .so instead of .dylib.
Moved new arithmetic tests into their own subdirectory arithmetic-cov so that the existing arithmetic test can run in non-batched mode as it uses printf which is still TBD for the batched version.
Fixed bug where shader ops weren't being cleaned up unless batched jit had occured.
Fixed compilation issue with newer LLVM versions and LLVMUtil::zero_if
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
@@ -5,13 +5,14 @@ This is a checklist for TSC's to review as part of the intake process. The TSC s
- Existing Project Governance
- [X] Project License exists ( [LICENSE.md](../LICENSE.md) ) and is OSI-approved
- [X] Any third-party components/dependencies included are listed along with their licenses ( [THIRD_PARTY.md](../THIRD_PARTY.md) )
- [X] Governance defined, outlining community roles and how decsions are made ( [GOVERNANCE.md](../GOVERNANCE.md] )
- [X] Governance defined, outlining community roles and how decisions are made ( [GOVERNANCE.md](../GOVERNANCE.md) )
- Closed by https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/pull/1137
- [X] Contribution Policy defined ( [CONTRIBUTING.md](../CONTRIBUTING.md) )
- Closed by https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/pull/1137
- [X] Code of Conduct defined ( existing or pull from [ASWF Sample Project](https://github.com/AcademySoftwareFoundation/aswf-sample-project/blob/master/CODE_OF_CONDUCT.md) )
- Closed by https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/pull/1137
- [] Release methodology defined ( [release.md](process/release.md) )
- [X] Release methodology defined ( [RELEASING.md](../docs/RELEASING.md) )
- Closed by https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/pull/1572
- New Project Governance
- [X] TSC members identified
- [X] First TSC meeting held
@@ -41,7 +42,7 @@ This is a checklist for TSC's to review as part of the intake process. The TSC s
- [X] New project announcement done ( https://www.aswf.io/osl-joins-aswf/ )
- [X] Project added to ASWF website and ASWF landscape
- Graduation
- []CII Badgeachieved
- [X]OpenSSF Best Practices Badgeachieved
- [ ] Committer Diversity established
- [ ] Commit/Contribution growth during incubation
- [ ] Committers defined in the project ( [COMMITTERS.csv](COMMITTERS.csv) or [COMMITTERS.yml](COMMITTERS.yml) )
@@ -50,4 +51,4 @@ This is a checklist for TSC's to review as part of the intake process. The TSC s
- [X] License scan completed and no issues found
- Minor issues were found but already fixed in PR https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/pull/1145
- [X] Code repository imported to ASWF GitHub organization
- [] Developer Certificate of Origin past commit signoff done and DCO Probot enabled.
- [X] Developer Certificate of Origin post commit signoff done and DCO Probot enabled.
- *fix/batched*: Assume BatchedRendererServices texture derivatives are in st space. [#1828](https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/1828) (by sfriedmapixar) (1.13.10.0)
- *fix*: Make backfacing shadeop indicate backfacing shader-global is needed [#1827](https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/1827) (by sfriedmapixar) (1.13.10.0)
- *deps*: Handle various OpenImageIO deprecations and changes in the run-up to OIIO 3.0 [#1834](https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/1834) [#1806](https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/1806) [#1838](https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/1838) [#1842](https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/1842) [#1850](https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/1850) (1.13.10.0)
- *deps*: Adjust to OIIO change to IC/TS API [#1848](https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/1848) (1.13.10.0)
- *deps*: Make finding bison work better on Apple Silicon Macs + homebrew [#1822](https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/1822) (1.13.10.0)
- *ci*: Mods to CI to deal with OIIO master raising dependency mins [#1833](https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/1833) (1.13.10.0)
- *ci*: Deal with CentOS 7 EOL and disappearance of yum mirrors [#1839](https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/1839) (1.13.10.0)
- *ci*: Fix GHA CI after they upgraded nodejs [#1837](https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/1837) (1.13.10.0)
- *admin*: Change release notice icon to use the icon vs the normal logo so it shows up better [#1777](https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/1777) (by John Mertic) (1.13.10.0)
Release 1.13.10.0 -- 1 June 2024 (compared to 1.13.9.0)
- *testrender*: Implement new Oren-Nayar model from OpenPBR [#1817](https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/pull/1817) (by Christopher Kulla)
- *docs*: Fix outdated or wrong repo URLs [#1811](https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/pull/1811)
Release 1.13.9.0 -- 1 May 2024 (compared to 1.13.8.0)
- *deps*: Support for LLVM-18 [#1773](https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/1773) (by مهدي شينون (Mehdi Chinoune) [#1803](https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/1803) (by Larry Gritz)
- *fix*: Make isconnected() work with downstream renderer "connections." [#1782](https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/1782) (by sfriedmapixar)
- *cleanup*: Change OSL_CONSTEXPR14 to constexpr [#1805](https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/1805)
* *fix(batched)*: Codegen bug for compref with varying index for batched shading [#1776](https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/1776) (by Alex Wells)
* *fix*: Print closure missing error message at compile time instead of run time. [#1781](https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/1781) (by sfriedmapixar)
* *int*: Change long-deprecated OIIO macro to new one [#1788](https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/1788)
* *ci*: Fix broken Macos-13 CI [#1780](https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/1780)
Release 1.13.7.0 -- 1 Mar 2024 (compared to 1.13.6.1)
### New minimum dependencies and compatibility changes:
* For GPU/OptiX mode, OptiX 6.0 support has been removed. For GPU rendering
with OptiX, a minimum of OptiX 7.0 is required, and the Cuda minimum has
been raised to 9.0 [#1737](https://github.com/AcademySoftwareFoundation/OpenShadingLangauge/pull/1737)
* The minimum version OpenImageIO has been raised from 2.2 to 2.4. [#1591](https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/pull/1591) [#1722](https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/pull/1722)
* The minimum CMake dependency has been raised from 3.12 to 3.15
* Add type information for needed attributes. [#1650](https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/pull/1650) (by Curtis Black) (1.13.4.0)
* Add API for building attribute getter free functions.
* Add missing members to ShaderGlobals in rend_lib.h [#1721](https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/pull/1721) (by Tim Grant) (1.13.5.0)
* OptiX direct callable API that owns groupdata buffer
[#1683](https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/pull/1683) (by Chris Hellmuth) (1.13.5.0)
* OptiX PTX pipeline overhaul [#1680](https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/pull/1680) (by Tim Grant) (1.13.5.0)
* Perf: Don't insert redundant run layer calls inside a basic block
[#1665](https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/pull/1665) [#1669](https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/pull/1669) [#1672](https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/pull/1672) (by Chris Hellmuth) (1.13.4.0)
* Fix NVPTX TargetMachine leak, etc. [#1763](https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/1763) (by Tim Grant)
### 🐛/🔧 Internals: fixes, improvements, and developer concerns
* Fix memory leak in oslc ASTvariable_declaration. #1576 (1.13.0.2)
* Fix memory leaks in LPE code in error conditions. #1593#1594 (1.13.0.3)
* Fix mismatched strides when calling OCIO for transformc with derivs [#1646](https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/pull/1646) (1.13.4.0)
* Set up ray types for testrender [#1648](https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/pull/1648) (1.13.4.0)
* Fix constant float values being converted to ints [#1674](https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/pull/1674) (by Declan Russell) (1.13.4.0)
* Don't try to ReParameter symbols not in the group [#1693](https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/pull/1693) (by Alejandro Conty) (1.13.5.0)
* Have ReParameter only copy data when it changes [#1698](https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/pull/1698) (1.13.5.0)
* Avoid deprecated OSLQuery ctr in test example [#1699](https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/pull/1699) (1.13.5.0)
* Zero derivs for interactive params when needed [#1700](https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/pull/1700) (by Alejandro Conty) (1.13.5.0)
* oslc bug fix: shader params whose default values involve "init ops" could be
incorrectly initialized if their values came from instance parameters set to
* The recently added SS::find_symloc can have const args [#1723](https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/pull/1723) (1.13.5.0)
* SymOverrideInfo bitfields should be the same type [#1745](https://github.com/AcademySoftwareFoundation/OpenShadingLangauge/pull/1745)
* Fix warnings about Qt compilation and enums [#1661](https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/pull/1661) (1.13.4.0)
* Changes to support fmt 10.0 [#1691](https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/pull/1691) (1.13.5.0)
* Raise OpenEXR minimum to 2.4 [#1734](https://github.com/AcademySoftwareFoundation/OpenShadingLangauge/pull/1734)
* Fix Partio build errors on pointcloud functions for MSVC [#1757](https://github.com/AcademySoftwareFoundation/OpenShadingLangauge/pull/1757) (by Alex Wells)
* Bump Cuda minimum to 9.0 [#1737](https://github.com/AcademySoftwareFoundation/OpenShadingLangauge/pull/1737)
* LLVM 16 and 17 support: LLVM opaque pointers support [#1728](https://github.com/AcademySoftwareFoundation/OpenShadingLangauge/pull/1728),
LLVM new pass manager support [#1729](https://github.com/AcademySoftwareFoundation/OpenShadingLangauge/pull/1729),
finish LLVM 16 and 17 support [#1730](https://github.com/AcademySoftwareFoundation/OpenShadingLangauge/pull/1730) (all by Brecht Van Lommel)
* Testing and Continuous integration (CI) systems:
* Dynamic analysis using address and leak sanitizers. #1581 (1.13.0.3)
*`testshade --help` now prints TextureSystem options and all hardware
information. #1584 (1.13.0.3)
* Improvements to SonarCloud and coverage analysis. #1596 (1.13.0.3)
#1607 (1.13.1.0)
* Disabling the dso version of testshade for the coverage analysis test.
#1604 (1.13.1.0)
* Be sure to run pointcloud tests when partio is found. #1611 (1.13.2.0)
* testshade: better testing of all closure parameter types. #1621
(1.13.2.0/1.12.8.0)
* Don't use the cached background map on the first bounce
[#1649](https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/pull/1649) (by Christopher Kulla) (1.13.4.0)
* Fix builds when USE_PYTHON=OFF [#1675](https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/pull/1675) (by Declan Russell) (1.13.4.0)
* Remove SonarCloud cache setup as it is now offered by default
* Test interactive parameter initialization in optix testrender [#1687](https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/pull/1687) (by Chris Hellmuth) (1.13.5.0)
* Fix GPU job not actually enabling OptiX [#1738](https://github.com/AcademySoftwareFoundation/OpenShadingLangauge/pull/1738) (by Brecht Van Lommel)
* Fix python-oslquery test failure on macOS [#1739](https://github.com/AcademySoftwareFoundation/OpenShadingLangauge/pull/1739) (by Brecht Van Lommel)
* Disable MacOS-11 CI [#1746](https://github.com/AcademySoftwareFoundation/OpenShadingLangauge/pull/1746)
* Skip analysis and scorecard workflows on push if only docs changed [#1744](https://github.com/AcademySoftwareFoundation/OpenShadingLangauge/pull/1744)
* For clarity, use the term 'runner' instead of 'os' [#1743](https://github.com/AcademySoftwareFoundation/OpenShadingLangauge/pull/1743)
* Test ABI compliance [#1748](https://github.com/AcademySoftwareFoundation/OpenShadingLangauge/pull/1748)
* Fix tests passing when test commands fail [#1733](https://github.com/AcademySoftwareFoundation/OpenShadingLangauge/pull/1733) (by Brecht Van Lommel)
* *ci*: Test against LLVM 17 [#1760](https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/pull/1760)
* Platform support:
* Include `immintrin.h` only when needed. #1605 (1.13.1.0)
* Changed snprintf formatting to satisfy some compilers.
*`doc/RELEASING.md` documents our release process and versioning policies.
#1572 (1.13.0.2)
* Improve documentation of `trace` function [#1671](https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/pull/1671) (by AidanWelch) (1.13.4.0)
* Update logos [#1705](https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/pull/1705), Move logos and add an icon-only logo [#1707](https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/pull/1707) (1.13.5.0)
### 🏢 Project Administration
* Alert slack "release-announcements" channel upon OSL release [#1740](https://github.com/AcademySoftwareFoundation/OpenShadingLangauge/pull/1740) [#1749](https://github.com/AcademySoftwareFoundation/OpenShadingLangauge/pull/1749)
[](https://www.youtube.com/watch?v=nmwMz1YnGPA "Open Shading Language Reel 2020")
If any other issue requires confidentiality that precludes a public question
or issue, you may contact the project administrator privately at
[lg@larrygritz.com](lg@larrygritz.com).
Contributing
@@ -416,12 +514,11 @@ Contacts, Links, and References
[Read or subscribe to the OSL development mail list](https://lists.aswf.io/g/osl-dev)
[Most recent PDF of the OSL language specification](https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/blob/master/src/doc/osl-languagespec.pdf
[Most recent PDF of the OSL language specification](https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/blob/main/src/doc/osl-languagespec.pdf
)
[OSL home page at SPI](http://openshadinglanguage.org)
[OSL home page](http://openshadinglanguage.org)
[Sony Pictures Imageworks main open source page](http://opensource.imageworks.com)
@@ -433,17 +530,22 @@ The current project leadership is documented in the
Many people have contributed features, bug fixes, and other changes to OSL
over the years: Steve Agland, Shane Ambler, Martijn Berger, Farchad
Bidgolirad, Nicholas Bishop, Solomon Boulos, Stefan Bruens, Stefan Büttner,
Matthaus G. Chajdas, Alejandro Conty, Thomas Dinges, Luke Emrose, Louis
Feng, Mark Final, Henri Fousse, Syoyo Fujita, Tim Grant, Larry Gritz, Derek
Haase, Sven-Hendrik Haase, John Haddon, Daniel Heckenberg, Thiago Ize, Matt
Johnson, Ronan Keryell, Chris Kulla, Elvic Liang, Max Liani, Adam Martinez,
John Mertic, Bastien Montagne, Steena Monteiro, Alexis Oblet, Erich Ocean,
Mikko Ohtamaa, Jino Park, Alexei Pawlow, Jay Reynolds, Declan Russell,
Patrick Scheibe, Alex Schworer, Jonathan Scruggs, Sergey Sharybin, Cliff
Stein, Stephan Steinbach, Esteban Tovagliari, Brecht Van Lommel, Alexander
von Knorring, Alex Wells, Roman Zulak. (Listed alphabetically; if we've left
anybody out, it is inadvertent, please let us know.)
Bidgolirad, Nicholas Bishop, Curtis Black, Rasmus Bonnedal, Solomon Boulos,
Stefan Bruens, Stefan Büttner, Matthaus G. Chajdas, Clark Chen, Alejandro
Conty, Damien Courtois, Dieter De Baets, Thomas Dinges, Daniel Dresser, Mads
Drøschler, Peter Ellerington, Luke Emrose, Louis Feng, Mark Final, Henri
Fousse, Stephen Friedman, Syoyo Fujita, Tim Grant, Larry Gritz, Nicolas
Guiard, Ole Gulbrandsen, Euan Haahr, Derek Haase, Sven-Hendrik Haase, John
Haddon, Niklas Harrysson, Daniel Heckenberg, Chris Hellmuth, Adrien Herubel,
Dan Horák, Thiago Ize, Matt Johnson, Ronan Keryell, Chris Kulla, Elvic Liang,
Max Liani, Adam Martinez, John Mertic, Bastien Montagne, Steena Monteiro, John
Mertic, Patrick Mours, Alexis Oblet, Erich Ocean, Mikko Ohtamaa, Jino Park,
Alexei Pawlow, Jay Reynolds, Declan Russell, Benoit Ruiz, Patrick Scheibe,
Alex Schworer, Jonathan Scruggs, Sergey Sharybin, Mark Sisson, Sandip Shukla,
Cliff Stein, Stephan Steinbach, Luya Tshimbalanga, Esteban Tovagliari, Brecht
Van Lommel, Thibault Vergne, Alexander von Knorring, Aidan Welch, Alex Wells,
Roman Zulak. (Listed alphabetically; if we've left anybody out, it is
inadvertent, please let us know.)
We cannot possibly express sufficient gratitude to the managers at Sony
Pictures Imageworks who allowed this project to proceed, supported it
| main | :white_check_mark: :construction: ALL fixes immediately, but this is a branch under development with a frequently unstable ABI and occasionally unstable API. |
| 1.12.x | :white_check_mark: All fixes that can be backported without breaking ABI compatibility. New tagged releases monthly. |
| 1.11.x | :warning: Only the most critical fixes, only if they can be easily backported. |
| <= 1.10.x | :x: No longer receiving patches of any kind. |
## Reporting a Vulnerability
If you think you've found a potential vulnerability in OSL, please report it
by emailing the project administrators at
[security@openshadinglanguage.org](security@openshadinglanguage.org). Only the
project administrators have access to these messages. Include detailed steps to
reproduce the issue, and any other information that could aid an
investigation. Our policy is to respond to vulnerability reports within 14
days.
Our policy is to address critical security vulnerabilities rapidly and post
The OSL Documentation Project is a work in progress.
The plan is to adopt the OCIO Documentation Project's methodology for maintaining and building documention of various kinds as it becomes the blueprint for the ASWF.
The plan is to adopt the OCIO Documentation Project's methodology for maintaining and building documentation of various kinds as it becomes the blueprint for the ASWF.
sed -i 's,^mirrorlist=,#,; s,^#baseurl=http://mirror\.centos\.org/centos/$releasever,baseurl=https://vault.centos.org/7.9.2009,' /etc/yum.repos.d/CentOS-Base.repo
set(ILMBASE_LIBRARIES${OPENEXR_IMATH_LIBRARY}${OPENEXR_IEX_LIBRARY}${OPENEXR_HALF_LIBRARY}${OPENEXR_ILMTHREAD_LIBRARY}${ILMBASE_PTHREADS}CACHESTRING"The libraries needed to use IlmBase")
set(OPENEXR_LIBRARIES${OPENEXR_ILMIMF_LIBRARY}${ILMBASE_LIBRARIES}${ZLIB_LIBRARIES}CACHESTRING"The libraries needed to use OpenEXR")
Source code for OSL consists of ASCII or UTF-8 characters.
The characters for space, tab, carriage return, and linefeed are
collectively referred to as *whitespace*. Whitespace characters
delimit identifiers, keywords, or other symbols, but other than that
have no syntactic meaning. Multiple whitespace characters in a row
are equivalent to a single whitespace character.
Source code may be split into multiple lines, separated by end-of-line
markers (carriage return and/or linefeed). Lines may be of any length
and end-of-line markers carry no significant difference from other
whitespace, except that they terminate `//` comments and delimit
preprocessor directives.
## Identifiers
*Identifiers* are the names of variables, parameters, functions, and shaders.
In OSL, identifiers consist of one or more characters. The first character
may be a letter (`A`-`Z` or `a`-`z`) or underscore (`_`), and subsequent
characters may be letters, underscore, or numerals (`0`-`9`). Examples of
valid and invalid identifiers are:
```
opacity // valid
Long_name42 // valid - letters, underscores, numbers are ok
_foo // valid - ok to start with an underscore
2smart // invalid - starts with a numeral
bigbuck$ // invalid - $ is an illegal character
```
## Comments
*Comments* are text that are for the human reader of programs, and
are ignored entirely by the OSL compiler. Just like in C++, there
are two ways to designate comments in OSL:
1. Any text enclosed by `/*` and `*/` will be considered a comment, even if
the comment spans several lines.
```
/* this is a comment */
/* this is also
a comment, spanning
several lines */
```
2. Any text following `//`, up to the end of the current line, will be
considered a comment.
```
// This is a comment
a = 3; // another comment
```
## Keywords and reserved words
There are two sets of names that you may not use as identifiers:
keywords and reserved words.
The following are *keywords* that have special meaning in OSL:
and break closure color continue do else emit float for if illuminance
illuminate int matrix normal not or output point public return string
struct vector void while
The following are *reserved words* that currently have no special meaning in
OSL, but we reserve them for possible future use, or because they are
confusingly similar to keywords in related programming languages:
bool case catch char class const delete default double
enum extern false friend
goto inline long new operator private protected
short signed sizeof static
switch template this throw true try typedef
uniform union unsigned varying virtual volatile
## Preprocessor
Shader source code is passed through a standard C preprocessor as a
first step in parsing.
Preprocessor directives are designated by a hash mark `#` as the first
character on a line, followed by a preprocessor directive name. Whitespace may
optionally appear between the hash and the directive name.
OSL compilers support the full complement of C/C++ preprocessing directives,
including:
```
#define
#undef
#if
#ifdef
#ifndef
#elif
#else
#endif
#include
#pragma error "message"
#pragma once
#pragma osl ...
#pragma warning "message"
```
Additionally, the following preprocessor symbols will already be
defined by the compiler:
| | |
| ------ | :------- |
| `OSL_VERSION_MAJOR` | Major version (e.g., 1) |
| `OSL_VERSION_MINOR` | Minor version (e.g., 9) |
| `OSL_VERSION_PATCH` | Patch version (e.g., 3) |
| `OSL_VERSION` | Combined version number = `10000*major + 100*minor + patch` (e.g., 10903 for version 1.9.3) |
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Clifford Yapp
