Fixes#2129
CUDA 13.2+ requires LLVM 22.1.2+ for the _NV_RSQRT_SPECIFIER fix.
Assisted-by: Claude Code / sonnet-4.6
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The time it takes to run a full testsuite was slowly creeping up to an
inconvenient time on CI. The most expensive test are the `render-*`
tests. Cut all of their resolutions in half to speed up CI runs.
Update a lot of reference images.
But after updating the ref images we need to fully pass CI (and pass
on my work machine and my personal laptop), I could see that there
were a number of `testsuite/render-*/ref/*.exr` files that did NOT
get updated in this process -- we must conclude that they are old
references that are no longer needed, so I deleted them.
I also noticed that testsuite/render-material-layer had no ref images
updated, and some git investigation reveals that this test that was
added in 2017 was NEVER in testing.cmake and thus never ran in CI
at all. So just remove it.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Analyzing the geomath and trig testsuite entires, each of them had
multiple reference outputs due to small precision differences between
platforms and optimization levels. By printing only 4 digits of
precision (and in one case of an unstable value to pass to tan, just
picking a different value), we are able to collapse those tests to one
reference output each.
only flushes near-zero values, not precision. With the reduced precision every
platform/backend produces identical text, so the four variant references are
removed and a single ref/out.txt remains. The JIT and C++
(OSL_DEBUG_OUTPUT_CPP=3) backends now produce byte-identical output and both
pass against it.
Assisted-by: Claude Code / Claude Opus 4.8
Signed-off-by: Larry Gritz <lg@larrygritz.com>
In main only, i.e. for OSL 1.16 release late in 2026, raise the
minimum OpenImageIO supported to 3.0.
That means that the mostly-2027 version of OSL will support building
against OIIO 3.2 (2027), 3.1 (2026), or 3.0 (2025), which I think is
enough range.
In the process, I shuffled some CI variants around. Removed a couple
variants that used OIIO 2.5, and bumped the job we use for testing a
debug build from being based on 2023 containers to 2026, and the job
we use for checking ABI to 2025.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Some of the translation of type codes to the abbreviations we use
internally for argument type signatures and internal function naming,
like 'f' for float, 'c' for color, etc., were repeated in various
place. This tries to remove some of the duplication.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
- According to ASWF, the technical charter should not be stored in
our repo, but just linked to its canonical ASWF home.
- Security addres in CONTRIBUTING had a bad copy/paste error and
incorrectly had the address for the openimageio security reporting.
- GOVERNANCE did not have the link to the meeting notes.
- README had a bad link, and did not link to the calendar. Spruced
up some related info and the section title.
- Typo fixes
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This commit fixes compilation and runtime errors of the OSL against
specific builds of LLVM 22.
When LLVM is compiled with LLVM_LINK_LLVM_DYLIB=OFF (default value) an
extra set of libraries needs to be used for Clang:
- The clangOptions library is needed because getDriverOptTable() used
from clangFrontend was moved from clangDriver to clangOptions.
- Similarly, printStats() used from clangSema has been moved to
clangAnalysisLifetimeSafety.
This linking error could be reproduced when LLVM is compiled with the
following options:
```
cmake \
-GNinja \
-DCMAKE_BUILD_TYPE=Release \
-DLLVM_INCLUDE_TESTS=OFF \
-DLLVM_INCLUDE_EXAMPLES=OFF \
-DLLVM_BUILD_LLVM_C_DYLIB=OFF \
-DLLVM_ENABLE_UNWIND_TABLES=OFF \
-DLLVM_ENABLE_ZSTD=OFF \
-DLLVM_ENABLE_ZLIB=OFF \
-DLLVM_ENABLE_PROJECTS=clang \
-DCMAKE_INSTALL_PREFIX=/opt/local/llvm/22.1.7 \
../llvm
```
A couple of notes:
- The linking issue can not be reproduced when using Homebrew's
LLVM 22 because it uses LLVM_LINK_LLVM_DYLIB=ON. This seems to be
leading to a situation when the linker issue is side-steps because
of the symbols that are pulled by the dynamic LLVM library.
- Passing LLVM_STATIC does not help with this case, since this flag
only affects the way LLVM libraries are handled and does not change
the way Clang libraries are handled:
`llvm-config --libfiles --link-static` does not print any
Clang-related libraries.
Since LLVM version 22 the VFS is to be explicitly created for the
CompilerInstance. Prior to version 22 this was done implicitly by the
LLVM when calling createFileManager(), but now this is no longer the
case, and LLVM has an assertion for it. The createVirtualFileSystem()
with default arguments solves this problem, and initializes
CompilerInstance's VFS the same way how it was done implicitly in the
older versions of LLVM. Without this, compilation of OSO triggers an
assert() in LLVM:
```
Assertion failed: (VFS && "CompilerInstance needs a VFS for creating FileManager"), function
createFileManager, file CompilerInstance.cpp, line 387.
```
It is unclear whether this was causing any actual issue with release
builds of LLVM, but it is better to use the API in the way it is
expected to be used and create the VFS.
Last but not least, this change also fixes the following assert on the
LLVM side when running OSL's test suite on macOS:
```
Assertion failed: ((isUIntN(8 * Size, Value) || isIntN(8 * Size, Value)) && "Invalid size"), function emitIntValue, file MCStreamer.cpp, line 135.
```
The value that trips on this assert statement comes from the
DW_AT_APPLE_major_runtime_vers that the code in DwarfDebug.cpp, and it
comes from the RuntimeVersion (RV) argument of the createCompileUnit().
This runtime value is supposed to indicate Obj-C runtime version which
is either 1 or 2 (value 2 is for Objective-C 2.0). Passing OSL version
as the runtime version does not fit single byte, and that is why the
assertion trips. This change follows Clang's code which sets the
runtime version to 0 for non-Objective-C languages. It is unclear what
side-effects this change could bring, but it is also a bit strange to
use OSL version for things that distinguish system runtime.
In order to be able to run OSL regression tests with LLVM compiled in
debug mode, LLVM_LINK_LLVM_DYLIB=ON had to be used. Without this, the
following assert in the LLVM triggers upon testrender startup:
```
: CommandLine Error: Option 'openmp-ir-builder-optimistic-attributes' registered more than once!
LLVM ERROR: inconsistency in registered CommandLine options
```
This is to do with the way how LLVM registers command line arguments.
It uses static variable initialization for this, and having LLVM as a
static library that is pulled via CLang into both oslcomp and oslexec
that are then pulled into testrender causes this situation to happen.
For some reason, it does not happen if LLVM is compiled as static but
release library.
The need of debug versions of OSL and LLVM libraries comes from the
need to compile Blender on Windows in Debug mode, and libraries need
to be compiled with a different CRT ABI.
Tested with the ctest with both LLVM 20 and 22 compiled in release.
With all these changes the test suite passes to the same degree for
LLVM 20 and 22, both Debug and Release. The only failing tests are
related to Python module, and those I did not yet manage to get passing
on a local development machine.
Signed-off-by: Sergey Sharybin <sergey@blender.org>
Adds agent configuration supporting Claude Code, Cursor, GitHub
Copilot, OpenAI Codex, and Opencode. A single AGENTS.md and contents
of .agents subdirectory serve as the canonical source for
instructions, skills, and other things that can be used across
multiple tools.
A script can be invoked by users to do the local setup of links or
references to allow them to use a particular tool:
`.agents/setup-agent TOOL` Currently supported tools by this script
include Claude Code, Codex, Cursor, GitHub Copilot, and OpenCode.
This initial commit includes two skills that I wrote and have used for
the last couple monthly releases: release-notes-update and
prepare-patch-release. I assume that we will add more over time as we
discover tasks that we want to make standard skill automation for.
Assisted-by: Claude Code / claude-sonnet-4-6
The AGENTS.md, Claude skills, and general scheme are entirely from me.
I used Claude to help me establish the setup script for the other
tools I don't happen to use.
This basic structure was first prototyped on OIIO and then copied to
OSL.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
When a parameter is interactive and connected, the connection should
win. OSL was incorrectly treating the interactive trait as taking
precedence, attempting to write into the read-only interactive buffer
during shader execution.
Fixed both single point and batch modes.
---------
Signed-off-by: Chris Hellmuth <chellmuth@gmail.com>
Store the new compile-time complexity metrics that were added by PR
#2085 (but only printed the the log as each group compiles) as fields
on ShaderGroup so they survive after optimization.
Expose them via getattribute(group, "stat:compiled_active_layers")
etc., and extend getstats() to emit a "Shader compilation stats,
post-optimized" section listing min/max/median and a leaderboard with
a ranked top-N list for each metric (controlled by the existing
"stat:rank_groups" option).
Tests: testsuite/compstats/ test covering both the getstats output and
the getattribute keys. Added a testshade --print-group-stats flag for
querying per-group stat keys.
Assisted-by: Claude Code / claude-sonnet-4-6
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We test on 3 Windows runner configurations currently -- Windows 2022
with VS2022, Windows 2025 with VS2022, and Windows 2025 with
VS2026. GitHub says the middle one will go away imminently (actually,
it won't "break", but it will redirect to Windows 2025 with VS2026,
which we already test). So take it out of our lineup, it's just waste.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Alongside the rest of the initialization for subpixel_radiance, an empty MediumStack is initialized. It stores the medium params with pointers in two arrays. One tracks the entry order and the other sorted based on priority. There is a priority handling scheme that will shuffle the incumbent medium pointers to give high priority (lower integer priorities) mediums lower indices. When processing BSDF closures, intersections with MxDielectric or MxGeneralizedSchlick may only be added when priority < current_priority or are both 0 (the precious priority). MediumStack exposes an integrate method that operates on aggregated parameters from all media sharing the same priority as the topmost medium; these aggregated parameters are stored on the stack and updated whenever media are added. A CDF built from these parameters is then sampled to select one of the overlapping media for scattering. At present, only Henyey–Greenstein phase functions are supported, implemented as a BSDF subclass and wraps the implementation in BSDL.
Signed-off-by: Owen O'Malley <theowen@email.com>
docs: Update security instructions to emphasize reporting via GitHub
The security@openshadinglanguage.org is still fine, but we prefer that
true vulnerability reports come via the GitHub security advisory
mechanism. (That makes it easy for us to turn them into CVEs when
needed, among other administrative niceties.)
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This change fixes crashes that could be observed on Ubuntu 24.04.
The crash happens in exception handling after rendering a scene
with OSL shaders.
This seems to be a bug in libgcc which was fixed in 14.3, but
Ubuntu 24.04 is shipped with 14.2. There is a report in the GCC
bug tracker:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=119151
Note that this is a runtime dependency, so it doesn't really matter
that a different GCC version was using during compilation as the
EH frames registration happens at runtime.
The easiest solution is to simply disable EH frames registration
in OSL's MemoryManager. This should not affect any runtime exception
unwinding as this MemoryManager is only used during JIT, so all the
EH frames are unregistered from the process when the JIT code is
actually executed.
The issue was originally noticed in Blender, here is the report with
some extra investigation and details:
https://projects.blender.org/blender/blender/issues/156348
Signed-off-by: Sergey Sharybin <sergey@blender.org>
OpenImageIO and OSL have similar CI infrastructure, mostly share a
design of such, and copy the files back and forth to synchronize from
time to time. Turns out it's been a while since the last sync, and a
lot of small improvements have accumulated on the OIIO side.
* Bump our CI workflow externally-hosted actions to the latest versions.
* Use shared-steps workflow for the old aswf container jobs.
* Split some of the steps of the shared-steps workflow into "old" and
"new" in cases where the 2022 ASWF docker containers containers cannot
run the latest actions, only update those to the most recent that are
able for those containers.
* Restore bleeding edge test to pybind11 master -- the reason we locked
it down has since been fixed on the OIIO side.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Periodic re-synchronization of build system components between the two
projects.
dependency_utils.cmake:
* Make dependency report more clear to read.
* Infastructure related to auto-build of missing dependencies, which
we don't yet do in the OSL side, but will eventually institute that
like we did on the OIIO side.
compiler.cmake:
* Warnings about clang versions so old that we no longer test them.
* Imroved ccache finding logic.
* Fix TIME_COMMANDS.
* Turn on CMAKE_EXPORT_COMPIE_COMMANDS to make a compile_commands.json.
* Various other insubstantial changes just to keep the two projects in
sync to make future diffs easier to understand.
gh-win-installdeps.bash:
* Accommodate newer python releases on Windows.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Mostly for conformance with OpenImageIO, since we share so many
scripts and management docmentation. But looking across ASWF project,
they almost all use "docs/" (OSL and OpenVDB were the only outliers).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
I found some places that still referred to the old PDF document in the
repo, but we haven't used that for a while and no longer build it.
Also remove some chunks of the Makefile that used to be responsible
for building the LaTeX docs, since those don't even exist in the repo
any more.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* When building openexr as a dependency for OSL, there is no need
to build the openexr command line utilities, saving a bit of build
time.
* Fix an error from PR #2074 where we tried to disable nightly CI runs
on forks -- we didn't make the right change to the Windows jobs, so
they were still running.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
fmtlib renamed its "master" branch to "main" yesterday. So our "bleeding edge" test that requests fmt "master" now fails.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We've discussed AI code assistant policy in multiple TSC meetings,
and on the OpenImageIO slack (technically it was a shared OIIO+OSL
discussion, but we had to pick a single place to have it). We iterated
on the policy document as an OIIO PR, so this PR here in OSL land is
just checking that same document into our repo.
* Duplicate the OIIO document. We will keep them in sync until such
time as the needs of the projects diverge or their communities have
different ideas about what policies to articulate.
* Appropriate references/pointers in supporting documents: README,
CONTRIBUTING, and PULL_REQUEST_TEMPLAGE.
I took the time to diff some of those documents and port other recent
changes from the OIIO side to OSL. These are mostly minor fixes or
improvements in wording, nothing substantive.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
While debugging some CI failures on Windows, I went down some wrong
roads that turned out not to be related to the actual problem... but
nevertheless are improvements I wish to keep. Here they are:
* Attempt to use the install-llvm-action from Windows wasn't able to
work (among other reasons) because we weren't passing along the
llvm_action_ver variable.
* For reasons I never quite understood, the llvm install action works
a lot better on Windows if done before the "build setup" step. So
I just moved that action earlier in the sequence.
* Some status messages about how we are installing LLVM.
* A correction in how we detect and print the pkg_ROOT environment
variable in the log generated by handle_package_notfound.
* Allow LLVM_PATH (set by that action) to be a proper hint for
FindLLVM.cmake.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
A recent change in OIIO main is to check that any auto-built
dependencies have a SHA-1 hash that matches what we think it should be
for that version. That works fine for a fixed version, but if we are
asking for OIIO to build from dependency master/main, as we do from
its "bleeding edge" CI variant, we need to tell it that it's ok to not
match the expected hash by setting
OpenImageIO_DEPENDENCY_BUILD_ALLOW_UNVERIFIED_TAGS=ON. So we need to
do this in the OSL bleeding edge variant, too.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
When using Clang 18<->22 with ld.lld, build fails with:
```
clang++ -Wl,--version-script=.../hidesymbols.map -o lib/liboslcomp.so.1.13.7 ...
ld.lld: error: version script assignment of 'global' to symbol 'test_shade' failed: symbol not defined
```
The issue is that test_shade does not belong to liboslcomp, instead it belongs to libtestshade,
and ld.lld actually checks that and fails. The minimal fix is to remove test_shade from hidesymbols.map,
the slightly better way is to create a separate symbols map for libtestshade.
See also: https://bugs.gentoo.org/929091
Signed-off-by: Sv. Lockal <lockalsash@gmail.com>
* Implement get_albedo() for BSDL imported BSDFs
We forgot to forward the albedo function to BSDL and also to
implement it for the latest MaterialX components added to BSDL
recently. This led to not very good sampling quality.
Signed-off-by: Alejandro Conty <aconty@imageworks.com>
* Incorporate diffuse MaterialX BSDFs to BSDL
This is just a code movement. The implementation was already there. Just
relocated to the BSDL framework the following items:
- burley_diffuse_bsdf
- oren_nayar_diffuse_bsdf
- translucent_bsdf
There are sampling noise differences that require updating the tests,
but otherwise images look identical.
Signed-off-by: Alejandro Conty <aconty@imageworks.com>
* Update testsuite refs and thresholds
Due to the last two commits (BSDL albedo fix and diffuse MTX code
movement) some references had to be updated due to noise differences.
I'm also following the policy of keeping as few refs as possible and
opening the thresholds so different platforms LSB differences are
accepted.
---------
Signed-off-by: Alejandro Conty <aconty@imageworks.com>
* Refactor install_homebrew_deps to match recent changes in the
version that's part of OIIO.
* Exclude building Qt and OpenVDB from the Intel Mac CI variant --
they seem to be hit or miss these days whether it is pre-built, and
if not, makes our CI on that variant take forever. We don't need it,
we test those packages on other platforms, including newer Macs.
* Bump llvm to 18 for the troublesome Intel case.
* Need to make sure OSL sets the OpenImageIO_CI variable also, so that
OIIO, when building, knows it's a CI run.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We don't test this in CI so it's probably gone undetected for a while,
but there are a few tests that differ slightly for the combination of
ARM-based Mac for debug builds. So check in a few more ref images so
that devs using that platform don't get spurious testsuite failures
for debug builds.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
For a few days now, the Windows CI variants have been almost always
failing due to unexplained inability to curl download the pre-built
LLVM that we stashed on a Google drive, which has worked flawlessly
for over a year.
J.F. Panisset suggested:
> Could be a wild goose chase, but a direction to look in with your
> Windows / curl issue would be SSL certificate verification. Windows
> uses some kind of magic process to update its store of recognized CA
> root certs, but since a Windows runner is a "fresh" Windows VM every
> time, it could be that it is missing an updated cert required to
> verify the https URL you are downloading from (assuming that's using
> a cert anchored to a root CA not in that base WIndows image). I've
> seen issues like that on Windows systems with limited Internet
> access. If you try curl --insecure to (temporarily) bypass
> certificate verification, are you able to download what you need?
And sure enough, adding --insecure clears it right up.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Consolidate sheen implementation from testrender and SPI into a single
BSDL-backed MaterialX lobe (`mtx::SheenLobe`), so sheen behavior is defined
in one place and reused consistently across paths.
- Move sheen logic into BSDL/MTX and wire it through the BSDL closure path
- Replace ad-hoc testrender-side handling with the new BSDL implementation
- Keep Conty/Zeltner mode selection inside the unified sheen lobe
- Update sheen tests and references to validate the new path
- Turn the furnace sheen test into a true energy-conservation check
(sheen layered over white diffuse should converge to flat gray)
AI Disclaimer: AI has been used to code review and bug hunt, but all the
code and fixes have been written by huomans.
Signed-off-by: Alejandro Conty <aconty@imageworks.com>
When compile_report >= 1, report several new post-optimization stats for the shader group:
* Number of active (non-unused) layers
* Maximum connection depth of the shader network (node count along the longest layer-to-layer chain, so a single unconnected node = 1 and a chain A->B->C = 3)
* Number of texture ops (texture, environment, gettextureinfo, texture3d)
* Number of noise ops
Note that these stats (number of layers, texture ops, noise ops) are the total number in the post-optimized shader, but not the number that will be run on any particular execution of the shader. Trivial example: the noise calls may all be inside an "if" statement with a condition that is never true in practice, but that can't be known ahead of time. In the other direction, a loop with noise calls in the body will underestimate how many calls are made in each shader run. So take with a grain of salt -- these stats are related overall shader code complexity, but not necessarily to execution cost.
Coded via pair programming with Claude Code, using the Claude Sonnet 4.6 model.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Intel icc is deprecated and hasn't had a release for a few years. It's
holding us back, both by making us work around an ever growing number
of icc bugs and limitation that will never be fixed, as well as not
allowing us to upgrade minimum versions of certain dependencies,
because icc can't correctly compile newer versions (as an example, it
cannot use a 'fmt' library newer than the oldest we support, 7.0).
So it's time to thank icc for its service and put it on the ice floe
for the polar bears to eat. This is of course in main (future OSL
1.16), and will not be backported to release branches, since we never
stop support of a dependency or toolchain of existing releases. People
requiring icc for whatever reason may keep using OSL 1.15 or older.
We will continue to support and test icx, the fully supported Intel
LLVM-based compiler.
This PR also removes some test reference output that was only needed
for icc.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The division (xclamped - edge0) / (edge1 - edge0) is not guaranteed
to be exactly 1 for xclamped == edge1 if the backend floating point
math has higher than expected error bounds. Eg. it is common on GPU
backends to use faster but less precise instructions for division.
The modified code is mathematically equivalent and it logically
guarantees the returned value is exactly 1.0 for x > edge1 > edge0.
Signed-off-by: Roeland Schoukens <roelandschoukens@hotmail.com>
Replace the standalone global ErrorHandler with the renderer's own
errhandler() so that error output is properly routed through the
renderer rather than a disconnected handler. This ensures errors during
shading are visible in the same context as the renderer that owns the
ShadingSystem.
Also replace some raw pointers with `new`/`delete` with
std::unique_ptr.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The ShadingSystem option "llvm_jit_fma" seems to advertise that it
switches between no fused multiply-add (the default, 0), versus
allowing fma to be generated in the JIT.
The underlying LLVM option is actually a 3-way switch about how to
utilized fused ops: Strict (no fused ops), Standard (allow fma but not
other fused ops), or Fast (allow other fused ops when it will speed
things up). Aside: "standard" is really confusing, as one might think
it means original standard IEEE 754 definitions that lacked fma.
These are not the same! We were using llvm_jit_fma to switch between
Standard and Fast, but upon closer examination, we now believe that it
should control a switch between Strict and Standard, and Strict should
be the default.
This seems to directly change the pesky render-microfacet test, which
has always seemed to have an unusual sensitivity to the specific
processor. Hopefully, users will find that across the board, it leads
to their renderers having fewer minor pixel differences from one
platform to another.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Clamp generalized Schlick F0/F90 to [0, 1] in the MTX Schlick Fresnel
constructor, and clamp the MTX conductor average-Fresnel fit to [0, 1]
to avoid invalid albedo estimates from the approximation.
Also update BSDL bootstrap linking to use Imath::Imath directly instead of
${ARNOLD_IMATH_TARGETS}.
Signed-off-by: Alejandro Conty <aconty@imageworks.com>
Unify DielectricRefl and DielectricBoth into a single DielectricBSDF<Fresnel>
template that handles both reflection-only and reflection+refraction modes
via a runtime `dorefr` flag. This replaces the old union of two separate
structs in DielectricLobe with a single `spec` member, simplifying the
code and enabling reuse with alternative Fresnel models.
Key changes:
- Merge DielectricRefl and DielectricBoth into DielectricBSDF<Fresnel>,
parameterized on the Fresnel type. The eval/sample methods branch on
`dorefr` internally. The existing DielectricReflFront, DielectricBothFront,
and DielectricBothBack now inherit from DielectricBSDF<DielectricFresnel>
for table-baking purposes.
- Propagate colored Fresnel through the dielectric eval/sample paths:
Fresnel eval() now returns Power instead of float, so reflection and
refraction weights carry the spectral information directly (needed for
Schlick's colored F0/F90).
- Add refraction_eta() accessor to DielectricFresnel so the templated
code can obtain the refraction IOR uniformly across Fresnel types.
- Introduce SchlickFresnel (inheriting DielectricFresnel) with generalized
Schlick F0/F90/exponent parametrization and TIR handling. SchlickLobe
reuses DielectricBSDF<SchlickFresnel> directly. The refraction IOR is
derived from F0 via the inverse Schlick approximation.
- Add BSDL_WRAP macro in testrender/shading.cpp to eliminate boilerplate
for wrapping BSDL lobes (MxConductor, MxDielectric, MxGeneralizedSchlick).
The old per-class MxGeneralizedSchlick via CLOSURE_*_PARAM macros is
replaced by the BSDL auto-registration path.
- Update test references and bump AA samples for the schlick-glass test
to reduce noise with the new implementation.
Signed-off-by: Alejandro Conty <aconty@imageworks.com>
Identical to AcademySoftwareFoundation/OpenImageIO/pulls/5024
There's something in pybind11 master at the moment that is crashing in
its destructors. It's been causing our "bleeding edge" test to fail for
over a week now. I'm tired of our test failing, so I'm locking down to
the last known working version. Will check back periodically and return
to testing against pybind11 master after they have fixed it.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Swap Adrien Herubel -> Thiago Ize for Autodesk (approved by TSC on
2025-12-04), and Clint Hanson -> Alexey Smolenchuk for DNEG (approved
by TSC on 2026-01-15).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
They got rid of 3.9 on the runners this week and broke CI.
Also I took the opportunity to update INSTALL.md to refect the
correct dependency versions we have tested.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Fix dudx, dudy, dvdx, dvdy, as well as dpdx, dpdy in testrender. Now, bump mapping from both UV-based mapping and 3D-based patterns works well. Mip-Map selection also works fine now. Images in Issue: #1978
This PR is intended to validate math and discuss tests.
Fixes#1978
This fix could potentially affect many other tests since dpdx, dpdy are changed.
I'm going to add tests representing bump and mipmapping in this PR.
---------
Signed-off-by: Alexey Smolenchuk <alexey.smolenchuk@gmail.com>
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Co-authored-by: Larry Gritz <lg@larrygritz.com>
- Mount the host's root under /host/root so it's visible in the
containers.
- Remove stuff we don't need to free lots of disk space.
- Tentatively go back to the free runner for the icx test.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Swap out deprecated sonar related action for the new one.
Also make some tweaks to the analysis workflow run conditions,
and update the actions/cache version.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This test started failing sporadically recently, with output images
that looked plausible, but differed slightly from the rest of the
checked-in reference images for this test. It only failed maybe 5% of
the time, so sometimes a CI run would pass, or sometimes fail just one
job variation, or very occasional two. Rerunning the failed job would
generally succeed, but sometimes fail again.
I'll cut to the chase: We were able to trace it to all failures
happening on GitHub runners that identified themselves as "Intel(R)
Xeon(R) Platinum 8370C CPU @ 2.80GHz". When subsequent runs of the
same job variation succeded, it was always because it landed on a
runner that identified itself as "AMD EPYC 7763 64-Core Processor".
We presume that some minor HW or SW differences (perhaps in a library
with an ISA-specific code path) resulted in LSB errors that conspire
to sample different light paths on a minority of pixels, leading to a
different pattern of specular "fireflies".
The expedient fix herein is simply to commit yet another reference
image that is considered to be a "pass".
Along for the ride, discovered during my debugging:
* Rename the label of one job variant, which was incorrect.
* Add comments to the test thresholding parameters (comments ported
from the analogous script in OIIO, making it easier to understand
next time one of us is figuring out this code).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
A week or two ago, the icx job variation stopped working, not enough
space to finish installing the packages containing icx and related
material. We didn't change, the Intel package (pegged to a specific
version from a couple years ago) didn't change, the ASWF docker
container (also old) didn't change, so we're assuming that the GHA
runner itself changed, leaving us less disk space, which is now not
enough.
The easiest way out of this for now is for the icx job variation to
use a paid bigger GHA runner with more "disk space". This means it can
only be enabled when running on the main repo, which uses the ASWF's
account that has paid runners. Sorry, CI can't currently test with icx
you people's individual forks until we figure out a way to require
less space. Until then, developers who don't have icx locally will
need to submit the PR to see how it does on icx.
A little bit of workflow magic was needed to ensure that this job
variation doesn't run at all on a developer's fork -- we don't want it
to run, and look like a failed test just because they don't have an
account that doesn't allow use of the big paid runners.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Switch CI back to the aswf/ci-osl:2025-clang18.3 container that
has OIIO 3.1, it will work fine with this change.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
batch: relax SIMD loop for certain operations
When building with a newer version of clang, I uncovered a few more
SIMD pragma loops fail to vectorize on clang.
Since it seems to depend on the compiler version, and I don't want to
disable attempted vectorization, I thought a better strategy was just
to disable the warnings when it fails to vectorize in those specific
places.
Add a "latest dependencies with clang" test, that's how I stumbled
across these in the first place.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
In all branches, the OptiX test job started failing several days ago,
corresponding to an update of the aswf/ci-osl:2025-clang18.3 container.
Reverting to aswf/ci-osl:2025-clang18.2 passes again!
Let's merge this fix so that our CI passes, and separately iterate
to figure out what went wrong with the newer container.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This adds LLVM 21 compatibility by adding the relevant code changes
guarded by `#ifdev`-statements for backwards compatibility.
Avoid deprecated LLVM calls for 21+
Additionally also add CI checks for this.
---------
Signed-off-by: Christian Heusel <christian@heusel.eu>
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Co-authored-by: Larry Gritz <lg@larrygritz.com>
Adds a new free function rs_allocate_closure which allows the renderer services to provide memory for closure storage, for both inbuilt (add/mul) and user defined closures.
The existing osl_* closure handling functions now use this new free function.
The CPU side fallback is to call through to the existing closure pool implementation. The GPU side fallback returns null.
testshade/testrender provide an example implementation showing how to implement a stack based closure pool for the GPU.
---------
Signed-off-by: Curtis Black <curtis.w.black@gmail.com>
When the ISA-specific dll is loaded dynamically by the batched shading system,
it was expecting the lib prefix to be there.
Signed-off-by: Alex Fuller <boberfly@gmail.com>
The supports_isa() function might be called from threads, and prior
to this change it might have added entries to the global sCpuFeatures
variable. This happened when detect_cpu_features() is called with
TargetISA::UNKNOWN and the code was looking for the best ISA. This
could lead to situation when the host CPU is ARM, and the probing
happens for AVX512.
This change ensures that the supports_isa() accesses sCpuFeatures
in the read-only manner.
This was originally noticed as unreliable OSL render tests in Blender
reported at https://projects.blender.org/blender/blender/issues/147642
Signed-off-by: Sergey Sharybin <sergey@blender.org>
Something must have changed on the GHA runners? Or a compiler version?
On one test job, we started failing this one test, the noise pattern
changed a few days ago, independently of any change to OSL repo.
Bumping up the comparison thresholds slightly appears to do the trick.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
I spotted two places where we used a *nonexistant* OSL_GCCVERSION
macro. This is never defined.
In platform.h, I think it's a simple typo and should be
OSL_GNUC_VERSION, which is the macro we actually use to hold the GCC
version, or 0 if it's not being compiled by GCC.
In wide_opcolor.cpp, it's odder and seems like also a cut-and-paste
error. At the very least it should be OSL_CLANG_VERSION since it's
part of a clause that also tests that clang < 14.0. But upon staring
at it, I can't help but wonder if this spot should *also* have a
clause `OSL_GNUC_VERSION ||` to make this simd loop macro work properly
for GCC, as it did in the other one in platform.h?
And should the one in platform.h include clang, or no?
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Our minimum gcc 9 and clang 5 lets us simplify a few preprocessor
conditionals.
Also give more clear cmake messages when too-old compilers are
encountered, for gcc, clang, and Apple clang.
Define OSL_GCC_ONLY_PRAGMA, which is handy in the future.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Drop support for llvm 11, 12, 13.
Simplify lots of conditional compilation.
With 14 as the minimum, the new pass manager and opaque pointers are
both supported, so we are able to get rid of all code related to
support of the legacy pass manager, as well as typed pointers. This
is a lot of removed code!
We also simplify the testing matrix as a result of having a less
expansive range of LLVM versions to consider.
When updating INSTALL.md, I noticed that we still had **bold** for
"new minimum dependencies" left over from when we first released 1.14.
So I'm unbolding them now, except for the ones we actually have
changed for 1.15.
I had originally intended to pull the minimum to 15, but actually that
makes it more difficult to continue CI testing vs VFX Platform 2022,
since the ASWF-docker container for 2022 only support up to LLVM 14.
And simultaneously, once I removed the code for legacy PM and typed
pointers, there was, in comparison, only a small amount of additional
simplification we could squeeze out of moving the minimum to llvm 15.
So for OSL 1.15, we will have a minimum llvm 14, and then for OSL 1.16
when we would feel more comfortable droping all VFX Platform 2022
support and move to the 2023 containers as our oldest testing
environment, we can bump to llvm 15.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
When building with:
- OptiX enabled (OSL_USE_OPTIX)
- Code Coverage disabled (NOT CODECOV)
- a parallel build
various components of the clang toolchain are likely to crash
as the testshade and libtestshade builds run in parallel and
can overwrite each other's intermediate results.
This chain adds explicit dependencies to serialize these builds,
avoiding toolchain crashes at a minimal performance cost.
---
Signed-off-by: Jean-Francois Panisset <panisset@gmail.com>
ci: Fix Windows CI
Recap: GHA Windows runners recently upgraded their MSVS to a version
whose std headers no longer get along with the LLVM 18 we had been
using for CI (which we download from the LLVM project's hosted binary
builds). But upgrading to LLVM 19 or 20 didn't help, because all of
the LLVM-hosted binaries of those releases were built with the flags
selecting the static runtime library, which is different from the
dynamic choice that's a more common default and which we used for both
our build of OSL and its other dependencies. You can't mix and match.
(The reason for this switch is related to their changing to build by
default using a certain malloc repacement library, which somehow leads
to needing the static runtime.)
To dig us out of this, Chris Kulla was kind enough to build us an
LLVM 20.1 from source, using the old dynamic runtime choice, and host
that as a zip file on a google drive.
This PR makes use of that as the LLVM we us for our Windows CI jobs,
and now we pass CI again.
This doesn't seem like a sustainable long-term solution, because if
the LLVM project is changing the default linkage, we don't want to
need to maintain separate custom builds just for us, surely we can
adapt by changing how compile flags on OSL and all of our
dependencies. But at least this gets us unstuck for right now.
Also as a reminder, our OSL Windows CI at the moment just does the
build, not a testsuite run. We'll tackle that separately.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We weren't testing against gcc-14 nor against C++20 in our test
matrix. Add them.
The fixes required were:
* Had to suppress some warnings gcc-14 complains about in LLVM's
headers.
* There were some tricky interactions between C++20 and fmtlib, which
is because when being compiled with C++ >= 20, fmt switches to a
bunch of tricky runtime checking that has different needs for
constexpr and consteval. I had to rearrange some things a bit to
make it happy simultaneously with both old and new.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This moves the dielectric implementation to BSDL and adds energy
conservation via Turquin style multiple scattering. That is, we create a
microfacet based reflection refraction model where we precumpute the
energy loss to a lookup table. Then at render time we renormalize the
lobe to remove the energy loss.
In addition we use Bounded Visible Normal sampling to avoid rejecting
reflection samples. This improves noise significantly at the cost of
losing valid refracted directions. This doesn't produce energy loss
thanks to the renormalization, but biases the the refraction for high
roughness and low IOR. The result is still plausible and very hard to
see which one is correct.
Also keeping the ability to layer a reflection only coating on top of
other BSDFs via layer().
Signed-off-by: Alejandro Conty <aconty@imageworks.com>
Add pybind11 3.0 to the CI test matrix.
Found a place where we defined a python method twice, fixed that.
Ran into some trouble where the Mac runners which used a pre-built
OIIO supplied by Homebrew, and apparently compiled against pybind11
2.x. I think maybe there are some corner cases where if you have two
python modules that share types (TypeDesc and a few more, in this
case), they should not be built with pre-3.0 and post-3.0 versions of
pybind11?
The quick fix was to switch to having OSL CI on Mac just build an OIIO
from scratch to ensure that it was using the same pybind11. But that
identified yet another problem (which I also think is a runner issue)
that required some minor adjustment to how OIIO auto-builds libpng.
There is a patch in review on the OIIO side for this. But since that's
not merged yet, I have doctored this PR to pull OIIO from my own
fork, the branch containing the fix, just to show that it works. After
the merge on the OIIO side, I will remove those customizations from
here before this PR is merged. (And delete this paragraph.)
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Share Shading Context when optimizing/jitting a shader vs. grabbing an additional context.
No need for a 2nd set of heaps/buffers when the existing context can be reused. Reduces memory footprint which increases cache efficiency, especially for threaded usage.
We can just swap the TexturePerThread info out for a null pointer and restore the previous one afterwards
For completeness we also process the errors in the context.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
These were discovered when moving SPI's renderer to the the current
release and switching to use the rs_ functions.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Turquin style multiple scattering is very simple, but also way noiser
than using a diffuse lobe for MS. This is because sampling microfacets
for reflection, the way we do now, is far from ideal at high roughnesses.
Switching to the reflection optimized sampling technique from "Bounded
VNDF Sampling for Smith–GGX Reflections" fixes all the trouble. This
sampling avoids sampling micronormals that are going to reflect the ray
under the surface (and therefore throwing away samples).
Paper:
https://gpuopen.com/download/publications/Bounded_VNDF_Sampling_for_Smith-GGX_Reflections.pdf
Signed-off-by: Alejandro Conty <aconty@imageworks.com>
Adds directives to prioritize the includes from $OpenImageIO_ROOT in case there is an older (or newer!) system install of OpenImageIO. Without this, OSL might use the includes in /usr/include/OpenImageIO yet link to the libraries in OpenImageIO_ROOT thus producing namespace clashes.
Signed-off-by: Clifford Stein <cstein@imageworks.com>
ShadingSystem() wasn't checking whether the offset returned by interactive_param_offset() was valid or not before performing the ReParameter() memcpy(). So, we could copy data to a location outside of a valid memory block and cause a crash.
Signed-off-by: Clifford Stein <cstein@imageworks.com>
I recently discovered that the caching of our ccache was either not
working at all, or was much less efficient/helpful than I thought. So
there are a number of improvements here:
- Split caching into separate restore/save, and rearrange the step
order so we save cache before tests run. This allows us to save the
ccache even for jobs with failed tests (where the job fails), or for
jobs that we purposely end early (maybe because we see failing
tests), not only when all tests finish and succeed. This makes
subsequent builds, that we do on the same branch while attempting to
fix failures, go faster by using the cache.
- Make subsequent pushes to the same PR or REF to cancel any previous
jobs, so if we push updates before the last CI run of the same
branch completes, it won't keep those going as useless zombie jobs.
- Change the cache key names we use to remove the github ref to
encourage cache reuse across different commits or branches (I
think), and to fix a couple errors where we had mis-named things and
were sharing caches between jobs when we shouldn't or vice versa.
- Be more careful about the location of the cache, making sure we were
for sure telling the GHA cache action to be saving/restoring the
same directory location that we told ccache to use as its cache.
- Use CCACHE_COMPRESSION env var to be extra sure we're using
compressed caches to be sure we get the best bang for buck on the
limited cache storage we're allowed.
In all, there were a number of subtle things broken. Give it a good
scrubbing. I'm not really sure exactly which one of these, or which
combination, was most responsible for getting things unstuck, but it
definitely works! Some results from two test pushes in a row I did
after these changes:
build deps build osl
VFXP 2024 / cold 4:12 6:11
VFXP 2024 / cached 1:42 (2.5x) 0:48 (7x)
ABI check / cold 4:11 6:14 + 5:38(abi ref)
ABI check / cached 0:57 (4.4x) 0:46 + 0:16 (11.5x)
bleeding edge / cold 8:12 9:57
bleeding edge / cached 1:39 (5x) 0:44 (13.6x)
It varies across jobs and from run to run, but let's call it something
in the neighbourhood of 4x speedup building dependencies and 10x
building OSL itself. It's not better because it doesn't speed up
linking, and not all of the dependencies' build systems are doing
things in a way that is amenable to ccache inserting itself into the
process. Still, this should be a big help in reducing the time of
our CI runs.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This moves the BSDF to libbsdl so we use the albedo tables
infrastructure. Adds roughness regularization and energy
compensation (multiple scattering, Turquin style).
I removed the thinfilm options that were not implemented and
don't exist in the MaterialX spec. Also I propose thinfilm to
be implemented as a decoration via vertical layering so we
don't repeat the parameters in every BSDF/closure.
Signed-off-by: Alejandro Conty <aconty@imageworks.com>
At long last, turn on full execution and testing for OptiX in our CI,
using the GitHub GPU-enabled runners.
This should greatly reduce the possibility of build breaks from not
actually testing OptiX rendering, not to mention making it possible to
make changes on the OptiX code path and have it confirmed by our usual
CI.
Previously, we were building against the OptiX headers, so at least we
wouldn't make an obvious build break, but we weren't executing the
testsuite on GPUs. So it was still possible to merge code that broke
that code path in practice. For PRs that looked obviously risky, I
would pull them to my work machine, build and test with real GPUs to
confirm. But this was a big pain, extra trouble for me, and very error
prone -- sometimes I didn't recognize that a PR needed that extra
testing, or sometimes I forgot entirely, and merged broken code.
Note that this depends on PAID GitHub runners with GPUs. The bill is
paid by ASWF, but it means those runners are only available on the
ASWF GitHub account -- that is, they will run for PRs once submitted,
but not when you merely push to your own account's fork, sorry. When
running on a non-ASWF account, it will select an ordinary free runner
and build OSL against the OptiX headers, but will not run the
testsuite since there is no GPU.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
NVIDIA recently set up a GitHub repo with the OptiX headers (with
version tags!), which is much better than the unreliable semi-secret
location we used to use.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
There was still some logic that was only needed during the period that
we supported both OptiX 6 and 7+.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Sony Pictures Imageworks BSDF library (working title BSDL) is here
published and introduced to testrender. It is a header only library
where everything is inlined and can be used on GPU. Although we haven't
kept the BSDFs up to date with the latest research in the last couple of
years, I hope you, at least, like the infrastructure. There are three
main items in this PR
1. Put BSDL as an INTERFACE cmake library in src/libbsdl.
2. Introduce the switch/case automatic pseudo virtual methods in
testrender to handle all the existing BSDFs.
3. As an example integrate SPI's ThinLayer BSDF in testrender and add
a test for it.
BSDL is a self-contained lib depending only on Imath. There is not even
a need to link to OSL and it can be used as a build only dependency.
Inside libbsdl directory there is a README with further information, but
documentation is the usual weak point with production code. We use this
code as-is directly for making movies with our in-house render, and we
welcome everybody to contribute theirs. The name "BSDL" is one of my
half-broken name ideas and I'm very much open to suggestions.
I recommend to first look at my changes to testrender, integration was
very easy. Then dive in the internals if you want. Credits for this code
go mainly to Chris Kulla and me (Alejandro Conty), but there are also
other smaller contributions from other Imageworks members, including
alumni Laura Lediaev who wrote the LTC sheen implementation.
Signed-off-by: Alejandro Conty <aconty@imageworks.com>
The batched libraries seem to require at least one symbol provided by liboslexec : `OSL::v1_14::Strings::null`. If this is not available, then attempts to do batch shading result in the following error at runtime :
```
ERROR: /home/john/dev/build/gaffer-1.6/lib/lib_b16_AVX512_noFMA_oslexec.so could not be loaded with error "/home/john/dev/build/gaffer-1.6/lib/lib_b16_AVX512_noFMA_oslexec.so: undefined symbol: _ZN3OSL5v1_147Strings4nullE"
LLVM_Util::call_function(osl_b16_AVX512_noFMA_allocate_weighted_closure_component_masked, args), requested function name doesn't exist in the current module!
/home/john/dev/gafferDependencies/OpenShadingLanguage/working/OpenShadingLanguage-1.14.5.1/src/liboslexec/llvm_util.cpp:4370: call_function: Assertion 'func' failed.
```
This came up in Gaffer, where we have a plugin that uses `liboslexec` to do geometry and image processing. Why does Gaffer require this fix but other clients such as `testshade` don't? My theory is that this is because `testshade` is an application, and since it links to `liboslexec` itself, the symbols are available to the batched library when it is opened. Whereas Gaffer does not link to `liboslexec`, but loads a plugin which does, meaning that `liboslexec` symbols do not become generally available. Using LD_PRELOAD to load `liboslexec` upfront also works, and I suspect loading the plugin with `DYLD_GLOBAL` probably would too, but both are more drastic than fixing the linking at source.
I don't know too much about how the batched libraries are loaded, so if there are better solutions I'm all ears.
Signed-off-by: John Haddon <jhaddon@cinesite.com>
It had escaped my notice that last month, GitHub ended support for a
single ISSUE_TEMPLATE.md, and now only supports an ISSUE_TEMPLATE
directory with individual templates. I can't imagine the engineering
reason for not being able to continue to support having just one
template as always.
Anyway, so this PR conforms to what they support, by copying (and
lightly modifying) the issue templates that OIIO has used for some
time.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The combination of cmake 4.0, changing homebrew default package
versions, and slight changes rolled out to the GitHub Actions runner
images conspired to break our CI for 2 of 3 Mac variants. Fixes:
* Apple: only add the -isysroot to the bitcode-generating clang++
command if CMAKE_OSX_SYSROOT is defined.
Plot twist -- this is no longer set by cmake starting with cmake 4.0!
* Fix incorrect warning message for apple clang vs newish llvm
in externalpackages.cmake.
* Remove homebrew unlinking of python@3.10 that warned in
install_homebrew_deps.bash. On the GHA runners, it was giving us
warnings because we were asking this package to be unlinked but it
wasn't installed. For future needs, add an env variable based way
to unlink individual packges.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Only use markdown -> readthedocs for the language specification. No
need to continue maintaining the LaTeX version, it only makes it
confusing for people who don't know which they should be updating.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Optional texture argument for colorspace (part 1 -- don't reject)
A while back, we proposed a new optional token/value parameter for the
texture functions: "colorspace", followed by a color space name, which
indicates that the texture name referenced is understood to be in that
colorspace (this is passed to the TextureSystem, which has known for
some time how to ensure the right transformations are applied as tile
data is read in).
We do not fully implement this yet! It's more involved than I have
time to get right by the scheduled OSL 1.14 non-beta release (which
was supposed to have been on Apr 1, oh no). Not wanting to delay
things further, this change at least causes the use of the new
parameter (which apparently, MaterialX has been using) to be silently
accepted and ignored, rather than generating runtime errors. We'll
come back right away for a more complete working implementation.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We're not LLVM 20 compatible yet, we'll deal with that separately
and not under the time pressure of a major releases that's overdue.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The tricky part is that several dependencies break because of cmake
4.0 dropping support for strict compatbility with cmake < 3.5. Have to
resort to some trickery with CMAKE_POLICY_VERSION_MINIMUM overriding
what those packages declare via their own cmake_minimum_required(),
and hope it works.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Bump to the latest in several cases. Also make sure the INSTALL.md
is in line with what we support and/or test.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The ptx cache key introduced in #1938 didn't include a shader's entry layer name in its cache key. The direct callable function generated during llvm construction includes this layer name, so we could get false positive cache hits when two identical networks with matching group names had different entry layer names. The end result is an optix compilation error. This PR extends the cache key to handle this scenario.
Signed-off-by: Chris Hellmuth <chellmuth@gmail.com>
ci: improve clang-format
Make a new run-clang-format.bash utility to issue the commands
for the right files. No "build" is requred, and the only dependencies
we need are the minimum it takes to run clang-format itself.
These changes reduce the approximate total elapsed time of a CI
clang-format job from 8:00+ to under 0:30.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Recent PR 1852 (Free functions for texturing, point clouds, and trace)
added host/device decorations to certain `osl_` functions that
previously were replicated in OptiX-based renderers. This means that
now the ones in those renderers will be seen as duplicate symbols.
The more clear intent that we'd like to work towards is that all the
`osl_` functions are meant to be "internal" to the liboslexec or the
bitcode it provides, and the `rs_` functions are the customization
points that the renderer is expected to supply.
Except... a bunch of the `osl_` functions that set parameters for
subsequent texture and trace calls do not yet have `rs_` customization
points. So we're actually not quite ready to change things.
To unbreak, then, this PR removes the `__device__` decorations (so
they are again safe/expected for GPU renderers to override them).
To keep testshade/testrender working, we add copies of them to
testshade/testrender specifically.
Like I said, eventually we'll add `rs_` customization points, and at
that point, we'll truly expect the `osl_` functions to be only supplied
by oslexec itself and not the renderers.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Fix issue #1929 with batched execution forcing interpolated integer arguments to be boolean when their default value is 0 or 1.
In actuality the renderer could assign any integer value, we update BatchedAnalysis to track the interpolated value as a write event
which will disqualify forcing of boolean as well as ensure future writes are masked properly.
Added new lockgeom test that reproduces the issue and verifies its fixed.
---------
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
Apparently, in LLVM 18, some things shifted between libraries and that
requires a slightly different link order. But only on Linux, or maybe
only with static libraries, so even though our Mac and Windows CI
tests have used LLVM 18, it wasn't symptomatic there.
We didn't discover this earlier because we didn't actually test
against LLVM 18 on any of the Linux CI runs. So we add that.
Also, to alleviate the pain in adjusting LLVM versions and often
needing to modify our build_llvm.bash script, I'm trying out a GHA
marketplace action KyleMayes/install-llvm-action, and it works well
and saved me time, so we may switch to that for other cases over time
as well.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
It was checking a "while" loops condition variable before the condition had executed causing false positive.
It was checking all elements of arrays causing false positives when only a subset of the array is actually being read from in "pointcloud_get", "spline", "splineinverse" calls. Implementation detail: in batched mode, to handle a varying count or knot count parameters a new version of osl_uninit_check_values_offset was added.
Fix message formatting for batched debug_uninit messages which was printing out pointer to the string instead of the strings themselves.
debug-uninit in testsuite was update to check for false positives for while(condition), spline, and splineinverse as well as correctly identifying uninitialized use of partially initialized arrays with spline and splineinverse.
Adding test for pointcloud_get was omitted due to local build configuration.
---------
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
This change adds a caching layer to the OptiX PTX pipeline that can skip LLVM generation and optimization. This helps improve scene build times for re-renders of scenes with large shader counts.
We generate a hash key from the optimized shadergroup, and depend on the renderer to provide a cache backend implementation. There is no overhead if the renderer doesn't explicitly opt-in to ptx caching.
A simple backend will be added to testshade/testrender in a follow-up PR.
As background information:
The sequence at runtime is group oso ->1-> runtime optimize by liboslexec ->2-> JIT to PTX via LLVM ->3-> driver converts PTX to actual executable GPU code on that hardware.
The "OptiX Cache" (part of OptiX & driver) speed up step (3) by not having the last step for optimized/JITed shaders it's encountered before.
This PR adds another cache to step (2), managed by OSL and/or the renderer internals, to allow you to skip the bulk of the work for that step for optimized shaders you've encountered already.
You still pay full price the very first time a shader is encountered, and that leads to terrible TTFP (time to first pixel). But this should take a big bite out of that in practice since it's very common to have encountered most shader configuration before. If the implementation the renderer provides is to store the cache persistently on disk or in a real database, it will be shared from run to run and possibly from user to user.
---------
Signed-off-by: Chris Hellmuth <chellmuth@gmail.com>
This changelist proposes improvements to the energy conservation between layered BSDFs in OSL.
The rational quadratic fit for GGX directional albedo from the MaterialX project has been integrated into `dielectric_bsdf` and `generalized_schlick_bsdf`, improving visual parity with MaterialX GLSL and MDL.
Signed-off-by: Jonathan Stone <jstone@lucasfilm.com>
---------
Signed-off-by: Jonathan Stone <jstone@lucasfilm.com>
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Co-authored-by: Larry Gritz <lg@larrygritz.com>
For any individual test running, the OSL_TESTSUITE_THRESH_SCALE
environment variable, if set, will scale the image comparison
thresholds used to decide if a test passes.
Furthermore, a special OSL_TESTSUITE_OPTIX_THRESH_SCALE environment
variable, if set, will cause *all* OptiX related tests to be set up
with scaled thresholds.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This PR adds an API to ask the shading system if derivatives are required for attributes needed by a shader group.
Signed-off-by: Curtis Black <curtis.w.black@gmail.com>
I think this stems from libclang (doing the C preprocessing step
for us) giving us relative paths on all platforms but Windows,
where it gives absolute paths -- with backslaches and C:\ and all.
I'm going to be conservative and not switch to relative paths accross
the board on Windows, but for now just restrict it to the builds we do
for CI, in order to make the related tests pass for Windows on our CI
(where we obviously don't want to have the reference output need to
bake in absolute paths of the place they may happen to execute on the
GHA runners).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
First step at getting full Windows CI: This PR gets us as far as doing
a Windows build of OSL for each Windows flavor on GHA. This is an
important part of making it much harder for us to inadvertently break
the build on Windows.
We currently only build, but we don't run the testsuite. This is
in progress and we don't pass all tests quite yet, but I didn't want
to wait any longer to get this much in our CI.
This is just about getting our own CI runs for Windows and still relies
on helper scripts being run as part of the CI to build or install the
dependencies before the main build starts. We want eventually to get
to the point where OSL's build system can itself auto-build all the
important dependencies if not found and it's not something most
users need to worry about.
A few notes about specific changes in this patch that came along
for the ride:
* Port many build_blah.bash scripts (or recent changes thereto) from
OIIO to OSL to synchronize them between the two projects and move
improvements made on the OIIO side to OSL's versions.
* Augment the build_llvm.bash script with logic to work on Windows,
too, using pre-built binaries associated with LLVM releases. Note
that not all of their releases have Windows binaries pre-built,
so you have to pick one of the right releases for it to work.
* Fix a problem with the way we propagate C++ std level flags from the
main build to the bitcode-generating pass. On Linux and Mac,
`-std=c++17` works identically on gcc/clang that we use for OSL
itself and the clang we use to generate precompiled bitcode. But on
Windows, the standard-setting flags for MSVC we use to build the cpp
is different from the flags needed for clang when compiling to
bicode.
* Modify the ci.yml to allow special branch names to exclude platforms,
to simplify iterating over many CI runs when only caring about one
platform. If the branch name contains the substring "windows-only",
the Linux and Mac CI test variants will be skipped, and analogously
for "macos-only" and "linux-only".
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Co-authored-by: Brecht Van Lommel <brecht@blender.org>
A while back, the changes of PR 1897 slightly changed the output
of this test under OptiX, so we're belatedly getting around to
updating the reference output.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
To solve a problem finding system headers when building with command line
tools instead of Xcode. Replaces some commented out code, as this should
be a more general solution to the same problem.
Adding arch and minversion for completeness, not because of a particular
known problem, but better to be sure everything matches up.
Signed-off-by: Brecht Van Lommel <brecht@blender.org>
Associate an (optional) displacement shader with each material. On startup, we execute
the displacement for each face-vertex that has a valid displacement shader assigned.
Signed-off-by: Chris Kulla <ckulla@gmail.com>
Ugh, I broke the testsuite under OptiX a while back and it took me a
while to notice. When I added new unit tests for color[24] and
vector[24], I used the format() call, and that's not supported in
OptiX mode. But printf is, so in this patch, I fix it all up and
now the tests pass in OptiX.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Related OIIO PR: https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/4567
Like in the corresponding OIIO PR, I'm proposing switching to a
2-level namespace scheme (basically, `OSL::v1_14` instead of the
current `OSL_v1_14) because that opens the door to possible future
schemes to preserve better ABI compatibility between minor releases.
I would like to squeeze this in before I finalize a 1.14 beta, so that
it doesn't change again during the beta period.
Also, I changed some macro names.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Modeled after what we recently did for OIIO.
Now the new build-steps.yml contains all the logic for the sequence
of how to build and test for any configuration, and the main ci.yml
has the specifications of all the different testing configurations.
The different platforms share the same build steps, so there's a lot
less needless repetition (and opportunity for divergence and mistakes).
Also add MacOS 15 to the CI suite.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Some last minute dependency adjustments before I cut a beta
for OSL 1.14.
* Raise CMake minimum to 3.19 (from 3.15). This is still very
conservative -- 4 years old, even more lax than our usual "3 years
back" rule.
Raising to 3.18 matches OIIO 3.0, and compared to 3.15, it adds Cuda
support as well as the `--debug-find` flag. Further raising to 3.19
-- which in 2025 seems prudent, even though it's beyond the version
chosen by OIIO 6 months ago -- adds more helpful features, including
presets, and Apple Silicon support.
* Raise OpenImageIO to 2.5 (from 2.4). The current release is 3.0, FYI.
* Bump LLVM minimum to 11.0 (from 9.0). This is a pretty conservative
move, still gives several years of back support, but brings a little
code simplification. I expect that next year's OSL will raise the
LLVM minimum by several more versions.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
After reduction of shared dependencies in #1890, there is an unresolved
OSLCompiler constructor symbol when linking liboslexec. Avoid including
the oslcomp_pvt.h header that causes this.
Signed-off-by: Brecht Van Lommel <brecht@blender.org>
* Move RELEASING.md into the dev docs area
* Document signed release tags, which we will start doing now
* Document release note generation
* Add git-cliff config file
* Update SECURITY version documentation
Closes#1856
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Updated README.md
Updated Documentation section with latest Siggraph course and companion repo links. Also added a sentence about the SWinOSL book.
Signed-off-by: Mitch Prater <mprater@laika.com>
Also a little be of typo fixing and docs updating, mostly from
comparing to the same docs in OIIO and picking the more thorough
description.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Identical in spirit to the coresponding PR for OpenImageIO.
The background here takes some explaining...
The ASWF's prepared Docker containers that reproduce a CentOS7-based
environment corresponding to VFX Platform 2021 and 2022 years contain
a glibc that is too old to run a newer version of "node" -- which
GitHub Actions themselves use. The old one is no longer receiving
support, and GitHub finally changed the policy that was allowing its
GHA runners to still run the old version that worked with those ASWF
containers. The new node version that is happy on the GHA runners
requires a glibc newer than the one in the containers. So that breaks
all our CI runs in 2021 and 2022 VFX Platform configurations.
See this thread, if you dare:
https://academysoftwarefdn.slack.com/archives/C0169RX7MMK/p1732574400981949
Anyway, Jean-Francois Panisset proposed and tested a fix, which lets
us continue working (for now) by downloading an old node implemenation
(which is happy with the old glibc) and installing it atop the
conainers, so it's this old one that the GH actions will use. It feels
like it's all held together by chewing gum and duct tape, but it
unbreaks our CI and hopefully it will keep working until all the
studios have switched fully to RHEL/Alma/Rocky 9.x and we no longer
care about testing on CentOS-7 based containers or supporting the VFX
Platform years that are too old to be relevant to the new OS that the
studios are using.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Some of the changes are specifically related to OIIO 3.0 requiring
OCIO and raising the minimum version it needs.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This PR is a continuation of #1829, updated to include the recently added triangle mesh support. It enables full path tracing support for the OptiX backend in testrender. We have tried to share code between the CPU and OptiX backends where practical. There is more sharing in this PR than there was in #1829, which should reduce the maintenance burden a bit.
ID-based dispatch
Virtual function calls aren't well supported in OptiX, so rather than using regular C++ polymorphism to invoke the sample(), eval(), and get_albedo() functions for each of the BSDF sub-types, we manually invoke the correct function based on the closure ID (which we have added as a member of the BSDF class).
```
#define BSDF_CAST(BSDF_TYPE, bsdf) reinterpret_cast<const BSDF_TYPE*>(bsdf)
OSL_HOSTDEVICE Color3
CompositeBSDF::get_albedo(const BSDF* bsdf, const Vec3& wo) const
{
Color3 albedo(0);
switch (bsdf->id) {
case DIFFUSE_ID:
albedo = BSDF_CAST(Diffuse<0>, bsdf)->get_albedo(wo);
break;
case TRANSPARENT_ID:
case MX_TRANSPARENT_ID:
albedo = BSDF_CAST(Transparent, bsdf)->get_albedo(wo);
break;
```
Iterative closure evaluation
Another key change is the non-recursive closure evaluation. We apply the same style of iterative tree traversal used in the previous OptiX version of process_closure() to the shared implementations of process_closure(), evaluate_layer_opacity(), process_medium_closure(), and process_background_closure().
Background sampling
We've included support for background closures. This includes an OptiX implementation of the Background::prepare() function. We've broken that function into three phases, where phases 1 and 3 are parallelized across a warp and phase 2 is executed on a single thread. This offers a decent speedup over a single-threaded implementation without the complexity of a more sophisticated implementation.
```
// from background.h
template<typename F>
OSL_HOSTDEVICE void prepare_cuda(int stride, int idx, F cb)
{
prepare_cuda_01(stride, idx, cb);
if (idx == 0)
prepare_cuda_02();
prepare_cuda_03(stride, idx);
}
```
Tests
I have enabled the render-* tests for OptiX mode. I've added alternative reference images, since the GPU output exceeds the difference threshold on many of the tests. But in most cases the difference between the CPU and GPU output is very small.
---------
Signed-off-by: Tim Grant <tgrant@nvidia.com>
Fix a variety of typos in the 2- and 4-component color and vector
helper structs. Very embarrassing!
Also embarrassing, a typo in the docs had said that the way to
overload the `!=` operator was with a function called
`__operator__ne__` while actually, all along it had been
`__operator__neq__`.
Comprehensive test of color2, color4, vector2, vector4.
New testing header: testsuite/common/shaders/osl-unittest.h
This has utilities to make it easier to make unit tests for shader
functions in our testsuite.
Various other minor fixes to these headers.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The recent free function change (#1852) put the OptiX paths in testrender and testshade in a non-working state (issue #1894). The bitcode for the free functions wasn't plumbed through for OptiX. This PR rolls the free function definitions into the existing "rendlib" bitcode and PTX. This is somewhat of a stopgap fix; eventually we'll want to handle the free function more explicitly.
Fixes#1894
Signed-off-by: Tim Grant <tgrant@nvidia.com>
* Reduce shared dependencies between liboslcomp and liboslexec by extracting some common utility functions to more standalone classes
Signed-off-by: Chris Kulla <ckulla@gmail.com>
OIIO 3.0 will finally warn aggressviely about some long-deprecated
ArgParse methods. Switch to the newer ones.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
OIIO 3.0 shifs some things around slightly and requires us to
include fmath.h directly in a place where we had been inadvertently
relying on it getting included transitively (but now is not).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The primary problem was that Imath's implementation of `operator[]`
took the address of just the `x` component and then used pointer
arithmetic. But that's undefined behavior in C++! Specifically, because
only vec.x is accessed, the compiler may see that .y and .z are never
accessed and not even bother materializing them in memory where the
pointer offset will access it. The fix (for now) is to cast to the whole vec
and take its address, which at least prompts the compilers to make sure
the entire vec is materialized in memory.
Once it was not failing with NaNs or infinite loops, there were still
3 tests that differed every so slightly in a few values for icx (more
lax math, maybe?). So had to add a few alternate reference images.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Provides free function API for:
* rs_texture
* rs_texture3d
* rs_environment
* rs_get_texture_info
* rs_get_texture_info_st
* rs_pointcloud_search
* rs_pointcloud_get
* rs_pointcloud_write
* rs_trace
Each of these match the existing signatures of the renderer services equivalents. A new function rs_trace_get is added to retrieve trace results and is intended to replace getmessage("trace", ...) which is currently not available on GPUs.
The fallback for each of these is to call the existing renderer services functions.
To achieve this, the helper structs TextureOpt, TraceOpt, NoiseOpt are removed from the host-only shading context, and are instead added into the LLVM generated code as stack allocations. These allocations only occur if the shader uses these features.
To allow these free functions to compile on GPUs, the texture per-thread context pointer is currently passed null. Future work will be needed to give this a proper home.
To make the pointcloud functions GPU friendly, they had to modified to not use variadic arguments and to avoid dynamic alloca.
---------
Signed-off-by: Curtis Black <curtis.w.black@gmail.com>
For Dev Days
#1862
Addressing a osltoy feature request, I've added the ability to specify custom search paths.
The changes consist of the following:
- Added a command line option `-I <path>` to specify search paths (`OIIO::ArgParse` unfortunately can't mirror `OSLCompiler`'s `-Ipath` syntax)
- OSLToyMainWindow now stores an array of `OSLCompiler` options alongside a flag indicating whether or not they should be regenerated before compilation. The only option currently able to be specified is `-Ipath`, but this should ease any future extension.
- Added a GUI component consiting of a window with a mutable list of the current set of search paths. This list is initially populated with any paths passed via the command line, but any modifications to the list will be reflected when the shader is next recompiled. Modifying this list does not currently force recompilation.
- Added an entry to the Tools menu to open the new GUI component.
- Added a line of output to the error window indicating what options the compiler was invoked with. I've defaulted to having it print for every run, but this may want to only print on failure.
## Tests
I did not add any automated tests for this feature, as is consistent with the rest of osltoy. I did, however, do some manual testing to make sure the functionality works as expected.
Shaders that failed to `#include` some file instead succeeded when the parent directory of that file was added to the list of search paths via both the command line and the GUI component. Respectively, when said search path is then removed via
the GUI component, any subsequent compilation attempts fail again. The GUI component otherwise behaves as expected.
---------
Signed-off-by: Maxwell Iverson <miverson@imageworks.com>
* Adapt OptixRaytracer to use triangle meshes.
* Refactor the accel build and pipeline creation.
* Remove the old quad & sphere code.
* Change globals_from_hit to use the mesh data.
Signed-off-by: Tim Grant <tgrant@nvidia.com>
* Update the OptiX reference images. Also, fix the outdated output for test_str_ops, which has been broken since the change to string hashes.
Signed-off-by: Tim Grant <tgrant@nvidia.com>
* clang-format
Signed-off-by: Tim Grant <tgrant@nvidia.com>
* Fix build issue when using OptiX 7.0.
Signed-off-by: Tim Grant <tgrant@nvidia.com>
---------
Signed-off-by: Tim Grant <tgrant@nvidia.com>
The MaterialX team noticed this issue here: AcademySoftwareFoundation/MaterialX#2016
Basically, if you add a medium through the anisotropic_vdf closure, it should not force the ior back to 1.0 as this IOR might have been changed by one of the microfacet closures. The IOR already defaults to 1.0 so cases that do not involve a microfacet closure are not affected.
Signed-off-by: Chris Kulla <ckulla@gmail.com>
testrender was originally envisioned as a tiny example renderer that should only handle spheres/planes. Over time several groups have expressed the wish for it to handle arbitrary geometry instead. This PR replaces the sphere and quad primitives with triangle meshes.
I am making use of the (embedded) rapidobj library to load models in the .obj format. The original .xml scene format remains, so that you can combine several models together as well as declare and assign the osl shader networks you want to these meshes. For backwards compatibility, spheres and planes are still supported via tessellation (you specify how many subdivisions you want). In the case of .obj scenes, the shader assignment can be done either by the mesh name or the material name.
I am submitting this to get the review kickstarted. The history of commits includes some fairly large test scenes that should be squashed away to go into the main repo. Files over 50Mb are recommended to use git lfs, but we probably want to figure that out as a different task (possibly handle it via a separate repo).
The handling of derivatives is not totally correct yet, but the behavior is compatible with the previous (incorrect) handling of derivatives we had before.
---------
Signed-off-by: Chris Kulla <ckulla@gmail.com>
This change fixes LLVM find package ignoring the library path provided
by LLVM_CONFIG and picking up a system-wide library instead.
This happens when a custom LLVM is compiled statically, and there is a
system-wide LLVM of the version which OSL expects.
The solution is to pass NO_DEFAULT_PATH to the find_library so that
the function only considers paths explicitly coming from the LLVM's
configuration executable.
It is a bit unclear what is the expected behavior of this module when
LLVM_CONFIG is not available. It is possible to limit possible side
effects of this change by only ignoring default paths if the LLVM
CONFIG executable is found.
In practice this issue happened in Blender's build environment where
HIP is used for OpenImageDenoiser: HIP toolchain pulls in LLVM system
wide. While it is not an issue on itself (as it is only a compile time
dependency for HIP support in OIDN) in conflicts with the logic in the
OSL's find_package(LLVM).
Signed-off-by: Sergey Sharybin <sergey@blender.org>
The GPU calling convention for variadic shadeops is to pass a pointer to the extra parameters as the op's final argument. Currently calls to osl_pointcloud_search don't follow this convention, so their ptx fails to compile due to argument count mismatches.
This PR doesn't implement the full correct behavior, it just adds a nullptr to the end of OptiX calls to osl_pointcloud_search. This makes everything at least compile, so scenes that have pointcloud operations can fail more gracefully on GPU (for now).
Signed-off-by: Chris Hellmuth <chellmuth@gmail.com>
Drop Python 2.7 support.
Upgrade from very old cmake FindPython to findPython3, update idioms.
Simplify a lot of cruft that was supporting python2, as well as
supporting build features we don't use (long ago, I needed some
special library naming conventions at SPI, etc., that we no longer
use).
Change the module filename from "OSL" to "oslquery" to reflect that
it's only covering oslquery and make room for other python bindings in
the future.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Add support for b4_SSE2 batched mode, enabling batched execution for all x86-64 CPUs that don't support AVX.
Quick tests were run to see that output with procedural and texture based materials looked ok and proper SSE2 batched code was being generated for wide/ functions. EDIT: Additionally, all tests pass now.
---------
Signed-off-by: Tuomas Tonteri <e90@tuomastonteri.fi>
On the OIIO side, there's a PR in review that changes the ImageCache
and TextureSystem APIs to return shared_ptr instead of raw.
(https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/4377)
This is destined for the upcoming OIIO 3.0 release.
This patch on the OSL side ensures that we build cleanly against both
OIIO 2.x and 3.0.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The convention in the single-point RendererServices is that the texture call returns
derivatives in st space, and they are transformed to xy space before returning
from the wrapper to RenderServices. This change makes BatchedRendererServices
follow the same convention.
---------
Signed-off-by: Stephen Friedman <sfriedma@pixar.com>
Some explanation is needed:
OIIO master recently added the ability to auto-build OCIO when a new
enough one is not found at its build time, and bumped the minimum OCIO
to 2.2.
Sometimes this is triggered by OSL's CI, when OCIO < 2.2, so building
OIIO also triggers building of OCIO.
All fine, except for the case when we are testing Intel's icx
compiler, which OCIO's build system does not recognize and fails to
build. So we make a little modification here that lets us, just for
that case, ensure that OCIO is pre-built with gcc even though we're
building OIIO and OSL with icx.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
For those who didn't see this get added in OIIO land a few months
back, the highlights are:
* set_utils.cmake -- variable setting utilities, including
replacements/wrappers for set(CACHE) and option() that have a few
extra bells and whistles, including that they allow environment
variables of the same name to override the default values.
* dependency_utils.cmake -- moved the prior contents of
checked_find_package.cmake here, added more features, including...
* An end-of-config dependency report that prints which dependencies
were found and their versions, which were found but were too old,
which were missing.
* The skeleton of an automatic local dependency build facility. The
short story is that if a dependency is not found or is too old AND
it can find a file called src/cmake/build_PACKAGE.cmake, it will run
that file and expect it to download and build the dependency on the
fly. I haven't added any of those scripts here yet, but on the OIIO
side we already have a few to build certain critical dependencies.
Future PRs to OSL will also add the missing pieces to build the
dependencies.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* fix: Handle OIIO deprecations
* int: Use ustring::strhash if it exists (not Strutil::strhash)
If it exists, ustring::strhash has some planned special properties.
* Deal with OIIO deprecations
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
... instead of long-deprecated parallel_options.
This is an API compatible change because OIIO itself has a paropt
constructor that takes a parallel_options, and I'll leave that
deprecated but present for a bit longer in OIIO.
For the sake of link compatibility, we leave the old function defined
in shade_image.cpp, so this should not present an ABI change. We'll
remove the old version later when it's safe.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This patch fixes a regression that went unnoticed during the ustringhash conversion.
Right now OSL expects strings to be represented at runtime as ustringhashes, but interactive parameters' values are still being stored as ustrings. This requires fixing in two locations, first the code that sets the initial value of an interactive parameter, and second the code that copies updated parameters into the interactive buffer.
Tests: New reparameter-strings test.
Signed-off-by: Chris Hellmuth <chellmuth@gmail.com>
This was breaking CI for us. Hard break for icc/icx tests, since those
REQUIRED a yum install of the intel compilers. Softer undetected break
for all the ASWF <= 2022 containers based on CentOS 7, which were not
failing outright but were failing to install certain optional
packages.
The solution (for now) is to configure yum to exclude the missing repo.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Various minor changes to eliminate reference to some long-ago
deprecated OIIO items (mostly which simply alias to std versions)
before then disappear for real.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This does not change the minimum C++ standard, which is still C++14,
even in main. But it does change the default in main to build in C++17
mode, and to get 14 you have to say so explicitly.
We do anticipate bumping the minimum before this fall's release, and
this is just a half step to get us ready. Maybe it will help shake
loose any compatibility problems and remind people on main that they
are using a branch that in a few months will have a new minimum C++.
This will not be backported to the release.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Add Zeltner-Burley sheen BSDF as a new mode of the MaterialX sheen closure.
Simplify math for computing tangent frame aligned with a specified tangent vector.
Signed-off-by: Chris Kulla <ckulla@gmail.com>
Fix the subsurface_bssrdf parameters to use effective color and radius instead of single scattering parameters
Signed-off-by: Chris Kulla <ckulla@gmail.com>
The backfacing shadeop now directly accesses the backfacing field
on the shader globals rather than calculating it from N and I.
This updates the shader-globals-needed query to reflect that
change.
Signed-off-by: Stephen Friedman <sfriedma@pixar.com>
Fixes to documentation of new oren_nayar_diffuse_bsdf parameters
* The new parameter was explained, but not added to the function
prototype in the docs.
* The semi-deprecated languagespec.tex was fixed, but it slipped our minds
that we've shifted the docs to markdown for ReadTheDocs (the real place
we should have changed was stdlib.md).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Implement new EON mode for Oren-Nayar closures (via an opt-in keyword arg for backwards compatibility)
* Add test for energy conserving Oren-Nayar (on half the sphere so we keep checking the old mode as well)
Signed-off-by: Chris Kulla <ckulla@gmail.com>
OptiX inlines all calls to optixTrace at module compilation time. Due to OSL's
lazy layer evaluation, if layer A calls layer B M times, and layer B calls
layer C N times, that can lead to M*N inlines of optixTrace if layer C contains
a trace operation.
In practice we've observed single trace ops being inlined hundreds of times,
leading to minutes-long shader compilations.
This patch adds a new option, lazytrace, to run layers with trace ops
unconditionally at the start of shader evaluation. This costs some potential
performance in cases where the trace layer would never be evaluated, but
removes all the compilation penalties that the inlining was presenting.
---------
Signed-off-by: Chris Hellmuth <chellmuth@gmail.com>
A few months back, PR #1733 seems to have switched the order that
testrender destroys the shading system versus the renderer (services).
This made some subtle bugs that were only symptomatic for GPU renders,
but it's because of the destructor order, where the shadingsystem's
dtr still references the renderer, which cannot be destroyed yet.
The clue is that the SS's constructor takes the RS pointer as an
argument. The RS, then, must have been constructed before the SS, and
therefore we should expect it to be a requirement for the RS to
outlast the lifetime of the SS. (Complex objects should be destroyed
in the opposite order that they were constructed, if they contain
references to each other.)
One code change is needed to avoid the sanitizer errors that the
incorrect change was originally meant to address: clear shaders when
SimpleRayTracer clears.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Fix references to old OIIO home
* Fix embarrassing release notes generating script botch that
accidentally used OpenImageIO URLs where OSL URLs were needed.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Replace the last few uses of boost with robin-map (which is already a dependency for OIIO, although it is new for OSL itself) and functions in OIIO.
Signed-off-by: Chris Kulla <ckulla@gmail.com>
We've been C++14 minimum for a long time now, we can just say constexpr.
Change a couple of stray OIIO_CONSTEXPR14 occurrences as well.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
fix: Fix typo
This is the only way that makes any sense -- define OIIO_HIDE_FORMAT
if it ISN'T already defined.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Especially for our "bleeding edge" test that builds from the current
master of several dependencies, sometimes our tests fail not because
of anything we've done, but because of a change in one of the
dependencies. To help track down exactly which change broke the
dependency, make sure our CI build logs print the specific commit hash
of those packages that we built.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Some methods and enums got renamed from LLVM 17 -> 18, etc.
Also some adjustment to warning messages about version mixing
between clang and llvm.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Remove the only use of boost::thread_specific_ptr.
This had already been marked as deprecated, and there is concensus that the overhead does not outweight the slight inconvenience of creating one object per thread
Signed-off-by: Chris Kulla <ckulla@gmail.com>
Background: Intel's icc is essentially discontinued and we think all
the renderers using OSL have moved on to the newer icx, but Alex Wells
still likes to sometime use icc for its better facilities for some
SIMD code generation sleuthing that he does. So we keep it in our CI
matrix to esure we don't break it for OSL until it is truly
unavoidable or he doesn't feel like he needs it anymore.
Some recent changes to OIIO's simd.h, from PR 4187 in that project,
seem to tweak what appears to be a compiler bug in icc -- bugs were
also noted in OIIO in the course of this PR and some workarounds were
put there just for icc to make OIIO's own tests pass. Well, once that
went in, OSL's CI tests for the icc case also started failing. It's
only texture derivatives that are broken, and in a weird wy that
smells like a compiler icc bug with SIMD generation similar to what we
discovered on the OIIO side.
Like I said, this is a discontinued compiler that no major renderers
are still using, so it doesn't seem worth tracking it down any further
to find a true fix. So let's just lock that one test case down to OIIO
2.5 (the current release) to avoid the changes that will eventually be
in OIIO 2.6/3.0 and hope that we fully retire icc before OIIO 2.5 is
no longer supported by OSL.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
OIIO recently pushed its master to be C++17 minimum. When using a C++17-minimum OIIO, you need to build OSL with C++17 as well. This broke a couple CI cases for OSL.
There were a few spots where we were hard-coded to C++14 on the OSL side rather than even using the C++ version specified for the OSL build, so these are some fixes here to address this.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The isconnected() returns 2 when used on an output that is connected to another downstream shader node -- this can be used to limit computation to only the outputs used. However, that misses the important case of where the renderer has declared a shader node output as a renderer output, which happens often when using OSL purely for pattern generation purposes. This change makes that case also return 2.
The isconnected test has been updated to include testing this functionality.
Signed-off-by: Stephen Friedman <sfriedma@pixar.com>
The (should be) tiny oslquery library was over-including and
over-linking beyond its actual needs.
* Remove`#include <OSL/oslconfig.h>`, which is a lot of setup, type
definitions, and many transitive header includes. Instead, just
include the very small number of headers and types actually needed
by the OSLQuery class definition.
* In particular, this removes includes of OpenEXR/Imath headers and
references to Imath types, the OIIO TextureSystem (!), and other
big things, from what oslquery.h needs to see.
* liboslquery can link against libOpenImageIO_Util (which it uses
primarily for ustring and TypeDesc), it doesn't need the full
libOpenImageIO, since it doesn't read or write image files.
* Remove some link references to Boost which isn't actually needed for
this library.
* This also lightens the load for the Python module for oslquery.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Currently, the GPU will crash when initializing non-string interpolated parameters with default values.
It looks like this is a regression from the ustringhash conversion in 12530f1 (#1732). I believe that before the phase 3 commit, we had global variables on GPU that we memcpy'd the defaults values from. Now after their removal, we're memcpy'ing from an illegal address.
As a proposed fix I removed the entire memcpy branch of the code and let interpolated parameters fall back to the default code path that stores the values directly. I'm open to instead guarding the removed block with "&& !gpu" making the patch have no effect on cpu, but leaned towards having unified code paths. Still, it's not an area of the code I'm very familiar with, any feedback would be appreciated.
Signed-off-by: Chris Hellmuth <chellmuth@gmail.com>
Prevent complications on the GPU by printing error messages at compile time instead of generating code to print them at runtime when a closure doesn't exist.
---------
Signed-off-by: Stephen Friedman <sfriedma@pixar.com>
* Have OSL indicate when it needs fliphandedness in shader globals.
* Add testsuite test to check that calculatenormal() triggers a globals_needed flipHandedness value.
---------
Signed-off-by: Stephen Friedman <sfriedma@pixar.com>
Fix bug in batched codegen of compref when the index is varying.
The loadvalue was left to defaulted to uniform load, which we can't store into a varying/wide result.
Fixed similar bug in getmessage, and added check for spline knotcount to be uniform (although varying support could be added).
This bug exposed a missed optimization in BatchedAnalyser where a early out (return) needs to change the conditional to be varying to support masking. However no check was being done to see if the loop control existed higher in the callstack. So now the execution scope tracks loop nesting depth to see if the current function is really in a loop or not. This allows the loop control flow to remain uniform (avoiding masking and more complex control flow).
Added nestloop-reg to testsuite to reproduce bug and verify its fixed.
---------
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
We didn't change anything on our end, but in the last few days, I
believe the GHA MacOS-13 runners have slowed done somehow, and so now
a few test are failing only because they don't complete within the
ctest timeout period for "small tests". So this just bumps up the
limit on this runner platform to allow the tests to pass.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Pointed out in MaterialX discussions, our spec was not clear what
round() should do in cases where the input value is exactly half way
between two integers. Our design (and implementaton) uses "round away
from zero" policy for that case, matching C/C++ standard round(). But
we were not clear about it in the docs.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Tweak results for ARM tests that have minor numerical differences and
update references.
Set parallel build level to nprocs+2 instead of hard-coding to be
better for the new runners having more cores than they used to.
Speed up macosx build of deps by using unity build of OIIO.
Fix runtest, was clobbering parameters to oiiodiff
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* New build option `CUDA_PREFER_STATIC_LIBS` tries to use Cuda's static
libraries if they can be found.
* Remove code related to nvrt library, it's no longer needed in OptiX 7.
* Changes in checked_find_package to allow a PREFER_STATIC_LIBS
optional argument. Turns out we didn't use this, but I think it's
useful functionality to keep now that I've written it.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Adds a new RendererServices API build_interpolated_getter which allows for building custom free functions that provide values for interpolated datas.
The new API is involved at material compilation time and allow the developers to provide specialized functions leveraging the information known at compile time. It will allow for many optimization opportunities for the compiler by replacing runtime branch with direct memory reads.
This PR is a followup to the API build_attribute_getter proposed in this PR: #1704
TestShade provides an example implementation showing how this compile time information can be used to select an appropriate function to use as the attribute provider, and how to configure the function signature.
Since the feature is enable by default when rs_bitcode is used, the existing testsuite ensure the feature is generating the same result as the previous API.
---------
Signed-off-by: uedaki <thiver.96@gmail.com>
* Delete the NVPTX target machine in the LLVM_Util destructor.
* Initalize m_nvptx_target_machine to nullptr.
* Workaround clang/CUDA12 texref issue.
* Remove the unnecessary null check before deleting the NVPTX TargetMachine.
---------
Signed-off-by: Tim Grant <tgrant@nvidia.com>
A recent change in OIIO master made some of the ustring methods
"trivial", and somehow that makes these variables warn about being
unused, which it did not do before.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We had been using clang-format 14. Bump to 17 as our official
reference standard for formatting.
Moved the clang-format CI test from an aswf container to bare Ubuntu
to make it run faster -- it doesn't need to do a real build, so
there's no point wasting 2 minutes or so downloading a container holding
library dependencies it won't need.
Also fix a duplicate line in .clang-format that newer clang-format
complained about.
Document that it's specifically clang-format 17 that is our formatting
standard
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Bump the "bleeding edge" test to LLVM 17, the "stable latest releases"
test to 16.
Cover clang 17 compilation with MacOS-13 build
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Move OSL library functions and remaining execution time interface from ustring to ustringhash.
Intent is all code for cpu or device just use ustringhash only. This allows us to remove
STRING_PARAM, ustringrep, and OSL_USTRINGREP_IS_HASH. All interfaces and library code
should explicitly use ustringhash or ustring,
or for library functions llvm can call ustringhash_pod or ustring_pod
Compile time strings used in the OSL Library are represented in strdecls.h
and will be available as ustring variables in the Strings namespace as well as
available as unstringhash constexpr variables in the Hashes namespace.
Compile time strings used by the Renderer should adopt a similiar strategy as
exemplified by TestShade which keeps its strings in rs_strdecls.h
and made available as ustring variables in the RS::Strings namespace as well as
available as unstringhash constexpr variables in the RS::Hashes namespace.
Added helper utility functions to generate ustring from ustringhash_pods
Constant string arrays: Previously, array copys required copying data via address
of symbol on host. To enable working on device, replaced memcpy with array allocation
and loading of values. Constant arrays are now present as
global arrays in final llvm module.
On the Batched side compatability/integration:
Batched OSL Library functions continue to use ustrings (not ustringhashes to reduce scope/risk).
Batched OSL library functions that call their scalar counterparts were adjusted
to pass arguments as ustringhash_pods and store results as ustrings
STRING_PARAMS, device_string.h, and string_hash.h: Built-in strings on the CUDA side are now accessed via Hashes namespace, instead of STRING_PARAMS and are
evaluated as ustringhashes. Similarly, DeviceString, StringParams, HDSTR are obsolete, and device_string.h is eliminated.
Hashes.h, which only contains OIIO's strhash and pvtstrlen functions (both formerly in string_hash.h), replaces all inclusions of device_string.h. Namespace Hashes in hashes.h
contains pre-compiled ustringhashes representing strings from strdecls.h.
Texture, dictionary, closure: Helper implementation functions are used by both scalar and batched versions. However, since scalar uses ustringhashes and
the batched version uses ustrings added a flag to inform functions on treating TypeDesc::STRING has either a ustring or a ustringhash
testing: Added testing of bitcode rendereservices for unoptimized OSL shader.
array-copy tests regression: Added new test to verify copying of arrays of unequal lengths
Signed-off-by: Steena Monteiro <steena.monteiro@intel.com>
* Deleted liboslexec/string_hash.h
Signed-off-by: Steena Monteiro <steena.monteiro@intel.com>
* Modified rs_get_attribute to use OpaqueExecutionContext and ustringhash.
Changed example programs--testshade, testrenderer, and osltoy--to use device friendly constexpr (RS|OSL)::Hashes and (RS|OSL)::Strings
variables with rs_strdecls.h vs constructing ustrings on the fly.
Signed-off-by: Steena Monteiro <steena.monteiro@intel.com>
* Fixed clang format issues; added uint64_t casts for constant64; removed unused variable; added array-copy to testsuite
Signed-off-by: Steena Monteiro <steena.monteiro@intel.com>
* Fixed clang format issues in llvm_instance.cpp
Signed-off-by: Steena Monteiro <steena.monteiro@intel.com>
* Added fix in llvm_assign_impl for assignment of non-constant scalar to an array
Signed-off-by: Steena Monteiro <steena.monteiro@intel.com>
* Fixed clang-format in simplerend.cpp.
Signed-off-by: Steena Monteiro <steena.monteiro@intel.com>
* Addressed review comments -- (1) Added ll.constant(ustring().cstr()) in llvm_gen and batched_llvm_gen and (2) Added comments for boolean arguments that indicate ustrings are treated as hashes
Signed-off-by: Steena Monteiro <steena.monteiro@intel.com>
* Addressed comments on Optix build failures
Signed-off-by: Steena Monteiro <steena.monteiro@intel.com>
* Resolved merge conflicts
Signed-off-by: Steena Monteiro <steena.monteiro@intel.com>
* Fixed clang-format
Signed-off-by: Steena Monteiro <steena.monteiro@intel.com>
* Fixed split-reg for batched
Signed-off-by: Steena Monteiro <steena.monteiro@intel.com>
* Replaced llvm_load_stringhash with llvm_const_hash; Changed format string to ustringhash_pod in osl_printf, osl_fprintf, osl_warning, osl_error stubs; Fixed ShaderGlobals' position, u, and v in optix_grid_renderer.cu.
Signed-off-by: Steena Monteiro <steena.monteiro@intel.com>
---------
Signed-off-by: Steena Monteiro <steena.monteiro@intel.com>
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Co-authored-by: Larry Gritz <lg@larrygritz.com>
This can come up if, for example, you build an OIIO using a static
Imath library that's more recent than what you're forced to use for
your OSL build. (Don't ask, you don't want to know.) So we need
to see half.h before we include any OIIO headers. But it's the OIIO
headers that adjudicated where we get half.h from, so we need to
be a little more explicit in, of all places, rend_lib.h of testrender.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Fixes#1755: Build error on MSVC with Partio
Explicitly mark unsupported/unused PointCould copy/move constructors and assignment operators as deleted.
Change SortedPointRecord index type from int to uint32_t and add static_cast during construction to avoid a warning.
Change SortedPointRecord index type from uint32_t to Partio::ParticleIndex to handle 64bit indices (where we can)
clang-format changes
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
Added OSL_INTEL_CLASSIC_PRAGMA(...) which only emits for Intel(r) Classic C++ Compiler, previously OSL_INTEL_PRAGMA did that.
As many Intel compiler specicif pragmas are now supported by icx and icc,
redefined OSL_INTEL_PRAGMA to emit for the classic and llvm based Intel compilers (icc and icx).
Updated some uses of OSL_INTEL_PRAGMA to OSL_INTEL_CLASSIC_PRAGMA as appropriate.
Fixed long icx 2023.1.0, 2023.2.0, 2023.2.1 build times for certain files by removing the "-mllvm -inline-threshold=100000" flag for all but 1 file avoiding the overhead of optimization passes being applied to inlined code that wasn't actually part of the SIMD loop that we truly needed inlined. icx now supports "#pragma forceinline recursive" which allows us to target just the SIMD loops to be inlined (as we did for icc).
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
This patch adds a new test to the CI matrix: verifying that we do not
break the ABI during releases. This is based on similar work done a
couple months ago in OpenImageIO.
Basically, in addition to building the current checkout, it also
builds a checkout of the designated "ABI standard" (generally, the
first non-beta release of a major version), then generates ABI dumps
of both versions and runs the ABI compliance checker using the tools
from https://github.com/lvc
Making two builds sounds expensive, but both builds are rigged to be
minimal, since only the libraries are needed for ABI checks (we can
turn off the binary tools, unit tests, python bindings, etc.), and of
course the test suite need not run. So actually, it's less expensive
than most of the test matrix entries, and it helps ensure that we
don't accidentally break ABI when we don't mean to.
I'm not sure how many false positives we'll have. We might need to
adjust the methodology slightly as we gain experience with this.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
When we bumped to cmake 3.15 minimum recently, we changed lots of
constructs like this:
if (VERBOSE)
message (STATUS "blah")
endif ()
to this:
message (VERBOSE "blah")
but I didn't realize that this was not enough, and to do it right
requires CMAKE_MESSAGE_LOG_LEVEL. So even when VERBOSE=ON, we weren't
actually getting all the verbose messages. Here, we fix it.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Fixes#1750
Batched shading crashes in point_cloud search if "index" is not passed as an attribute.
Fixed by allocating space for indices on stack before calling into underlying search algorithm which requires them (thus crashed if not provided).
Added 2 new tests to exercise it with no indices provided and with indices array provided but not long enough.
testsuite/pointcloud/rdcloud_varying_mismatch.osl
testsuite/pointcloud/rdcloud_varying_no_index.osl
Added alternative test results for
testsuite/pointcloud/ref/out_rdcloud_varying_mismatch.macos.tif
testsuite/geomath/ref/out.icx.alt.txt
Format change for wide_oppointcloud.cpp
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
This is probably the last time we'll regen the PDF docs, since we
are mid-transition to ReadTheDocs. We will probably not directly
distribute a PDF version with 1.14 or subsequent releases.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Stephen Friedman alerted me to the fact that on Windows, bit fields
within a struct will only be combined if they are the same type. We
were a bit sloppy in the defintion of SymOverrideInfo, and that was
causing the struct to end up a different size in Windows. I think the
solution is that all those bit fields should be `unsigned int`.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The MacOS 11 CI seems to be broken on GitHub's runners. I'm not sure
why, but since it's old and getting various deprecation warnings from
Homebrew and GH, I'm inclined to just disable it rather than work hard
to keep this old version of the OS running.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Convert the LaTeX docs to Sphinx + Myst Markdown extension
I think this is in decent shape. It's going to need a lot of thorough
proofreading to ensure that thing got lost or mangled on the way from
the original docs. I suspect we will have several follow-up PRs to fix
thing.
After we're reasonably happy with it, we'll eventually remove the
.tex and the pdf.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
It gave a VERY old link to the SPI CLA, which we no longer use. In
fact, because we have CLA enforcement through the CLA bot, there is
not really a need to have it in the checklist... nobody can forget.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Add a GHA workflow that is triggered up on drafting a release, which
will echo a notice to the ASWF slack `#release-announcements` channel.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Bump minimum supported LLVM version to 17
* Rename `makeArrayRef` to `toArrayRef` to avoid `llvm::makeArrayRef`
deprecation warnings.
* Minor required changes related to `llvm::Align`.
* Disable `initializeInstrumentation` and `createPruneEHPass` which
were remove and appear to serve no purpose now.
* Add assert when using default optimization levels with legacy pass
manager, which is no longer supported.
This depends on both opaque pointers and new pass manager support.
Signed-off-by: Brecht Van Lommel <brecht@blender.org>
* Keep CMAKE_CXX_STANDARD warning for LLVM 16
* Enable LLVM 17 too, adjust for changes in new pass manager PR
* Fix OptiX data layout mismatch with LLVM 16+
i128:128 was already in the NVPTX backend since LLVM 6, the docs
this was taken from seem outdated. Unclear it started failing now,
but the data layout seems clearly incomplete without this.
* Modify CI to run the bleeding edge test on LLVM 16
* Manually make global variable names valid for PTX
* Fix build with older LLVM
---------
Signed-off-by: Brecht Van Lommel <brecht@blender.org>
Co-authored-by: Larry Gritz <lg@larrygritz.com>
Co-authored-by: Tim Grant <tgrant@nvidia.com>
Based on Brecht's tests, we realize that OSL doesn't work at all with
Cuda 8.0 (which didn't support the C++14 that we use), so we're
raising the minimum to 9.0. And there are some flaky tests with 9 and
10, so we're making the recommended minimum be 11.0.
It's not our policy to change the minimum versions of dependencies on
release branches, but I think we should backport this to the release
branch. I believe what this change is doing is not removing actual
support for old versions, but rather, correctly documenting what has
been working (or not) for quite some time.
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Co-authored-by: Brecht Van Lommel <brecht@blender.org>
Use the absolute path to the Python executable to avoid picking up
a different executable with the same version.
Signed-off-by: Brecht Van Lommel <brecht@blender.org>
In #1668 USE_OPTIX changed to OSL_USE_OPTIX, but CI was not updated.
Also fix use of CPU SIMD flags for compiling CUDA bitcode
---------
Signed-off-by: Brecht Van Lommel <brecht@blender.org>
PR #1563 accidentally changed tests to accept failing commands by
default. (This seems to have been a mistake -- a temporary change
added when debugging that LG neglected to remove before submitting
the PR.)
This can cause tests to pass even though the program crashed, when
a previous run created the correct output file.
---------
Signed-off-by: Brecht Van Lommel <brecht@blender.org>
LLVM 16 removes support for default optimizations levels (O1, O2, O3)
from the legacy pass manager, which are used by OSL. This adds supports
for the new pass manager while keeping the legacy pass manager working
for older LLVM versions.
* Enable new pass manager for LLVM 15+
* Both default and custom optimization levels have been implemented
* But the performance and correctness beyond passing tests has not been
verified
* `ptx_compile_group` still uses the legacy pass manager
* Fix unecessary alias analysis and wrong #endif placement
* Fix LLVM 16 compatibility
* Ensure running prevent bitmasks pass is reentrant, and initialize once
* Make compiling with the new and legacy pass manager mutually exclusive
LLVM 17 removes even more legacy pass manager functions. With these
changes LLVM 17 works as well.
* OptiX fix, avoid using legacy pass manager for opt command since it does
not support default -O levels anymore
* Only enable new pass manager for LLVM 16+
---------
Signed-off-by: Brecht Van Lommel <brecht@blender.org>
* LLVM opaque pointers support
New LLVM versions require opaque pointers, that is you can no longer tell
from a pointer value which data type it points to.
* Remove deprecated GEP and op_load methods without type, and refactor
code to always pass the type.
* Add element type argument to op_scatter and op_gather.
* Eliminate pointer type comparison which no longer work with
opaque pointers.
* Keep interactive params marked as uniform in batched analysis, so
we can deduce uniform/varying correctly from just the symbol
* Enable opaque pointers on LLVM 15, except when using OptiX
OptiX opaque pointers were not tested yet and left disabled. It is
likely broken due to the sneaky conversion in
`BackendLLVM::llvm_store_value` between `int64` and `char*`, but we
can no longer compare the pointer type. This function is called from
many places and it's unclear to me when this happens.
According to a4e779c4 this code is only temporary, so this may resolve
itself as the ustring refactoring gets completed.
Signed-off-by: Brecht Van Lommel <brecht@blender.org>
* Fix debug_uninit and pointcloud batched testsß
Signed-off-by: Brecht Van Lommel <brecht@blender.org>
* Variable renaming and comment for clarity
Signed-off-by: Brecht Van Lommel <brecht@blender.org>
* Add/restore debug names for groupdata access
Make batched backend more consistent with the regular backend.
Signed-off-by: Brecht Van Lommel <brecht@blender.org>
* Rename scalar_type to component_type
Signed-off-by: Brecht Van Lommel <brecht@blender.org>
* Alternative interactive param fix based on feedback from Alex
Signed-off-by: Brecht Van Lommel <brecht@blender.org>
* OptiX fixes
Signed-off-by: Brecht Van Lommel <brecht@blender.org>
* Use opaque pointers only for LLVM 16+
To avoid potentially breaking LLVM 15 with OptiX.
Signed-off-by: Brecht Van Lommel <brecht@blender.org>
---------
Signed-off-by: Brecht Van Lommel <brecht@blender.org>
The unconditional dereferencing of the sg pointer is only necessary if
texture_thread_info is not supplied. And for that matter, the
texture_thread_info is only necessary if the texture_handle is not
supplied.
So rearranging the nesting skips a bunch of the work, and the need
for the caller to supply the sg if they have the texture_thread_info,
nor for them to need texture_thread_info if they already have the
handle.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Fixes#1716: Crash assigning boolean expression to int in batched mode
Modifying BatchedAnalysis::establish_symbols_forced_llvm_bool to excluded renderer outputs to fix the issue.
Added new test compassign-bool to recreate the issue and verify it is fixed.
Comprehensive Fix for Issue 1716: Crash assigning boolean expression to int in batched mode
Rearchitected BatchedAnalysis::establish_symbols_forced_llvm_bool to track a BoolStatus per Symbol evaluating all operations that write to the symbol. Only if all ops that write to it always produce a logically boolean result will the BoolStatus be set to Yes.
There is another set of ops that could produce a logically boolean result if their inputs were also boolean. In that case the BoolStatus is set to Maybe and its input symbols tracked (as they may not have a known BoolStatus yet) and deferred for a follow on pass.
After the initial pass is made, if there were any deferred Symbols, we can check their dependencies to see if they have a known BoolStatus and we can set the Symbol's BoolStatus to Yes or No depending on those input dependencies. If the BoolStatus of any dependencies is still Unknown, we just defer it again. We continue making passes over deferred Symbols until we have established a known BoolStatus for them all. Afterwards we use forced_llvm_bool(true) on the Symbols where BoolStatus is Yes.
Improved support for storing forced_llvm_bool Symbol values by automatically converting from int->bool and then a vector of bools (mask) to the platforms native mask type (might be vector of integers on platforms without native mask support).
Fixed bug in BatchedAnalysis where arrays of closures were not being marked as varying, only closures where.
Allow GroupData to hold actual boolean or native mask data type vs. integers when a Symbol is forced_llvm_bool.
Improved batched GroupData layout calculation to use llvm offset and alignment calculations based on the actual DataLayout (might consider upgrading the scalar code gen to do the same).
Fixed bug where results of partial results from RendererServices::get_userdata allowed initops to execute unmasked effectively overwriting the results from RendererServices::get_userdata. Issue stemmed from op_useparam not being modelled properly by BatchedAnalysis.
Added new test compassign-bool to recreate the issue and verify it is fixed.
Added new test closure-layered to recreate the issue and verify it is fixed.
Added new test userdata-partial recreate the issue and verify it is fixed.
Fixed bug in analysis of forced_llvm_bool for getattribute, whose return value is always a bool but result parameter when it is an integer should NOT be forced_llvm_bool.
Added test getattribute-shading to recreate the issue and verify its fixed.
Added testsuite/getattribute-shading/ref/out-scalar.txt to support builds that exclude batched.
---------
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
This bump brings access to the following interesting features (not a
comprehensive list):
- 3.13 Nov 21, 2018
- -B, -S let you specify source and build directories without the "cd" dance
- target_link_options()
- ctest --progress / CTEST_PROGRESS_OUTPUT
- 3.14 Mar 14, 2019
- cmake --build now supports -v/--verbose
- file(CREATE_LINK )
- 3.15 Jul 17, 2019
- env CMAKE_GENERATOR can be used instead of -G
- when using --build, the --target can now list multiple targets (and also can be `-t`)
- --install can be used separately without running through the whole build
- message() now supports NOTICE, VERBOSE, DEBUG and TRACE.
- CMAKE_FIND_PACKAGE_PREFER_CONFIG variable
- Enable C++20 for Apple Clang
Make sure all CI cases have a new enough cmake
Use message(VERBOSE ...) feature to simplify lots of `if` blocks.
Clean up code for older cmake versions
Freshen some of our CI build_<dependency>.bash scripts.
Note that this will NOT be backported to the current release branch,
but it will be part of the upcoming 1.13 family of releases.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We were overzealous in switching everything to ustringhash. The
filename argument to get_texture_handle should allow taking a ustring.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
On SPI Arnold, we are seeing GroupData as a bottleneck for GPU performance. Regardless of whether the buffer is backed by local or global memory, it's off-chip and slow to access.
This change moves qualifying parameters out of the GroupData struct and onto the stack. On GPU this makes those parameters more likely to be stored in registers, leading to better performance.
Qualifying parameter details
The qualifying parameters are all the output parameters minus three exceptions.
A typical output parameter's lifetime is scoped to the execution of the instance it is associated with. The parameter is initialized at the beginning of instance execution, accessed during the body of the instance, and after the instance body its final state is broadcast to all its downstream connections. This lifetime makes output parameters a good fit for stack storage.
The three exceptions:
- output parameters can be connected upstream and written to before instance execution
- output closures are initialized at the beginning of shader execution
- output parameters can be renderer outputs and read after instance execution
These are all identifiable at code generation time and can be flagged as disqualifying.
Improvements
On internal scenes, this change improves performance by about 15% on GPU. On CPU there aren't any performance regressions, but we don't see significant gains either. It also improves ptx generation and compilation by about 10%, and slightly lowers our memory footprint due to the GroupData size reduction.
Tests
Exceptions 2 & 3 are covered by existing tests. A new test is added for exception 1, and for the case of successful optimization. This patch was also run through our internal shading testsuite.
---------
Signed-off-by: Chris Hellmuth <chellmuth@gmail.com>
## Description
This PR is an overhaul of PTX compilation in OSL. The goal is to minimize the size of the generated PTX to help reduce time spent in subsequent stages (e.g., module creation, pipeline linking, etc.).
### rend_lib
The main functional change is in how the functions defined in rend_lib.cu are handled. In the current setup, rend_lib.cu is compiled to a standalone PTX file using nvcc. The functions are considered `extern` in generated shaders, and the dependencies are satisfied when the final pipeline is linked together. Since the functions are `extern`, they are not available for optimization when each shader is generated. This prevents the inlining of small functions, which incurs a lot of function call overhead in terms of PTX size and run-time cost.
So instead of compiling rend_lib.cu separately, it is compiled to LLVM bitcode along with the rest of the "shadeops" sources. This bitcode is used to seed the LLVM Module for each shader. This makes the definitions available when each shader is being generated and optimized, which allows the optimizer to make better decisions. It also allows for better control of inlining decisions which affect the size and quality of the generated PTX.
### shadeops
Another significant functional change is in how the shadeops functions (including those defined in rend_lib.cu) are treated. In the current pipeline, each shader must carry along with it definitions for all non-inlined shadeops functions that are used. This bloats the size of the PTX for each shader, and results in many duplicate copies of those functions existing in the final linked pipeline.
In the new pipeline, the unified shadeops+rend_lib bitcode is compiled to PTX using the offline `llc` tool. This PTX is used by the renderer to create a single "shadeops" module that provides the definitions for those functions. This makes it possible to drop the definitions for those functions from the PTX for each shader, which can significantly reduce the size of the generated PTX.
### quirks
This PR contains some interesting "quirks" which are largely fallout from compiling rend_lib.cu using clang instead of nvcc:
- The clang-generated PTX is processed by a Python script to transform some function & symbol declarations to better match what was generated by nvcc. It might be possible to add the right decorators to the C++/CUDA code to get the correct visibility for these symbols, but I was not successful in doing so.
- It was necessary to change the function signatures for some of the rend_lib functions to use `void*` instead of the actual OSL types (e.g., `OSL::Color3*`). Trying to use the actual types produces an LLVM assertion failure, for reasons that I don't understand.
- I have enabled flush-to-zero (FTZ) for clang-compiled code, to better match the behavior of nvcc. I added a CMake variable to enable or disable FTZ in the event that somebody wants to disable it.
- I have added some alignment hints on pointer accesses in rend_lib.cu. clang does not appear to be able to deduce the alignment of pointers in many cases, which turns things like `memcpy` and `memset` into long series of byte operations. Functions like `osl_get_matrix` are bloated substantially if the hints aren't provided.
---------
Signed-off-by: Tim Grant <tgrant@nvidia.com>
This is just for future-looking OSL 1.13 releases, it won't be
backported to the 1.12 release branch.
Remember that by the time we release OSL 1.13 as a supported release
family, OIIO will also have done its annual release, of 2.5. So our
minimum of OIIO 2.4 means that we will be supporting the current
release, as well as the prior year's release (and, of course, we
always stay compatible with OIIO's trunk as it evolves). So we're
really keeping compatibility with 3 major releases, which seems like
plenty to juggle.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Adds a new RendererServices API build_attribute_getter which allows for building custom free functions that provide values for attributes, effectively replacing both get_attribute and get_array_attribute.
This new API is invoked at material compilation time to allow render developers to provide specialized functions capable of taking advantage of known compile time information. This will allow for many optimization opportunities, such as replacing run-time map lookups with direct memory reads from known locations, remove brancing required for checking if derivatives are needed, branching on type-conversions if required, etc.
This PR currently implements this approach for attributes, a future PR could do the same for user-data.
Tests
TestShade provides an example implementation showing how this compile time information can be used to select an appropriate function to use as the attribute provider, and how to configure the function signature.
---------
Signed-off-by: Curtis Black <curtis.w.black@gmail.com>
* Add the missing thread_index and shade_index members to the copy of ShaderGlobals in testshade/cuda/rend_lib.h.
* Reorder the ShaderGlobals members in rend_lib.h to match the definition in shaderglobals.h.
---------
Signed-off-by: Tim Grant <tgrant@nvidia.com>
The string situation is still a bit tricky on the OptiX side. There's
a full overhaul of that coming separately as we continue to shift to
an all-ustringhash world.
But in the mean time, we were getting some testsuite failures that
were because of a mismatch between times we think of a gpu-side
ustring as a char* and when it's a uint64. IN FACT, they are always
the hashes, but we backed ourselves into a corner where sometimes that
64 bit pattern is passed around as a pointer, ugh.
This small patch attempts to catch this case and just do the
appropriate cast to compensate. It's a band-aid, a more comprehensive
fix is coming, as I said.
This gets us down to just ONE filing optix test case, which I will
tackle separately.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
---------
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Certain fmt library versions don't automatically know how to format
atomics. Explicitly load them to turn into regular ints to avoid new
build errors introduced by the recent journaling changes.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Intel's servers have again updated the default icx that you get when
you install from their yum repository, and the new one seems
incompatible with our libstdc++ toolchain. So lock down on 2023.1.0
for now so that our CI can pass.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Move from cryptically named ASWF/images to more descriptive ASWF/logos.
Add an icon only version of the logo.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Remove leftover logic from when we used TravisCI.
* Pick a default for LLVM_DISTRO_NAME so it works when the script is run
standalone when not on GitHub Actions CI.
* Switch the logic about file location to enumerate only the couple of
old versions that uses the old download location, rather than trying
to enumerate an ever-growing set of modern releases that use the current
download location.
* Comment o say that when LLVM 10+ is the floor, the old case can go away
entirely.
Also bump the CI "bleeding edge" test to use LLVM 15.0.6.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The recently merged PR 1702 fails all the OptiX tests. I tracked down
the problem and it's that the patch was a bit too aggressive about
removing things no longer needed. The intent was that even if you
don't use the new journaling, nothing should break. But it does break
for OptiX, and this patch restores some of the functions 1702 removed
(like the implementation of osl_printf) and some other necessary
stubs.
This takes us from failing all ~150 OptiX oriented tests to failing
only 4, which I believe are unrelated to 1702 and are merely revealed
by them. I will fix those in a separate PR.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Journaling Algorithm for error, warning, fprintf
Removes dependence on ShadingContext and OIIO for logging error, warning, printf, and fprintf calls
for use on CPU and device, and tracks logging calls per thread index and shade index.
All calls to shader execution track thread index; ShaderGlobals contains shade_index (for processing
fmt specification calls) and thread_index for allocating memory per thread.
* Add new customization points in RendererServices for reporting errors, warnings, printf, and fprintf.
These customization points accept a fmtlib style format specifier (vs. printf style) whose argument types are represented as EncodedTypes (encodedtypes.h) and can be subsequently decoded via a utility function.
Removes dependence on ShadingContext and OSL::ErrorHandler for logging error, warning, printf, and fprintf calls
for use on CPU and device during Shader execution. ShadingContext and OSL::ErrorHandler remain utilized during shader compilation/JIT.
Journal Buffer approach: each thread gets its own chain of pages within the shared journal buffer for recording errors, warnings, etc. By maintaining a separate chain per thread we can record errors (etc.) in a non-blocking fashion. Shade index is recorded for use by error reporter to sort messages but it is not used at present.
Using a Journal buffer is up to a Renderer. The Renderer use the new customization points (rs_errorfmt, rs_filefmt, rs_printfmt, rs_warningfmt or their virtual function equivalents) to interact with a journal::Writer and journal buffer owned by its RenderState. The Renderer would of have to allocate and initialized the journal buffer with the requested page size and number of threads before executing a shader. After a shader is done executing, the contents of the journal buffer can be transfered back to the host and processed with a journal::Reader, which reports them through a journal::Reporter's virtual interface. Intent is for Renderers's to use the Journal buffer and provide their own overriden version of the journal::Reporter. Testshade provides example usage. For legacy purposes, the default virtual RendererServices will route errors to ShadingSystem's OSL::ErrorHandler, but intent is for Renderer's to switch over to using a Journal buffer or their own custom logging.
NOTE: OSL library functions as well use Renderer Service free functions can directly call OSL::filefmt, OSL::printfmt, OSL::errorfmt, OSL::warningfmt which wrappers to the underlying free functions (who may record the messages into a journal buffer).
Added a new hashcode datatype in typespec.cpp; represented as "h" in function arguments in builtindecl.
To print closures added a new OSL function osl_closure_to_ustringhash to return a ustringhash_pod value directly.
Shader execute functions now require a zero-based thread_index to identify the calling thread; used for journalling.
RendererServices::get_texture_info parameter ShadingContext * changed to ShaderGlobals * (aka ExecutionContext).
ShaderGlobals now tracks both shade_index and thread_index.
As we look to hide/remove ShaderGlobals in the future, we still need a context object to replace the role the ShaderGlobals struct is currently serving (just minus the actual shader global variables). As a first step towards this goal, we introduce the concept of an ExecutionContext and OpaqueExecutionContext. These just alias to ShaderGlobals currently, but start to establish the idea of hiding its contents. Instead of direct memory access to the ExecutionContext/ShaderGlobals new getter functions have been added that accept an OpaqueExecutionContext and provide the requested data. When inlined there should be no performance penalty, and reliance on the ShaderGlobals implementation detail will be reduced. IE:
inline const Vec3 & get_P(OpaqueExecContextPtr oec);
template<typename RenderStateT> RenderStateT* get_rs(OpaqueExecContextPtr oec);
inline int get_raytype(OpaqueExecContextPtr oec);
Updated free function renderer services and opcolor to use the ExecContext, but left the rest alone to minimize size of this PR.
We introduce opfmt.cpp that provides shim function for llvm_gen to call, where the shim adapts datatypes from llvm to c++ forwarding calls to the free function renderer service error, warning, print, fileprint functions.
Split llvm_gen_printf into llvm_gen_printf_legacy, for string format calls or when using optix, and the new llvm_gen_print_fmt which will convert the printf style specifier to the fmtlib format that the Renderer Service customization points are expecting.
fprintf by default no longer writes out to a file, is routed to journal::Reporter::report_file_print which could be overriden (example in TestShade to maintain existing behavior). However for security reasons, we do not recommend allowing OSL shader's direct file system but instead just an annotated log entry.
To maintain existing behavior of disallowing output of repeated errors/warnings over a history window, we have provided 2 classes that could be used during reporting: TrackRecentlyReported and Report2ErrorHandler. TrackRecentlyReported can turn on/off limitting errors and warnings and control the history capacity. Report2ErrorHandler simply reports errors to an existing OSL::ErrorHandler to allow for easy integration. However we expect Renderers to not use Report2ErrorHandler and provide their own to directly interact with their logging systems.
---------
Signed-off-by: Steena Monteiro <steena.monteiro@intel.com>
Our latest license scan from LF reveailed that the usual copyright and
license declaration was missing on many shader source files in
testsuite. This was never intentional, it was simply overlooked for
certain test files. Just add the standard notice to those files.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Like other oslcomp source files that are duplicated in oslexec, this
duplication should only be done for shared library builds where the
symbols are hidden from each other.
Signed-off-by: Brecht Van Lommel <brecht@blender.org>
Sometimes an interactive param is used in a way that gets
flagged as needing derivs. But we were neither allocating space
for that nor zeroing them. This resulted on memory corruption.
This fixes incorrect renders in a production context where params are
often tagged as needing derivs.
Signed-off-by: Alejandro Conty <aconty@imageworks.com>
OIIO's 2.3 family is obsolete and no longer updating, so we removed
the branch marker. (Branch markers should always indicate places where
development may happen.)
But we still reference that named branch in a couple places for OSL's CI,
so just replace it with the immutable tag of the last release in the 2.3
family.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
On gpu, the renderer currently owns a shader's groupdata params buffer so that it can pass its pointer to both the init and entry functions.
The downside of this approach is that in order to avoid dynamic memory allocation, the renderer must commit to a buffer size at build time. Using a conservative size to accommodate potentially large shaders can lead to gigabytes of unnecessary memory footprint.
This patch adds a new OptiX direct callable alternative to the existing init and entry callables. This new function allocates a perfectly-sized groupdata params buffer, so that the gpu only pays for the memory it needs, and then it calls init and entry.
Particularly large shaders can require larger buffers than there is space on the cuda stack. To handle this case, there is a new option, "max_optix_groupdata_alloc". Any shader requiring a groupdata buffer larger than this value will not allocate its own buffer, and instead use the pointer passed in by the renderer (presumably coming from a global memory pool).
Signed-off-by: Chris Hellmuth <chellmuth@gmail.com>
For GPU in particular, this prevents needless transfers to the device
when no values actually change.
Also, keep track of some statistics about how much ReParameter is
being used. (Which also help to verify that this change works as
intended.)
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Sometimes, after optimization, symbols tagged as interactive
get removed because they are unused. But ReParameter is failing
with errors because it goes looking for the removed symbol to
master, which is, of course, not interactive. This patch avoids
that so the symbol is not found and we just return false.
Signed-off-by: Alejandro Conty <aconty@imageworks.com>
Tighten up some of the explanations and update obsolete parts of the
CONTRIBUTING document.
This doesn't change any of our policies or procedures. But some advice
was out of date or no longer applicable (for example, now that we use
clang-format, we do not need to detail all the formatting rules, they
are enforced automatically).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The Intel yum repo renamed this package a couple weeks ago, and it
broke our CI. Update to the new name.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This change updates testrender's optix implementation of reparameter,
and adds a new test.
Previously we were grabbing the pointers from incorrect SBT records, so
this moves them into a global lookup table.
Signed-off-by: Chris Hellmuth <chellmuth@gmail.com>
BREAKING CHANGE: to RendererServices ABI (including for CPU) and to
the renderer-side setup when using OptiX.
This overhauls the implementation of how interactively-editable
parameters work, where they live in memory, and get it all working on
GPU/OptiX so that renderers can support interactive adjustment of
those params without recompiling shaders.
The basic gist is as follows:
* We continue work to finish making a clean separation between
"interpolated" parameters and "interactive" (editable) parameters.
* Interpolated params are collected and put into a separate memory
area -- a separate per-group allocation on both the CPU and GPU
(where applicable). It needs to remember the offset into this arena
where each of the interpolated parameters resides. These allocations
and eventual release are taken care of by the OSL shading system,
they live in the ShaderGroup. When the group is set up, this block
of memory is initialized with the correct initial values of the
params and are ready to go.
* The implementation of ReParameter writes to this special memory
area also, that's how it works now (both CPU and GPU).
* How does the OSL library know how to allocate, free, and copy to the
device memory? It doesn't! Instead, we add new RendererServices
methods `device_alloc()`, `device_free()`, and
`copy_to_device()`. It's up to the renderer to provide those, so
that the OSL library doesn't itself need to know about the Cuda
runtime. These are trivial, there's really only one implementation
that makes sense, and you can copy it from the ones in testshade and
testrender.
* Interactive parameters are NOT constant folded during runtime
optimization.
* The shader entry points themselves now take an extra parameter in
the main call -- this will be the pointer to the beginning of the
shader group's interactive parameter arena.
* When JITing, references to interactive parameters know to retrieve
them from their designated offset into the interactive parameter
area.
* This means that the renderer-side OptiX/Cuda code is responsible for
adding this extra pointer parameter to the call to the shader entry
points. You can see how this is done in the testshade and testrender
Cuda code.
* It's up to the renderer to figure out how to make the OptiX hit program
aware of the interactive parameter pointer for that particular material,
in order to pass it to the osl shader entry point. The way I did it in
testshade and testrender is using a field in the struct that's given
to each entry of the shader binding table and can be retrieved on the
OptiX side via optixGetSbtDataPointer(). In testshade/testrender, a
data pointer already existed which wasn't used. In a real renderer, you
may need to add a field or come up with whatever other way you want
to somehow get this pointer, which can be retrieved via
shadingsys->getattribute(shadergroupptr, "device_interactive_params",
TypeDesc::PTR, &myptr);
you can see how I do that in optixraytracer.cpp (testrender) and in
optixgridrender.cpp (testshade).
A number of other things you will see that's worth calling out:
I added a device_ptr utility template that is just a wrapper around a
device side pointer that makes it hard to accidentally dereference it
on the host side.
Since I was changing RendererServices anyway, I also remove unused
register_global, fetch_global, global_map which were unused. They were
leftovers from the way we handled strings in OptiX 6.x.
Encapsulate cuda global symbol name mangling into
BackendLLVM::global_unique_symname(). I did this early on, turns out
it wasn't necessary, but I still like that encapsulation, so I'm
keeping it.
I bumped the 3rd set of digits in the version to reflect that the
changes in RendererServices break ABI. This is only in main, it
obviously cannot be backported to a release branch.
All tests pass for scalar and batch and optix.
I added a new simple reparam test, and renamed the old reparem to
reparam-array. Oddly, the reparam-array test doesn't work properly on
optix (it had never been tried before), but it also failed in optix at
main -- so it's not related to this patch! Putting that on my list of
other oddities to investigate later. It may just be a quirk of
testshade, I'm not really sure yet.
Added to BackendLLVM (and batched) a llvm_ptr_type(TypeSpec) method
that returns the LLVM type of a pointer to the specified type.
Note: This patch doesn't account for the face that a parameter marked
"interactive" could prevent a shader from correctly building for the GPU
because it used the kind of construct that is fine in shader source code but
only will work on GPU if it can be resolved to be a constant by the time
we get done with the runtime optimization (as pointed out by Stephen
Friedman. We'll come back to the problem later with a more robust and
automatic fix -- and if we are lucky, Stephen will have the opportunity to
upstream the approach he already has.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Following #1657, it turns out we forgot to zero initialize the userdata derivatives for the optix path.
This leads to some rendering glitches caused by the memory garbage left in those derivatives.
Signed-off-by: Pascal Lecocq <pascal.lecocq@gmail.com>
Co-authored-by: Pascal Lecocq <plecocq@imageworks.com>
Oops, I noticed that OSL was still using deprecated OIIO::simd type
aliases (float4, int4, bool4) instead of the modern vfloat4, vint4,
vbool4.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
As Fabrice Macagno pointed out on the oiio-dev mail list, we weren't
setting the correct variable name CMAKE_CXX_VISIBILITY_PRESET.
And, oops, this also revealed that there were some things that needed
to be declared public that were not (because, specifically, they
needed to be called across library boundaries by the per-ISA dynamic
libraries used by batch shading!).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This was apparently something I put in to track down a possible
llvm-side problem many years ago. I haven't seen the error message in
recent memory (and it was only enabled at SPI anway), so I guess it
was a red herring or has long since been fixed. Get this cruft out of
here.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This avoids some possible confusion with other projects, or if this
is built as a subproject.
Also, change from the C++ side having an OSL_USE_OPTIX defined -- only
for building OSL itself -- to having it always defined in oslconfig.h
and set to 0 or 1 depending on whether OptiX support was turned on at
build time. This allows app-side use of OSL headers to know if it's
dealing with an OptiX-capable OSL build.
Clean up OSL_USE_OPTIX and OSL_USE_BATCHED symbol defs
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This issue was caused by 8c52d48 . The problem is that the constant float values are being turned into ints. The PR restores the previous behavior and preserves the float format.
Note from LG explaining:
When outputting the oso and using fmt library to write the value, {} (equivalent to {:g}) prints the "simplest" text representation that, when re-parsed as a float, will give you the SAME binary float value. So the "true" value of -9.99999996e+11 is printed as -1000000000000, which when reparsed will give you the closest float to that, which is the original value. (Remember that the value of -1e12 is not exactly representable as a float.)
But if both will parse to the same float, why does the shader print -2147483648.000000 instead of -999999995904.000000? That's the interesting part.
The answer, and the real core of the problem, is that when we parse the oso file it sees -1000000000000 and thinks it's an int, lexically. Now, once it gets to figuring out what to do with that, if the initial value of a float parameter looks like an int, it's still no problem, it will convert the int to the equivalent float.
Except that in this case, because the text that looked lexically like an int is out of range for a 32 bit int, it gets INT_MIN, so by the time it's converted back to float, it's -2147483648.000000
Your fix works (at least in this instance) because it forces the value to print in a way that lexically can only be a float and not an int.
Signed-off-by: Declan Russell <declan.russell@autodesk.com>
This issue was caused by 5efe15a. Essentially, we now use Python_EXECUTABLE to execute the serialize-bc.py script. Therefore, we have to run find_package(python) so that the Python_EXECUTABLE variable gets setup.
Signed-off-by: Declan Russell <declan.russell@autodesk.com>
This includes two fixes:
1) When registering symbols that need user data, sort the entries in
the set so the layer number is ignored. A needed udata iteam shouldn't
depend on the layer and separating them makes find_userdata_index()
sometimes find an index with different derivs status.
2) osl_bind_interpolated_param() is memcpy'ing derivs that might not
be there, yielding corrupted derivs and possibly a crash.
Signed-off-by: Alejandro Conty <aconty@imageworks.com>
My initial implementation of the run layer check optimizations didn’t account for the fact that init ops for symbols have their llvm code generated before we build basic blocks from the OSL IR.
This caused the code to be wrong in two ways: 1) the set of known run layer calls was being reset between instances too late in the pipeline, so a useparam inside an init op would actually be checking the previously compiled layer’s set, and 2) even when reset earlier, it assumed valid basic block ids existed for the init ops when they did not.
So, if a layer had a useparam inside one of its init ops that generally matched the same code location as a useparam in the previously compiled layer, the run layer check could be incorrectly omitted.
I’ve updated the code so that the default optimization behavior is reverted to the way it was before #1665, and added a new option opt_useparam that can be turned on to enable a fixed, restricted version of the optimization that only applies to code in the main section.
I'm working on a test that properly triggers the issue.
Signed-off-by: Chris Hellmuth <chellmuth@gmail.com>
Take into account whether a parameter can change due to interactivity
when comparing parameters for "mergeability".
This proposed patch fixes a corner case we were seeing during live rendering with ReParameterization.
We were experiencing a problem when ss->ReParameterize() wasn't successfully applying parameter changes for us.
In our situation, a shader's parameter, "A", was marked with lockgeom=0 (ie. "live" or "editable") and was set via ss->Parameter(). This parameter was connected (through a series of copies and network links) to another shader's parameter, "B", downstream whose initialized value (ie. the one specified in the .osl/.oso file) matched A's set value. When this happened, A's shading layer was getting marked as "mergeable" and deleted, and we wouldn't see any updates to A via "reparameterization" take effect.
Signed-off-by: Clifford Stein <cstein@imageworks.com>
We can currently query which attribute names and scopes are requested in a shader. This PR extends that idea to also allow querying the attribute types. This now more closely matches the equivalent available queries for user data.
For example, this shader:
```
shader Shader
{
color c;
getattribute("user_color", c);
string s;
getattribute("user_string", s);
}
```
allows us to do this:
```
int nattr = 0;
if (shadingSys->getattribute(shaderRef.get(), "num_attributes_needed", nattr) && nattr)
{
// this already works
OIIO::ustring* names = nullptr;
shadingSys->getattribute(shaderRef.get(), "attributes_needed", OSL::TypeDesc::PTR, &names);
// this is the added feature
OIIO::TypeDesc* types = nullptr;
shadingSys->getattribute(shaderRef.get(), "attribute_types", OSL::TypeDesc::PTR, &types);
// nattr: 2
// names: ["user_color", "user_string"]
// types: [OIIO::TypeColor, OIIO::TypeString]
}
```
The motivation behind this feature is to perform additional type validation at material compile time when determining what primvars can be provided to the shader.
One side-effect of this change to be aware of is that, as mentioned in the code comments, if the same name is requested multiple times with different types, it will now be reported multiple times. Hopefully this is not too much of an issue as the same behaviour also occurs when an attribute is requested in multiple scopes, or when user data is requested with multiple types.
Signed-off-by: Curtis Black <curtis.w.black@gmail.com>
* Switch lockgeom to interpolated and interactive
Establish two new metadata hints about shader parameters:
- "interpolated" means that the parameter comes from the geometry
somehow and may be bound with a different value at different
points on the surfaces, or different values for different
primitives that share the material. This corresponds to concepts
known variously as "geometric primitive data/variables", "user
data", or "vertex variables", depending on the renderer.
- "interactive" means that the user desires the parameter to be
allowed to change when the scene is interactively rerendered,
without incurring a full recompile of the shader. (Variables not
so marked might still be capable of being changed, but not
necessarily without a re-JIT of the shaders.)
Both of these require that the parameter not be fully constant-folded
or optimized away during runtime optimizeation or JIT, and thus
supplant the old "lockgeom=0" hint. But their semantics may differ in
other ways, so we thought it was worth introducing a separation, and
at the same time, finally fixing the mis-design of "lockgeom" (which
in retrospect is both a confusing name and also has the 0|1 sense
backwards, thwarting even my own intuition every time I use it).
We introduce new variants of `ShadingSystem::Parameter()` that changes
the optional `bool lockgeom` parameter for a new enum `ParamHints`
that allows for future expansion with more hinted properties. For
now, the old calls still work (lockgeom=false is the same as passing
ParamHints::interpolated) and this does not introduce any break to
backwards ABI compatibility at this time.
In this PR, "interpolated" is basically the replacement for the old
lockgeom (though the sense is reversed). The "interactive" hint is put
in place but doesn't do anything especially useful at the moment that
is different from interpolated. A few more things are coming down the
pike in subsequent PRs: (1) Make interactive work and fully hook it up
to ReParameter(), including on GPU. (2) Make a new global SS option
that says if you're building shaders for an interactive render or an
offline render. (If offline, the even "interactive" hinted params
should be constant-folded.)
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This patch is a very simple part one of work to optimize the number of conditional run layer calls that OSL makes, which take the form:
if (!groupdata->run[parentlayer])
parent_layer([...]);
After examining our slowest JIT’ing shaders on gpu, @tgrant-nv showed us that these run layer calls were significantly bloating our ptx and slowing down their compilation. Many of them are unnecessary because static analysis can show we are guaranteed to have already run the layer in question.
For example, reading different fields from a connected struct like below will often generate multiple conditional run layer calls:
```
struct Pair {
float x;
float y;
};
shader example_shader(Pair p = {})
{
if ([…]) {
float xx = p.x;
float yy = p.y;
[…]
}
}
```
Since this is not really a performance issue on cpu (runtime overhead is just checking the conditional bit), OSL doesn’t make a huge effort to optimize away these redundant cases. Each useparam op will be careful not to insert more than one per layer, and code that is called unconditionally in the main section of the shader will also not have redundant checks, but there are plenty of examples like above that are not currently removed.
This patch takes a basic first step, adding metrics to track run layer call counts and removing duplicate run layer calls inside a basic block, covering cases like the example above.
On my set of production test scenes, this alone removes anywhere from 20-50% of total run layer calls, with our biggest shaders seeing the most improvement. Total ptx line reduction varies between 3-20%, and optix module compilation (jit time) is reduced between 1-15%.
Signed-off-by: Chris Hellmuth <chellmuth@gmail.com>
But note: OSL does not yet work with LLVM 16.
Because homebrew recently upgraded its default llvm to 16.0, there may
be users for whom builds now break. The solution is:
brew install llvm@14
cmake ... -DLLVM_ROOT=/usr/local/opt/llvm@14
Signed-off-by: Larry Gritz <lg@larrygritz.com>
For LLVM 15+, we need to disable opaque pointers. We did this for
regular bitcode generation in llvm_macros.cmake, but neglected to make
the corresponding change for copmiling to ptx in cuda_macros.cmake.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
LLVM 16 was just released. I'm pretty sure we won't build properly
against it because of the phase-out of typed pointers in LLVM IR, so
warn prominently so that anybody building against it who has problems
will know why.
Also, LLVM itself has bumped their toolchain requirements with the
release of 16. Technically, I'm not sure if that extends to projects
using the LLVM APIs, but warn anyway and hope for the best. If it
fails, at least the warning will supply an explanation.
At a later time, we'll need to upgrade the code on our side to have the
option of using opaque pointers so we can run with LLVM 16+.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
When generating calls to osl_bind_interpolated_param in llvm_instance.cpp, the userdata_has_derivs field is set to true if the parameter's derivatives are needed, but the code does not take into account that on gpu there is no space allocated for these derivatives.
This leads to an incorrect derivatives pointer getting passed into the renderer, which the renderer will use to write into unrelated groupdata fields.
For our renders, getting the correct image was worth the 1-2% inflation of groupdata struct size that we observed by simply re-enabling gpu userdata derivatives. If desired, adding an option to turn them on/off is straightforward, but I didn't want to preemptively bloat the code if it's not necessary.
Signed-off-by: Chris Hellmuth <chellmuth@gmail.com>
In particular, `${LLVM_ROOT}/bin` will end up being (incorrectly!)
`/bin` if LLVM_ROOT is empty. That is bad if some other llvm-config
happens to be in /bin.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The runtime optimization does a great job of constant folding, and
we'd pushed it there because, with instance parameter values and
shader network connections known, it can find so much more opportunity
to optimize than we could in oslc.
However, there is one pesky problem, which is that shader parameters
that are initialized to even simple expressions such as 3+1 end up
with "init ops", which although later constant folded by the time the
shader is JITed, make it impossible to know the value via OSLQuery.
So this patch just takes the simplest cases -- certain `int OP int`
and `float OP float` expressions involving literal constants, and
performs the operation as it's parsing the code. So
int val = 3 + 1;
actually just immediately is turned into `val = 4` instead of making
an "add".
To reiterate, the add would never have happened while executing the
shader -- at runtime in the renderer, when it's time to optimize and
JIT the shader, it would know it's a constant 4 value. This is
strictly about making oslc directly output an .oso file that knows
that parameter `val` has default value 4 instead of throwing up its
hands and saying "it's math code that will be evaluated later."
Signed-off-by: Larry Gritz <lg@larrygritz.com>
In particular, 1 prints a short version of the compile report (just
the fact that it happened and the time), >1 prints more verbose info.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Stop using the deprecated, soon to be removed, ubuntu-18.04 runners
and switch to 20.04.
* Bump the "latest" test openexr to 3.1.6
* Update INSTALL.md to document some things we're already testing and
are known to work.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We had turned it off in one function to preserve formatting, but
failed to turn it on at the end of the function. This led to several
small functions not being clang-formatted, now fixed.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* testrender: Don't use the cached background map on the first bounce (camera ray)
* testrender: Change the default resolution of the background so that importance sampling is enabled by default
Signed-off-by: Chris Kulla <ckulla@gmail.com>
Apparently some varieties of fmod(triple,float) -- which were not
documented in the spec but were intended to work and did allow that
combination in oslc -- were not fully implemented in the runtime.
* Add all these tests to the arithmetic-reg test, to fully test all
possible argument type and unif/vary/const combinations once and for
all for fmod. Also add some more rudimentary testing to miscmath.
* Implement the missing bits and/or avoid the code paths that rely
on the unimplemented parts.
The implementation of fmod for funny -- both for batch and non-batch,
it's got one code path that just directly generates code and doesn't
need any functional implementation in llvm_ops at all, and a second
code path that uses the functional implementations to work around
(different, in each case!) what appears to be bugs or limitations in
long-ago LLVM versions we don't need any more. I'm trying to assess
whether those are still needed at all in the oldest versions of LLVM
we are using, and if not, I may amend even more simplification to this
whole thing.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Try to set ray types correctly for rays -- a little naive now, only
using "camera", "shadow", and "diffuse" for all rays (leave
distinguishing between reflection, refraction, diffuse, glossy for
another day, and currently testrender doesn't support subsurface or
displacement, but maybe someday).
Also, for testshade, improve efficency -- don't decode raytype on
every point. For each individual shade, testshade was converting the
name of the raytype from a string to a ustring, then decoding to a
bitfield. Pull it all out of the loops by computing the bitfield once.
Adjust glass shaders to not change behavior based on raytype:
The code path that was supposed to suppress noisy caustic rays never
worked properly before because we never set the ray type in
testrender. Now that testrender sets it -- but does not distinguish
between the different types of secondary rays -- the code path is
active, but incorrect, and causes tests to fail. Just remove the
offending clause for now and return to the old behavior of the shader
(no attempt to eliminate caustic rays). Later, we should revamp the
glass shader entirely by switching to the newer MaterialX-inspired
closures.
Note: when you specify a resolution in the Background command in our
scene xml, it will run the shader out of any ray context at all (therefore
having no raytype) in order to populate an environment map that will be
used as needed. If you omit the resolution, no importance sampling is
used at all, so it's much noisier. I think the right approach is to do both --
populate the map for importance sampling, but run the shader on the rays
themselves to get higher quality visible results.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We were getting an exception thrown on the OCIO side, because we were
passing 0 for the y stride (which, us humans know, should have been fine
since there was only one row, so the stride shouldn't have mattered).
Now pass the right stride even though we don't need to step from row
to row.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Make sure LLVM_ROOT/bin (either as a cmake var or as an env var) end
up in the hints list when searching for llvm-config.
Without this, one of our Mac CI jobs was ending up with clang 14 and
llvm 15, and that was causing problems. The particular test was
supposed to be using 14 for both.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This makes some minor changes to the RS APIs to add an additional
optional parameter to get_texture_handle so it takes a `TextureOpt*`
to accommodate a similar change on the OIIO TextureSystem side
from https://github.com/OpenImageIO/oiio/pull/3761
The purpose here is to add color management options to texture calls.
However, please note that this PR only reserves a public API change we
know we'll want, but it does not fully implement the color management.
Further work will be needed, as follows:
* Current calls to RS::get_texture_handle should be updated to pass
the TextureOpt*. (This is a formality for future-proofing in case
future TS or alternate TS implementations want to bake the color
space choice into the texture handle itself. But the OIIO change
doesn't currently rely on that at the moment.)
* When the texture.h symbol `OIIO_TEXTURESYSTEM_SUPPORTS_COLORSPACE`
exists, we are building against an OIIO that has the new int field
`colortransformid` in the TextureOpt and TextureOptBatch structures.
We should fill this in with a nonzero value when we want the texture
call to be doing a color transformation as it reads texture tiles.
* Use the new TextureSystem:get_colortransform_id() to get that color
transform ID given two named from/to color spaces. We agreed that
it's a reasonable restriction that the color space names should be
known by the end of runtime optimization, so that should be constant
foldable.
* Recognize new optional argument(s) to the texture() call family that
lets the caller specify the presumed color space of the texture,
and use that value (with constant folded lookup of the ID, above)
to set the value in the TextureOpt.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Ported from OIIO. This lets us not have to depend on the specific
definition in OIIO's fmath.h, and in particular not worry about
whether it's a new enough version of OIIO that the old bit_cast is now
marked as deprecated.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
It seems something in the latest builds of the aswftesting/ci-osl:2022
containers has changed, and explicitly running /usr/bin/yum (instead of
the broken /usr/local/bin/yum) fixes it for now. This was completely
breaking the icc & icx tests, which require yum to install the Intel
compilers.
Also, bump up to the -clang14 containers for those tests, just to
get to a more recent LLVM.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Fixed batched support for closures with array types.
Enabled testsuite/userdata-custom/BATCHED (follow up to PR1620/PR1621).
Added modified version of PR1625 (Fix Issues with Derivative of getattribute) from danieldresser-ie
changing default behavior of MaskedData<DataT>::assign_all_from_scalar(const void *) to zero out Dx Dy when DataT has derviatives.
Added testsuite/userdata-custom
Fixed bug in testshades handling of userdata with derivatives.
Added bool LLVM_Util::is_type_array(llvm::Type* type)
llvm::Type* LLVM_Util::element_type_of(llvm::Type* array_type)
Fixed icx compiler warning with src/liboslexec/osogram.y
Fixed unreferenced function warning in src/testrender/shading.cpp
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
To save build time, we had OSL's Mac CI tests use the homebrew
installation of openimageio. But... Homebrew recently switched its
default python to 3.11, which didn't match the versions we're testing
OSL against here and broke the python bindings, so need to make a few
adjustments.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Related to https://github.com/OpenImageIO/oiio/pull/3653
Kai Pastor (dg0yt) points out that the recent change I made both in
OIIO and OSL to make the version a cache variable has some unexpected
side effects that can lead to unexpeted behavior by inadvertently
retaining a version number upon recompiles if the cache itself was not
cleared, and this can happen unexpectedly even when the user isn't
trying to override the version.
This patch changes the behavior as follows:
You must use the special `-DOSL_VERSION_OVERRIDE=...` to override the
version.
* if you do so, you are playing with fire and get what you deserve
if you misuse it or change branches and don't remember to clear
the cache.
If you do a normal build (without trying to override the version),
* you will get the real version number because it IS NOT cached
* if you switch branches, you'll always get the right version number
* ...but switching branches without clearing the cache can still, as
always, have other subtle behavior differences versus doing a
truly fresh build from scratch.
So in short, it still allows a way to override the version if you
really want to, but if you are not trying to do that, you can no
longer get yourself inadvertently into unexpected trouble.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This is another fix I'm attempting without really understanding all this LLVM stuff.
I started by adding a simple test case that attempts to add two closure parameters, where one has not been set. This crashes or returns random values from uninitialized memory, matching a problem I ran into while trying to get batched mode working in Gaffer.
I investigated some, and found that add_closure_closure is assuming closure pointers are stored with the width of the current batch, but build_llvm_init is creating uniform closures. I checked with @sfriedmapixar, who confirmed that all the batched closure code is assuming wide closures, so it seems build_llvm_init is what needs fixing.
Adding a `sym.make_varying();` to build_llvm_init appears to fix the problem ... I really don't know enough about the overall structure to know if this is the right place to make this fix, or if it could have other consequences, but it's enough to get this test passing, so hopefully someone who does know the background can review it.
Signed-off-by: Daniel Dresser <danield@image-engine.com>
Lately, it has been experiencing intermittent availability problems
that causes our CI to fail while setting up dependencies.
What happens if we don't use the problematic ubuntu-toolchain-r at all?
Nothing, it seems. So eliminate it.
I think that once upon a time, it was necessary to add this repo to
get certain new packages? But maybe they are mainstream enough now
that it's not necessary to add this repo? In any case, it doesn't seem
to hurt anything, it's one less moving part that can't fail if it's
not there.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
When evaluating varying closure keyword parameters in batched execution,
the value of the first lane in the batch would be used for the whole
batch.
Signed-off-by: Daniel Dresser <danield@image-engine.com>
Reminder: phase 1 changed the RendererServices APIs to be based
completely on ustringhash for public method calls, not ustring.
This PR starts along the path of phase 2.
In oslconfig.h, I define a new type `ustringrep` which is the
in-memory/in-shader CPU-side representation of a ustring. Currently,
it's still equivalent to a ustring.
I also define a type called `ustring_pod`, which is the "plain old
data" representation inside a ustringrep (i.e., a const char* when
it's a ustring, and a uint64_t when it's a ustringhash). For reasons
not important, it's exceedingly hard to pass a true
ustring/ustringhash as parameters to or from LLVM-jitted code without
running afoul of LLVM's type checking, so it's just easier to pass the
underlying data type.
So the main work of phase 2 is to change everything related to shader
data from a ustring into a ustringrep, and things passed as const
char* into passing a ustring_pod. In some sense, it's just renaming
and type conversion plumbing. When that's done, phase 3 is to throw
the switch and change the definition of ustringrep to ustringhash (and
of course fix everything that's broken).
This PR does not complete phase 2, but takes a big bite out of it.
It's an arbitrary bite, not much rhyme or reason to which specific
parts I did so far. Just working on it piece by piece and felt that
this was a place where much has been done, everything still works, and
it's a reviewable chunk without getting too much out of control.
Areas I worked on are related to range_check, raytype, getattribute,
bind_interpolated_param, uninit_check, range_check, naninf_check,
getmessage/setmessage, some texture setup, pointcloud.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Changes:
* Scorecards and analysis -- allow new to cancel running old
* Scorecard only runs on main repo, no need to run on forks
* Nightly CI only runs on main repo, no need to run on forks
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This seems to have shifted just enough things around to make
clang-format want to re-jigger a few things.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Wow, this has been lurking for a while. Our Findpartio.cmake tries
looking for a partio exported config, but if not found, falls back
to a simple code to find it manually.
All was fine with the fallback code, which if found would set both
"partio_FOUND" and "PARTIO_FOUND". But the config file, if found --
only for the newer partio versions -- only set partio_FOUND. And in
that circumstance, we were not seting up the pointcloud tests!
Although the partio tests were running in CI (thankfully), I was not
running them locally on my machine with a new-ish partio.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Address -Winconsistent-missing-override clang warning
* Fix ptx definitions of OSL_CONSTANT_DATA arrays
LLVM 14 introduced two ptx regressions to our file-scoped static variables:
1) They gained the .visible directive
2) They stopped being optimized out when unused
This lead to duplicate symbol errors when building an OptiX pipeline with
multiple shaders.
We attempted a fix using anonymous namespaces in #1561, but kept the static
keyword, resulting in a partial solution.
Removing static generates ptx that matches LLVM 12 for the affected variable
definitions.
Signed-off-by: Chris Hellmuth <chellmuth@gmail.com>
The first step of the great ustringhash conversion: This changes all
RendererServices methods (i.e., callbacks to the renderer from the
shader JITed code) that used to take ustring parameters, switched to
take ustringhash instead.
The second step -- which I will tackle after this part is merged --
will be to fully switch the in-memory representation of strings during
shading execution to ustringhash on the CPU (thus matching how we've
been doing it on GPU and making a number of data stuctures and
representations identical/shared for CPU and GPU).
But in the mean time, this lets renderers get started changing their
RendererServices specializations. Just like with OSL's internal
implementation, it may be easier for renderers to break the job into
phase I (rendererservices) and phase II (in-memory rep).
Some notes and guideposts:
* In oslconfig.h, define ustringrep to alias to either ustring or
ustringhash, depending on a new (but temporary) CMake variable
`OSL_USTRINGREP_IS_HASH`, which is OFF for now, meaning that
ustringrep is still ustring.
* Also in oslconfig.h, several helper conversion functions among
ustring, ustringhash, ustringrip.
* RendererServices and BatchedRendererServices (and their various
subclasses in testshade and testrender): change the method
signatures from ustring to ustringhash for all methods that are
reachable from shader code.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Looking at the code coverage report and trying to get a higher
coverage amount, employing two strategies: (a) remove unused code
that is not covered because it's... unused; and (b) test things
that are used but aren't in the test run. Details follow.
Remove deprecated version of LLVM_Util::make_function.
Disable contents of dump_struct_data_layout when OSL_DEV is not
defined, it will never be called in any other circumstances -- it's
strictly a debugging tool, not part of the shipping library.
When OSL_USE_OPTIX is not enabled, exclude compiling contents of
ptx_compile_group.
get_alignment can also be OSL_DEV only as well as simplified
Arbitrarily pick a couple of render tests to use llvm_opt levels 12
and 13 to verify that they work correctly. (That's a couple hundred
lines of pass setup code that wouldn't otherwise be included in the
coverage analysis, nor tested in the testsuite.) This also required us
to add the --llvm_opt command to testrender, as it is in testshade.
Make the analysis run build for avx512 as well for more code coverage.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The immintrin.h header is required for the implementation of the
popcount and coutr_zero function only when the Intel compiler is used.
Move the inclusion accordingly. The current state breaks build on
platforms that don't provide the header, but provide the alternative
means (gcc/clang built-ins, etc).
Signed-off-by: Dan Horák <dan@danny.cz>
We compile testshade.cpp twice -- one for a special "does it compile
as a DSO" check but that isn't run from the testsuite. This was
causing the coverage analysis to think nothing in testshade.cpp (at
least one copy of it) was getting run, which is technically true, and
counting all of its lines as untested for the coverage analysis. Just
exclude that funny dso build from the coverage case.
Also exclude anything in src/build-scripts from coverage analysis,
that's not code that ships, so to speak.
We're trying to chip away at untested code bit by bit to get our
coverage to over 80% of lines. We're getting close.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Define std::hash for ustring and ustringhash, when not using an OIIO
new enough to do so.
Define std::less for ustringhash, when not using an OIIO new enough
to have operator< defined for ustringhash (will be fixed separately).
No need now for ustringHash, which is also unwisely similar to
ustringhash, just use std::hash instead.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Reduce dependency on ShadingContext and ShadingSystem objects
for accessing the ShadingSystem and consequently the ColorSystem. In its current form
it is not device-friendly.
We introduce a self-contained pvt::ShadingStateUniform that maintains a proper
state and is independent of the ShadingContext and ShadingSystem. We add an
opaque pointer to ShadingStateUniform in the ShaderGlobals structure.
Update uses of ColorSystem in opcolor.cpp to go through
ShaderGlobals::shadingStateUniform. Also update uses of m_commonspace_synonym
and m_unknown_coordsys_error in opmatrix.cpp to go through
ShaderGlobals::shadingStateUniform.
NOTE: The ShadingStateUniform is private and an OSL implementation detail and
does not change the public interface other than adding an opaque pointer to
ShaderGlobals.
* Fixed LLVM representation to accommodate ShaderGlobals::shadingStateUniform.
* Added missing Copyright Contributors license to shading_state_uniform.h
* Removed m_commonspace_synonym data member
Moved ShadingStateUniform closer to its use in osl_prepend_matrix_from
* Fixed clang-format diffs. Added void* shadingStateUniform to ShaderGlobals in rend_lib.h. Needed for proper working.
* Added shadingStateUniform to wide_opmatrix and batched_shader_globals.h
Adjusted batched struct data layout check in codegen.
* Reordered headers to include shading_state_uniform.h with the other headers
from liboslexec
Signed-off-by: Steena Monteiro <steena.monteiro@intel.com>
Some explicit SIMD loops were technically illegal because each lane was not truly independent, fixed by adding "reduction" clause. Some Mask objects had to be replaced with just "int" to allow it to be specified in the reduction clause.
icx v2022.2.0 does apppear to have some regressions for large amounts of Dual code in a couple use cases, added alternative(improved) Single Statement Assignment (SSA) version of OIIO::fast_sincos which allowed unit tests to pass. Consider updating OIIO fmath.h to use SSA coding style (which would allow this workaround to be removed).
Another regression with transform_normals was avoiding by introducing an OSL_UNLIKELY(condition) to generate a branch vs. masked execution, this change is a general improvement and should be left in.
Avoid flex_bison produced warning in osogram.cpp "variable 'osonerrs' set but not used".
Due to a number of regressions that appear to be fixed in next revision, choose to skip icx 2022.2 and continue CI for the time being with icx 2022.1 until the next release.
Because multiple (possibly newer) versions of oneapi may be installed, use a config file to specific a specific compiler and tbb version NOTE: Individual oneapi components may independent version numbering.
For centos images, use a config file to specificy a specific oneapi compiler and tbb version
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* For analysis, USE_QT=0 will disable osltoy from building (since we
don't/can't test it in the testsuite, it's unexecuted but really
shouldn't count against our code coverage metric).
* Exclude Imath duplicate headers from Sonar analysis.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Traditionally, OSL has supported building against the current and also
the previous supported version of OIIO, but not necessarily any older.
With OIIO 2.4 released (any day now), and OSL "main" is actually going
to be next year's supported release, it seems reasonable to change the
forward-looking development of OSL to have a minimum of OIIO 2.3,
which means that OSL-next will support OIIO 2.3, 2.4, and the
currently-in-progress 2.5 (or whatever it's called by that point).
The main reason to do this bump is because we'd like to depend on the
ustringhash class, and possibly other things we will probably be able
to easily backport to 2.3 but not to 2.2.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* A variety of changes to get a clean build and passing tests when
using LLVM 15.0.
* I've noticed problems when using LLVM 15 but building with earlier
clang, so the cmake scripts now print a warning in that case, so if
users encounter trouble they have a hint about what to do to fix it.
* For our CI tests on Mac, force the MacOS-11 test to use llvm 14,
and the MacOS-12 test to use llvm 15.
IMPORTANT TO-DO / CAVEATS:
1. When doing JIT at optlevel 12 or 13, I had to disable a number of
passes that don't seem to exist anymore in LLVM 15. This is enough
to get it working, and to be honest, I don't know if anybody uses
these opt levels. But we need to revisit this, because I don't
know if there these are cases where the names of the passes merely
changed or that new passes take their place (in which case we
should add the new passes, not stop after merely disabling the
deprecated ones). For that matter, optlevel modes 11, 12, 13 are
supposed to match what clang does for -O1, -O2, -O3, and that
changes from one release to the next, so we should probably revisit
this list and make sure it's matching clang's current choices
(which I assume are crafted and periodically revised by clang's
authors).
2. LLVM 15 switches to "opaque pointers". It still supports typed
pointers... for now. But as you can see in the diffs, I had to
disable a variety of deprecation warnings and take some other
actions to put LLVM 15 in the old "opaque ptr" mode to match our
use of LLVM <= 14. But this is only temporary, because the typed
pointer mode is expected to be removed entirely in LLVM 16. So at
some point in the next few months, we are going to need to support
opaque pointers in our use of LLVM. (Also note: for a while, we're
going to have a bunch of ugly `#if` guards or other logic to
support both opaque pointers for users with llvm >= 16 and also
typed pointers for users with llvm <= 14, until such time as our
LLVM minimum is at least 15, which I expect is not a reasonable
assumption for at least a couple years.)
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Several places in leparse.cpp were found by static analysis to have
memory leaks -- results of calling _parse() that, in the case of
early exists in error conditions, never got deleted.
I also did some other modernization: changed NULL to nullptr, changed
some loops to more compact range-for, utilize some Strutil helpers
that will be more efficient than the alternative std::string
manipulation.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This was meant to work all along, which is why this patch is so small!
But a tiny detail -- not explicitly passing info about the `--help`
argument -- made ArgParse substitute its own default, which exits
after printing the usage info. But, as I apparently meant to do all
along, if you specify it explicitly, you get to handle it, and the way
we handled it alsp printed the extra information.
Anyway, with this patch, `testshade --help` not only prints all the
testshade command options, but also it prints some information at the
end about some of the build settings of OSL, versions of key
dependencies, hardware info, and a run-down of the ShadingSystem
options. This can be helpful for debugging.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
direnv (https://github.com/direnv/direnv) can be used for local developer workflow enhancements. direnv uses .envrc file for configuration. Because .envrc is usually specific to a developer workflow, it is generally not committed.
Signed-off-by: Mark Sisson <5761292+marksisson@users.noreply.github.com>
- There are a handful of tests we disable entirely with because they
are irrelevant or just cannot be made to be clean (and that's ok).
- Tests with sanitizers run slow, so we increase timeouts, and also
set AA to 1 for render tests to speed them up. We rig it so a
non-matching output won't fail, and that's ok because every other CI
test is checking the results; here we have no reason to believe
results will differ from other tests, we just need to run with
sanitizers turned on.
- In testing.cmake, we rig texture tests to disable the leak
sanitizer, because we can't eliminate some completely benign leaks
from the texture system, and because they are called from JITed
code, we don't get a clean stack trace of things that happen in OIIO
and therefore we just can't enumerate the allowed exceptions. So we
trust that OIIO itself tests the texture system adequately and is
able to ignore the benign "leaks."
- We already submitted PRs and merged the little fixes that were
discovered by enabling address and leak sanitizers. This is just the
final bit to turn it on for CI runs from here on out.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Rather than specify the version in the usual project() statement,
project (OSL VERSION 1.13.0.0 ...)
this patch decouples it
set (OSL_VERSION "1.13.0.0" CACHE STRING "Version")
project (OSL VERSION ${OSLVERSION} ...)
thus allowing the possibility of overriding the version number at
build confuguration time (`cmake -DOSL_VERSION=3.1.4.1`).
This should be used with extreme caution. Probably never used by most
people. I don't need to describe all the ways that things can go
sideways if you make your build of x.y.z.w actually be arbitrarily
different from the x.y.z.w that everyone else uses and that is
documented.
That said, there are two scenarios I have in mind for which this
ability is very useful:
1. Sometimes, we add or deprecate API features "in advance" by having
an `#if` enable or disable code based on the version, essentially
putting in place changes that can't be enabled yet because they
would break compatibility rules, but we want to integrate the code
now and have it become active upon reaching a particular future
version in which that incompatibility will be allowed. The version
number override makes it easy to test this by quickly doing a build
"as if" we had already bumped the version number to see what will
happen when we actually do.
2. Every once in a while we need to do an internal build that, to
conform to version compatibility rules, needs to appear to still be
part of a past version family. This lets individual sites play some
games with version number overrides rather than have to create new
branches and tags for certain one-off internal releases.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This is a tricky situation to describe. Consider this shader:
shader test (float A = 0.5,
float B = A)
{
printf("A = %g, B = %g\n", A, B);
}
By default, A is 0.5 and B gets A's value, also 0.5. If you give A a
different instance value, like 1.0, then B will also get the same
value, they'll both be 1.0. If you give B an instance value, that
will be used for B rather than copying A's value, like this:
$ testshade -param B 1.0 -shader test test
A = 0.5, B = 1
But there is a corner case bug: if we assign B an instance value of
0.0, it will incorrectly fall into a special case where it thinks it
gets a default value, only coincidentally because 0.0 is in an unused
slot for the default value (unused because it has no default -- its
default comes from "init ops", that assignment).
$ testshade -param B 0.0 -shader test test
A = 0.5, B = 0.5
!!!
Reading the patch should be self-explanatory for what the bug is and
how we fix it.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Signed-off-by: Larry Gritz <lg@larrygritz.com>
I've tried to write a complete set of step-by-step instructions for
everything about how to coordinate and deploy a release. This is a
living document, we will amend over time. I'm sure there must be
things I've forgotten.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The little ReadEvent class had two members that are never set when
OSL_DEV is not defined, nor are they accessible (the accessor
functions also are disabled based on OSL_DEV). It looked to the static
analyzer like the data members were uninitialized, which they
were. Add OSL_DEV guards around the declarations of those members as
well, so they aren't even present in the struct when not needed for
this debugging mode.
Food for thought: another alternative is to keep the members but just
remove the OSL_DEV guards entirely -- just let the two int data
members be used all the time, not just for OSL_DEV. It's too small to
have any perf penalty, and it unclutters the declaration by removing
all those `#ifdef` lines.
For that matter, I also wonder if there's any real point to having
private members and accessor functions for this tiny internal-only
class, and maybe it's better to just let it be compact declaration
of a simple `struct` with three directly accessible data members.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
In testshade, there was a "loop" that wasn't really a loop, and it
was getting flagged by static analysis as being fishy. This patch
rewrites it as an "if".
Also take the opportunity to change from stream output to OSL::print
formatted output.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We had hard-coded longer allowable timeouts for particular testsuite
tests. But for the static analysis runs, it takes extra long and we
need a way to tell it to increase the timeouts even more, and to
exclude the longest running tests from the analysis coverage run (it's
not necessary to run those tests, they don't increase coverage enough
to justify being an hours long run).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Signed-off-by: Larry Gritz <lg@larrygritz.com>
In the course of our runtime optimization of shader networks, we
generate lots of new constant symbols (for example, as we optimize an
"a = b + c" into "a = newconst" if b and c can be determined to be
constants). The new constants are named with a numerical suffix doled
out in a sequental manner.
I've always thought that maybe it would be helpful for me when
debugging optimizer output if the name of the symbol also revealed the
value of the constant it held (when possible). Never did anything
about it.
Now it turns out that this has an application for JIT to PTX, where
the PTX cache can save compilation-to-PTX time if it encounters LLVM
IR that is exactly the same as it has seen before and is cache. If
the constants are generated in a different order, it generates
non-functional, purely text differences in the code we generate, and
this means the PTX cache doesn't function as efficiently as possible.
So this change alters the compiler-generated constant naming for two
cases: int constants, and string constants. The int constants append
the value of the integer (using '_' instead of '-' for negatives), and
the string constants append the hexidecimal value of the ustring hash
for that string. In both cases it does this without using or
incrementing the `m_next_newconst` counter that it uses to
sequentially name the other types that can't easily encode the value
to generate a unique name.
It's really to help the PTX cache, but as a side benefit, it will
be slightly easier for me to make sense of the optimized code when
I need to read it with my own eyes while debugging.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Today's branching of OIIO 2.4 and subsequent changing of its master
branch to think it's "2.5" seems to have triggered an automatic fuse
that annotates a particular function in OIIO as deprecated for version
2.5 and beyond. OSL was still calling the old version, so some
immediate changes are necessary to call the correct new name when
using a new OIIO.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Exclude osltoy from coverage analysis, since we don't have a way to
run it from CI
* Run batch shading for analysis to get better code coverage
* llvm_gen_select: extra check that we can't overrun memory of cond[]
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Change the languagespec to document the new MaterialX-synchronized
closures.
* Document the old ones as "deprecated" -- still supported for 1.x but
assumed to be on the slate for removal in OSL 2.x.
* Change stdosl.h so that the new closures, instead of being seen only
if MATERIALX_CLOSURES is defined, to being enabled by default, but
(for now) can be disabled if NO_MATERIALX_CLOSURES is defined.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Removing the suffix ids and callseq comments generated by global counters
leads to deterministic ptx in a multithreaded environment.
Signed-off-by: Chris Hellmuth <chellmuth@gmail.com>
* Minor changes to fix a couple more warnings given by SonarCloud
static analysis.
* Get the code coverage enabled, it wasn't running properly.
* testrender: allow secret TESTRENDER_AA to override command line --aa
* runtest.py: allow secret OSL_TESTSUITE_SKIP_DIFF to force skipping
the reference comparison step.
* For analysis run, set TESTRENDER_AA=1 to keep the tests from being
prohibitively expensive, and OSL_TESTSUITE_SKIP_DIFF=1 to keep it
from failing because the aa=1 images are different.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We had been using clang/llvm 12 and when trying out 14 I started
getting test failures with OptiX complaining about duplicate
symbols. They all seemed to be the things we declared as
OSL_CONSTANT_DATA. It looks like there's a difference in llvm 12 vs 14
in how the generated PTX marks its symbol visibility.
The solution (thanks to a clever suggestion by Alex Conty) was just to
enclose those declarations in anonymous namespaces. Works like a
charm!
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This allows the refraction index to be detected even if a medium closure
is not present. Additionally, the medium detection now traverses layer()
closures.
Signed-off-by: Chris Kulla <ckulla@gmail.com>
There seems to be just a teensy bit of nondeterminism or
platform-variance in this one test that we haven't tracked down the
source of yet, that moves some specular fireflies around. Smells like
maybe an uninitialized variable somewhere in testrender? Because it
doesn't happen on any other test and we haven't detected it in our
real renderer, we don't currently suspect that it's an actual error
in oslexec.
Big hack: Re-run failed "render" tests up to 5 additional times before
giving up.
This makes me feel dirty. But it makes tests pass for now. Eventually we
will find the embarrassing testrender bug that is responsible for the
occasionally different specular sampling noise.
Also slightly bump up the error tolerance for this test.
Also add one alternative that seems to frequently pop up.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Didn't properly detect the disallowed use of arrays for the false
clause because we checked `t` twice.
Yay for static analysis!
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Another round of clang-format
* Bump the standard clang-format we use for CI from clang11 to clang14.
This will make some minor additional changes.
* Due to a mismatched target name, we weren't actually checking the
formatting in CI on all the files we thought we were, and therefore
not assuring that certain code was correctly formatted. So there is
some fixup needed, which is included here.
* And a bunch of files we didn't try to do before, either.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The idea is to stop propagating derivs through BSDFs, which we
never did properly. Instead we trace rays with radius + spread,
which we turn into OSL derivs in SG when we hit a surface. Then
when bouncing off a surface we use the BSDF roughness to set the
spread. In addition we also:
* Return BSDF samples/eval as a small struct Sample(). This
avoids output parameters in the methods.
* Stop passing a full ShaderGlobals to the BSDF methods. Pass
a viewing direction wo instead so they could run in a simpler
context.
Signed-off-by: Alejandro Conty <aconty@imageworks.com>
This simplifies the logic around the ior which drives reflection/transmission (fresnel effect)
from the ior which drives refraction (and total internal reflection). Setting both iors equal to each other
gives the expected physical behavior.
Also tweak the bsdf constructor to support variadic arguments which simplifies a bunch of code.
Add a new test to validate the refractive version of generalized_schlick_bsdf.
Signed-off-by: Chris Kulla <ckulla@gmail.com>
Implement get_albedo for the sheen closure, based on fit derived by the MaterialX team
Add optional mode to testrender to visualize the result of the get_albedo method. This can be useful for validation of the get_albedo method.
Signed-off-by: Chris Kulla <ckulla@gmail.com>
* Implement reflective microfacet based MaterialX closures
Add test cases for dielectric, conductor and generalized schlick closures
Signed-off-by: Chris Kulla <ckulla@gmail.com>
* Implement refraction case in microfacet closures
Signed-off-by: Chris Kulla <ckulla@gmail.com>
* Update reference images
Signed-off-by: Chris Kulla <ckulla@gmail.com>
* Relax diff threshold on render tests
Signed-off-by: Chris Kulla <ckulla@gmail.com>
It printed the u,v coordinates where the error was first discovered.
This is not necessarily deterministic, of course!
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Add support for layer closure with dielectric reflections as top layers, and a test with nested layer closures
* Add support for sheen in the layer closure
Signed-off-by: Adrien Herubel <adrien.herubel@autodesk.com>
* Extend the bsdf interface in testrender to use Color3 instead of float in preparation for MaterialX closures which are almost all tinted
* Add stub parameters and IDs for all MaterialX closures
* Register all MaterialX closure types
* Implement most MaterialX closures, reusing the existing implementations for now
This commit is mainly intended to do the busy work of adding the new closures, no new BSDFs are actually implemented yet. This means that some of
the closures are handled in a very rudimentary way, ignoring some key parameters. The intent is to improve support for these over time.
Signed-off-by: Chris Kulla <ckulla@gmail.com>
The sampler implementation in testrender was very close to the 2020 article from Brent Burley. This patch
catches up to best practices, including a faster owen scrambler from Nathan Vegdahl and recognizing that
the index scrambling can be achived by the same owen scrambler method (which saves a bunch of logic and runs
faster). Finally, avoid the redundant bit reversals and use compiler intrinsics when available to benefit
from hardware instructions (for ARM or GPUs). Finally the integer hash constants were updated to the latest
from the hash prospector project.
While more optimizations are certainly possible, this is a reasonable balance between quality and speed.
Update reference images for new sampler
Signed-off-by: Chris Kulla <ckulla@gmail.com>
* GPU string overhaul
Big string cleanup, yet again, for GPU. We still had this tricky
double indirection happening with strings for OptiX -- the string was
a pointer to a "global", which in turn held the hash. No, throw all
that out. Just directly store strings as the ustring hash on
OptiX. No extra indirection. Many fewer differences between how things
are handled on OptiX versus CPU (except for the fact itself that the
"string" holds the hash rather than the char*).
Some details worth noting:
* LLVM_Util loses "type_string()" and calls it "type_ustring()" to
emphasize it's the type we're using to represent a ustring, and
definitely not a C or C++std string. Also, LLVM_Util itself now lets
the caller tell it whether it prefers that the representation of a
ustring be the charptr or the hash, and then handles most of the
details internally. The BackendLLVM, then, makes that decision early
on, charptr for CPU and hash for GPU.
* Several more spots in LLVM_Util and BackendLLVM have been plumbed to
pass around llvm IR SSA variable names to try to make the resulting
LLVM IR more readable when I'm trying to debug it. (This was
strictly debugging scaffalding aid for me, but comes along for the
ride because it's useful to keep.) I also added a
BackendLLVM::llnamefmt to format strings, which only does the
formatting when the shading system knows it's going to save the IR
for inspection, and otherwise (that is, in production) it does not
do the string formatting or pass the names along, so it's not doing
all that string manipulation when no human is going to be looking at
the IR dumps.
* A lot of clunky OptiX-only code disappears.
* Some side effects of my reducing complexity of the IR as I debugged:
there were times we were generating redundant pointer casting in the
IR, such as grabbing a pointer, then casting it to a void*, then
casting back to some other kind of pointer. In some cases I reduced
that quite a bit. Again, this was mostly just a scramble to make the
IR more readable while I debugged, but it probably has a minor
performance gain as well since it reduced somewhat the sheer amount
of IR we generate and pass to the optimizer and JIT.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Remove register_string entirely from RendererServices
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Remove more register_string remnants
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* comment typo
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Set up a SECURITY.md explaining how to confidentially report
vulnerabilities.
* Create scorecards.yml workflow to run OSSF scorecard to check supply
chain security issues.
* Use exact SHA's for actions versions to guard against hijacked
actions repos that might try to substutite something else at the
version tag we use.
* Make sure actions we use don't get write perms on our repo.
* Tried to use harden-runner action to ensure no actions on the runner
overwrite source code or have unauthorized outgoing net connections
during the GHA run, but currently leaving it disabled because of too
many false positives -- it flags every dependency download as a case
of altering source code. Which it technically is, but it doesn't
matter for any jobs that aren't producing any artifacts that will be
consumed or relied on by users.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Most LLVM API IR calls that generate ops allow an optional name of the
llvm symbol that will hold the result of the operation. By default, it
will make one up, such as "%37", "%38", etc. Needless to say, these
are difficult to understand when reading volumes of IR dumped for
debugging purposes.
So in this patch, I'm starting to expose the ability for our own code
to pass names through our wrapper functions. I needed this for some
deep debugging.
I haven't instrumented the entire set, just trying it on for some of
the specific areas I needed to debug. But this is a useful first step.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This makes OptiX 7.0 the minimum version we accept, and greatly
simplifies a lot of code we were separately maintaining for 6 and 7.
This will go into main AFTER the release branch for 1.12 is split off.
The dropping of OptiX 6 will be a 1.13 (or whatever we call it) change.
Don't build the optix modules at all when not USE_OPTIX.
Totally remove stringtable.
Other simplifications stemming from these.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The implementation of simplex noise we've been using is public domain,
but looks like when we pasted the license notice into our repo, there
was one stray sentence (probably left over from an earlier license
scheme Stefan Gustavson used?) that makes reference to GPL. But the
software and its repo are NOT -- it's very clear that it's public
domain: https://github.com/stegu/perlin-noise/blob/master/LICENSE.md
So remove this confusing sentence.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We'd been waiting until the minimum OIIO requirement was high enough
to include support for the newer syntax.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
In a few of my big clang-format revisions, I was over-zealous, and did
not realize that I was making the command line option parsing unreadable.
Revert those secions.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Recent OIIO and fmt changes slightly changed this one place's output
with some version/platform combinations. Also make it match exactly
between the example-deformer and example-batch-deformer tests.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
fmt master seems to have just deprecated the default ability of
fmt::sprintf("%s") to printing anything that allows stream
output. This breaks use of Strutil::sprintf formatting of ustrings.
This isn't yet in any tagged release of fmtlib, but I'm trying to get
ahead of it.
The quick fix is to just change those handful of places that still use
sprintf convention with nonstandard types to use std::format style
calls, which we are slowly converting to anyway. (And maybe a couple
other spots while I was at it.)
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Banish the last of the sprintf calls (except the handful that need
to be there, because such formatting is still supported in shaders
themselves, so something needs to understand them).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Another semi-random collection of conversions. Bit by bit.
These are all internals, none of it involves public interfaces.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Use unity build to speed up OIIO build (for sufficiently new OIIO).
* Build OIIO with STOP_ON_WARNING=OFF, so that if a compiler upgrades
and makes new warnings for OIIO, that doesn't break OSL's CI build.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Add support for closures to the batched interface.
The renderer interface for closures, including registering
and processing them remains unchanged.
The Ci variable in BatchedShaderGlobals has moved from
the uniform portion to the varying portion, which may
require changes to any renderer code that was utilizing
the BatchedShaderGlobals.
To utilize closures with the batched interface, you register
the closures per usual, then once shading has finished,
the Ci in BatchedShaderGlobals will contain an Block of pointers
to the roots of a closure trees that can be processed
independently just like in the normal single-point interface.
This initial implementation is largely functional, however,
there may be more optimization gains available in the code
that copies the run-time SIMD values and scatters them
to the separate closure-data locations.
One remaining un-implemented feature is passing
_closures_ via get/setmessage() when using the batched
shading interface, and the locations where that would
need to be implemented are marked in the source.
Signed-off-by: Stephen Friedman <sfriedma@pixar.com>
* Address batched closures review comments:
* Remove unneded Block specializations for pointers.
* Rename null variable to null_value.
* remove a 0 default param to rop.llvm_void_ptr in batched_llvm_gen.cpp:llvm_gen_mul
* Optimizations to masking and loading in wide_opclosure.cpp
* Fix LLVM11+ builds by using llvm_vecor_type instead of llvm::VectorType::get
Signed-off-by: Stephen Friedman <sfriedma@pixar.com>
* Remove unneeded oslclosure.h in wide.h now that we aren't explicitly instantiating the templates.
Signed-off-by: Stephen Friedman <sfriedma@pixar.com>
The Intel OneAPI LLVM-based C++ compiler, as we download it from Intel
for our CI, bumped from 2022.0.0 to 2022.1.0.
New warning popped up -- newest Intel LLVM-based compiler warns about
overriding certain floating point math flags.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Fix warnings from stdint on windows by pre-including the standard header
* Make sure register_JIT_Global is a public symbol since it needs to be called by client code
* Fix how llvm-as, llvm-link and python are invoked by cmake so they can be properly found on Windows platforms
Signed-off-by: Chris Kulla <ckulla@gmail.com>
There's no rhyme or reason to this particular set of changes... I just
picked a few files and did the obvious conversion. Will do more in
subsequent PRs.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
A few minor changes were needed to address new warnings.
clang14 warns if you use bitwise `|` or `&` in an `if`, but there are
some places where we do that purposely to avoid branches.
Disable clang14 warnings about bitwise ops for all of liboslexec.
Also some batch shading related vectorized loops fail to vectorize
with clang14 -- though they worked fine with clang12 -- and so we
needed to mark those as "complex" SIMD loops. Is this a regression
with clang14?
Revise simd loop pragma for wide_transformc due to llvm14 regression
Signed-off-by: Larry Gritz <lg@larrygritz.com>
In order to move towards shader execution on devices, we need to replace
host side interactions with new concepts that can exist on the device (or host).
The current virtual function based RendererServices exemplifies this dependency
on host side objects.
To break the dependency we introduce free function based renderer services (rs_free_function.h)
To simplify its introduction, we only added the renderer service free functions required by opmatrix.cpp.
Goal is to have a single opmatrix that can be compiled for host or device (eventually).
Renderers can provide llvm bitcode representation of their renderer service
free functions to the ShadingSystem. During shader compilation/JIT the
renderer's bitcode is combined with opmatrix.cpp bitcode along with the JIT'd shader network.
The idea is the combined LLVM IR representation can then be lowered to the target
(be it on the host or device[SPIRV|PTX]). Thus a renderer's free function implementation can be shared.
opmatrix.cpp now only calls the free functions (not virtual RendererServices).
However liboslexec on the host contains an rs_fallback.cpp which forwards
calls through the existing virtual RendererServices for seamless compatibility
when a renderer does not provide bitcode for the renderer services.
Testshade has been modified to optionally to provide rs_bitcode,
utilizing a new Cmake macro, EMBED_LLVM_BITCODE_IN_CPP, to turn rs_simplerend.cpp (the free function implementation)
into bitcode.
Free functions can be enabled via --use-rs-bitcode flag in testshade.
Free functions need access to RenderState values because they lack a this
pointer like the RendererServices virtual methods have. Added a RenderState
containing minimal set of values required by free functions.
Testshade/SimpleRenderer is responsible for setting RenderState values.
Added ShadingSystem attribute rs_bitcode to hold renderer services free function bitcode (provided by the renderer)
Renderers using global variables (like ustring's) not in strdecls.h must register their global variables' addresses
using OSL::register_JIT_Global (const char *global_var_name, void *global_var_addr).
This is necessary for the JIT process to resolve global variables that are embedded in RendererServices bitcode.
A validation step occurs that emits an error with a list of unmapped global variables.
RendererServices can reference strings with the STRING_PARAM(varname) macro.
By default, STRING_PARAM(varname) works for any string in strdecls.h.
For any string NOT in strdecls.h, it will map to a global variable with a prefix of "RS_".
For example, STRING_PARAM(raster) maps to a global ustring RS_raster.
NOTE: C linkage with a "RS_" prefix is used to allow for unmangled
non-colliding global symbol names, so its easier to pass them to
OSL::register_JIT_Global(name, addr) for host execution
NOTE: the STRING_PARAMS macro adapts to OSL_HOST_RS_BITCODE macro
to utilize the RS_ prefix. RS_ prefixed versions of all OSL::Strings
intances have been created by rs_free_function.h, so the same STRING_PARAMS
macro can be used for renderer service or OSL strings.
NOTE: We expect the handling of strings to change in the future possibly being replaced by Handles entirely
It is the renderer's responsibility to instantiate any NEW ustring global variables with the proper RS_ prefix;
It is the renderer's free function implementation to import/extern any of these NEW global variables with the proper RS_ prefix;
For testshade, a rs_strdecls.h has been added and is utilized to instance the RS_ global variables and register their addresses with OSL in simplerend.cpp.
rs_strdecls.h is also used by rs_simplerend.cpp (free function implementation) to create the externs with the RS_ prefix.
NOTE: DO NOT instantiate global variables inside the free function renderer services, because we strip global constructors as part of the pruning process.
Added LLVM_Util::validate_global_mapping to identify unresolved globals.
Added utility macro __OSL_STRINGIFY(string_value) in oslversion.h.in so the
proper naming prefix is used and we don't have multiple implementations around.
Added liboslexec build step to produce bitcode containing OSL library functions that depend on free functions (here, opmatrix.cpp)
Added a new Cmake EMBED_LLVM_BITCODE_IN_CPP function in llvm_macros.cmake that replaces the LLVM_COMPILE macro
and provides for both linking and serializing multiple OSL cpp files.
Fixed cuda.macro to use new location of serialize-bc.py in src/build-scripts.
Fixed formatting issues.
Disabled erasing llvm.global_ctors from LLVM module because of issues on MacOS Apple Clang.
Left as future work.
Added better error checking for module_from_bitcode().
Signed-off-by: Steena Monteiro <steena.monteiro@intel.com>
New classes of warnings triggered when enabling option "opt_warnings":
* Creation of new strings mid-shader.
* Access to the characters of strings mid-shader.
We already warned about texture access by name that could not be
turned into a handle.
In all of these cases, we only warn *after* runtime optimization, so that shader
idioms in the source code that appear to do these forbidden things will not warn
as long as they can be reduced to safe operations in the course of the runtime
optimization step.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
checked_find_package:
* Optional SETVARIABLES gives a list of vars to set to TRUE if the package
is found.
* Optional DEBUG flag turns on extra debugging information about how the
package is found.
* Optional PREFER_CONFIG uses a dependency's exported config, if available,
rather than our Find module.
* Global ALWAYS_PREFER_CONFIG option can be set to ON to always prefer
config over Find modules for all dependencies.
fancy_add_executable:
* -DENABLE_INSTALL_<executable>=OFF lets you turn off the installation
of an executable.
other:
* Change the target name for doing the clang-format over the code to
"run-clang-format", because "clang-format" is an imported target name
of the libclang exported cmake (which we may wish to use soon).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Pass some extra headers to NVCC_COMPILE to make sure that the PTX files are regenerated whenever those headers are changed.
Signed-off-by: Tim Grant <tgrant@nvidia.com>
* Don't generate PTX for rend_lib.cu when using OptiX 6.
Signed-off-by: Tim Grant <tgrant@nvidia.com>
* Fix the callable program signature for the init and group functions in the OptiX wrappers.
Signed-off-by: Tim Grant <tgrant@nvidia.com>
* Fix issues with the LLVM arguments for osl_printf when targeting CUDA.
Signed-off-by: Tim Grant <tgrant@nvidia.com>
* Fix llvm_gen_construct_triple for OptiX. The 'to' argument was not handled correctly.
Signed-off-by: Tim Grant <tgrant@nvidia.com>
* Improve support for CUDA osl_get_matrix.
Signed-off-by: Tim Grant <tgrant@nvidia.com>
* Add basic support for named transforms to testshade/testrender.
Signed-off-by: Tim Grant <tgrant@nvidia.com>
API changes we had to take into account:
* TargetRegistry.h location
* No more DisableTailCalls field in PassManagerBuilder.
Needed to update the ref image for render-microfacet test, some sparklies changed.
Looks like the new LLVM probably JITs to ever so slightly different math code, tickling
some LSB differences that at 1 sample per pixel, results in some different sampling
directions leading to fireflies. We decided to just commit a new ref image and move on.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
In our generated OSLConfig.cmake, we inserted a find_dependency(Imath)
that accidentally demanded the OpenEXR_VERSION (should have been
Imath_VERSION). That works fine as long as Imath version is the same
(or higher) than the OpenEXR version. But if the installed OpenEXR
version is somehow ahead of the Imath version, it will fail.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
A small instalment on the long-term shift from printf-style formatting
to std::format style formatting.
Define preprocessor OSL_INTERNAL, which is defined when compiling OSL
itself, but will not be defined for downstream projects using the OSL
headers.
When building OSL, enable the symbol OIIO_ERRORHANDLER_HIDE_PRINTF,
which causes ErrorHandler methods using the old conventions to be
marked as deprecated and generate warnings.
Clean up all the calls revealed by those warnings. (And
opportunistically clean up some other things that happened to be in
the same files.)
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Also change the one exception mentioned to a std one, matching
Imath 3.x and insulating us from any of the old Iex custom exceptions
from Imath 2.x.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Changed a function signature in OIIO master that threw us off here
just a tad, as there was no direct type conversion the compiler could
figure out on its own. Easy adjustment gives it the right hint.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This eliminates a huge amount of redundancy in our CI descriptions --
instead of the entire CI logic replicated for each test in its own
universe, we group tests into (a) all the ones based on ASWF VFX
Platform containers, (b) non-VFXP-year oddballs run on Ubuntu, (c) Mac
tests, (d) clang-format verification. Each has the build logic
expressed once, and a short list of variables that change for each
individual test of that category.
This organization is much easier to read, maintain, modify tests, or
add new test variants, and cuts the size of the ci.yml file by over 50%.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Avoid many clang vectorization workarounds for OSL_INTEL_LLVM_COMPILER_VERSION
Added Makefile option VEC_REPORT to turn on generation of optimization reports
Reduce impact of REQUIRE_INF_NAN for icx by explicitly honoring inf, nan, and signed zeros vs. disabling fast-math
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
Add compiler support for Intel(r) oneAPI DPC++/C++ Compiler (based on LLVM technology).
Detect if CMAKE_CXX_COMPILER_ID matches "IntelLLVM" and set CMAKE_COMPILER_IS_INTELCLANG=1
NOTE: CMAKE_COMPILER_IS_INTEL will not be set, that is reserved for the classic Intel(r) C++ Compiler.
Define OSL_INTEL_LLVM_COMPILER to hold an encoded Intel(r) LLVM Compiler version
(e.g. 20220000), or 0 if not an Intel(r) LLVM Compiler.
Define OSL_INTEL_CLANG_VERSION to hold the encoded Clang version the
Intel(r) LLVM Compiler is based on (e.g. 140000),
or 0 if not an Intel(r) LLVM compiler.
Ensure OSL_CLANG_VERSION is when its a vanilla clang compiler, not apple or Intel(r) oneAPI DPC++/C++ Compiler.
Added OSL_ANY_CLANG which is set for any clang based compiler.
Removed OSL_NON_CLANG_COMPILER.
Updated uses of OSL_NON_CLANG_COMPILER with more specific and understandable combinations of OSL_ANY_CLANG, OSL_INTEL_CLANG_VERSION, and OSL_INTEL_COMPILER.
Make sure our math calculations are consistent,
especially division. Different compilers may have choose to use
reciprocal division loosing precision causing slightly different
results which can lead to aliasing differences when running the testsuite
Disabled reciprocal division for all projects.
if -ffast-math is enabled to to the underlying compiler, inf's and NaN's may not be handled properly (by design)
When building liboslexec, ensure any source files that require proper INF or NaN handling have additional command line flags passed if needed (as the default behavior maybe -ffast-math):
shadingsys.cpp
wide/wide_shadingsys
wide/wide_optest_float
Enable unitialized Matrix44, because all of its members get overwritten anyway, in impl_transform_normal_masked.
Added alternative test results where needed, wsing '.icx.' as shorthand for the Intel(r) oneAPI DPC++/C++ Compiler in alternative test result filenames.
Changed some tests to use the pretty(value) which clamps values close to 0 to be 0 to avoid lots of different alternative rest result files when using printf.
Added "--center" option to several regression tests to avoid aliasing in results between SIMD batched and scalar execution.
* Added icx-vp2022 to CI plan.
* To better clarify the classic vs. LLVM based Intel compilers and that they have version #'s not just 0 or 1,
renamed OSL_INTEL_COMPILER to OSL_INTEL_CLASSIC_COMPILER_VERSION
renamed OSL_INTEL_LLVM_COMPILER to OSL_INTEL_LLVM_COMPILER_VERSION
Removed unnecessary FMT_VERSION and OPENIMAGEIO_CMAKE_FLAGS from the icx-vp2022 worklow, added back USE_SIMD.
Fix spelling of losing.
Added OSL_OMP_COMPLEX_SIMD_LOOP(...) which is enabled or disabled in platform.h based on which compilers/versions can handle vectorizing complex nested control flow. This makes it cleaner to document which loops need to vectorization conditionally controlled and centralizes where it is managed.
* Removed extra fp-model=consistent for icx-vp2022 workflow
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
Slight reworking of -Werror logic to let the default behavior be
changed by setting a cmake variable. In particular, this allows us to
make a CI build disable -Werror when we need to.
Previously, having the "OSL_CI" env variable set trumped all else and
forced -Werror. Now it just contributes to the default behavior, but
setting `-DSTOP_ON_WARNING=OFF` and cmake config time can still
disable it.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Additionally, allow custom command line arguments to run after each
image is loaded, but before the comparison.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Turn on warnings about OIIO::Strutil::format and ustring::format
We should be using Strutil::sprintf / ustring::sprintf, or upgrading
entirely to Strutil::fmt::format / ustring::fmtformat.
Some day, the plain "format" will be deprecated, then disappear, then
reappear as compatible with std::format (as are fmt::format/fmtformat now).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The str() method was not part of the final C++17 std::string_view.
To ensure future interchangeability with std::string_view, change the
few places where we call str() in the OSL code base.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Track that the backfacing() shadeop requires N and I shader globals.
* Test ensuring that N and I are reported as needed shader globals when a backfacing() shadeop is used.
Signed-off-by: Stephen Friedman <sfriedma@pixar.com>
* Implement Batched pointcloud_search, pointcloud_get, and pointcloud_write.
Updated behavior of scalar pointcloud_search and pointcloud_get to not code gen branch with code gen runtime error for maxpoints/num_points larger than array size. Instead now it codgens a clamp of the maxpoints/num_points to the arraysize. This allows code to continue executing, and if accessed outside of the range, existing codgen range checks will still emit a runtime error.
Extracted class PointCloud defnition and some helper functions to pointcloud.h so it can be shared between batched and scalar implementation.
Moved PointCloud out of anonymous namespace to OSL::pvt so a single implementation (including static maps) could be shared between batched and scalar implementation.
Refactored implementation of
template<typename DataT, int WidthT> struct Masked;
template<typename DataT> struct Ref;
to flatten class hierarchy and fix bug where derivatives of unbounded arrays could not be constructed, IE:
MaskedDx<int[]> myDeriv(ptr, arraylength);
Added MaskedData::assign_val_lane_from_scalar
Renamed MaskedData::assign_val_from to MaskedData::assign_val_from_wide
Renamed MaskedData::assign_all_from to MaskedData::assign_all_from_scalar
Updated BatchedAnalysis to indicate that results of pointcloud_search, pointcloud_get, and pointcloud_write are always varying regardless of the symbols passed in.
Updated debugging in BatchedAnalysis to emit unmangled symbol names.
Properly implement (was broken previously) BatchedBackendLLVM::llvm_zero_derivs(const Symbol& sym, llvm::Value* count) including using code gen loop to reduce code bloat.
* Fixed warnings about BatchedRendererServices<WidthT>::pointcloud_get and BatchedRendererServices<WidthT>::pointcloud_write not returning a value.
* Fixed unused variable "ok" in batched pointcloud_write.
Fixed undefined type when building with USE_PARTIO=0
* Update pointcloud tests to avoid non-deterministic results for unsorted results and out of range max_points/num_points.
Added support for NOOPTIMIZE file tag for batched regression tests.
Made array-length-reg NOOPTIMIZE so its code gen would actually execute vs. constant fold.
Fixed bug for storing arraylength result in varying result symbol.
* Fix missing virtual function overrides of MyBatchedRendererServices in example-batched-deformer
* Change varying point cloud tests to use pixel centers in order to avoid situations where point cloud could provice multiple data points at equal distances causing non-determinism.
* More changes to avoid non-determinism in pointcloud test results
* Add alternate reference images for point cloud results that were coming back slightly different for essentially identical distances.
* Fixed bug in llvm_gen for raytype where the wrong type was passed as a parameter for a string.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
(many ported/replicated from various recent OIIO PRs)
* CI: bump to latest dependency releases
* Bugs fixed in build_opencolorio.bash, it botched some of the
directory choices when customizing where to place and build
it. Also, switch to the more standard BUILD_SHARED_LIBS cmake
variable to control whether to build dynamic or static libraries.
* Correctly handle the cmake variable VISIBILITY_MAP_FILE being set to
"" on the command line as a way to disable using the visibility map
file entirely.
* On Linux, also set link flags `--exclude-libs ALL` to hide symbols
from any embedded static library dependencies, to prevent clashes
with other versions of those libraries that may be compiled into a
DCC app.
* Some other minor touch-ups in compiler.cmake.
* Refresh build_openexr.bash with the latest changes from OIIO.
* CI workflow: revise failed build artifact storage to capture cmake
files.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
New OIIO guards the contents of its Imath.h with the
`OIIO_IMATH_H_INCLUDED` preprocessor symbol. This lets us in OSL
pre-define it to absolutely ensure that OSL headers are including
exactly the Imath headers they want, not accidentally getting the ones
that OIIO includes.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
OIIO's string_view::c_str() was cute, but it's a little dicey and it's
not present in the C++20 std::string_view. We didn't use it in a ton
of places, so replace with other idioms so that in the future we are
better able to interchange with or switch to std::string_view (when
our minimum supported C++ standard is high enough).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Here was the construct it was confused by:
struct Data {
float height;
};
Data foo(Data a, float b)
{
return Data; // <----- what's this madness?
}
Curiously, this revealed a source code error in one of our other tests,
oslc-err-struct-print, which had not been detected previously!
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Added Wide<const int, WidthT> wide_shadeindex, void* userdata_base_ptr, void* output_base_ptr to BatchedExecutor interface (and underlying implementation) that allows each data lane
of a batch to utilize a different shade index in combination with a uniform userdata and output base pointer.
Fixed bug in batched version of llvm_gen_compassign where index was being loaded as uniform value even when it was varying (really just caused assert/extra output in Release).
For scalar and batched, avoid generating code for init ops or default values when a sym_loc is found because unlike get_userdata, when supplied sym_loc can't fail.
Improved LLVM_Util::op_scatter to handle scalar destination data types, previously only wide destination were required but to handle scatter results from SOA wide data layout to AOS
output placement layout scalar destinations need to be supported.
Updated BatchedRendererServices to be less abstract replacing several pure virtual functions with default noop versions to better match behavior of RendererServices.
Changed testshade to utilize output placement by default in batched mode.
Added NOSCALAR option to testing framework to allow for batched only tests.
Added testsuite/example-batched-deformer to exercise userdata placement and simple example of utilizing batched interface.
Added alternate output for example-batched-deformer to account for some rounding differences.
Added comment to note that the batched implementation assumes that SymLocationDesc::offset is readable at point 0 (because we clamp masked off lanes to 0 vs. adding conditionals).
Renamed local variable wide_index_to_output to wide_index_for_userdata to more accurately describe how it is used.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
Implement batched versions of isnan, isinf, and isfinite.
Added batched regression testsuite ieee-fp-reg.
NOTE: its really difficult to test isnan(value) == true,
as its unclear to actually generate a NAN in OSL because
most functions are "safe" and check/avoid NAN's
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
We were passing along, in the exported cmake config files, a demand
for downstream clients to use a C++ standard version equal to that
which we used to build OSL itself. But that's not necessary. We may
choose to build with C++17 (for example), but all of our public APIs
are safe to use for C++14.
Split the build-time options to CMAKE_CXX_STANDARD for the standard we
build this project with, and DOWNSTREAM_CXX_STANDARD for the standard
we are declaring is the minimum to *use* the library.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Enable BATCHED execution of most testsuite entries that don't use closures or point cloud.
Added batched implementation of hash function.
Updated batched implementation to use the symbol.get_int(), symbol.get_float(), and symbol.get_string() methods vs. *(int*)symbol.data() approach.
Fixed bug in batched code gen where mangled symbol names were output as part of debug uninit warnings and range checking.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Fixed reparameter in batched execution.
Updated batched behavior for unlocked geom shader output parameters to function like scalar version.
Fixed batched bug when non-entry layers are executed and execution masks are compared producing a <i1x16> instead of i1 for use in conditional operation.
Added more reparameter test cases to cover arrays and arrays of aggregate types.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Enabled command line assignable user data for batched execution in testshade, which fixes testsuite/userdata-passthrough
Changed print_outputs for batched execution to not use << operator which includes parenthesis, but instead output each componenet with spaces to match the non-batched print_outputs.
renamed MaskedData<...>::assign_from(void* ptr_wide_data) to assign_val_from(const void* ptr_wide_data) to better reflect that only the value is assigned, and Dx and Dy are NOT assigned.
Added MaskedData<...>::assign_all_from(const void* ptr_data) to take a scalar DataT and broadcast it to all lanes of the MaskedData, masked off lanes are NOT overwritten, and value, Dx, Dy are assigned.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Apply clang-format to wide.h
Add some paranetheses to conditional to avoid warning.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Wrapped all printf arguments with pretty() in testsuite/geomath/test.osl to avoid small floating point differences from affecting printed results in test.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Add testsuite/geomath/ref/out-alt.txt to allow for minute floating point differences with some compiler/platform combinations.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
We used to build MaterialX shaders on our end. Now it's all done in
the MaterialX project itself, and along the way, we failed to keep in
sync with them. (And even if we had "kept up", what does that even mean?
We want OSL to be independent of any particular MX release, but the MX
shaders may change from release to release. Better to be entirely on
their side of the fence.)
We turned off building these shaders by default a long
time ago, now let's finally remove the last remnants and urge everyone
to rely on the MaterialX project for this material.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Added MaskedData::assign_from(void *ptr_wide_data) to perform a masked copy of wide data at some address.
Added enum class RenderServices::TraceOpt::LLVMMemberIndex to correlate C++ data members with LLVM type generated by BatchedBackendLLVM::llvm_type_batched_trace_options()
Fixed bug in BatchedBackendLLVM::llvm_load_arg where array types where only properly broadcasting uniform to varying arguments for the 0th element of the array.
For batched implemented trace options as a LLVM type with direct code gen to alloca, init default values, and set optional values vs. callbacks. Same code could be resused for non-batched impl.
Added BatchedMessageBuffer to context and BatchedMessageList to wide_opmessage.cpp to allow a shared buffer to be cleared by the ShadingSystem while allowing target specific compilation of SIMD import/export of message values into the list to compile in wide_opmessage.cpp.
Extended testshade's SimpleRenderer and BatchedSimpleRenderer to implement trace and getmessage to allow new tests to exercise some basic functionality.
Added new batched regression tests:
message-reg
trace-reg
Added new test: message-no-closure which is a simplied version of message with no closures
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Temporarily disable simd compilation of message import and export data to test if that is cause of crash in bleeding edge CI workflow.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Temporarily disable more simd compilation of batched message test if that is cause of crash in bleeding edge CI workflow.
Temporarily run only message-reg test for bleeding edge CI workflow
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Temporarily run only message-reg.regress.batched.opt for bleeding edge CI workflow
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Fix alignment of MessageBlock allocation so that reads/writes can be assumed aligned to Block<> data members
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Re-enable disabled simd statements from trouble shooting bleeding edge CI.
Re-neable all tests for bleeding edge workflow.
Applied clang-format to wide_opmessage.cpp
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* pybind11 2.9.0 release -- bump the vfx2022 and "latest" CI tests to
use it.
* Adjust other CI test pybind11 versions to get good coverage across all
the pybind11 releases, and to synchronize the vfx platfom tests with
the pybind11 release that was current for its respective year.
* Sync build_scripts/build_pybind11.bash with the latest in OIIO, including
bumping the default pybind11 version we download to 2.8.1.
* Opportunistic touch-up of warnings about llvm version to bring
into sync with our actual current requirements.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Among other cruft that existed to support versions 7 and 8, this lets
us completely get rid of PrePromoteLogicalOpsOnBitMasks custom pass,
which is only needed for LLVM < 8.
Also get rid of some cmake checks for LLVM versions that require
C++14 or higher -- not necessary now that C++14 is our minimum.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Add batched support for dict find, dict next, dict value.
Implementation takes advantage of uniform values and only does lookup once and broadcasts results if necessary.
When nodeid is varying, it falls back to simple loop over active data lanes performing lookups.
* Added new regression test xml-reg.
Enabled BATCHED execution of existing xml test
* Add new wide_opdictionary.cpp missed in original commit
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
Maybe somebody else is building this as a subproject or has a variable
called "CI"?
* For our CI runs, set an env variable "OSL_CI" (not just generic "CI").
* Default STOP_ON_WARNING to be off for supported releases, only on by
default for main branch where we're actively developing. Because you
want downstream users who are building supported releases to not get
tripped up when their compiler upgrades and there's a new warning.
* Make sure STOP_ON_WARNING is true for our CI runs, even for release
branches -- but now check OSL_CI, not CI, in case somebody else has
set that.
Fixes#1428
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Make PROJECT_VERSION_RELEASE_TYPE be a cache string, so individual
build configs or sites can override it.
* Make OSL_SUPPORTED_RELEASE be an overrideable option, and make
OSL_DEV_RELEASE be driven by it, rather than (possibly incorrectly)
independently set.
* Change OSL_IS_SUBPROJECT to PROJECT_IS_TOP_LEVEL (and with the opposite
sense), to match CMake 21+ behavior. And no need to set it ourselves if
using cmake 21+.
* All of the clang-format stuff should only apply if PROJECT_IS_TOP_LEVEL.
* Add a TIME_COMMANDS option to help debug performance of the build.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Add batched support for 2d texture calls, gettextureinfo.
To enable, added:
bool RenderServices::is_udim (TextureHandle *texture_handle)
TextureSystem::TextureHandle* BatchedRendererServices::resolve_udim_uniform(...,TextureSystem::TextureHandle*, float S, float T)
void BatchedRendererServices::resolve_udim(...,TextureSystem::TextureHandle*,Wide<const float> wS, Wide<const float> wT, Masked<TextureSystem::TextureHandle*> wresult)
which allows batched code gen to resolve_udim's for varying S & T and bin the resulting TextureHandle* and make calls to get_texture_info_uniform with a single resolved udim texture handle, then broadcast the results to data lanes with matching TextureHandle *'s.
Added extensive new regression tests:
gettextureinfo-reg
gettextureinfo-udim-reg
texture-opts-reg
Enabled BATCHED execution of existing tests:
gettextureinfo-udim
gettextureinfo
texture-alpha-derivs
texture-alpha
texture-blur
texture-connected-options
texture-derivs
texture-errormsg
texture-firstchannel
texture-interp
texture-missingalpha
texture-missingcolor
texture-simple
texture-smallderivs
texture-swirl
texture-udim
texture-width
texture-withderivs
texture-wrap
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Removed some commented out includes
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Add conditional compilation dependent on OIIO version to enable RendererServices::is_udim
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Fix clang-format issues
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Fix bug in testing.cmake where non-existant OPENIMAGEIO_VERSION was being checked, when it should have been OpenImageIO_VERSION.
Added quiet option to oiio-tool calls in regression tests to allow produced out.txt to match between baseline and batched runs.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Added optional silent=False argument to oiiotool in test suite, allowing us to avoid generating warnings for file already exists that was causing differences when comparing baseline/out.txt and out.txt in regression tests
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Added missing batched texture option rnd for the llvm code gen representation
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Fixed compilations issues with OIIO 2.4 by enabling "rnd" batched texture option.
Fixed some other compilation issues when OSL_DEV is defined.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Cleared up some comments around populating rnd.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Add support for batched texture3d.
Discovered mismatch between documentation, llvm_gen implementation, and oslc when passing user derivatives to texture3d.
oslc and documentation expect optional parameters "vector dpdx, vector dpdy, vector dpdz",
however llvm_gen and optexture implementation only accept "vector dpdx, vector dpdy"
and use 0 for dpdz when calling into the texture subsystem.
This commit includes changes to oslc to make the compiler match the implementation of accepting only "vector dpdx, vector dpdy".
Enable BATCHED execution of testsuite/texture3d
Add new regression test textur3d-opts-reg
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Revert oslc to make the compiler match the documentation of accepting only user inputs that include dpdz: "vector dpdx, vector dpdy, vector dpdz".
Fixed llvm_gen and optexture to properly extract user dpdz from texture3d options or initialize a Vec3(0) and pass through to texture subsystem.
Modified batched_llvm_gen and wide_optexture to properly extract user dpdz from texture3d options or initialize a Wide<Vec3>(0) and pass through to texture subsystem.
Updated texture3d-opts-reg test to pass dpdz for calls taking user derivatives.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Added batched support for environment.
Added BatchedRendererServices::environment virtual method
Added new test environment
Added new regression test environment-opt-reg
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* clang-format of batched_rendererservices.h
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Rename testsuite/environment-opts-reg to testsuite/texture-environment-opts-reg to line up with a different PR adding testsuite/texture-environment.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Removed some environment option test for "wrap" as that is an unsupported option for environment.
Added test and handling for "time" option to batched texture3d. Current choice is to just silently ignore "time" as currently supported 3d formats do not support time.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Removed redundant environment test as texture-environment covers it.
Enabled BATCHED execution of texture-environment
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Extended test timouts further for arithmetic-reg.regress.batched.opt and transform-reg.regress.batched.opt which were causing spurious CI timeouts.
Reduced some multipliers of user supplied derivs to environment tests to reduce differences in results.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
Co-authored-by: Larry Gritz <lg@larrygritz.com>
It was documnted for texture(), but not for the others, for which it
also has worked for many years.
Thanks for Alex Wells for pointing this out.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
I recently bumped the ctest per-test timeout to 300s because one of the
batched shading regression tests was occasionally taking longer than the
prior limit of 240s.
But there is a better way: Individual tests can have their timeouts
set individually via set_tests_properties(TIMEOUT), so I raised the
timeout to 300s for all tests whose names contain "-reg", also kept it
rather long (the old 240s) for the long render tests that are all named
"render-". Then for everything else, we can actually lower the default
to 60s, since no other tests should be long-running.
(For the curious, the reason these batched regression tests are so
expensive is that they test a bunch of combinations, and they first
run the test in the non-batch scalar mode, then again in batched mode,
and compare the results to ensure that the batch code never deviates
from scalar. This makes them take at least twice as long as they
ordinarily would. But also remember that these are only run for the
few CI test matrix entries where we are specifically testing batch
shading.)
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Increase testing timeout, some of the batch shading regression tests
seem to take longer than the prior 240s limit.
* Turn off OpenVDB when using the osl:2021 in combination with OpenEXR
3.x -- the version of OpenVDB in the 2021 container is too old to
understand OpenEXR 3.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Use LLVM 13 / clang 13 for the VFX Platform 2022 test.
* Add MacOS-11 test
* Poke around how to use arbitrary LLVM versions via Conan. (Works! But
I have not yet enabled it.)
Signed-off-by: Larry Gritz <lg@larrygritz.com>
OIIO master recently removed support for Field3D, which is rarely used
and the project is effectively unmaintained and isn't able to build
against the newer OpenEXR releases that we want to encourage people to
use.
So the primary purpose of this patch is to switch our unit test for
texture3d from using a field3d file (and thus would not run when using
newer OIIO) to a roughly equivalent test using openvdb (which is
supported, assuming OIIO is built in the presence of OpenVDB).
But, ha, I discovered two things, either one of which means that we
had actually not been unit testing texture3d at all lately: (1) To
discover whether the installed OIIO supported openvdb (and therefore
whether to bother running the test), we were relying on a totally
nonexistant OPENIMAGEIO_BIN cmake varaible. I don't know what that was
supposed to be (did I mean OIIOTOOL_BIN? but that also is not a
thing), but anyway, it caused it to always skip the test. (2) Oh boy,
the OIIO that we were relying on for may tests, provided by the ASWF
docker containers, was an OIIO that was not built in the presence of
OpenVDB (even though our OSL testing containers DO have OpenVDB), so
again it skipped the test because it needs that OIIO-side support for
texture3d() to know how to read openvdb files. Fix this by building
OIIO from scratch with openvdb support when doing the "VFX Platform
2022" test, rather than relying on the one in the container.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Because they can fail with some locales. Replace with OIIO::Strutil
parsing functions that are locale-independent.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Renaming the master branch to main is imminent. Here are the few
places where we refer directly to branch names that will no longer
exist.
While searching for "master/", I also noticed a couple files that
had the wrong project's copyright boilerplate on it (a lot of the
cmake scripts I write and submit to both projects simultaneously).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This is primarily for the sake of mingw, but also is the proper kind
of stream to pass to OIIO::Filesystem::open.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Use a cmake idiom so that you need an explicit -DCMAKE_INSTALL_PREFIX=
in order to write over /usr/local or any potentially sensitive area.
In the absence of this being set explicitly, it will configure the
cmake build so that an "install" ends up in ./dist (which is what the
Makefile wrapper did all along).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Added
template <int M00, int M01, int M02, int M03,
int M10, int M11, int M12, int M13,
int M20, int M21, int M22, int M23,
int M30, int M31, int M32, int M33, int DivisorT>
struct sfm::StaticMatrix44
to model a matrix whose elements are known at compile time. Internally it utilizes proxys instead of floats in order to optimize for special cases of 0, 1, -1 elements and not even generate math when possible. The StaticMatrix44 is used by wide_opspline.cpp to create specialized functions for each of the spline basis types. A dispatch function routes the compile time optimized spline or spline inverse routine.
Altered splineimpl.h to provide "int basis_type_of(StringParam basis_name)" to allow the batched implementation to map a string to spline enum value used as an index in the dispatch function to functions optimized for each type of spline basis (instead of generic 4x4 matrix implementation)
Added "T sfm::select_val(bool cond, const T left, const T right)" as alternative to the ternary operator which appeared to possibly take address of its arguments which prevented privatization of those variables when compiled inside a SIMD loop.
Added note to BatchedAnalysis implementation identifying that local variables scope is lost between .osl and .oso and when a function is inlined multiple times, its local variables could be treated as varying when they could have been uniform if somehow identified as a local variable by the Analyzer.
Enable BATCHED testing of:
spline-boundarybug
spline-derivbug
spline-nonuniformknots
spline
splineinverse-ident
splineinverse
Added new BATCHED_REGRESSION tests:
spline-reg
splineinverse-knots-ascend-reg
splineinverse-knots-descend-reg
Added sfm::Inverter to remove nested loop behavior of batched splineinverse as OIIO::invert has a loop which was called from inside the search interval loop, created a nested loop. Under SIMD/SIMT the nested loop could execute incoherently. Instead we choose to use the class sfm::Inverter to manage state between is_bracketed_by(min,max), meant to be called from the search interval loop, and bracketed_invert(min,max) meant to be called after exiting the search interval loop, avoiding a nested loop that could be executed at different interval's for each SIMD/SIMT lane/thread.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
Patrick Hodoul pointed out that we were forcing OpenColorIO to
initialize and read configs as soon as the ShadingSystem was
created. In this patch, we're changing to a lazy creation approach.
We remove the `OCIOColorSystem m_ocio_system` member and the
ocio_transform() method from ShadingSystem. Put them in the
ShadingContext instead. This allows lock-free access because each
context is used by only one thread at a time.
Within the OCIOColorSystem, make the `OIIO::ColorConfig` be a
shared_ptr rather than a member, so it can be created lazily. If the
Context's OCIOColorSystem needs a config and doesn't have its own yet,
it asks the ShadingSystem, which allocates one the first time (and
only that one time).
So, in short, SS allocates *one* shared ColorConfig, the first time
any context asks for one. The Context only asks the SS for this once
per context (i.e., once per thread). Individual shades just use the
ColorConfig of their context, not needing to lock for it (except for
that brief lock the first time any context asks for the very first
ColorConfig that's needed).
Some things were guarded by `OIIO_HAS_COLORPROCESSOR`, which is silly
because we no longer support OIIO versions so old that it's not
defined. But then OSL `undef`ed it for Cuda. I replaced these guards
by a more straightforward direct test of whether it was Cuda code.
I added a sample OCIO config to testsuite so we can use it for tests.
In particular, the new testsuite/ocio -- test using OCIO for color
transforms.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Fixes 1416
In OSL, function parameters must be marked as `output` or they are not
supposed to be writable. In issuing errors when a write is performed on
something not writable, we missed an interesting case:
```
void g(output float b) {
b = 1; // ok, since b is marked as output
}
void f(float a) {
g(a); // <--- should be disallowed, g is writing to f's non-output param
}
float x;
f(x);
```
This patch catches cases like these. The new error looks like this:
`test.osl:22: error: cannot write to non-output parameter "a", as passed to function g() as arg 1 (output parameter "b")`
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Add batched SIMD implementation for all color operations.
Added batched llvm code gen for: llvm_gen_blackbody, llvm_gen_luminance, llvm_gen_transformc (including code gen of loop to identify unique "from" and "to" space combinations to call library function with).
Made ColorSystem const correct.
Pulled much of ColorSystem's implementation into src/liboslexec/opcolor_impl.h so it could be inlined in both opcolor.cpp and wide_opcolor.cpp.
To enable SIMD fast path for ColorSystem::blackbody_rgb added new methods to expose underlying lookup table optimization.
bool can_lookup_blackbody(float T /*Kelvin*/) const;
Color3 lookup_blackbody_rgb (float T /*Kelvin*/) const;
Color3 compute_blackbody_rgb (float T /*Kelvin*/) const;
When can_lookup_blackbody() returns true
then lookup_blackbody_rgb can be safely called,
otherwise the expensive compute_blackbody_rgb is required.
Updated implementation of ColorSystem to succesfully vectorize on multiple compilers, optimized lookup table data types and indexed dereferences to generate prefered 32bit indexed gathers vs 64bit gathers which require multiple instructions and registers.
To improve code generation and less chances of aliasing issues, changed nested M[i][j] operator access to Matrix in ImathMatrix.h to directly access underlying 2d array M.x[i][j] also changed Vector operator V[i] to directly access V.x, V.y, V.z in OSL/IMathx/IMath*.h
Moved helper testIfAnyLaneIsNonZero from wide_opmatrix.cpp to be testIfAnyLaneIsNonZero in OSL/wide.h
Added sfm::min_val to provide an implementation of min which returns values not references to the original through a ternary (which can cause issues with vectorization). Added Clang specific sfm::min_val and sfm::max_val implementations to assist in vectorization.
Enabled BATCHED execution of existing testsuites: blackbody, color, transformc, wavelength_color
Added new testsuite BATCHED regression tests: blackbody-reg, color-reg, luminance-reg, transformc-reg, wavelength_color-reg
Added sfm::min_val to provide an implementation of min which returns values not references to the original through a ternary (which can cause issues with vectorization). Added Clang specific sfm::min_val and sfm::max_val implementations to assist in vectorization.
* Adopt OIIO calls to fast_cbrt and fast_expm1 for blackbody computation.
Use single precision for bb_spectrum object/operator and manually implement std::pow(wlm,-5.0f) to better enable vectorization.
Reduced masked code generation for rgb_to_hsl.
Removed duplicated code by implementing ColorSystem::blackbody_rgb(T) in terms of ColorSystem::can_lookup_blackbody(T), ColorSystem::lookup_blackbody_rgb(T), and ColorSystem::compute_blackbody_rgb(T). Removed some casting to uint16_t that was unnecessary.
Removed some test shaders that were not intended to be promoted.
* Implement batched llvm code generation for ops: andor, bitwise_binary_op, clamp, get_simple_SG_field, isconstant, select, unary_op, mix
Fixed bug in BatchedAnalysis where the complement operator was being treated as always having a boolean result. In reality the result of complement or other bitwise operation would only be boolean if its input parameters were forced to be boolean (left TODO note for future improvement).
Fixed bug in printf where integer who is forced_llvm_bool() is formatted as a float was not converted to an integer first.
Enabled BATCHED for execution for tests: and-or-not-synonyms, isconstant, logic, select.
Expanded testsuite/shaderglobals to exercise/access all shader global data members.
Add regression tests: andor-reg, bitwise-and-reg bitwise-or-reg bitwise-shl-reg
bitwise-shr-reg bitwise-xor-reg, complement-reg, mix-reg, select-reg
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
Remember that our current 'master' is really aiming for a release in
late 2021 at the earliest, and thus will be a version primarily
intended to be used widely in 2022. Release branches never drop
supported dependencies, so the 1.11 branch will continue to support
OpenEXR all the way back to 2.0.
VFX Platform guidance is to support the current year, plus up to 3 years
prior, so that means that for 2022-ish software, we should support back
to 2019 dependency ecosystem.
For OpenEXR, 2019 state of the art was 2.3 -- that's still dreadfully
old, and is no longer getting even critical or security bug fixes. But
it still means that we can comfortably upgrade our advertised minimum
OpenEXR from 2.0+ to 2.3+ and eliminate some build and dependency
logic that was dedicated to dealing with 2.0..2.2.
We're also raising the OpenEXR/Imath "recommended minimum" (build-time
warning, but not rejection) to 2.4. Even that is shaky, but the fact
that it's still in wide use and was officially (though IMHO unwisely)
the VFX Platform 2021 recommendation, means we can't yell at people
too much about it.
It is possible that by the time we make a supported OSL 1.12 release,
we will bump even further to OpenEXR 2.4 as the minimum. We'll make
that decision later. It would depend on our being confident that even
though 2.3 is technically within the 3 year window we try to support,
we cannot discern that anybody is actually using it (who would also
need the most recent OSL in the same app).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Added batched llvm code gen for: llvm_gen_blackbody, llvm_gen_luminance, llvm_gen_transformc (including code gen of loop to identify unique "from" and "to" space combinations to call library function with).
Made ColorSystem const correct.
Pulled much of ColorSystem's implementation into src/liboslexec/opcolor_impl.h so it could be inlined in both opcolor.cpp and wide_opcolor.cpp.
To enable SIMD fast path for ColorSystem::blackbody_rgb added new methods to expose underlying lookup table optimization.
bool can_lookup_blackbody(float T /*Kelvin*/) const;
Color3 lookup_blackbody_rgb (float T /*Kelvin*/) const;
Color3 compute_blackbody_rgb (float T /*Kelvin*/) const;
When can_lookup_blackbody() returns true
then lookup_blackbody_rgb can be safely called,
otherwise the expensive compute_blackbody_rgb is required.
Updated implementation of ColorSystem to succesfully vectorize on multiple compilers, optimized lookup table data types and indexed dereferences to generate prefered 32bit indexed gathers vs 64bit gathers which require multiple instructions and registers.
To improve code generation and less chances of aliasing issues, changed nested M[i][j] operator access to Matrix in ImathMatrix.h to directly access underlying 2d array M.x[i][j] also changed Vector operator V[i] to directly access V.x, V.y, V.z in OSL/IMathx/IMath*.h
Moved helper testIfAnyLaneIsNonZero from wide_opmatrix.cpp to be testIfAnyLaneIsNonZero in OSL/wide.h
Added sfm::min_val to provide an implementation of min which returns values not references to the original through a ternary (which can cause issues with vectorization). Added Clang specific sfm::min_val and sfm::max_val implementations to assist in vectorization.
Enabled BATCHED execution of existing testsuites: blackbody, color, transformc, wavelength_color
Added new testsuite BATCHED regression tests: blackbody-reg, color-reg, luminance-reg, transformc-reg, wavelength_color-reg
Added sfm::min_val to provide an implementation of min which returns values not references to the original through a ternary (which can cause issues with vectorization). Added Clang specific sfm::min_val and sfm::max_val implementations to assist in vectorization.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Adopt OIIO calls to fast_cbrt and fast_expm1 for blackbody computation.
Use single precision for bb_spectrum object/operator and manually implement std::pow(wlm,-5.0f) to better enable vectorization.
Reduced masked code generation for rgb_to_hsl.
Removed duplicated code by implementing ColorSystem::blackbody_rgb(T) in terms of ColorSystem::can_lookup_blackbody(T), ColorSystem::lookup_blackbody_rgb(T), and ColorSystem::compute_blackbody_rgb(T). Removed some casting to uint16_t that was unnecessary.
Removed some test shaders that were not intended to be promoted.
* Added additional test result for black body because the change from double to single precision caused a few results to differ slightly (max delta 0.0000534058)
* Incorporated changes from PR to "fix color conversion from RGB to xyY"
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
This change now requires renderers to manually link in the compiled rend_lib.cu
PTX code when using OptiX 7.
Signed-off-by: Clifford Stein <cstein@imageworks.com>
Changed the prototype of osl_get_attribute callback in liboslexec in
order it takes a copy of the attribute TypeDesc casted into a long long
instead of taking a pointer to it. This allows to make it work properly
on GPU else we have an invalid pointer.
Signed-off-by: nicolas <nicolas@isotropix.com>
Fix problem with unresolved symbols when multiple shaders call certain (#1401)functions on the GPU (ie. osl_printf()).
This fixes a problem I was seeingon the GPU: if I had two shaders that contained a printf(), then the second shader group would report osl_printf as being an unresolved function (the first shader group was fine). If I looked at the PTX for the second shader group, osl_printf was tagged as a "weak" function. Removing the "weak" attribute resolves the problem.
I'm not sure why osl_printf and rend_get_userdata should be different from the other functions in rend_lib.cu.
This is easily reproduced in testsuite/testoptix by adding a printf() to checkerboard.osl and microfacet_test.osl.
Signed-off-by: Clifford Stein <cstein@imageworks.com>
Added clang pragma and gcc attribute to disable Fused Multiply Add for nonAffineInverse to allow offline compilers to produce results closer to those of LLVM IR compilation (which has FMA disabled by default) which produced the reference images of the testsuite.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
Newly added functionality in TextureSystem (only in OIIO >= 2.3.7)
allows us to add a new flavor of gettextureinfo that takes texture
coordinates, and in the case of UDIM textures, will retrieve the info
for the particular UDIM texture that will be used for the given
position:
int gettextureinfo (string texturename, string paramname,
float s, float t, output TYPE destination)
NOTE: Use with caution, this will be more expensive than ordinary
calls to gettextureinfo that will tend to constant-fold effectively,
since the s,t coordinates are runtime-determined and won't be known
for runtime optimization.
If built against older versions of TextureSystem that lack support for
the new is_udim() and udim_resolve() methods, this new gettextureinfo
will simply use the old gettextureinfo() without coordinates, and thus
will not be able to retrieve position-specific texture info.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Most important: make sure we have a proper VFXPlatfom-2022 like test,
with Python 3.9 and OpenEXR/Imath 3.1.
Fix a few of the "name" entries for tests to match the set of versions
actually present on those images.
Remove apt-get install of libboost-regex-dev, since we use only
std::regex now that we're no longer using gcc4.8.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Similar to a recent change in OIIO... For historical reasons, our
wrapper makefiles assume that we build in `build/ARCH[.variant]`, so
maybe build/linux64, build/linux64.debug, etc.
Get rid of this extra bit of hierarchy, it no longer serves us and
it's contrary to modern cmake practice.
Also used this as an excuse to remove some SPI build script
customization that we no longer use.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This is analogous to the "output placement" that we added recently, but
for input of userdata.
Reminder: "userdata" means shader parameters marked as lockgeom=0 that
are expected to be supplied on a per-shade basis, such as interpolated
vertex data on geometric primitives. The primitive data binds by name to
matching shader parameters.
This patch concerns how the shader gets this per-point data.
Old school: in the middle of the shader, when userdata is needed, the
shader calls osl_bind_interpolated_param(), which makes a callback to
RendererServices::get_userdata(), which... does something magical and
renderer-specific to retrieve the data by name. This is expensive and
also challenging to implement on the GPU (which is why it remains
unimplemented).
This patch changes the methodology to the following, which is somewhat
analogous to how we did output placement:
0. It is assumed that there is an arena of memory where the userdata
is going to be, and the pointer to its start is passed down to the
shader via execute() (just like how the shaderglobals and output
arena pointers are passed), along with an index of the current
shade point.
1. The renderer pre-declares SymLocationDesc records for the userdata,
which specifies the offsets and strides within the arena where the
data can be found.
2. Prior to launching the shader execute(), the renderer is expected
fill in those values in the userdata arena for the points to be
shaded.
3. When JITing the part of the shader that initializes those userdata
parameter values, we simply JIT a simple data copy directly from
the computed position within the arena (offsetting the right amount
for this data, at this shade index) -- just a couple of
instructions per shade, no function calls, no RendererServices
callbacks, no name lookups!
4. There is no 4. That's it.
See the changes to osldeformer.cpp for an example of how this looks from
the renderer side. It's really straightforward.
Currently, if there is no SymLocationDesc for a lockgeom=0 parameter
that needs initialization, it will fall back to the old callback. We
may fully deprecate the old way, but not for a while, so this is back
compatible for now.
This seems to work, passes all tests, doesn't break anything, and works
for the osldeformer example!
There is follow-up work to do: I think we can pretty easily do the
same thing for ShaderGlobals ("make your own SG struct"), named marix
retrieval, and other attributes. Will do those in subsequent PRs if
this meets everybody's approval.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Well intentioned attempt to ensure that Cuda runtime library was in
the library path when tests were run, but it clobbered the existing
LD_LIBRARY_PATH.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Add batched code gen to generate a loop to calls to OSL library noise functions with binning of varying noisetypes and options. For example if half of a SIMD batch uses gabor option "bandwidth" of 10 and the other half default "bandwidth" of 1, then gabornoise will be called twice (once with bandwith option set to 10 and a mask with the 1st half of SIMD lanes enabled, then a 2nd time with bandwith option set to 1 and a mask with the 2nd half of SIMD lanes enabled.
Added batched OSL noise library functions which compile with outer loop explicit vectorization. NOTE: Perlin and some other noises have SIMD intrinsics that work on multiple data elements (block vectorization). When the the active number of lanes in a batch is below a tunable threshold, the legacy block vectorization version is called. This avoids the worst case scenarios of 1 active SIMD lane executing, and allows the 3-4 wide x,y,z data element version to run instead.
Add previosly omitted /Qopenmp-simd and -fopenmp compiler flags along with -DOSL_OPENMP_SIMD to cause OpenMP outer loop explicit SIMD vectorization of loops in batched OSL libraries.
Fix Mask::invoke_foreach to be OSL_NOINLINE, it was mistakenly set to OSL_FORCEINLINE
Update oslnoise.h to access target specific library sfm namespace
Improve compile time for 4d perlin noise with Dual arguments by using a 2 iteration loops to avoid repeating a large amount of code.
Changed ShadingSystemImpl::count_noise(void) to count_noise (int number=1) to allow batched version to update the call count with the number of active lanes in the current mask.
Fixed sfm::scalar_gabor3 and sfm::scalar_pgabor3 to not use the loop array index when storing partial results, when vectorizing an aliasing issue on the stack caused incorrect results.
Enabled BATCHED execution of all noise tests.
Added new tests: noise-generic, pnoise-generic
Added new BATCHED_REGRESSION tests: noise-gabor-reg, noise-reg, pnoise-reg
Avoid issue with constant folding in hashnoise with different compilers causing slightly different results from sin & cos to cause massively different results due to aliasing in hashnoise.
* Add openmp-simd and target compiler flags when CMAKE_COMPILER_IS_GNUCC for batched target specific liboslexec libraries
Move filter_gabor_kernel_2d from gabornoise.h to .cpp to avoid unused function warning.
* Revert to reference 4d perlin noise for Clang vectorization.
* Utilize reference filter_gabor_kernel_2d which uses its own gabor_mul_m22_v2(const Matrix22& m, const Vec2& v) instead of relying on Imath version which differs in naming between OSL/matrix22.h and <Imath/ImathMatrix.h>
Move filter_gabor_kernel_2d back from gabornoise.cpp to .h to share it between gabornoise.cpp and sfm_gabornoise.h.
Disable Clang vectorization of strlen because resulting binary causes crashes in testsuite (TODO: investigate)
* To avoid "may be used uninitialized in this function [-Werror=maybe-uninitialized]" messages add extra (unnecessary) initialization, as its a unlikely fallback case shouldn't affect performance much.
* Disabled GCC optimizations for nonAffineInverse to allow results to match LLVM-IR version for Matrix44 inversion.
Implemented better optimization for transform normal when matrix is uniform.
* Mark nonAffineInverse definition as OSL_HOSTDEVICE to match its forward declaration.
* Added environment variable TESTSHADE_LLVM_JIT_FMA to force FMA generation for JIT'd code, needed to produce reference i mages for tests.
Fixed warning message that was accessing a null pointer when testshade requests FMA without specifiying a target.
Added out_LLVM_JIT_FMA.tif image for hashnoise tests whose results vastly differ for small changes in input values, so we need a 2nd test image to compare against when FMA is enabled for LLVM JIT.
* Rearranged local stack variables inside of sfm::simplexnoise4 to instantiate Vec4 locals closer where they are used. This was necessary to get correct results with gcc (even with vectorization disabled), suspect issue with register overflow or stack stomping.
* Fix issue with impl_transform_normal_masked failing to vectorize when only the result is varying.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
OIIO 2.3 will rearrange some header files slightly. If you need to use
ImageCache, you must include imagecache.h (duh!). We were accidentally
relying on some leaked definitions in other header files, which will
soon be removed.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Implement batched versions algebraic, trigonometric, and transcendental OSL library functions.
Add extensive regression tests to exercise binary and ternary operations with combinations of uniform, varying, and constant arguments.
Added some reusable xmacros for shaders in the testsuite/common/shader's directory to create unary, binary, and ternary test shaders.
Increase timeout of CI tests from 180 seconds to 240 seconds to allow comprehensive regression tests to complete.
Alternative is to break regression tests into separate test directories so the individually finish faster.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
This feature (used, very possibly, only by me, for debugging) lets me
force OSL to write an archive that contains the oso files and a
serialized version of the shader network.
If no explicit name is given (by the "archive_filename" attribute), it
picks a name based on the shader group name. But that may not be a
safe string for a valid filename. We already eliminated anything
before the last slash, but it turns out that a colon (':') and pipe ('|')
can also really mess things up. So this small change protects against that.
For the actual commands sent to system(), escape special characters
in the strings and enclose any arguments that come from the user in
double quotes. That prevents shenanigans like asking for an archive
filename called "; rm -r *". By enclosing in double quotes, we'll
end up with a badly named filename produced by tar, rather than
possibly an arbitrary command being executed.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Background: OIIO now has errorf that are explicitly printf-like, and
errorfmt that are explicitly std::format-like, and also an old style
error() that is ambiguous. The ambiguous calls are (at this moment)
aliases to the printf ones, but will be deprecated at some point (and
may, later when it seems safe, get reborn as the std::format compliant
behavior).
This roots out and removes the last few calls to the old ambiguous
ones from the OSL codebase.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Update the "bleeding edge" CI test to gcc-11, and also LLVM 12,
which we hadn't tested before.
* A "latest releases" CI test uses gcc10 and LLVM 11.
* Turns out that nothing else on the OSL side needed to be fixed to
accommodate gcc 11 (though you need a very up-to-date OIIO if you
want to compile it with gcc11 unless you turn warnings off).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Implement batched matrix ops, including transform, determinant, transpose, and construction and getmatrix with spaces, multiplication, division, negation.
Moved helpers det2x2, det3x3, and det4x4 from opmatrix.cpp to Imathx.h so they could be shared with the wide_opmatrix.cpp.
Enabled BATCHED testing for matrix and transform testsuites
Added new BATCHED_REGRESSION testsuites array-reg, matrix-arithmetic-reg, matrix-compref-reg, matrix-reg, transform-reg to more fully exercise combinations of uniform and varying data types and indexing variables.
* Removed #if 0 code block of determinant match that was moved to Imathx.h
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
As discussed at the TSC meeting and on the mail list, I won't merge this
change until June 1. It will only be in master (future OSL 1.12), not
backported to any release branch.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Implement batched loop control flow (including, break, continue) along with early outs (return, exit). Also add batched support for printf and other string operations. Implement batched support for shadingsystem ops like bind_interpolated_param, get_userdata, rangecheck, uninitcheck_values, and nancheck. Implement batched support for negate, min/max, arraycopy, compassign, mxcompref, mxcompassign, .Added BATCHED_REGRESSION tests to the suite to exercise control flow and string ops and compare scaler and batched results. Enabled various tests to also execute in BATCHED mode. Enable unoptimized BATCHED mode testing, as constfolding was avoiding any work in optimized mode for some tests.
Fixed bug in testshaded batched mode that inverted population of x and y shader global values, but only noticed in order of side effects like printf in shaders.
Added buffered file support to ShadingSystem to allow batched fprint to write out to files in the correct order.
Fixed bug in BatchedAnalysis which was not handling unlocked geom correctly by requiring unlocked input parameters be varying inorder to accept Wide results from BatchedRendererServices::get_userdata.
LLVM_Util had previously added support managing masks, and its strategy should
be reviewed before trying to understand this commit's changes:
// Overview of masking strategy
//
// For if/then based on a conditional, the result of the conditional is
// conditionals written to a symbol that is a wide boolean. To handle
// nesting we will keep a stack of conditional masks. When a new
// conditional mask is pushed, it is combined with the previous mask on
// the top of the stack. It is the top of the mask stack that is used
// as the mask for any instructions requiring masking. When a
// conditional scope is exited, we just pop the top of the conditional
// mask stack.
//
// Loops are handled by pushing their conditional result onto the
// conditional mask stack, and popping it back off at the end of the
// loop.
//
// Handling of early outs (exit, return, break, continue) is done by
// storing a mask representing the lanes which are going to be
// deactivated (or continue executing) to an allocated location in the
// stack. We actually have to store these early out masks, so that we
// can apply it to the conditional mask stack. As that stack unwinds,
// any early out masks are applied to the top of the conditional mask
// stack. As there are different types of early outs that apply to
// different control flow scopes and multiple types can be present at
// the same time for different data lanes, we must store these early
// out masks at separate locations on the stack.
//
// To handle 'exit', we allocate a 'shader mask' that is initially
// populated with a startMaskValue representing the active lanes in a
// shader global batch. For example perhaps only 7 of 16 data lanes
// are populated, the startMaskValue would store that value and push it
// onto the conditional mask stack. We require a location on the stack
// for this 'shader mask' in order for an 'exit' statement to modify it
// and cause a subset of the lanes to be deactivated. If an exit was
// processed, the 'shader mask' is where it will be stored and pulled
// from when applying to the unwinding conditional mask stack.
//
// To handle 'return', we allocate a 'function mask' that is initially
// populated with a startMaskValue representing the active lanes when
// an inlined function was called. This startMaskValue should just be
// value at the top of the conditional mask stack. As function calls
// can be nested, we maintain a stack of function masks. We require a
// location on the stack for each 'function mask' in order for a
// 'return' statement to modify it and cause a subset of the lanes to
// be deactivated. If an return was processed, the 'return mask' is
// where it will be stored and pulled from when applying to the
// unwinding conditional mask stack. When a function scope is exited,
// the function mask is popped so any return's processed would apply to
// the outer function mask vs. the one just exited.
//
// To handle 'break', we use the condition symbol's location on the
// stack to directly disable data lanes for the loop execution. As we
// already had a location to store the conditional symbol's mask, no
// need to allocate or store a separate one. However code will need to
// reload the condition's value when a break 'could have been' called
// vs. relying on a conditional result already in a register.
//
// To handle 'continue', we allocate a 'continue mask' that is
// initially populated with a 0 representing the deactived lanes by
// 'continue' operations. We require a location on the stack for each
// 'continue mask' in order for a 'continue' statement to modify it and
// cause a subset of the lanes to be deactivated. If an continue was
// processed, the 'continue mask' is where it will be stored and pulled
// from when applying to the unwinding conditional mask stack. When a
// loop's scope is exited, the continue mask is popped so it would't be
// applied to an outer loop.
//
// The logic of when and how to actually apply these masks is handled
// by the caller, tracking/storing the different stacks and masks is
// handled here. And queries exist to see how many early outs
// operations have been processed, these counts can be used by the
// caller to determine which to generate code to apply each of them.
In our case BatchedBackendLLVM is the caller and during code gen will be calling:
push_shader_instance(llvm::Value* startMaskValue) and pop_shader_instance()
push_function_mask(llvm::Value* startMaskValue) and pop_function_mask()
push_masked_loop(llvm::Value* location_of_control_mask, llvm::Value* location_of_continue_mask) and pop_masked_loop()
And to handle if/then/else be calling
push_mask(llvm::Value *mask, bool negate, bool absolute) and pop_mask()
By checking masked_exit_count(), masked_return_count(), masked_break_count(), and masked_continue_count()
before and after code gen of a function call, then block, else block, loop body, etc.
we can detect if a exit, return, break or continue existed in the generated scope
and then can call apply_exit_to_mask_stack(), apply_return_to_mask_stack(),
apply_break_to_mask_stack(), or apply_continue_to_mask_stack() when needed.
Add utility function "foreach_unique" to provide scalar execution of a Functor
for each unique value in the a Wide data block along with a mask identify which
data lanes match the unique value.
Add utility function "unproxy(const T &elem)" to extract value type from a proxied
element. This is alternative to static_cast<ValueType>(elem), and better self documents.
Fixed compatibility bug where llvm 9+ asserted that bool was not a valid datatype for llvm::ConstantDataVector::getSplat.
Added LLVM_Util::test_mask_lane(llvm::Value *mask, llvm::Value * lane_index) to test a mask with a variable lane index.
* Updated OIIO::vformat calls to OIIO::sprintf.
Conditionally compile buffered_file_output_mutex to avoid unused variable warning when not building with USE_BATCHED.
Ran couple files through clang-format.
* Initialize local variable mask_slot to avoid maybe-uninitialized errors
* Fixed paths to additional oiiodiff that runs when a test fails to find a match to use the /baseline subdir when performing a BATCHED_REGRESSION test.
Improved control-flow-test to use 0.5 default value instead of 0 to tell difference between actual assignment and 0's on the heap.
* Changed some control-flow-reg tests to use non-zero default values to tell difference between uninitialized heap and setting of default values.
Changed order of parameters so that any initops that contain an "exit" are the final parameter so the exit doesn't skip assigning default values. Otherwise those parameters would be uninitialized data and cause intermitent CI failures.
* Deleted extraneious cmakefile settings for clang
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* "SOVERSION" option was set twice, oops!
* Stray "OIIO" in a name that should have been this project's name.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This PR adds closure declarations for the MaterialX PBS nodes. A discussion of their design has been ongoing in a thread at the MaterialX GitHub repo. The full history of this discussion can be found here: https://github.com/materialx/MaterialX/pull/614
Signed-off-by: Niklas Harrysson <niklas.harrysson@gmail.com>
This fixes CI in the Mac test. Something about the Homebrew upgrade to llvm 12.0.0_1 made this fix necessary to build.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Up until now, client apps declare which output variables are "renderer
outputs" before shader groups are compiled, and also after a shader is
executed, it is up to the app to extract the individual output values
and do with them what it needs to do. Those outputs live on the shader
heap (the temp storage used to run the shader), and information about
where it lives on the heap can be retrieved via find_symbol() and
symbol_address().
New paradigm:
Various inputs and outputs of shaders live in different memory arenas,
and for those inputs/outputs, the app declares (prior to shader group
compilation) which arena it lives in, its offset within the arena, and
the stride between entries (for different shades, such as different
rays in flight, different points on a mesh, pixels in an image, etc.).
The shader JITs into code that automatically pull inputs from their
sources and copy outputs to their destinations with no need for client
app code to call find_symbol or symbol_address or do any extra data
copies.
This symbol location data is communicated to the shading system through
new calls to add_symlocs() passing a new struct called a SymLocationDesc,
which has the aformentioned arena, offset, stride, etc.
Right now, the only supported arena is Outputs, and the actual call
signature to execute(), and to the shader JIT code itself, includes
extra parameters for the base pointer to the output arena (this could
conceivably change from shade to shade) and the integer shade index of
this shade.
I've tried to set this up in a general way, so that additional arenas
will be added later -- places where ShaderGlobals can be customized,
UserData, etc. Also there are placeholders for being able to talk about
absolute addreses and offsets into the heap, though I don't use them
at the moment. Some of the flexibility may be later removed as we learn
more about how we want to use this.
To show it in action, I have modified the "osldeformer" example to use
this new method to place the outputs directly where we want them.
It's just storing results into an array. The output arena base pointer
is the start of the array, the offset of the single output is 0, and
the stride is the size of each array element (one 3-float point).
As a much more complex example, testshade also uses the new technique
(unless you use --no-output-placement, but I'm not sure how long I'll
keep that). This is a good example of storing to images. Testshade thinks
of itself as shading pixels in an image, so there is an ImageBuf for every
renderer output (think of it like an AOV). The output arena base pointer
is the address of the (0,0) pixel of the ImageBuf for the first AOV; the
"offset" of each output variable is the address difference between that
AOV's first ImageBuf pixel and the arena base pointer, the stride is the
pixel data size for each output, and the shadeindex is the pixel index of
that shade.
It all works. Crazy!
OK, not quite all. Here's what hasn't been done yet:
* testrender does not yet use the new API (only testshade)
* GPU (have only tried CPU so far)
* Batch shading -- I don't want to mess with this while Alex is
still integrating all his work. We'll go back and catch the batch
shading up after he's all done.
These will come in subsequent PRs.
All the old calls should still work, so this ought not to be a source
compatibility break for renderers using OSL. Not at this point, anyway.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Left out some imath targets in liboslnoise
* Conform a couple libraries to conventions of the ${local_lib} notation
we use in the othe rlibraries.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This leaves the minimum C++ at 11, but bumps the default (if not set by
the user via CMAKE_CXX_STANDARD) from 11 to 14.
This is meant to start identifying any problems people have switching
to 14. We anticipate a hard switch to 14 minimum before the next OSL
major release. This change is in master only (i.e. future OSL), we would
never change minimum or default C++ standard in the supported release
branches (currently OSL 1.11.x).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Introduce cmake option USE_BATCHED to enabling building batched SIMD shader execution for a comma separated list of batch sizes, ISA's, and noFMA: (0, b8_AVX, b8_AVX2, b8_AVX2_noFMA, b8_AVX512, b8_AVX512_noFMA, b16_AVX512, b16_AVX512_noFMA)
When USE_BATCHED is set, -DOSL_USE_BATCHED=1 will be passed allowing for source files to conditionally compile in batched execution support.
When USE_BATCHED is set, cmake variable BUILD_BATCHED=True and projects can conditionally add batched sources to be built.
Updated OSL source with #ifdef OSL_USE_BATCHED/#endif were needed.
Improved/streamlined liboslexec/CMakeLists.txt to iterate over targets in USE_BATCHED vs. hardcoding all variants.
Updated _OSL_SUPPORTS_* defines to match USE_BATCHED values simplying creating compiler definitions in Cmake.
Added new BATCHED_REGRESSION testing mode, when a file named BATCHED_REGRESSION exists in a testsuite folder, the run.py will be executed and its output files copied to "./baseline". The same run.py is then run again but with environment variable TESTSHADE_BATCHED=1, which cause testshade to execute in batched mode. Then the results/image comparison happens between the batched output files and those in "./baseline". The "./ref" dir is never used in a regression test.
Presence of BATCHED_REGRESSION prevents non-regression tests from being created for a folder.
Converted arithmetic-cov to be a BATCHED_REGRESSION test.
Removed testsuite/arithmetic-cov/ref image files.
Renamed ShadingSystem::supports_batch_execution_at(int width) to ShadingSystem::configure_batch_execution_at(int width) to better identify its behavior: based on currently set attributes for llvm_jit_target and llvm_jit_fma, test if current machine is capable of supporting batched execution at the specified width. If no specific llvm_jit_target was requested, sets llvm_jit_target and llvm_jit_fma to the supported configuration for the requested width. Returns true if supported, false otherwise
Fix bug in batched Matrix*Matrix code generation where the order of arguments were being reversed (swap was intended to transform float*Matrix to a Matrix*float function call, but accidentally happened for Matrix*Matrix)
* Moved define of OSL_USE_BATCHED to oslconfig.h.in
Fixed order of includes in batched*.cpp so oslconfig.h is higher
* Renamed testsuite/arithmetic-cov to testsuite/arithmetic-reg to better represent it is a regression test.
Changed #ifdef OSL_USE_BATCHED to #if OSL_USE_BATCHED and rely on it being set to 0 or 1 by <OSL/oslconfig.h>
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
Generating code for prepare_closure and gen_closure was crashing.
The code now matches the implementation of the removed LLVM function.
Signed-off-by: Brecht Van Lommel <brecht@blender.org>
Ever since adding OpenEXR 3.0 support, we had some cases where
installations that built the 2.x IlmBase and OpenEXR portions
separately and installing to separate areas, had trouble being found
properly.
Remove the need to include OpenEXR/OpenEXRConfig.h
Also make some minor changes to ci.yml to make sure we are documenting
the right openexr versions used for each test.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
It seems that our liberal use of cmake argument `--config` should only
be used in conjunction with the `--build` flag, not for the initial
configuration step. Apparently this was just silently ignored for a
long time, but for CMake 3.20, it is an error and breaks a lot of our
CI cases and our build_blah.bash scripts.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Added building of ISA specific shared libraries lib_b16_AVX512_oslexec, lib_b8_AVX2_oslexec, lib_b8_AVX512_oslexec, lib_b8_AVX_oslexec to house precompiled OSL library functions that execute over batches of 8 or 16 in SIMD for the ISA. Compiler flags for OpenMP simd code gen and ISA targets has been added for Intel(r) C++ Compiler (ICC) and CLang (newer versions of GCC 6+ might be possible, but untested).
Implement batched llvm code gen for: generic function calls, useparam, compare ops, addition, subtraction, multiplication, division, modulus, assignment, component reference, construct triple, construct color, derivative extraction. Stubbed out all other code gen functions with TBD asserts.
Populate OpDescriptiors with valid wide version of llvm-generating routine
Added wide_opalgebraic.cpp which uses X-macros (instead of #define like llvm_ops.cpp) to define wide(batched) versions of OSL library functions: sqrt, inversesqrt, floor, ceil, trunc, round, sign, abs, fabs, fmod, and step.
The X-macro wrappers follow a pattern of manufacturing a target specific library function name with enough parameter types embeded in its name to uniquely identify it (vs. other versions). Then it declares local Wide<T> or Masked<T> wrappers that convert any void */char * parameters to references to Block<T,WidthT> data blocks of wide SOA data. Then an explicit OpenMP simd loop iterates over the data lanes and extract a local scalar values from the Wide|Masked wrappers, then the scalar implementation of the library function is then inlined using the local scalar values. Finally the result is writen back out to the data lane inside the Wide|Masked wrapper. This paradigm allows scalar implementations to be resused inside simd loops and avoid having to use intrinsics or assembly. It also allows the same implementation to be recompiled for different target ISA's and varius Widths (8|16). The build system will create copy of each wide_*.cpp to a target and batch size specific named b(8|16)_(AVX512|AVX2|AVX)_wide*.cpp and build it with different -D__OSL_TARGET_ISA and -D__OSL_WIDTH values which inturn will manufacture uniquie function names. Sometimes scalar algorithms/functions can be refactored to provide better performance when executing inside a SIMD loop. sfmath.h (SIMD friendly math) houses these alternative math functions, although many improvements have already been moved into OIIO as they benefit (or do no harm) to scalar code gen.
Made ShadingContext remember the ShaderGroup it just optimized. This allows symbol queries without actually jitting or executing a shader.
Improved TestShade to not actually execute the shader during setup_output_images, but to instead explicitly JIT scalar or batched version of the ShaderGroup (primarily to make sure JIT happens during the "setup" stage vs. lazily later).
Fix TestShade to explicitly set the number of OIIO worker threads to avoid overhead (and debugging confusion) of OIIO thread pools being created even when "-t 1" was requested.
Modified ShadingSystem to only perform group_post_jit_cleanup (delete operations of shader group) only if both scalar and wide JIT's have occured or if RendererServices doesn't support batching. Without this changed the operations were being deleted before a batched JIT could occur.
Extended testsuite framework to look for file named "BATCHED" which causes another run of the test with TESTSHADE_BATCHED=1
Added testsuite new tests with BATCHED enabled for passing tgh shaderglobal values, and increased coverage of arithmetic tests with reference images for float, color, point, vector, normal data types along with Dx Dy results.
Added utility macros __OSL_CONCAT, __OSL_CONCAT3, ..., __OSL_CONCAT10 to be able to easily manufacture function names.
Added macro __OSL_WIDE_PVT to give each target specific library its own namespace avoiding collisions should multiple libraries be loaded.
Added sfm::negate(const T &x) with optimized implementation.
Disabled some unreferenced functions warnings for ICC and removed some unused functions from batched_analysis.cpp
Updated BatchedBackendLLVM to match behavior of BackendLLVM by configuring its LLVM_Util based on ShadingSystem attributes.
Disable clang format for X macro based building of initializer arrays to prevent clang format from reordering the #include files.
Fix control flow in factory function TargetLibraryHelper::build to not trigger assert unnecessarily.
Limit list of OSL library functions in builtindecl_wide_xmacro to just those we have implemented so far because all functions listed must exist in the target specific library for it to successfully be loaded and resolved.
Added LLVM_Util::op_zero_if(llvm::Value *cond, llvm::Value *v) which allows its implementation to work around an LLVM issue where expensive instructions to produce the value (div, sqrt, etc) are duplicated (once with a mask, once without).
Fix bug in ShadingSystem::supports_batch_execution_at where jit_fma was being accidentally negated causing rest of logic to fail.
Implement ShadingSystem::BatchedExecutor<WidthT>::jit_group
* In Cmake, changed ISA specific target libraries from SHARED to MODULE so that on OSX they will be named .so instead of .dylib.
Moved new arithmetic tests into their own subdirectory arithmetic-cov so that the existing arithmetic test can run in non-batched mode as it uses printf which is still TBD for the batched version.
Fixed bug where shader ops weren't being cleaned up unless batched jit had occured.
Fixed compilation issue with newer LLVM versions and LLVMUtil::zero_if
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
Removed use of options.PrintMachineCode since this has been removed as
of llvm 12. Converted uses of DebugLoc::get to DILocation::get since
it has been deprecated and removed as of llvm 12.
PrintMachine code will only be used if llvm is lower than version 12.
TODO: Needs to be re-added in the new way for LLVM12. (LG just doesn't
have time to figure this out right now.)
Pass adjustment as various things have been deprecated or changed
names. (LG)
Signed-off-by: Euan Haahr <euanhaahrmail2@gmail.com>
Signed-off-by: Larry Gritz <lg@larrygritz.com>
In PR 1346, we overlooked the fact that OSLQuery::init() is defined
*differently* in liboslquery and the separate copy that used to be
in liboslexec.
In this patch, we remove the oslquery-side version (and in fact
coalesce it into the constructor, the only place it was used). So now
instead of being replicated differently in the two libraries, it only
exists in oslexec. That constructor, since it takes a ShaderGroup* as
an argument, only makes sense to call if you have a full ShadingSystem
nearby.
Add ShadingSystem::oslquery(), deprecate OSLQuery(ShaderGroup*).
This eliminates the weirdness of having this constructor undefined in
liboslquery, moves the functionality to the ShadingSystem.
Because of ABI compatibility issues, we have to merely deprecate (but
not remove) the OSLQuery functionality for now.
Avoid assertions for shaders with params that are arrays of indeterminite length
Signed-off-by: Larry Gritz <lg@larrygritz.com>
liboslquery and liboslnoise exist as separate libraries for apps that
only need to query shader parameters or use OSL's noise functions,
respectively, but dont want to pull in the rest of OSL and all its
heavy dependencies (including LLVM). But liboslexec also needs that
functionality and so separately put the individual modules inside.
This was causing some problems for a tricky use case, and I can't
think of a good reason for it to be done this way now. So in this
patch I remove the redundant modules and just make liboslquery and
liboslnoise into *link dependencies* of liboslexec. None of this
should affect downstream projects, I don't think; linking against
liboslexec ought to just automatically pull in those other libs.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* strings are no longer copied to the GPU and are instead passed around
as ustring hashes. This means we no longer need to copy string pointers
CUDA/PTX code. This should result in faster compilations because
strings are defined at "compile time" instead of "runtime".
* OSL's shaders' printf() output is now handled on the host/CPU
* testshade and testrender now use a separate optixLaunch() to set up
global variables
* removal of runtime compilation of CUDA (libnvrtc dependency eliminated!)
which was used to handle the setting of global variables in OptiX 7's
CUDA programs.
Signed-off-by: Clifford Stein <cstein@imageworks.com>
* Separate the CI stages into separate steps. (As was prototyped in OIIO)
* Combine the dependency-building scripts for CentOS and Ubuntu into a
single script.
* Get rid of cruft related to the CI systems we no longer use (Appveyor,
Travis, CircleCI).
* Get rid of some needless CI verbosity
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This speeds up CI builds by 50% or more, for PRs that are minor
changes by reusing the compiled results from previous runs. Most of
the remaining time cannot be squeezed down -- it's the instantiation
of docker containers and running the testsuite itself. But the build
(including of dependencies we build from scratch) is dramatically
faster.
At the same time, remove some commented-out test matrix cases. When/if
we re-enable those tests, we can add those back.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Add implementation for BatchedBackendLLVM. BatchedBackendLLVM implements the same set of functionality as BackendLLVM does, except it's implementation and many of it's methods handle wide data type (for symbols identified as varying (is_uniform()==false). Many methods exist for (re)use by batched_llvm_gen and batched_llvm_instance.cpp. A common theme is additional "bool is_uniform" parameter to identify if the operation we are generated code for is uniform or not. We cannot rely on the Symbol itself, as an operation will be varying if just 1 of its inputs is varying. This means that when uniform symbols are loaded as arguments, they will need to be widened/broadcast to partcipate in the varying operation. Additionally "bool index_is_uniform" is used to generate gather/scatters vs. a wide load when symbols are varying. Please remember the underlying "wide" data types are Structure of Arrays (SOA) and all address calculations have been modified inside BatchedBackendLLVM to work with SOA's.
Introduce helper class FuncSpec to encapsulate building function names that include a batch width and target ISA along with uniform and varying parameters.
Introduce TargetLibraryHelper interface and factory method to construct a Concrete TargetLibraryHelper
that provides a prefex string that all function calls will be prefixed with
and correctly initialize a function map for the shading system with
the functions from the target ISA library. Intent is batch size and target ISA specific shared libaries will be built and the correct on be loaded based on host machine.
Introduce TempScope. Any calls to getOrAllocateTemp during the lifetime of a TempScope
will be associated with the latest TempScope on the stack and
when that TempScope's lifetime ends, any temp allocations will
be marked unused and will be available for reuse by the next
call to getOrAllocateTemp.
Introduce liboslexec/wide/define_opname_macros.h to define macros to consistently build OSL library function names including batch width, target isa, 0 to 5 arguments, and optionally masking.
Added builtindecl_wide_xmacro.h which is the equivalent of builtindecl.h for identifying the function names and signatures of OSL library functions that exist in batch size and target ISA specific shared libraries.
* Added missing initialization of OpDescriptor::llvmgenwide
Added friend declaration for ConcreteTargetLibraryHelper to use private base class TargetLibraryHelper
* Replaced some older copyright notices with latest one.
Moved llvm::Value* arrind assignment in batched_backendllvm.cpp from innerloop.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Rename CLA files so they no longer say "draft."
* Change CONTRIBUTING.mv to correctly describe the new CLA process,
remove language related to the old SPI CLAs. Also some touchups to
README.md.
* Update the project_intake.md to check of various things that we've
completed.
* Change all references to the old imageworks GitHub URLs to the new
ASWF ones. (This touches comments at the start of basically every file.)
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* readme and documentation for windows build is added
* some fixes
* remove pythonpath
* add .vscode
* windows installation document is added to new directory
* add windows build script to new doc folder
* remove .vscode folder by adding it to .gitignore
* remove .vscode manually
* update document format
* change the name of the documentation to Readme.md
* add relative image path
* image now is ok
* remove copyright problems
* remove my website name
* add appropriate copyright notices to all the files
* a little fix
* Pixar copyright is added to the python file
* Modified Apache 2.0 License is added to THIRD_PARTY.md
* remove my website from the doc
* revert changes on THIRD-PARTY.md
* Modified Apache 2.0 License is reproduced
Introduce BatchedAnalysis which processes operations similar to code gen in order to build up relationships between operations and symbols controlling conditional control flow. Using these relationships and information on which shader globals are varing and which symbols renderer outputs, varying can be pushed down to just the symbols that will require it. Also analysis to determine which symbols can be forced in llvm to be boolean vs integers.
The idea is simple, but understanding & managing the relationships is complex, however it is entirely encapsulated in BatchedAnalysis.cpp. The main goal is to build up a multimap "m_symbols_dependent_upon" which maps a key symbol to all other symbols that due to operations, conditions, loops, function calls, early out control flow (break, continue, exit) would be forced to be varying if that key symbol where varying. We can use this map to recursively push varying of symbols down from shader globals, renderer outputs, and operations whose results are implicitly varying.
Several custom data/acceleration structures have been created to efficiently model these relations ships with their key feature being that their iterators are never invalidated as additional items are pushed and popped such that one can always iterate from a historical position back to the root:
DependencyTreeTracker, FunctionTreeTracker. Additonally LoopStack tracks loop nesting and aids in detecting cyclical reads.
Analysis is also performed to identify which symbols are read at a different conditional scope than they were written to and might require masking. Symbols written to and read from within the same (or deeper nesting) conditional scope do not require masking. A conditional scope is defined as the stack of of conditions evaluated to reach an operation and is managed by the DependencyTreeTracker.
When adding new ops, or when changing implementation or behavior of
existing ops, the following functions should be evaluated to see if
they need to be updated to handle to handle the new op or change:
bool are_op_results_always_implicitly_varying(ustring opname);
bool are_op_results_always_implicitly_uniform(ustring opname);
bool does_op_implementation_require_masking(ustring opname);
SymbolPtrVec * protected_shader_globals_op_implicitly_depends_on(ShaderInstance &inst, Opcode & opcode);
If there is a new or change in control flow op's are made,
a more detailed evaluation will be needed as their behavior
is hard coded vs. lookup based implementation of the
functions above.
RuntimeOptimizer uses BatchedAnalysis before it coalesces temporaries so that varying and uniform symbols do not get merged.
* Add ShadingSystem attribute "opt_batched_analysis"(defaults to false) which triggers additional analysis during runtime optimization to set Symbol::is_uniform, Symbol::forced_llvm_bool(), and Opcode::requires_masking. Only coalesce_temps for symbols whose is_uniform() and forced_llvm_bool() values match. Update testshade to set opt_batched_analysis when batched execution is requested.
* Initialize opt_batched_analysis to true when RendererServices provides a BatchedRendererServices for WidthOf<16> or WidthOf<8>.
Changed testshade from explicitly enabling opt_batched_analysis to explicitly disabling it when not in batched execution mode.
Removed test for opt_batched_analysis from coalesce symbols pass and just rely on a Symbol's is_uniform() and forced_llvm_bool() to be set correctly.
OIIO removed "RB-2.1" branch marker because it's not expecting any
further development in the obsolete 2.1 branch. We still test against
it, though, so replace its designation in our CI with the specific
branch tag of the last OIIO 2.1 release.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Add storage to Symbol for analysis result bool forced_llvm_bool().
Results of a compare_op and other ops with logically boolean
results under certain conditions could be forced to be represented
in llvm as a boolean <i1> vs. an integer <i32>. This simplifies
code generation, and under batched execution is a requirement
to make efficient use of hardware masking registers by allowing a
vector of bools <16 x i1> vs. integers <16 x i32>. However the
underlying OIIO::TypeDesc as well as OSL does not support bools,
therefore they need to be promoted to integers when interacting with
other integer op's.
The value of forced_llvm_bool() is currently only respected during
batched execution. Forced bools should not be coalesced with regular
ints, only other forced bools.
* Nicolas Guiard is a new TSC member representing Isotropix
* Chris Kulla has had a new affiliation for a while now
* Clean up a bit of language left over from when this document was
just a proposal.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Add ShadingSystem attribute "searchpath:library" (similar to searchpath:shader) that is used to identify directories where target ISA optimized OSL shared libraries may exist.
Remove testshade command line parameters of "path" and "librarypath" as they were never used and can still be specified with "options".
During batched analysis we will be figuring out which symbols need to be varying and which can be uniform during batched execution, we will also be figuring out the minimum number of operations which require masking as well as storing/caching some other operation specific flags used later in batched code generation. We choose to store these analysis results inside Symbol and Opcode directly so that they are efficiently accessible and stable under optimization passes than might invidate OpcodeVec.
Modeled after recent similar changes in OIIO.
* Detect if OSL is being built as a subproject, and if so, set
`OSL_IS_SUBPROJECT` (a.k.a. `${PROJECT_NAME}_IS_SUBPROJECT`).
* Move all the testing logic out of the main CMakeLists.txt (and a few
testing-related macros from oiio_macros.cmake) and into a new
src/cmake/testing.cmake.
* Move all of the packaging logic out of the main CMakeLists.txt and
into a new src/cmake/packaging.cmake.
* Make all the testing and packaging inactive if this is being built
as a subproject. Also disable the clang-format target for subprojects.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Upon reading cmake docs mre carefully, it's clear that when you pass a
version to find_package (which does much of the heavy lifting inside
checked_find_package), it DOESN'T MEAN you want that version or
later. It means you want to find a version that is "compatible" with
the named version. The meaning of "compatible" is up to the package.
Some packages define it as being >= the requested version. Others
define it as being the same major release, or same major AND minor
release, or exact.
But generally speaking, we only care about the minimum version (e.g.,
"I want FooBar 8.0 or later"). And it's not guaranteed that is what
the package's compatibility will do. In fact, FooBar may be using
compatibility mode "SameMajorVersion", which will make our find of
FooBar succeed for 8.0, 8.1, 8.2, but if a user encounters FooBar 9.0,
our build will reject it, even though it's probably fine.
So I augmented our checked_find_package wrapper to take optional
VERSION_MIN to give the real minimum version, when that's the semantic
we really want (in which case we would omit the ordinary version
number used for the compatibility check and let VERSION_MIN do all the
work). There's also an optional (and usually omitted) VERSION_MAX that
you can use if you are sure that we must be less than a certain
version (the max is exclusive, you must be <, not <=, whereas the min
is inclusive >=).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Some minor refactoring of the recently introduced get_vec3 introduced
a bug into the constant folding of + and - for mixed floats and
triples: the constant produced was of the type of the first operand,
instead of the result. This fixes.
Added handy helper to make vector constant (and disambiguate it from
the one that makes a color constant).
* Add Symbol::wide_dataoffset() to track offset in heap to wide version of symbol's data
Add flag to Context to track if last execution was batched
Add template<int WidthT> ShadingContext::Batched interface to group together batched versions of execute_init, execute_layer, and execute methods.
Instantiate ShadingCOntext::Batched<8> and ShadingContext::Batched<16>.
Refactor std::vector<std::pair<ErrorHandler::ErrCode, std::string> m_buffered_errors to use proper ErrorItem class with optional Mask to identify which data lanes the Error message applies to.
Added struct ErrorBatch and std::vector<ErrorBatch> m_buffered_error_batches to track which entries in m_buffered_errors came from batched execution as those ranges need to processed multiple times, once per active data lane.
Refactored ShadingContext::process_errors into a templated helper to correlate batched and not batched errors effectively serializing the message output up to the ShadingSy
stem.
Modify ShadingContext::symbol_data to use the symbol's wide_dataoffset() when last execution was batched.
Add RunLLVMGroupFuncWide to model a a function pointer to a batched verson of a JIT'd shader function that accepts batched shader globals and an integer representing the mask (active data lanes) to execute on.
Added ShadingSystemImpl::Batched<int WidthT> interface to group together batched methods jit_group, and jit_all_groups
Instantiate ShadingSystemImpl::Batched<8> and ShadingSystemImpl::Batched<16>
Added ShaderGroup::batch_jitted to track if a group had been jitted for batching (separately from scalar jit)
Added ShaderGroup::llvm_groupdata_wide_size to track amount of heap space needed for a wide version of the group data structure.
To hold onto wide versions of llvm functions side by side with scalar, added llvm_compiled_wide_version, llvm_compiled_wide_init, llvm_compiled_wide_layer.
Add plumbing through ShadyingSystem into Context to support Batched Execution.
* Removed ErrorBatch as it was deemed unnecessary (unreachable usage scenario), which simplifies processing buffered errors.
Changed bool m_execution_is_batched to int batch_size_executed, so we know exactly how many data lanes to process errors for.
Fixed some clang-format issues in osl_pvt.h
After talking this over with Luke Emrose it seems unwise to allow
merging of layers that produce renderer outputs.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Most failures in this function returned false, but if it got as far as
parsing but failed that stage, it ended up returning true (success),
which is clearly wrong.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
As OpenEXR works toward their 3.0 release, which includes splitting
Imath (the vector, matrix, math, and 'half' stuff) out into a separate
repo, their top of tree master finally breaks our "bleeding edge"
build.
This is a little tricky, because now our support of this dependency is
attempting to span 3 eras: 3.0 with the full split of openexr from
imath, recent openexr+ilmbase 2.x that supplies reliable exported
cmake configs, and older 2.x that predates cmake configs. Eventually
we'll phase out older versions and clean up.
Fixes in this patch generally include:
* Add a new CI test of VFX2021 + openexr/imath master. (This will go away
when 3.0 is released and that's what we use for the main 2021 case.)
* For new enough versions of OpenEXR, use the exported config file and
targets, rather than the old style OPENEXR_LIBRARIES style variables.
We have kind of a mixed idiom use because we're still supporting older
versions that predate use of the config files.
* Deal with the fact that the Imath headers now live inside
`<Imath/ImathBlah.h>` rather than `<OpenEXR/ImathBlah.h>`. But of
course, you need to start including things just to find out the
version. It's tricky. The logic is to look at
`<OpenEXR/OpenEXRConfig.h>` to determine the version we're dealing
with, then include Imath stuff from one or the other depending on
what we found. I rearranged quite a few includes to ensure that all
this logic only happens in a couple spots and the rest gets included
transitively. As it turns out, quite a few of our `#include`s of
Imath headers were redundant.
* OSL's exported config needs to look for OpenEXR.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Two new preprocessor directives that oslc will respond to:
#pragma error "my message"
#pragma warning "my message"
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Added 64bit specific popcount call for GCC/Clang in Mask<int WidthT>::count()
* Implement BatchedSimpleRenderer for TestShade
* Introduce template<int WidthT> struct BatchedShaderGlobals
to store upto WidthT instances of ShaderGlobal data. The data for a batch is broken into underlying uniform and varying data structures. Uniform contains a fixed set of shader global values that are shared between all instances. Varying has Block<DataT, WidthT> members so each instance can hold a different value. The Block data structure is specialized to hold the underlying data type in a SIMD friendly Structure of Arrays (SOA) data layout but provides an array subscript interface to a proxy that can import/export the DataT in its orignal data layout.
Introduce template<int WidthT> class ShadingSystem::BatchedExecutor as an interface to execute batches of shader globals through a shader network. BatchedExecutor provides similar execute methods that accept a BatchedShaderGlobals and batchSize instead of individual ShaderGlobal(s). This is just the interface, batched backend implemention to follow in subsequent pull requests.
Introduce template<typename DataT, int WidthT> Wide to hold a reference to Block<DataT, WidthT> and provide access to the underlying SOA data layout. Wide can handle array types and const correctness. The Wide adapter is lightweight and can be copied by value, used as function parameter.
Once batched execution finishes, the resulting symbol address can be passed to a Wide<DataT, Width> to access the resulting data in a type safe manner.
Updated testshade with command line option "--batched" causing batched execution of 16 or 8 instances through the shader network and to extract the results. A later pull request will add batched interface to renderer services.
* Add new file src/include/OSL/batched_shaderglobals.h
* Disambiguate fabs call to fabs(double) so it doesn't try OSL::fabs which has no overload for double
* clang-format of batched_shaderglobals.h
* Add BatchedRenderServices, Mask<int WidthT> to represent active lanes during SIMD execution, BatchedTextureOptions, Masked<typename DataT, int WidthT>
* Removed unnecessary .c_str(), changed call to ::fabs to std::fabs
* Removed BatchedRendererServicesBases in favor of simple virtual functions is_overriden_*().
Improved some comments for BatchedRendererServices.h
Workaround some QT macro issues.
* Add BatchedRendererServices implimentation which mainly asserts that its methods are never directly called.
Fixed up issues with OSL_USING_DATA_WIDTH(WIDTH) macro.
* Removed Dz Structure Of Arrays wrappers as OSL doesn't currently support Dz.
* clang-format of mask.h and wide.h
* Added GCC and MSVC specific intrinsics for Mask<int WidthT>::count() and ::first_on()
Moved documentation for Wide, Masked, and other classes in wide.h from top overview to be near class definition.
Fixed some typos.
* [cmake] fixed a compilation error when generating bitcode for Cuda files.
Basically the problem is that Boost::thread does some compile time checks that rely (when compiling with MSVC) on the runtime lib being set (/MD switch for instance) And the command that generates the bitcode doesn't specify this, and is not compatible with the switch (clang++ doesn't support MSVC switches, clang-cl is the one that does, but clang-cl doesn't support -O3 and other switches ...)
The easiest solution is to just disable the compile checks from Boost::threads by defined the macro used to guard the checks header.
* [cmake] silenced a ton of warnings about dllimport attributes being ignored on Windows, when generating bitcodes
* [cmake] fixed MSVC's command line warning about overwriting a /W argument.
* better fix for the dllimport attribute not being supported when compiling for Cuda
- the OSL export macros now check for __CUDACC__
- added comments in the CMake to remember to restore the warnings when the 3rd parties' export macros are also fixed.
* clang-format fix in export.h
* Bug fix: optimizer interaction output param that also is userdata
There's a "pass-through parameter" idiom, where an output param gets
its initial value from an input, it gets modified by the shader, but
it's also an output.
If the modification is optimized away -- say, if it was originally
`theparam *= scale` and scale is 1 -- then it may be that you end up
with no ops in the shader that read from the parameter. If the
parameter was hooked up to userdata, then get_userdata will never be
called to retrieve the value (because it's usually lazy, only doing
the callback the first time it's needed), and so the output value of
parameter will be its parameter default value in the shader, NOT the
value of the interpolatd variable, which was never retrieved and
copied into it, and also that a shader consisting of *only* that
particular flavor of useparam is not marked as "does nothing" (thus
skipping any code generation at all).
This patch adjusts the logic somewhat so that if that output parameter
is connected downstream or is a renderer output, then we definitely
initialize it with a useparam op, even if it is otherwise apparently
not read from within the shader.
Added a new test for this specific case.
Also necessitated update to reference output of debug_uninit test,
only because the extra useparam op caused some of the instruction
numbering to change.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* testshade: allow arbitrary command line-specificed userdata
To ease testing, add `--userdata` command line option to testshade,
which constructs a ParamValueList that will be used to look up
"userdata" from the shader.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The version of Mask::first_on turned out not to have all the necessary
clauses fleshed out, breaking on Windows. So I refactored to get it
right.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The specific combination of Mac + Python 3.8/3.9 + the lastest
pybind11 release causes a crash in the testsuite/python-oslquery
test. It works on other platforms and with older versions of pybind11.
I've tried to understand exactly what's going wrong, and have come up
empty and can't justify spending more time on it. It's suspiciously
similar to several recently "fixed" pybind11+python interaction bugs,
I suspect there are a couple edge cases they missed and that it will
be fixed eventually in a future pybind11 release (or possibly a future
python patch, there does seem to be a problem that pybind11 uncovered
in the cpython code).
In the mean time, a slight rewrite of the loop avoids the problem for
the purposes of our CI tests.
Also a few adjustments to switch back to python 3.9 after some recent
Homebrew fixes that had made us revert back to 3.8 for a while.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Added 64bit specific popcount call for GCC/Clang in Mask<int WidthT>::count()
* Introduce template<int WidthT> struct BatchedShaderGlobals
to store upto WidthT instances of ShaderGlobal data. The data for a batch is broken into underlying uniform and varying data structures. Uniform contains a fixed set of shader global values that are shared between all instances. Varying has Block<DataT, WidthT> members so each instance can hold a different value. The Block data structure is specialized to hold the underlying data type in a SIMD friendly Structure of Arrays (SOA) data layout but provides an array subscript interface to a proxy that can import/export the DataT in its orignal data layout.
Introduce template<int WidthT> class ShadingSystem::BatchedExecutor as an interface to execute batches of shader globals through a shader network. BatchedExecutor provides similar execute methods that accept a BatchedShaderGlobals and batchSize instead of individual ShaderGlobal(s). This is just the interface, batched backend implemention to follow in subsequent pull requests.
Introduce template<typename DataT, int WidthT> Wide to hold a reference to Block<DataT, WidthT> and provide access to the underlying SOA data layout. Wide can handle array types and const correctness. The Wide adapter is lightweight and can be copied by value, used as function parameter.
Once batched execution finishes, the resulting symbol address can be passed to a Wide<DataT, Width> to access the resulting data in a type safe manner.
Updated testshade with command line option "--batched" causing batched execution of 16 or 8 instances through the shader network and to extract the results. A later pull request will add batched interface to renderer services.
* Add new file src/include/OSL/batched_shaderglobals.h
* Disambiguate fabs call to fabs(double) so it doesn't try OSL::fabs which has no overload for double
* clang-format of batched_shaderglobals.h
* Add BatchedRenderServices, Mask<int WidthT> to represent active lanes during SIMD execution, BatchedTextureOptions, Masked<typename DataT, int WidthT>
* Removed unnecessary .c_str(), changed call to ::fabs to std::fabs
* Removed BatchedRendererServicesBases in favor of simple virtual functions is_overriden_*().
Improved some comments for BatchedRendererServices.h
Workaround some QT macro issues.
* Add BatchedRendererServices implimentation which mainly asserts that its methods are never directly called.
Fixed up issues with OSL_USING_DATA_WIDTH(WIDTH) macro.
* Removed Dz Structure Of Arrays wrappers as OSL doesn't currently support Dz.
* clang-format of mask.h and wide.h
* Added GCC and MSVC specific intrinsics for Mask<int WidthT>::count() and ::first_on()
Moved documentation for Wide, Masked, and other classes in wide.h from top overview to be near class definition.
Fixed some typos.
Bad cut and paste made this all wrong. Fix.
Also rename the template's AlignmentT -> Alignment to help reduce
confusion because we usually use the T end to indicate a template
parameter that's a type.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
It can't run the tests, because there is no GPU on the GHA
instance. But at least we can tell if a PR will breake the build
itself. Later we will find a way to execute these via ASWF's AWS
instances that are for this purpose.
Includes the following changes:
* Download optix headers when needed. The ASWF docker container we're
running on already has Cuda.
* Use CUDA_TARGET_ARCH CMake variable instead of hard coding in
llvm_util.cpp.
* Bump cuda target default to sm_60.
* Make CI failure artifacts include CMakeCache.txt for inspection.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Change bleeding edge CI test to use LLVM 11.
A variety of changes were necessary to the code in LLVM_Util:
* `#if` guards for differing calls related to VectorType::get(),
CreateShuffleVector.
* constant-generating functions now return llvm::Constant* instead of
llvm::Value*.
* New wide_constant(width,value) functions for when you want a width other
than the default vector_width.
* Renamed llvmVectorGet -> llvm_vector_type to be more symmetric with
other methds we have.
* Simplified places that could have made calls to constant(),
constant_bool(), wide_constant(), llvm_vector_type(), and
call_function(), but instead did more complex llvm calls by hand.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
With just a few minor adjustments, we're able to eliminate these two
headers (and everything else they pull in) from every client of
oslexec.h. For the purposes of oslexec.h's declarations, all uses are
either pointers or references, so we only need the names
forward-declared, but we don't need the implementations revealed to
everybody including oslexec.h.
In particular, this hides the RendererServices and BatchedShaderGlobals
from a lot of translation units, should cut down on compile time and
reduce exposure of certain semi-internals to only the modules that
need them.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Introduce template<int WidthT> struct BatchedShaderGlobals
to store up to WidthT instances of ShaderGlobal data. The data for a batch is broken into underlying uniform and varying data structures. Uniform contains a fixed set of shader global values that are shared between all instances. Varying has Block<DataT, WidthT> members so each instance can hold a different value. The Block data structure is specialized to hold the underlying data type in a SIMD friendly Structure of Arrays (SOA) data layout but provides an array subscript interface to a proxy that can import/export the DataT in its orignal data layout.
Introduce template<int WidthT> class ShadingSystem::BatchedExecutor as an interface to execute batches of shader globals through a shader network. BatchedExecutor provides similar execute methods that accept a BatchedShaderGlobals and batchSize instead of individual ShaderGlobal(s). This is just the interface, batched backend implemention to follow in subsequent pull requests.
Introduce template<typename DataT, int WidthT> Wide to hold a reference to Block<DataT, WidthT> and provide access to the underlying SOA data layout. Wide can handle array types and const correctness. The Wide adapter is lightweight and can be copied by value, used as function parameter.
Once batched execution finishes, the resulting symbol address can be passed to a Wide<DataT, Width> to access the resulting data in a type safe manner.
Updated testshade with command line option "--batched" causing batched execution of 16 or 8 instances through the shader network and to extract the results. A later pull request will add batched interface to renderer services.
* Add new file src/include/OSL/batched_shaderglobals.h
* Disambiguate fabs call to fabs(double) so it doesn't try OSL::fabs which has no overload for double
* clang-format of batched_shaderglobals.h
* Removed unnecessary .c_str(), changed call to ::fabs to std::fabs
* Simplify implementation of TypeSpec::is_triple, more efficient.
* Add TypeSpec::is_triple_based(), analogous to all the other
is_X/is_X_based.
* Bug fix: Symbol::get_string()'s assertion was checking whether it was
a string, but should have checked if it was string_based.
* Add Symbol::get_vec3(), analogous to get_int/float/string.
* Add Symbol::coerce_float and coerce_vec3, like get but with some type
coercions.
* Add Symbol::dataptr() as synonym for data(). This is for "forward
compatibility", a future PR will make more widespread use of this.
* Lots and lots of places were error-prone and verbose casts are replaced
with the new methods, and some logic simplified.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Repeat of https://github.com/OpenImageIO/oiio/pull/2741
One of our CI matrix entries recently started failing due to changes
in somethign on GHCI's Ubuntu image. Fix by switching this matrix
to use the appropriate ASWF docker image.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Bring back the idea of a Findpugixml.cmake file because it turns out
that (for pugixml <= 1.10) the exported config files are not reliable.
* Fix CI scripts to build PugiXML (with a few different versions). We
previously were relying on the embedded version for all CI tests, which
masked problems associated with using an external PugiXML.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Added OSLEXECPUBLIC attributes to inner classes of LLVM_Util to fix
linker errors on MSVC
* do not include cxxabi.h on non GNUC platforms
* fixed windows linker issues
- Link with Version.lib (system lib) to prevent missing symbols in
ClangDriver.lib
- Do not define OSL_EXPORTS for oslcomp/oslquery as they should import
(not export)
Co-authored-by: Dieter De Baets <dieter.baets@unity3d.com>
* OSO re-entrant parser.
* OSL re-entrant parser.
* Formatting
* Use homebrew flex & bison for OSX
* Remove old oslparse()/osoparse() extern declarations.
* Set minimum versions of Flex and Bison
Add logic to flexbison.cmake to prefer the Homebrew version if it's
found (and if no BISON_ROOT is set) because the system install in
/usr/bin is too old to properly build with the new reentrant parser
directives.
Co-authored-by: rzulak <rzulak@imageworks.com>
This is good for optimization, allowing "de-virtualization" of function
calls in cases when the compiler can be sure that there can be no further
subclassing.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This patch drops support for OIIO 2.0, making 2.1 the minimum as we
move forward on the way to OSL 1.12 (or 2.0?). The current public
supported release of OIIO is 2.2, so this means that we are still
supporting both the current and the previous year's OIIO major
releases, as is our usual policy (that is, for an OSL that won't
be an official release until some time in 2021).
This patch does not change the supported OIIO releases of prior OSL
releases. In particular, the current release family, OSL 1.11, will
continue to support OIIO 2.0. We never change minimum dependencies
during the course of a release series.
One way this becomes a big simplification is that OIIO 2.1 started
exporting cmake configurations directly, and thus we no longer need
our copy of FindOpenImageIO.cmake, and can jettison some old
conventions that it used for older OIIO releases.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Move things around -- add new SS::getattribute queries for the
dependencies, SIMD build flags and runtime capabilities, and
Cuda/OptiX versions. Then use these queries as the implementation for
reporting them in the statistics output.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
testshade --help or --runstats now print additional info about the
dependency versions, hardware capabilities enabled at build time, and
hardware capabilities detected at runtime. This should help debugging
of user experiences as an easy way to find out what OSL thinks it's
working with in terms of its environment.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Extra char prevented the "release type" from being amended onto the
version string.
Also, change the OSL_INTRO_STRING to use the preferred convention
of "Open Shading Language" rather than "OpenShadingLanguage".
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Work by Alex Wells.
When generating the code to copy one layer's outputs to a downstream
layer's inputs, we can skip the copy entirely if we already know that
the downstream layer will not be run.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
Work of Alex Wells.
Like we do already for "for" loops, when generating extra ops to test for
uninitialized values, "do-while" loops also need to skip testing of their
condition variable (which we know has not been written yet, under most
circumstances).
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
Work of Alex Wells, Intel.
Added custom optimization passes to improve AVX/AVX2 code generation
which are automatically added when targeting the host:
* PrePromoteLogicalOpsOnBitMasks
* PreventBitMasksFromBeingLiveinsToBasicBlocks
Added support for experimental llvm_optimize: 10,11,12,13. Optlevels
10, 11, 12, 13 explicitly create optimization passes. They are
stripped down versions of clang's -O0, -O1, -O2, -O3. They try to
provide similar results with improved optimization time by removing
some expensive passes that were repeated many times and omitting other
passes that are not applicable or not profitable. Useful for
debugging, optlevel 10 adds next to no additional passes.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Switch from old cmake macros that were marked obsolete in cmake 3.12 to
the new FindPython.cmake module.
Instead of defaulting to python 2.7, default to whatever is found.
But user can still set PYTHON_VERSION variable to request a specific
one in case there are multiple.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Luke Emrose noticed that for texture calls that retrieved alpha, if
derivatives were later taken *only* of the alpha but not the main
color result of the texture call, those derivs would always end up 0,
the texture call not noticing that it needed to call the version of
texture that returns derivatives.
This patch provides a fix. It was a simple matter of it testing only
whether main result derivs were needed, when determining which variety
of texture must be called.
Curiously, this was already done correctly for texture3d!
Added testsuite/texture-alpha-derivs to test this specific situation.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Work by Alex Wells.
This will be used extensively by the SIMD batched shading.
* Added support for masked operations by managing a stack of masks
that were created by conditionals or control flow. Detailed
explanation of the Masking Strategy in llvm_util.h.
* Added support for logically masked functions (handling of masked
return from inside OSL functions)
* Added support for masked loops (handling of masked break and or
masked continue).
* Added support for masked exit (handling of masked exit from entire
shader).
* Added support for management of when masking is required with a
ScopedMasking object to allow nesting during code gen.
* op_store (and other store variations) respect the current masking
requirement and current top of the mask stack (not just the top
entry, but a combined view of the entire mask stack).
* Added support for results of conditional operations to remain
boolean. Most masking operations are expecting vector of bool
values, not a vector of integers., e.g. when vector width is 16, for
scalar or <16 x i1> for wide. When a conditional result is used by
another operation a numerical value it is automatically converted to
an integer.
* Added `llvm::Type *type_native_mask() const;` because different
ISA's may have different representations of a mask from llvm's
vector of bits that comparison operations emit. And we need varying
boolean symbols to have the correct data type (size) on the stack
and in data structures.
* Added gather and scatter operations.
* Added helpers to convert from bool<->int, native_mask<->llvm_mask
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Periodic comparison of analogous build scripts in OIIO & OSL and
synchronize to ensure that my current most favored idioms are in both.
Split checked_find_package into separate file for easier
synchronization between projects.
Bumped OpenEXR versions to latest patches.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Rename LICENSE -> LICENSE.md for uniformity with other projects
and our own documentation.
Rename LICENSE-THIRD-PARTY.md -> THIRD-PARTY.md because it was
confusing GitHub's autodetection of the license (this makes many
tools, including third part such as the ASWF's landscape fail to
correctly categorize this project as BSD-3-Clause).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
There's a bug in the current Homebrew llvm 10.0.1 where
it fails to link because of missing libxml2.tbd. It's fixed
upstream in LLVM, but not yet fixed in Homebrew. Hopefully
a subsequent 10.0.x or 11.x will fix it, but for now, just
fall back to llvm@9.
Also issue a warning in the build scripts if this condition
is detected, so if a user encounters it on their system, they
have a hint about how to address it.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This new file contains the hashes of commits whose only function was
to reformat large amounts of code.
Ordinarily, this would make for confusing 'git blame' output, by
making old code written by author A appear to have a new modification
date and authorship by B, when in fact we know that B just did some
reformatting. This can make it especially hard to discern code history
after after commits that establish the use of "clang-format".
A new git feature (requiring git >= 2.23) allows pointing to a file
containing hashes of commits to be ignored by git blame:
git blame --ignore-revs-file .git-blame-ignore-revs
To avoid having to remember this all the time, consider adding this to
your git config:
git config blame.ignoreRevsFile .git-blame-ignore-revs
Note that git supports this feature but does not dictate the name of
the file. I'm just following the convention I'm seeing pop up in other
projects of calling it ".git-blame-ignore-revs".
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We used to disable fmt and fall back to tinyformat, when using Cuda.
The new OIIO 2.2 removes tinyformat and the OIIO_USE_FMT preprocessor
symbol. So make sure that we also tell fmt to avoid using the int128
type, not supported by Cuda. That seems to be enough to let it use
the rest of fmt without trouble.
Work by Alex Wells.
Separate OSL optimization from JIT by adding optional parameter
`bool do_jit = true` to all optimize_group* methods. This will allow
optimization without JIT'ing for the host, or optimizing once but
JIT'ing for multiple back ends. It will be necessary if one will be
JIT'ing only for batched execution, we would not want to pay the cost
of JIT'ing a scalar version as well (but still could if we wanted both
scalar and batched versions simultaneously).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
description:Generate or update release notes for a patch, minor, or major release, or just to update main. Run git-cliff, organize and edit the output per project conventions, and insert into CHANGES.md.
argument-hint:[patch|minor|major|main] [prev-tag]
---
Generate release notes for an OSL release.
Arguments: `$ARGUMENTS`
- First argument (optional): release type — `patch` (default), `minor`, `major`, or `main`.
- Second argument (optional): previous release tag (e.g. `v3.1.2.0`). If omitted, find the most recent tag automatically. If the release type is `main`, instead of a tag, look for the last commit at which the CHANGES.md file was updated.
## Steps
1.**Determine the previous tag** if not provided:
```
git describe --tags --abbrev=0
```
or look at recent tags: `git tag --sort=-version:refname | head -10`.
However, if the "release type" is `main`, instead of a tag, just find
### New minimum dependencies, toolchain, and compatibility changes:
### ✏️ OSL Language, standard library, and oslc compiler (for shader writers):
### ⛰️ API changes and new ShadingSystem features (for renderer writers):
### ☀️ testshade/testrender/osltoy improvements
### 🐛/🔧 Internals: fixes, improvements, and developer concerns
### 🏗 Build/test/CI and platform ports
* CMake build system and scripts:
* Dependency and platform support:
* Testing and Continuous integration (CI) systems:
### 📚 Notable documentation changes:
### 🏢 Project Administration
### 🤝 Contributors
```
Note that the section outline may already be present, in which case you
only need to fit items into the existing category outline.
Rules:
- Group commits into sections; within each section, cluster related items together.
- When needed, expand terse one-liners into enough prose that users understand what changed and why it matters.
- For `feat:` commits, make sure the feature is explained sufficiently — don't just copy the commit subject.
- For `api:` or `api!:` commits, clearly call out what changed in the public API.
- Include PR links and author attribution for every entry.
- The notes should "tell the story" of the release, not just be a dump of commit subjects.
- We aim to make major/minor releases approximately in October 1 of each year. If the anticipted release date is already in the file, don't change it. If it is not present, ask for confirmation of the planned release date.
### For updating release notes in main:
Rules:
- Generally, follow the rules for "major/minor releases", except as noted in other items below.
- Don't apply a new skeleton of category listings unless they are not present for the upcoming release.
5. **Insert the formatted notes** into `CHANGES.md` in the appropriate place (as detailed below). Leave the existing content intact.
- When updating `main` or preparing a `major` or `minor` release, insert the updates in the top section for the upcoming major/minor release. Insert a new set of category sections only if it's not already present.
- When doing a `patch` release, insert the changes immediately above the last patch release of that branch, so that CHANGES.md lists releases in descending numerical (version) order.
- When porting a set of release notes from a release branch into main, or from an older (obsolete) release branch to the current release branch, insert it into the right place to maintain overall descending order.
6. **Double check that the notes are adequately descriptive.** (See more
detailed description of this step below.)
7. **Forward-port release notes from release branches if needed**
8. **Show a summary** of what was inserted and ask the user to review before finalizing.
## Forward-porting release notes
Release notes may have been generated independently in main, and release
branches. When updating one branch, we ensure that any changes from older
branches have been incorporated.
- When preparing `patch` release notes, check the CHANGES.md file in the
"dev-X.(Y-1)" branch for the previous minor release family to identify any
X.(Y-1).Z patch release notes that are not reflected in the current release
notes that we are updating.
- When updating `main` or doing a `major` or `minor` release, check the
CHANGES.md file in both the "dev-X.(Y-1)" and "dev-X.(Y-2)" branches.
Check whichever is newer of the local and remote copy of that branch, since
the local one may have had release notes or its version in CMakeLists.txt
updated, but not yet pushed to GitHub.
- If any patch releases are present in the older dev branches checked, insert
the release notes for those patch releases into the right positions in the
current release notes that we are updating.
- If a change is forward-ported in this manner and the same PR is an update in
the current set of changes we are updating as our main task, document that
in the current set of notes using the following convention: in the line of
the notes, the explanation of the version where it appears should reflect
the first version of all branches where it appeared, for example, `(3.2.0.0,
3.1.3.0, 3.0.8.0)` to indicate that the patch was added to each of those
versions. The versions should be listed in descending order.
## Useful abbreviations for category labels
| Abbrev | Meaning |
|--------|---------|
| build | CMake/build system |
| deps | Changes to accommodate dependency or toolchain changes |
| ci | CI/GitHub Actions |
| docs | documentation |
| int | internal/refactor |
| test | testsuite or unit tests |
| HEADER.h | Developer utilities in a public header file |
## Combining PRs into single entries
To be more concise and easier to read, within a release's notes, related
PRs/commits can be combined into a single bullet-point line, which would look
YOU MAY DELETE ALL OF THIS IF YOU ALREADY HAVE A DESCRIPTIVE COMMIT MESSAGE!
This is just a template and set of reminders about what constitutes a good PR.
But please look over the checklist at the bottom.
If THIS TEXT is still in your PR description, we'll know you didn't read the
instructions!
### Description
<!-- Please provide a description of what this PR is meant to fix, and -->
<!-- how it works (if it's not going to be very clear from the code). -->
## Tests
### Tests
<!-- Did you / should you add a testsuite case (new test, or add to an -->
<!-- existing test) to verify that this works? -->
## Checklist:
### Checklist:
<!-- Put an 'x' in the boxes as you complete the checklist items -->
- [ ] I have read the [contribution guidelines](../CONTRIBUTING.md).
- [ ] I have previously submitted a [Contributor License Agreement](http://opensource.imageworks.com/cla/).
- [ ] I have updated the documentation, if applicable.
- [ ] I have ensured that the change is tested somewhere in the testsuite (adding new test cases if necessary).
- [ ]My code follows the prevailing code style of this project.
- [ ]**I have read the guidelines** on [contributions](https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/blob/main/CONTRIBUTING.md) and [code review procedures](https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/blob/main/docs/dev/CodeReview.md).
- [ ]**I have read the [Policy on AI Coding Assistants](https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/blob/main/docs/dev/AI_Policy.md)**
and if I used AI coding assistants, I have an `Assisted-by: TOOL / MODEL`
line in the pull request description above.
- [ ]**I have updated the documentation** if my PR adds features or changes
behavior.
- [ ]**I am sure that this PR's changes are tested in the testsuite**.
- [ ]**I have run and passed the testsuite in CI***before* submitting the
PR, by pushing the changes to my fork and seeing that the automated CI
passed there. (Exceptions: If most tests pass and you can't figure out why
the remaining ones fail, it's ok to submit the PR and ask for help. Or if
any failures seem entirely unrelated to your change; sometimes things break
on the GitHub runners.)
- [ ]**My code follows the prevailing code style of this project** and I
fixed any problems reported by the clang-format CI test.
- 3 blank lines between free functions/classes; 1 between class methods; max 1
blank line inside a function body
-`//` for regular comments; `///` Doxygen for public API
- If a file has a strong local style, imitate that.
## C++ guidelines
- Error handling: prefer `bool` returns, explicit status propagation, and
`errorfmt()`-style reporting — not exceptions — consistent with the
surrounding subsystem
- Preserve API, ABI, and behavior compatibility unless the task explicitly
requires a break
- For hot paths: no hidden allocation in inner loops or parallel regions;
precompute outside hot paths
## OIIO utility types (prefer over raw C++ equivalents)
The [OpenImageIO](http://github.com/AcademySoftwareFoundation/OpenImageIO) (OIIO) project is a key dependency of OSL and the two projects are co-developed. Many utilities shared by both projects live in OpenImageIO.
Prefer C++17 `std` and Imath types except as noted below. Avoid introducing
new third-party dependencies without a strong reason. Rely liberally on these
classes and headers from OIIO:
-`OIIO::TypeDesc` - light-weight description of simple types (aliased
as `OSL::TypeDesc`)
-`OIIO::ustring` and `OIIO::ustringhash` - light-weight unique string heavily
used in OSL. (Aliased as `OSL::ustring` and `OSL::ustringhash`).
@@ -4,27 +4,28 @@ This is a checklist for TSC's to review as part of the intake process. The TSC s
- Existing Project Governance
- [X] Project License exists ( [LICENSE.md](../LICENSE.md) ) and is OSI-approved
- [X] Any third-party components/dependencies included are listed along with thier licenses ( [THIRD_PARTY.md](../THIRD_PARTY.md) )
- [X] Governamce defined, outlining community roles and how decsions are made ( [GOVERNANCE.md](../GOVERNANCE.md] )
- Closed by https://github.com/imageworks/OpenShadingLanguage/pull/1137
- [X] Any third-party components/dependencies included are listed along with their licenses ( [THIRD_PARTY.md](../THIRD_PARTY.md) )
- [X] Governance defined, outlining community roles and how decisions are made ( [GOVERNANCE.md](../GOVERNANCE.md) )
- Closed by https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/pull/1137
- [X] Contribution Policy defined ( [CONTRIBUTING.md](../CONTRIBUTING.md) )
- Closed by https://github.com/imageworks/OpenShadingLanguage/pull/1137
- Closed by https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/pull/1137
- [X] Code of Conduct defined ( existing or pull from [ASWF Sample Project](https://github.com/AcademySoftwareFoundation/aswf-sample-project/blob/master/CODE_OF_CONDUCT.md) )
- Closed by https://github.com/imageworks/OpenShadingLanguage/pull/1137
- [] Release methodology defined ( [release.md](process/release.md) )
- New Project Goverance
- Closed by https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/pull/1137
- [X] Release methodology defined ( [RELEASING.md](../docs/RELEASING.md) )
- Closed by https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/pull/1572
- New Project Governance
- [X] TSC members identified
- [X] First TSC meeting held
- [X] TSC meeting cadence set and added to project calendar
- [] CLA Approved ( if used ) ( [CCLA](ccla.md) and [ICLA](icla.md) )
- [X] CLA Approved ( if used ) ( [CCLA](CLA-corporate.md) and [ICLA](CLA-individual.md) )
- Project charter ( [charter.md](charter.md) )
- [X] Approved by TSC
- Closed by https://github.com/imageworks/OpenShadingLanguage/pull/1137
Download the appropriate CLA from the links above, print, sign, and rescan
it (or just add a digital signature directly), and email it to:
opensource@imageworks.com
Our CLA's are based on those used by Apache and many other open source
projects.
**After the repo is moved to github.com/AcademySoftwareFoundation**
OSL uses EasyCLA for managing CLAs, which automatically checks to ensure
CLAs are signed by a contributor before a commit can be merged.
* If you are an individual writing the code on your own time and
you're SURE you are the sole owner of any intellectual property you
contribute, you can [sign the CLA as an individual contributor](https://github.com/communitybridge/easycla/blob/master/contributors/sign-a-cla-as-an-individual-contributor-to-github.md).
* If you are writing the code as part of your job, or if there is any
The OSL project grows and thrives from assistance from Contributors.
Contributors include anyone in the community who contributes code,
documentation, or other technical artifacts that have been incorporated into
the projects repository.
the project repository.
Anyone can be a Contributor. You need no formal approval from the project,
beyond the legal forms.
### How to Become a Contributor
* Review the coding standards to ensure your contribution is in line
with the project's coding and styling guidelines.
* Review the coding standards and [contributing guidelines](CONTRIBUTING.md)
to ensure your contribution is in line with the project's coding and styling
guidelines.
* Sign the Individual CLA, or if you are employed by an organization that
might have any claim to IP you create, have your organization sign the
Corporate CLA.
* Submit your code as a PR with the appropriate DCO sign-off.
* Submit your code as a PR with the appropriate DCO sign-off on each commit.
The easiest way to do this is to ensure that you commit your code with `git
commit -s`.
## Committers
@@ -75,9 +78,10 @@ request that the TSC restore them to active Committer status.
## Technical Steering Committee
The Technical Steering Committee (TSC) has final authority over this
project. As defined in the project charter, TSC responsibilities include,
but are not limited to:
The Technical Steering Committee (TSC) has final authority over this project.
As defined in the project [technical charter](https://github.com/AcademySoftwareFoundation/foundation/blob/main/project_charters/open-shading-language_charter.md),
in addition to committer activities, TSC responsibilities also include, but
are not limited to:
* Coordinating technical direction of the Project.
@@ -125,20 +129,21 @@ represents the project at all ASWF TAC meetings.
### TSC Members
(Proposed list, subject to revision, some are placeholders while their
organizations decide who should be the permanent representative)
Current voting members of the TSC are:
* **Chair and TAC repreentative**: Chris Kulla - Sony Pictures Imageworks
* **Chief Architect**: Larry Gritz, Sony Pictures Imageworks
* **Chair and TAC representative**: Chris Kulla - Epic Games
* **Chief Architect**: Larry Gritz - Sony Pictures Imageworks
* Luke Emrose - Animal Logic
* Eric Enderton - NVIDIA
* Stephen Friedman - Pixar
*Derek Haase - DNEG
*Adrien Herubel - Autodesk
* Lee Kerley - Sony Pictures Imageworks
*Nicolas Guiard - Isotropix
*Thiago Ize - Autodesk
* Lee Kerley - Apple
* Mitch Prater - Laika
* Alexey Smolenchuk - DNEG
* Brecht Van Lommel - Blender
* Alex Wells - Intel
### TSC Nomination and Succession
@@ -178,6 +183,9 @@ her absence. Meetings may also be streamed online where appropriate;
connection details will be posted to the osl-dev mail list in advance of the
scheduled meeting.
The biweekly TSC meetings can be found on the [ASWF Calendar](https://calendar.openshadinglanguage.org),
just click on the OSL TSC meeting entries to get the Zoom link.
Items are added to the TSC agenda which are considered contentious or are
modifications of governance, contribution policy, TSC membership, or release
process, in addition to topics involving the high-level technical direction
@@ -201,9 +209,11 @@ The TSC may invite additional persons to participate in a non-voting
capacity.
The meeting chair is responsible for ensuring that minutes are taken and
archived in the project repository.
archived.
Due to the challenges of scheduling a global meeting with participants in
several time zones, the TSC will seek to resolve as many agenda items as
possible outside of meetings on the public mailing list.
possible outside of meetings on the public mailing list or through GitHub
issues, discussions, and pull requests.
Meeting notes (usually very brief) can be found at: https://docs.google.com/document/d/1yf0bG6eoE2EvKZBNZX3nskdTvu99ADTDTNOknCDJd1I
[](https://www.youtube.com/watch?v=nmwMz1YnGPA "Open Shading Language Reel 2020")
which use RenderMan<sup>®</sup> in the examples but which are primarily about writing OSL shaders.
There is also the [Shader Writing in Open Shading Language book](https://www.routledge.com/Shader-Writing-in-Open-Shading-Language-with-RenderManr-Examples/Prater/p/book/9781032421100).
This covers all the material from the Siggraph course but in far greater detail. It also includes much more information about how OSL operates and covers a much broader range of shader examples.
Contact & reporting problems
----------------------------
Simple "how do I...", "I'm having trouble", or "is this a bug" questions are
best asked on the [osl-dev developer mail
list](https://lists.aswf.io/g/osl-dev).
That's where the most people will see it and potentially be able to answer
your question quickly (moreso than a GH "issue").
Bugs, build problems, and discovered vulnerabilities that you are relatively
certain is a legit problem in the code, and for which you can give clear
instructions for how to reproduce, should be [reported as
| main | :white_check_mark: :construction: ALL fixes immediately, but this is a branch under development with a frequently unstable ABI and occasionally unstable API. |
| 1.15.x | :white_check_mark: All fixes that can be backported without breaking ABI compatibility. New tagged releases monthly. |
| 1.14.x | :warning: Only the most critical fixes, only if they can be easily backported. |
| <= 1.13.x | :x: No longer receiving patches of any kind. |
## Reporting a Vulnerability
If you think you've found a potential vulnerability in OSL, please
report it to the maintainers. Include detailed steps to reproduce the issue,
and any other information that could aid an investigation.
The best way to report a vulnerability is to file a GitHub [security
The OSL Documentation Project is a work in progress.
The plan is to adopt the OCIO Documentation Project's methodology for maintaining and building documentation of various kinds as it becomes the blueprint for the ASWF.
There is much work to be done here.
Of particular interest are these topics. Please consider adding any information you might have about any of these to the repo:
* Any material at all for people learning to write OSL shaders (beyond the formal language spec, which is not intended to be a tutorial or cookbook for writing shaders).
* Example .osl code and/or .h files that are:
1. Generally useful, but do not warrant inclusion into the OSL source itself.
2. Tutorial in nature.
* Robust installation or build instructions for Windows and macOS.
* Guide for how to integrate OSL into an application such as a renderer.
If you have some relevant content and you'd like to discuss its inclusion, please use the OSL Documentation discussion:
## **Building, Using and Debugging OSL in Windows**
---
#### Copyright Contributors to the Open Shading Language project.
#### SPDX-License-Identifier: CC-BY-4.0
---
### **Contents**
1. Introduction
- Experimental
- My Setup
- Inspiration
2. Preparing Environment
- Cleaning Up the PATH
- Set-Up needed Environment Variables
- Set Flags based on your needs
3. Building (Release, Debug)
- Running the code based on your config
- Be Patient
- Errors
4. Running
5. Development in Windows
---
### **1. Introduction**
This document is prepared for those who have problem in building OSL in Windows OS, this is completely custom setup. And it is developing to make it compatible with all situations.
(`Disclaimer:` This Process is completely Experimental and it worked on my system, maybe you get error with changing some configs, during this process).
At a first glance it seems to be complicated, but it is very easy to do, bare with me.
So it would be great to share our experiences to reach to best way of building in Windows (and of course other Operating Systems) to save our time and energy to focus on OSL development and OSL Shader writing.
### **1.1 Experimental**
After a lot of efforts on manual building OSL on Windows, Finally I found a clean way to do that, I explained every steps I did, I hope it works for you too. This method is stricktly related to Environment Variables you create and Flags you set to the command and need patient and hard disk.
I've built OSL with minimal needed packages, for extra packages, they should be set as flag (**script is under development**)
### **1.2 My Setup**
This is my setup I have on my PC for building OSL, so you need to install some software before start.
**OpenShadingLanguage:** cloned from latest master branch
**LLVM:** 11.0.0 C++ is set on C++14
**CLANG:** 11.0.0 C++ is set on C++14
**WinFlexBison:** 2.5.22
**Zlib:** 1.2.11
**JPEG:** 2.0.5
**TIFF:** 4.1.0
**PNG:** 1.6.35
**PyBind11:** 2.6.0
**LibRaw:** 0.20.2
**OpenImageIO:** 2.1.20
**OpenColorIO:** 1.1.0
**OpenEXR/**IlmBase: 2.5.3
**GLUT:** 3.2.1 freeglut on windows (this is needed for partio) (unfortunately Partio didn't build, it has a problem) (I’ve created a PR for it in Disney github repo)
**Disney Partio:** latest from master (zip file) (Skip this for now) (I didn't build OSL with Partio)
**PugiXML:** 1.10
---
### **1.3 Inspiration**
After lots of manual building dependencies one by one and fix errors and things I got inspired by `PIXAR USD INSTALLATION METHOD` and borrowed its structure to build OSL.
so I studied its python build script and found we can use same logic for installing OSL, fortunately after changing and adding OSL needed packages and some custom setup it worked fine
It still needs some other works to make it complete for all dependencies (optional).
We will keep Pixar name on it as inspiration code (or we can keep their copyright)
You can Run the installer from: `yourpath/OpenShadingLanguage/docs/build_install/windows/build_osl.bat`
---
### **21. Preparing Environment**
There are bunch od prerequisites for building OSL that I explained in following sections.
### **2.1 Cleaning Up the PATH**
Be sure you don't have any custom item added to PATH in your global Environment Variables of Windows
You can do it:
`Start Menu > Edit the system environmment variables > Environment Variables > User Variables/System Variables`
so you should not have any path that refer to previous osl installation folders
Recommendation: remove added path for cmake, git, python, Qt, llvm
it should be clean now.
### **2.2 Set-Up needed Environment Variables**
For running batch scripts without any problem I recommend to create some base environment variables.
**Needed Environment Variables: (Key/Value based on my installation folder)**
Open `docs/build_install/windows/osl_env_vars_setup.bat` in your editor and edit paths' values based on your installation folders.
### **2.3 Set Flags based on your needs**
The main build launcher is here `docs/build_install/windows/build_osl.bat`
For now we can not assign pre-installed llvm so we have to build all together at once and put all of them in one directory.
I am working on it to see how we can use external llvm installation without error.
**Table of flags we can assign to the command**
| Flag | State | Can Use |
| ------------- | --------------- | ------- |
| --osl | Implemented | Yes |
| --python | Implemented | Yes |
| --zlib | Implemented | Yes |
| --llvm | Implemented | Yes |
| --clang | Implemented | Yes |
| --pugixml | Implemented | Yes |
| --openexr | Implemented | Yes |
| --tiff | Implemented | Yes |
| --jpeg | Implemented | Yes |
| --png | Implemented | Yes |
| --flex | Implemented | Yes |
| --bison | Implemented | Yes |
| --opencolorio | Implemented | Yes |
| --openimageio | Implemented | Yes |
| --libraw | Implemented | Yes |
| --pybind11 | Implemented | Yes |
| --Ptex | Implemented | No |
| --openvdb | Implemented | No |
| --partio | Implemented | No |
| --ffmpeg | Not-Implemented | No |
| --field3d | Not-Implemented | No |
| --opencv | Not-Implemented | No |
| --gif | Not-Implemented | No |
| --heif | Not-Implemented | No |
| --squish | Not-Implemented | No |
| --dcmtk | Not-Implemented | No |
| --webp | Not-Implemented | No |
---
### **3. Building (Release, Debug)**
We Prepared some batch files to do it for you automatically.
Based on Environment Variables you set in previous sections you can just call them and wait for it to be built and installed.
**My Method:**
- Open Powershell
- Go to: yourPath/OpenShadingLanguage/docs/build_install/windows
- Run build_osl.bat release
### **3.1 Running the code based on your config**
Based on the config (`Release` or `Debug`) that you send for installer it will create directory in `BASE_LOCATION` for you
`BASE_LOCATION/osl_release` for Release mode
`BASE_LOCATION/osl_debug` for Debug mode
**IMPORTANT:** `Good to know you can not have Release and Debug files in one directory at the same time, it makes conflict and give you weired errors. So keep them separate.`
### **3.2 Be Patient**
You should be patient because the whole process (based on your PC config) takes time between 30 minutes to 1 hour
And it takes:
**Release:** `~8Gb` of your hard disk (after installation) (Download size is `~350Mb`)
**Debug:** `~90Gb` of your hard disk (after installation) (Download size is `~400Mb`)
so please set a location to `BASE_LOCATION` that have enough space.
### **3.4 Errors**
If you got any error in any step of building process, you can remove the related package's build directory from `BASE_LOCATION/build` and run the installer again.
If you got any error please report to the forum of OSL.
---
### **4. Running**
We Prepared some batch files to do it for you automatically. Based on Environment Variables you set in previous sections you can just call it.
**Recommendation:** use `Powershell`
**TIP:** If you add your osl installation folders (both release and debug) in global Windows Environment Variable, you will get error.
So you can add release path or debug path in your global environment variables. Or create and use batch scripts.
For adding path you should add these two paths to your global env variables:
• PATH : `<osl install dir>\bin`;
• PATH : `<osl install dir>\lib`;
For example, you can open osltoy in releae mode with running `osltoy_release_launcher.bat`
if you have question, feel free to ask.
---
### **5. Development in Windows**
After editing your code, when you want to see your changes you should build osl again. But this time, it is so faster than before, (if you keep build folders) So just need to rebuild it in release mode.
**Release mode**:
- open powershell
- cd OpenShadingLanguage repo
- cd build_script
- run build_osl.bat release
- run osl_release_sln.bat
- example: set osltoy as default project
- run it in release x64 mode
**Debug mode**:
- open powershell
- cd OpenShadingLanguage repo
- cd build_script
- run build_osl.bat debug
- run osl_debug_sln.bat
- example: set osltoy as default project
- run it in debug x64 mode
If you open Visual Studio solution (sln) from release or debug, you can build it from inside it after your code editing
I recommend using: (in **powershell**)
- **osl_release_sln.bat** or **osl_debug_sln.bat**
now you can use Visual Studio for debugging.
---
**TIP about OSLToy:**
If you load a shader in `osltoy` and when hit Recompile, it is crashed and closed, probabely your output color attribute name is not `Cout`
It is mandatory to have at least one `Cout` output Variable in any shader, Otherwise osltoy will crash.
That's it
TIP: this process is getting better, I will update the document soon
**Conclusion:**
For easy installation you just need to setup your Environment Variables in `yourPath/OpenShadingLanguage/docs/build_install/windows/osl_env_vars_setup.bat` based on your paths
It is a custom setup, for example all dependencies are built before osl, because I had lots of issues on using external installation (like llvm)
and I tried this method on llvm 7, 8 and 9, all with error, so my current build setup is on llvm 11 and C++14, it can be changed in code. (I am trying to make it easy to use for changing configs with arguments)
Please take a look at docs and let me know my mistakes.
This method worked fine for me for release, debug and debugging in visual studio.
I am trying to build OSL on `Linux` with same method. If it worked fine I will Update the build system.
Use of "AI coding assistants" is permitted on this project, with the following
guidelines and principles.
Summary of our core values:
- Human must always be in the loop, and is the responsible party for
the contents of a PR (including fully understanding and being able
to explain, defend, and modify it in response to review comments).
- Interact with the project and community yourself, not by agent.
- Disclose what tools you used and how. At a minimum, we require an
"Assisted-by: TOOL/MODEL" line in the commit comments and PR description.
- Don't waste maintainer's time with low quality PRs.
The long version:
### Human authorship and interaction with the project
**The human making a PR is considered the author** and is fully responsible
for the code they submit.
**No PRs should be submitted, reviewed/merged, or deployed without a human
fully in the loop.** You may use agents locally to help you write code (and
docs, tests, etc.), but you may not use agents to autonomously interact with
the project community. Submit your own PRs, write your own discussion
comments, respond to community members in your own words, approve or reject
code in review using your own brain.
Code autonomously submitted without an identifiable human author and
responsible party will be immediately closed, and the account associated with
the submission may be banned from future project participation.
**DO NOT "vibe code."** This term refers to a pure prompt-to-behavior cycle
where the human accepts un-inspected code based on the program behavior alone.
We don't want this. Code that must be maintained by the project should be
truly understood. Project code is ideally the result of discussion / pair
programming between the author and assistant, but even if the machine is doing
most of the work, all code should be approved and fully understood by the
author prior to PR submission.
### Professionalism and quality
**The usual high quality level of PRs should be maintained regardless of tools
used.** This includes not only code, but also the design, testing, PR
preparation, and the author's ability to explain and defend the code and
respond to questions and requests for changes. We never want to hear "but the
AI..." as an excuse for anything.
**PRs should always be [reviewed](CodeReview.md) and approved by someone other
than their author if at all possible.** This is especially true of code with
machine-generated components, because of the additional risk that the
submitter/operator may not fully understand what they did not write
themselves. This is part of the reason for AI tool disclosure -- to ensure
that another pair of eyes is on it. AI code review may be requested for
additional input, but the AI cannot give "approval", nor merge code itself.
Even when using a coding assistant to write the code, you are strongly
encouraged to write your own PR description, in your own words, if for no
other reason than to force yourself to understand the code well enough to
describe it to others.
### Disclosure
**Disclosure is required.** Patches that were written with the aid of AI
coding assistants must have, at a minimum, the following line in the commit
comment and PR description body:
Assisted-by: TOOL / MODEL
Ideally, to the extent practical, the PR description should also have a brief
description of how the tool was used, including the gist of key prompts or
summarizing the direction of the dialog. A full log is not necessary; a short
summary will do.
There are several reasons for this disclosure/documentation, even though it
will sometimes be inconvenient:
- Reproducibility: We want even automated work to be reproducible by others in
principle, just like a scientific paper.
- Education: We want to actively teach each other what tools, prompts, or
methods are successful at making high quality results for the project.
- Metrics: We would like the ability to look back and compare assisted vs
non-assisted code for things like defect rate (how often were new bugs
introduced or commits needed to be reverted), whether people are more likely
to write comprehensive tests when using assistants, etc.
- Compliance: Many key users and developers are in companies where they need
to disclose how AI tools were used in their work, and in other software that
they use. Let's make it easy for them.
- Insurance: Should future legal decisions radically change the IP status of
AI-generated code, or if particular models are implicated in copyright
violations, we want a way to find out which contributions might need to be
revisited in order to bring the project back into compliance.
### Intellectual Property
Regardless of how the code came to be -- from your head alone, from a friend,
from Stack Overflow, from a blog post, from reading other code bases, or from
a coding assistant -- we expect the terms of the
[DCO](https://developercertificate.org/) and CLA to apply, and for the author
to take reasonable care that code is not copied from a source with an
incompatible license.
You may find that your confidence about complying with the DCO+CLA depends on:
- If you provided a detailed specification and how much you edited or guided
the results yourself.
- Whether you are modifying or extending existing code versus writing large
amounts of entirely new code.
- The degree to which the generated code seems to fit into our structure and
idiomatic style, and therefore seems unlikely to be copied from elsewhere.
- Whether your tool has guardrails to prevent answers that are too similar to
existing code, for example as claimed by [Claude](https://privacy.claude.com/en/articles/10023638-why-am-i-receiving-an-output-blocked-by-content-filtering-policy-error) and [Copilot](https://docs.github.com/en/copilot/how-tos/manage-your-account/).
### Extractive submissions
**Maintainer time and attention are precious commodities**, and use of coding
assistants is not an excuse to submit poor PRs or to externalize costs onto
maintainers/reviewers (such as responsibility for understanding, testing, or
fixing PRs that the human author does not have a full understanding of).
Maintainers are free to take counter-measures against submitters of sub-par
PRs, or other violations of this policy, up to and including banning habitual
abusers from future participation in the project.
**We discourage use of AI tools to fix GitHub issues labeled as "good first
issue" or for "Dev Days" work.** Cultivating and educating new contributors is
important, and as such, we do not want people to swoop in and use automated
tools to trivially solve tasks that were curated specifically for somebody to
actually learn from. We expect each developer to implement at most one "good
first issue" per Dev Days occasion, and we encourage you to solve the problem
with your own brain to maximize the learning experience.
### Exceptions
This AI tool use policy is not meant to encompass cases such as:
- "Smart auto-complete", spell-checking, grammar checking, or other uses that
aren't really contributing substantively to authorship.
- Use of LLMs to explain code or learn about the codebase, answer basic
programming questions, help with background research, or audit the code
for bugs that are subsequently confirmed and fixed by people.
- Language translation for non-fluent English speakers or other accessibility
accommodations.
- Trivial or de-minimis fixes such as fixing a typo, obviously wrong variable
use, etc.
- Reviewing your own code for mistakes prior to submitting a PR (as long as it
isn't making the fixes for you).
### Creative contributions -- out of scope
This project repository contains code for tools, and their documentation. We
don't really directly host any artistically creative work here, so we have not
addressed additional policy issues that might come up in such a context.
If you are representing a different project that is using this document for
guidance or inspiration for your own policies, please be aware that you may
have needs that are not covered by these policies but deserve well-crafted
guidance.
### References and inspiration
Our policy has been informed and inspired by the following efforts in other communities:
- [LLVM AI Tool Policy](https://llvm.org/docs/AIToolPolicy.html)
- [Fedora Project policy on AI-assisted contributions](https://communityblog.fedoraproject.org/council-policy-proposal-policy-on-ai-assisted-contributions/)
- [Linux Foundation policy on Generative AI](https://www.linuxfoundation.org/legal/generative-ai)
- [Rust policy on rejecting burdensome PRs](https://github.com/rust-lang/compiler-team/issues/893)
- The METR paper [Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/)
- [GitHub Blog: Rethinking open source mentorship in the AI era](https://github.blog/open-source/maintainers/rethinking-open-source-mentorship-in-the-ai-era/)
For monthly patch releases, we don't have any formal alpha/beta/RC stages.
For the annual minor or major releases we have a staged release:
- `alpha` is everything after we have branched, in which we think all the big
new features have been added, and while we will make continued changes and
fixes, we are aiming for increased stability and to minimize additional
breaking changes prior to the release.
- `beta` should be about one month prior to the final release date. We create
a tag `v1.2.3.4-beta` and announce the beta (see below "Making the
release"). Once we tag a beta, we try very hard not to allow any further
changes that change the API or are not ABI-compatible (i.e., safe fixes
only).
- `RC` or release candidates around one week prior to the final release date.
This is a trial run, hopefully identical to the final release (except for
removing the RC label and making a new tag). The only changes allowed after
RC are fixes to the most critical bugs or build breaks that would be bad for
users to encounter. If critical problems are found with RC1, they should be
fixed and immediately re-tagged as RC2, etc. All other fixes should be
postponed until the next month's patch release.
### Making the release
A final release is made on the scheduled date, provided the latest RC has
survived for at least a few days of testing without finding any critical
problems. Don't make the release final until you are sure there are no
truly critical bugs or build breaks that users will encounter.
The following are the steps for making the release:
1. Double check the top-level CMakeLists.txt to ensure the right version
number and type of version designation (i.e.,
`PROJECT_VERSION_RELEASE_TYPE` should be set appropriately).
2. Edit CHANGES.md to reflect the correct date of the release and ensure it
includes any last-minute changes that were made during beta or release
candidate stages. Please see the section "Generating release notes" below.
3. Push it to **your** GitHub fork, make sure it passes CI.
4. Tag the release with a signed tag:
$ git tag -s v1.2.3.4 # add a signed tag
$ git tag -v v1.2.3.4 # verify the tag
Be sure that the tag also reflects if it's an alpha (v1.2.3.4-alpha),
beta (v1.2.3.4-beta), release candidate (v1.2.3.4-RC1), or developer
preview (v1.2.3.4-dev). If it's a final release, just use the plain
version number (v1.2.3.4).
Note that the tag signing requires you to have a GPG key set up and
associated with your GitHub account. Please see [GitHub's signed tagging instructions](https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-tags) for more information.
If you get errors messages "gpg: signing failed: Inappropriate ioctl for device", then the solution is to first:
$ export GPG_TTY=$(tty)
5. If this will now be the recommended stable release, move the `release`
branch marker to the same position.
6. Push it to GitHub: `git push aswf release --tags`
(This example assumes "aswf" is the name of the remote for the GitHub
@@ -73,7 +71,7 @@ from __future__ import print_function, absolute_import
###
### Only one main doc region will be output by the `docdeep` program,
### specified with the `-d` command line argument. Any other doc-regions
### will not be included in the documention output of that run.
### will not be included in the documentation output of that run.
###
### However, one doc-region may *include* the text of another doc-region
### as follows:
@@ -114,7 +112,7 @@ from __future__ import print_function, absolute_import
### For pre-code comments, the contiguous comment region associated with
### a declaration immediately precedes the declaration. Such a comment
### set has its first line start with `///>`. For post-code comments,
### the documentaiton comments follows the code declaration, and this is
### the documentation comments follows the code declaration, and this is
### designated by having its first line start with `///<`. I like to
### just remember that the `<` or `>` points in the direction of the code
### declaration that the comment applies to.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Larry Gritz
