Take into account whether a parameter can change due to interactivity
when comparing parameters for "mergeability".
This proposed patch fixes a corner case we were seeing during live rendering with ReParameterization.
We were experiencing a problem when ss->ReParameterize() wasn't successfully applying parameter changes for us.
In our situation, a shader's parameter, "A", was marked with lockgeom=0 (ie. "live" or "editable") and was set via ss->Parameter(). This parameter was connected (through a series of copies and network links) to another shader's parameter, "B", downstream whose initialized value (ie. the one specified in the .osl/.oso file) matched A's set value. When this happened, A's shading layer was getting marked as "mergeable" and deleted, and we wouldn't see any updates to A via "reparameterization" take effect.
Signed-off-by: Clifford Stein <cstein@imageworks.com>
This patch is a very simple part one of work to optimize the number of conditional run layer calls that OSL makes, which take the form:
if (!groupdata->run[parentlayer])
parent_layer([...]);
After examining our slowest JIT’ing shaders on gpu, @tgrant-nv showed us that these run layer calls were significantly bloating our ptx and slowing down their compilation. Many of them are unnecessary because static analysis can show we are guaranteed to have already run the layer in question.
For example, reading different fields from a connected struct like below will often generate multiple conditional run layer calls:
```
struct Pair {
float x;
float y;
};
shader example_shader(Pair p = {})
{
if ([…]) {
float xx = p.x;
float yy = p.y;
[…]
}
}
```
Since this is not really a performance issue on cpu (runtime overhead is just checking the conditional bit), OSL doesn’t make a huge effort to optimize away these redundant cases. Each useparam op will be careful not to insert more than one per layer, and code that is called unconditionally in the main section of the shader will also not have redundant checks, but there are plenty of examples like above that are not currently removed.
This patch takes a basic first step, adding metrics to track run layer call counts and removing duplicate run layer calls inside a basic block, covering cases like the example above.
On my set of production test scenes, this alone removes anywhere from 20-50% of total run layer calls, with our biggest shaders seeing the most improvement. Total ptx line reduction varies between 3-20%, and optix module compilation (jit time) is reduced between 1-15%.
Signed-off-by: Chris Hellmuth <chellmuth@gmail.com>
But note: OSL does not yet work with LLVM 16.
Because homebrew recently upgraded its default llvm to 16.0, there may
be users for whom builds now break. The solution is:
brew install llvm@14
cmake ... -DLLVM_ROOT=/usr/local/opt/llvm@14
Signed-off-by: Larry Gritz <lg@larrygritz.com>
For LLVM 15+, we need to disable opaque pointers. We did this for
regular bitcode generation in llvm_macros.cmake, but neglected to make
the corresponding change for copmiling to ptx in cuda_macros.cmake.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
LLVM 16 was just released. I'm pretty sure we won't build properly
against it because of the phase-out of typed pointers in LLVM IR, so
warn prominently so that anybody building against it who has problems
will know why.
Also, LLVM itself has bumped their toolchain requirements with the
release of 16. Technically, I'm not sure if that extends to projects
using the LLVM APIs, but warn anyway and hope for the best. If it
fails, at least the warning will supply an explanation.
At a later time, we'll need to upgrade the code on our side to have the
option of using opaque pointers so we can run with LLVM 16+.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
When generating calls to osl_bind_interpolated_param in llvm_instance.cpp, the userdata_has_derivs field is set to true if the parameter's derivatives are needed, but the code does not take into account that on gpu there is no space allocated for these derivatives.
This leads to an incorrect derivatives pointer getting passed into the renderer, which the renderer will use to write into unrelated groupdata fields.
For our renders, getting the correct image was worth the 1-2% inflation of groupdata struct size that we observed by simply re-enabling gpu userdata derivatives. If desired, adding an option to turn them on/off is straightforward, but I didn't want to preemptively bloat the code if it's not necessary.
Signed-off-by: Chris Hellmuth <chellmuth@gmail.com>
In particular, `${LLVM_ROOT}/bin` will end up being (incorrectly!)
`/bin` if LLVM_ROOT is empty. That is bad if some other llvm-config
happens to be in /bin.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The runtime optimization does a great job of constant folding, and
we'd pushed it there because, with instance parameter values and
shader network connections known, it can find so much more opportunity
to optimize than we could in oslc.
However, there is one pesky problem, which is that shader parameters
that are initialized to even simple expressions such as 3+1 end up
with "init ops", which although later constant folded by the time the
shader is JITed, make it impossible to know the value via OSLQuery.
So this patch just takes the simplest cases -- certain `int OP int`
and `float OP float` expressions involving literal constants, and
performs the operation as it's parsing the code. So
int val = 3 + 1;
actually just immediately is turned into `val = 4` instead of making
an "add".
To reiterate, the add would never have happened while executing the
shader -- at runtime in the renderer, when it's time to optimize and
JIT the shader, it would know it's a constant 4 value. This is
strictly about making oslc directly output an .oso file that knows
that parameter `val` has default value 4 instead of throwing up its
hands and saying "it's math code that will be evaluated later."
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Stop using the deprecated, soon to be removed, ubuntu-18.04 runners
and switch to 20.04.
* Bump the "latest" test openexr to 3.1.6
* Update INSTALL.md to document some things we're already testing and
are known to work.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We had turned it off in one function to preserve formatting, but
failed to turn it on at the end of the function. This led to several
small functions not being clang-formatted, now fixed.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
In particular, 1 prints a short version of the compile report (just
the fact that it happened and the time), >1 prints more verbose info.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Make sure LLVM_ROOT/bin (either as a cmake var or as an env var) end
up in the hints list when searching for llvm-config.
Without this, one of our Mac CI jobs was ending up with clang 14 and
llvm 15, and that was causing problems. The particular test was
supposed to be using 14 for both.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Apparently some varieties of fmod(triple,float) -- which were not
documented in the spec but were intended to work and did allow that
combination in oslc -- were not fully implemented in the runtime.
* Add all these tests to the arithmetic-reg test, to fully test all
possible argument type and unif/vary/const combinations once and for
all for fmod. Also add some more rudimentary testing to miscmath.
* Implement the missing bits and/or avoid the code paths that rely
on the unimplemented parts.
The implementation of fmod for funny -- both for batch and non-batch,
it's got one code path that just directly generates code and doesn't
need any functional implementation in llvm_ops at all, and a second
code path that uses the functional implementations to work around
(different, in each case!) what appears to be bugs or limitations in
long-ago LLVM versions we don't need any more. I'm trying to assess
whether those are still needed at all in the oldest versions of LLVM
we are using, and if not, I may amend even more simplification to this
whole thing.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We were getting an exception thrown on the OCIO side, because we were
passing 0 for the y stride (which, us humans know, should have been fine
since there was only one row, so the stride shouldn't have mattered).
Now pass the right stride even though we don't need to step from row
to row.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Ported from OIIO. This lets us not have to depend on the specific
definition in OIIO's fmath.h, and in particular not worry about
whether it's a new enough version of OIIO that the old bit_cast is now
marked as deprecated.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
It seems something in the latest builds of the aswftesting/ci-osl:2022
containers has changed, and explicitly running /usr/bin/yum (instead of
the broken /usr/local/bin/yum) fixes it for now. This was completely
breaking the icc & icx tests, which require yum to install the Intel
compilers.
Also, bump up to the -clang14 containers for those tests, just to
get to a more recent LLVM.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Fixed batched support for closures with array types.
Enabled testsuite/userdata-custom/BATCHED (follow up to PR1620/PR1621).
Added modified version of PR1625 (Fix Issues with Derivative of getattribute) from danieldresser-ie
changing default behavior of MaskedData<DataT>::assign_all_from_scalar(const void *) to zero out Dx Dy when DataT has derviatives.
Added testsuite/userdata-custom
Fixed bug in testshades handling of userdata with derivatives.
Added bool LLVM_Util::is_type_array(llvm::Type* type)
llvm::Type* LLVM_Util::element_type_of(llvm::Type* array_type)
Fixed icx compiler warning with src/liboslexec/osogram.y
Fixed unreferenced function warning in src/testrender/shading.cpp
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
To save build time, we had OSL's Mac CI tests use the homebrew
installation of openimageio. But... Homebrew recently switched its
default python to 3.11, which didn't match the versions we're testing
OSL against here and broke the python bindings, so need to make a few
adjustments.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Related to https://github.com/OpenImageIO/oiio/pull/3653
Kai Pastor (dg0yt) points out that the recent change I made both in
OIIO and OSL to make the version a cache variable has some unexpected
side effects that can lead to unexpeted behavior by inadvertently
retaining a version number upon recompiles if the cache itself was not
cleared, and this can happen unexpectedly even when the user isn't
trying to override the version.
This patch changes the behavior as follows:
You must use the special `-DOSL_VERSION_OVERRIDE=...` to override the
version.
* if you do so, you are playing with fire and get what you deserve
if you misuse it or change branches and don't remember to clear
the cache.
If you do a normal build (without trying to override the version),
* you will get the real version number because it IS NOT cached
* if you switch branches, you'll always get the right version number
* ...but switching branches without clearing the cache can still, as
always, have other subtle behavior differences versus doing a
truly fresh build from scratch.
So in short, it still allows a way to override the version if you
really want to, but if you are not trying to do that, you can no
longer get yourself inadvertently into unexpected trouble.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This is another fix I'm attempting without really understanding all this LLVM stuff.
I started by adding a simple test case that attempts to add two closure parameters, where one has not been set. This crashes or returns random values from uninitialized memory, matching a problem I ran into while trying to get batched mode working in Gaffer.
I investigated some, and found that add_closure_closure is assuming closure pointers are stored with the width of the current batch, but build_llvm_init is creating uniform closures. I checked with @sfriedmapixar, who confirmed that all the batched closure code is assuming wide closures, so it seems build_llvm_init is what needs fixing.
Adding a `sym.make_varying();` to build_llvm_init appears to fix the problem ... I really don't know enough about the overall structure to know if this is the right place to make this fix, or if it could have other consequences, but it's enough to get this test passing, so hopefully someone who does know the background can review it.
Signed-off-by: Daniel Dresser <danield@image-engine.com>
Lately, it has been experiencing intermittent availability problems
that causes our CI to fail while setting up dependencies.
What happens if we don't use the problematic ubuntu-toolchain-r at all?
Nothing, it seems. So eliminate it.
I think that once upon a time, it was necessary to add this repo to
get certain new packages? But maybe they are mainstream enough now
that it's not necessary to add this repo? In any case, it doesn't seem
to hurt anything, it's one less moving part that can't fail if it's
not there.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
When evaluating varying closure keyword parameters in batched execution,
the value of the first lane in the batch would be used for the whole
batch.
Signed-off-by: Daniel Dresser <danield@image-engine.com>
Reminder: phase 1 changed the RendererServices APIs to be based
completely on ustringhash for public method calls, not ustring.
This PR starts along the path of phase 2.
In oslconfig.h, I define a new type `ustringrep` which is the
in-memory/in-shader CPU-side representation of a ustring. Currently,
it's still equivalent to a ustring.
I also define a type called `ustring_pod`, which is the "plain old
data" representation inside a ustringrep (i.e., a const char* when
it's a ustring, and a uint64_t when it's a ustringhash). For reasons
not important, it's exceedingly hard to pass a true
ustring/ustringhash as parameters to or from LLVM-jitted code without
running afoul of LLVM's type checking, so it's just easier to pass the
underlying data type.
So the main work of phase 2 is to change everything related to shader
data from a ustring into a ustringrep, and things passed as const
char* into passing a ustring_pod. In some sense, it's just renaming
and type conversion plumbing. When that's done, phase 3 is to throw
the switch and change the definition of ustringrep to ustringhash (and
of course fix everything that's broken).
This PR does not complete phase 2, but takes a big bite out of it.
It's an arbitrary bite, not much rhyme or reason to which specific
parts I did so far. Just working on it piece by piece and felt that
this was a place where much has been done, everything still works, and
it's a reviewable chunk without getting too much out of control.
Areas I worked on are related to range_check, raytype, getattribute,
bind_interpolated_param, uninit_check, range_check, naninf_check,
getmessage/setmessage, some texture setup, pointcloud.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Changes:
* Scorecards and analysis -- allow new to cancel running old
* Scorecard only runs on main repo, no need to run on forks
* Nightly CI only runs on main repo, no need to run on forks
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This seems to have shifted just enough things around to make
clang-format want to re-jigger a few things.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Wow, this has been lurking for a while. Our Findpartio.cmake tries
looking for a partio exported config, but if not found, falls back
to a simple code to find it manually.
All was fine with the fallback code, which if found would set both
"partio_FOUND" and "PARTIO_FOUND". But the config file, if found --
only for the newer partio versions -- only set partio_FOUND. And in
that circumstance, we were not seting up the pointcloud tests!
Although the partio tests were running in CI (thankfully), I was not
running them locally on my machine with a new-ish partio.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Address -Winconsistent-missing-override clang warning
* Fix ptx definitions of OSL_CONSTANT_DATA arrays
LLVM 14 introduced two ptx regressions to our file-scoped static variables:
1) They gained the .visible directive
2) They stopped being optimized out when unused
This lead to duplicate symbol errors when building an OptiX pipeline with
multiple shaders.
We attempted a fix using anonymous namespaces in #1561, but kept the static
keyword, resulting in a partial solution.
Removing static generates ptx that matches LLVM 12 for the affected variable
definitions.
Signed-off-by: Chris Hellmuth <chellmuth@gmail.com>
The first step of the great ustringhash conversion: This changes all
RendererServices methods (i.e., callbacks to the renderer from the
shader JITed code) that used to take ustring parameters, switched to
take ustringhash instead.
The second step -- which I will tackle after this part is merged --
will be to fully switch the in-memory representation of strings during
shading execution to ustringhash on the CPU (thus matching how we've
been doing it on GPU and making a number of data stuctures and
representations identical/shared for CPU and GPU).
But in the mean time, this lets renderers get started changing their
RendererServices specializations. Just like with OSL's internal
implementation, it may be easier for renderers to break the job into
phase I (rendererservices) and phase II (in-memory rep).
Some notes and guideposts:
* In oslconfig.h, define ustringrep to alias to either ustring or
ustringhash, depending on a new (but temporary) CMake variable
`OSL_USTRINGREP_IS_HASH`, which is OFF for now, meaning that
ustringrep is still ustring.
* Also in oslconfig.h, several helper conversion functions among
ustring, ustringhash, ustringrip.
* RendererServices and BatchedRendererServices (and their various
subclasses in testshade and testrender): change the method
signatures from ustring to ustringhash for all methods that are
reachable from shader code.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Looking at the code coverage report and trying to get a higher
coverage amount, employing two strategies: (a) remove unused code
that is not covered because it's... unused; and (b) test things
that are used but aren't in the test run. Details follow.
Remove deprecated version of LLVM_Util::make_function.
Disable contents of dump_struct_data_layout when OSL_DEV is not
defined, it will never be called in any other circumstances -- it's
strictly a debugging tool, not part of the shipping library.
When OSL_USE_OPTIX is not enabled, exclude compiling contents of
ptx_compile_group.
get_alignment can also be OSL_DEV only as well as simplified
Arbitrarily pick a couple of render tests to use llvm_opt levels 12
and 13 to verify that they work correctly. (That's a couple hundred
lines of pass setup code that wouldn't otherwise be included in the
coverage analysis, nor tested in the testsuite.) This also required us
to add the --llvm_opt command to testrender, as it is in testshade.
Make the analysis run build for avx512 as well for more code coverage.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The immintrin.h header is required for the implementation of the
popcount and coutr_zero function only when the Intel compiler is used.
Move the inclusion accordingly. The current state breaks build on
platforms that don't provide the header, but provide the alternative
means (gcc/clang built-ins, etc).
Signed-off-by: Dan Horák <dan@danny.cz>
We compile testshade.cpp twice -- one for a special "does it compile
as a DSO" check but that isn't run from the testsuite. This was
causing the coverage analysis to think nothing in testshade.cpp (at
least one copy of it) was getting run, which is technically true, and
counting all of its lines as untested for the coverage analysis. Just
exclude that funny dso build from the coverage case.
Also exclude anything in src/build-scripts from coverage analysis,
that's not code that ships, so to speak.
We're trying to chip away at untested code bit by bit to get our
coverage to over 80% of lines. We're getting close.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Define std::hash for ustring and ustringhash, when not using an OIIO
new enough to do so.
Define std::less for ustringhash, when not using an OIIO new enough
to have operator< defined for ustringhash (will be fixed separately).
No need now for ustringHash, which is also unwisely similar to
ustringhash, just use std::hash instead.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Reduce dependency on ShadingContext and ShadingSystem objects
for accessing the ShadingSystem and consequently the ColorSystem. In its current form
it is not device-friendly.
We introduce a self-contained pvt::ShadingStateUniform that maintains a proper
state and is independent of the ShadingContext and ShadingSystem. We add an
opaque pointer to ShadingStateUniform in the ShaderGlobals structure.
Update uses of ColorSystem in opcolor.cpp to go through
ShaderGlobals::shadingStateUniform. Also update uses of m_commonspace_synonym
and m_unknown_coordsys_error in opmatrix.cpp to go through
ShaderGlobals::shadingStateUniform.
NOTE: The ShadingStateUniform is private and an OSL implementation detail and
does not change the public interface other than adding an opaque pointer to
ShaderGlobals.
* Fixed LLVM representation to accommodate ShaderGlobals::shadingStateUniform.
* Added missing Copyright Contributors license to shading_state_uniform.h
* Removed m_commonspace_synonym data member
Moved ShadingStateUniform closer to its use in osl_prepend_matrix_from
* Fixed clang-format diffs. Added void* shadingStateUniform to ShaderGlobals in rend_lib.h. Needed for proper working.
* Added shadingStateUniform to wide_opmatrix and batched_shader_globals.h
Adjusted batched struct data layout check in codegen.
* Reordered headers to include shading_state_uniform.h with the other headers
from liboslexec
Signed-off-by: Steena Monteiro <steena.monteiro@intel.com>
Some explicit SIMD loops were technically illegal because each lane was not truly independent, fixed by adding "reduction" clause. Some Mask objects had to be replaced with just "int" to allow it to be specified in the reduction clause.
icx v2022.2.0 does apppear to have some regressions for large amounts of Dual code in a couple use cases, added alternative(improved) Single Statement Assignment (SSA) version of OIIO::fast_sincos which allowed unit tests to pass. Consider updating OIIO fmath.h to use SSA coding style (which would allow this workaround to be removed).
Another regression with transform_normals was avoiding by introducing an OSL_UNLIKELY(condition) to generate a branch vs. masked execution, this change is a general improvement and should be left in.
Avoid flex_bison produced warning in osogram.cpp "variable 'osonerrs' set but not used".
Due to a number of regressions that appear to be fixed in next revision, choose to skip icx 2022.2 and continue CI for the time being with icx 2022.1 until the next release.
Because multiple (possibly newer) versions of oneapi may be installed, use a config file to specific a specific compiler and tbb version NOTE: Individual oneapi components may independent version numbering.
For centos images, use a config file to specificy a specific oneapi compiler and tbb version
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* For analysis, USE_QT=0 will disable osltoy from building (since we
don't/can't test it in the testsuite, it's unexecuted but really
shouldn't count against our code coverage metric).
* Exclude Imath duplicate headers from Sonar analysis.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Traditionally, OSL has supported building against the current and also
the previous supported version of OIIO, but not necessarily any older.
With OIIO 2.4 released (any day now), and OSL "main" is actually going
to be next year's supported release, it seems reasonable to change the
forward-looking development of OSL to have a minimum of OIIO 2.3,
which means that OSL-next will support OIIO 2.3, 2.4, and the
currently-in-progress 2.5 (or whatever it's called by that point).
The main reason to do this bump is because we'd like to depend on the
ustringhash class, and possibly other things we will probably be able
to easily backport to 2.3 but not to 2.2.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* A variety of changes to get a clean build and passing tests when
using LLVM 15.0.
* I've noticed problems when using LLVM 15 but building with earlier
clang, so the cmake scripts now print a warning in that case, so if
users encounter trouble they have a hint about what to do to fix it.
* For our CI tests on Mac, force the MacOS-11 test to use llvm 14,
and the MacOS-12 test to use llvm 15.
IMPORTANT TO-DO / CAVEATS:
1. When doing JIT at optlevel 12 or 13, I had to disable a number of
passes that don't seem to exist anymore in LLVM 15. This is enough
to get it working, and to be honest, I don't know if anybody uses
these opt levels. But we need to revisit this, because I don't
know if there these are cases where the names of the passes merely
changed or that new passes take their place (in which case we
should add the new passes, not stop after merely disabling the
deprecated ones). For that matter, optlevel modes 11, 12, 13 are
supposed to match what clang does for -O1, -O2, -O3, and that
changes from one release to the next, so we should probably revisit
this list and make sure it's matching clang's current choices
(which I assume are crafted and periodically revised by clang's
authors).
2. LLVM 15 switches to "opaque pointers". It still supports typed
pointers... for now. But as you can see in the diffs, I had to
disable a variety of deprecation warnings and take some other
actions to put LLVM 15 in the old "opaque ptr" mode to match our
use of LLVM <= 14. But this is only temporary, because the typed
pointer mode is expected to be removed entirely in LLVM 16. So at
some point in the next few months, we are going to need to support
opaque pointers in our use of LLVM. (Also note: for a while, we're
going to have a bunch of ugly `#if` guards or other logic to
support both opaque pointers for users with llvm >= 16 and also
typed pointers for users with llvm <= 14, until such time as our
LLVM minimum is at least 15, which I expect is not a reasonable
assumption for at least a couple years.)
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Several places in leparse.cpp were found by static analysis to have
memory leaks -- results of calling _parse() that, in the case of
early exists in error conditions, never got deleted.
I also did some other modernization: changed NULL to nullptr, changed
some loops to more compact range-for, utilize some Strutil helpers
that will be more efficient than the alternative std::string
manipulation.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This was meant to work all along, which is why this patch is so small!
But a tiny detail -- not explicitly passing info about the `--help`
argument -- made ArgParse substitute its own default, which exits
after printing the usage info. But, as I apparently meant to do all
along, if you specify it explicitly, you get to handle it, and the way
we handled it alsp printed the extra information.
Anyway, with this patch, `testshade --help` not only prints all the
testshade command options, but also it prints some information at the
end about some of the build settings of OSL, versions of key
dependencies, hardware info, and a run-down of the ShadingSystem
options. This can be helpful for debugging.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
direnv (https://github.com/direnv/direnv) can be used for local developer workflow enhancements. direnv uses .envrc file for configuration. Because .envrc is usually specific to a developer workflow, it is generally not committed.
Signed-off-by: Mark Sisson <5761292+marksisson@users.noreply.github.com>
- There are a handful of tests we disable entirely with because they
are irrelevant or just cannot be made to be clean (and that's ok).
- Tests with sanitizers run slow, so we increase timeouts, and also
set AA to 1 for render tests to speed them up. We rig it so a
non-matching output won't fail, and that's ok because every other CI
test is checking the results; here we have no reason to believe
results will differ from other tests, we just need to run with
sanitizers turned on.
- In testing.cmake, we rig texture tests to disable the leak
sanitizer, because we can't eliminate some completely benign leaks
from the texture system, and because they are called from JITed
code, we don't get a clean stack trace of things that happen in OIIO
and therefore we just can't enumerate the allowed exceptions. So we
trust that OIIO itself tests the texture system adequately and is
able to ignore the benign "leaks."
- We already submitted PRs and merged the little fixes that were
discovered by enabling address and leak sanitizers. This is just the
final bit to turn it on for CI runs from here on out.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Rather than specify the version in the usual project() statement,
project (OSL VERSION 1.13.0.0 ...)
this patch decouples it
set (OSL_VERSION "1.13.0.0" CACHE STRING "Version")
project (OSL VERSION ${OSLVERSION} ...)
thus allowing the possibility of overriding the version number at
build confuguration time (`cmake -DOSL_VERSION=3.1.4.1`).
This should be used with extreme caution. Probably never used by most
people. I don't need to describe all the ways that things can go
sideways if you make your build of x.y.z.w actually be arbitrarily
different from the x.y.z.w that everyone else uses and that is
documented.
That said, there are two scenarios I have in mind for which this
ability is very useful:
1. Sometimes, we add or deprecate API features "in advance" by having
an `#if` enable or disable code based on the version, essentially
putting in place changes that can't be enabled yet because they
would break compatibility rules, but we want to integrate the code
now and have it become active upon reaching a particular future
version in which that incompatibility will be allowed. The version
number override makes it easy to test this by quickly doing a build
"as if" we had already bumped the version number to see what will
happen when we actually do.
2. Every once in a while we need to do an internal build that, to
conform to version compatibility rules, needs to appear to still be
part of a past version family. This lets individual sites play some
games with version number overrides rather than have to create new
branches and tags for certain one-off internal releases.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This is a tricky situation to describe. Consider this shader:
shader test (float A = 0.5,
float B = A)
{
printf("A = %g, B = %g\n", A, B);
}
By default, A is 0.5 and B gets A's value, also 0.5. If you give A a
different instance value, like 1.0, then B will also get the same
value, they'll both be 1.0. If you give B an instance value, that
will be used for B rather than copying A's value, like this:
$ testshade -param B 1.0 -shader test test
A = 0.5, B = 1
But there is a corner case bug: if we assign B an instance value of
0.0, it will incorrectly fall into a special case where it thinks it
gets a default value, only coincidentally because 0.0 is in an unused
slot for the default value (unused because it has no default -- its
default comes from "init ops", that assignment).
$ testshade -param B 0.0 -shader test test
A = 0.5, B = 0.5
!!!
Reading the patch should be self-explanatory for what the bug is and
how we fix it.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Signed-off-by: Larry Gritz <lg@larrygritz.com>
I've tried to write a complete set of step-by-step instructions for
everything about how to coordinate and deploy a release. This is a
living document, we will amend over time. I'm sure there must be
things I've forgotten.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The little ReadEvent class had two members that are never set when
OSL_DEV is not defined, nor are they accessible (the accessor
functions also are disabled based on OSL_DEV). It looked to the static
analyzer like the data members were uninitialized, which they
were. Add OSL_DEV guards around the declarations of those members as
well, so they aren't even present in the struct when not needed for
this debugging mode.
Food for thought: another alternative is to keep the members but just
remove the OSL_DEV guards entirely -- just let the two int data
members be used all the time, not just for OSL_DEV. It's too small to
have any perf penalty, and it unclutters the declaration by removing
all those `#ifdef` lines.
For that matter, I also wonder if there's any real point to having
private members and accessor functions for this tiny internal-only
class, and maybe it's better to just let it be compact declaration
of a simple `struct` with three directly accessible data members.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
In testshade, there was a "loop" that wasn't really a loop, and it
was getting flagged by static analysis as being fishy. This patch
rewrites it as an "if".
Also take the opportunity to change from stream output to OSL::print
formatted output.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We had hard-coded longer allowable timeouts for particular testsuite
tests. But for the static analysis runs, it takes extra long and we
need a way to tell it to increase the timeouts even more, and to
exclude the longest running tests from the analysis coverage run (it's
not necessary to run those tests, they don't increase coverage enough
to justify being an hours long run).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Signed-off-by: Larry Gritz <lg@larrygritz.com>
In the course of our runtime optimization of shader networks, we
generate lots of new constant symbols (for example, as we optimize an
"a = b + c" into "a = newconst" if b and c can be determined to be
constants). The new constants are named with a numerical suffix doled
out in a sequental manner.
I've always thought that maybe it would be helpful for me when
debugging optimizer output if the name of the symbol also revealed the
value of the constant it held (when possible). Never did anything
about it.
Now it turns out that this has an application for JIT to PTX, where
the PTX cache can save compilation-to-PTX time if it encounters LLVM
IR that is exactly the same as it has seen before and is cache. If
the constants are generated in a different order, it generates
non-functional, purely text differences in the code we generate, and
this means the PTX cache doesn't function as efficiently as possible.
So this change alters the compiler-generated constant naming for two
cases: int constants, and string constants. The int constants append
the value of the integer (using '_' instead of '-' for negatives), and
the string constants append the hexidecimal value of the ustring hash
for that string. In both cases it does this without using or
incrementing the `m_next_newconst` counter that it uses to
sequentially name the other types that can't easily encode the value
to generate a unique name.
It's really to help the PTX cache, but as a side benefit, it will
be slightly easier for me to make sense of the optimized code when
I need to read it with my own eyes while debugging.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Today's branching of OIIO 2.4 and subsequent changing of its master
branch to think it's "2.5" seems to have triggered an automatic fuse
that annotates a particular function in OIIO as deprecated for version
2.5 and beyond. OSL was still calling the old version, so some
immediate changes are necessary to call the correct new name when
using a new OIIO.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Exclude osltoy from coverage analysis, since we don't have a way to
run it from CI
* Run batch shading for analysis to get better code coverage
* llvm_gen_select: extra check that we can't overrun memory of cond[]
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Change the languagespec to document the new MaterialX-synchronized
closures.
* Document the old ones as "deprecated" -- still supported for 1.x but
assumed to be on the slate for removal in OSL 2.x.
* Change stdosl.h so that the new closures, instead of being seen only
if MATERIALX_CLOSURES is defined, to being enabled by default, but
(for now) can be disabled if NO_MATERIALX_CLOSURES is defined.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Removing the suffix ids and callseq comments generated by global counters
leads to deterministic ptx in a multithreaded environment.
Signed-off-by: Chris Hellmuth <chellmuth@gmail.com>
* Minor changes to fix a couple more warnings given by SonarCloud
static analysis.
* Get the code coverage enabled, it wasn't running properly.
* testrender: allow secret TESTRENDER_AA to override command line --aa
* runtest.py: allow secret OSL_TESTSUITE_SKIP_DIFF to force skipping
the reference comparison step.
* For analysis run, set TESTRENDER_AA=1 to keep the tests from being
prohibitively expensive, and OSL_TESTSUITE_SKIP_DIFF=1 to keep it
from failing because the aa=1 images are different.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We had been using clang/llvm 12 and when trying out 14 I started
getting test failures with OptiX complaining about duplicate
symbols. They all seemed to be the things we declared as
OSL_CONSTANT_DATA. It looks like there's a difference in llvm 12 vs 14
in how the generated PTX marks its symbol visibility.
The solution (thanks to a clever suggestion by Alex Conty) was just to
enclose those declarations in anonymous namespaces. Works like a
charm!
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This allows the refraction index to be detected even if a medium closure
is not present. Additionally, the medium detection now traverses layer()
closures.
Signed-off-by: Chris Kulla <ckulla@gmail.com>
There seems to be just a teensy bit of nondeterminism or
platform-variance in this one test that we haven't tracked down the
source of yet, that moves some specular fireflies around. Smells like
maybe an uninitialized variable somewhere in testrender? Because it
doesn't happen on any other test and we haven't detected it in our
real renderer, we don't currently suspect that it's an actual error
in oslexec.
Big hack: Re-run failed "render" tests up to 5 additional times before
giving up.
This makes me feel dirty. But it makes tests pass for now. Eventually we
will find the embarrassing testrender bug that is responsible for the
occasionally different specular sampling noise.
Also slightly bump up the error tolerance for this test.
Also add one alternative that seems to frequently pop up.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Didn't properly detect the disallowed use of arrays for the false
clause because we checked `t` twice.
Yay for static analysis!
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Another round of clang-format
* Bump the standard clang-format we use for CI from clang11 to clang14.
This will make some minor additional changes.
* Due to a mismatched target name, we weren't actually checking the
formatting in CI on all the files we thought we were, and therefore
not assuring that certain code was correctly formatted. So there is
some fixup needed, which is included here.
* And a bunch of files we didn't try to do before, either.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The idea is to stop propagating derivs through BSDFs, which we
never did properly. Instead we trace rays with radius + spread,
which we turn into OSL derivs in SG when we hit a surface. Then
when bouncing off a surface we use the BSDF roughness to set the
spread. In addition we also:
* Return BSDF samples/eval as a small struct Sample(). This
avoids output parameters in the methods.
* Stop passing a full ShaderGlobals to the BSDF methods. Pass
a viewing direction wo instead so they could run in a simpler
context.
Signed-off-by: Alejandro Conty <aconty@imageworks.com>
This simplifies the logic around the ior which drives reflection/transmission (fresnel effect)
from the ior which drives refraction (and total internal reflection). Setting both iors equal to each other
gives the expected physical behavior.
Also tweak the bsdf constructor to support variadic arguments which simplifies a bunch of code.
Add a new test to validate the refractive version of generalized_schlick_bsdf.
Signed-off-by: Chris Kulla <ckulla@gmail.com>
Implement get_albedo for the sheen closure, based on fit derived by the MaterialX team
Add optional mode to testrender to visualize the result of the get_albedo method. This can be useful for validation of the get_albedo method.
Signed-off-by: Chris Kulla <ckulla@gmail.com>
* Implement reflective microfacet based MaterialX closures
Add test cases for dielectric, conductor and generalized schlick closures
Signed-off-by: Chris Kulla <ckulla@gmail.com>
* Implement refraction case in microfacet closures
Signed-off-by: Chris Kulla <ckulla@gmail.com>
* Update reference images
Signed-off-by: Chris Kulla <ckulla@gmail.com>
* Relax diff threshold on render tests
Signed-off-by: Chris Kulla <ckulla@gmail.com>
It printed the u,v coordinates where the error was first discovered.
This is not necessarily deterministic, of course!
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Add support for layer closure with dielectric reflections as top layers, and a test with nested layer closures
* Add support for sheen in the layer closure
Signed-off-by: Adrien Herubel <adrien.herubel@autodesk.com>
* Extend the bsdf interface in testrender to use Color3 instead of float in preparation for MaterialX closures which are almost all tinted
* Add stub parameters and IDs for all MaterialX closures
* Register all MaterialX closure types
* Implement most MaterialX closures, reusing the existing implementations for now
This commit is mainly intended to do the busy work of adding the new closures, no new BSDFs are actually implemented yet. This means that some of
the closures are handled in a very rudimentary way, ignoring some key parameters. The intent is to improve support for these over time.
Signed-off-by: Chris Kulla <ckulla@gmail.com>
The sampler implementation in testrender was very close to the 2020 article from Brent Burley. This patch
catches up to best practices, including a faster owen scrambler from Nathan Vegdahl and recognizing that
the index scrambling can be achived by the same owen scrambler method (which saves a bunch of logic and runs
faster). Finally, avoid the redundant bit reversals and use compiler intrinsics when available to benefit
from hardware instructions (for ARM or GPUs). Finally the integer hash constants were updated to the latest
from the hash prospector project.
While more optimizations are certainly possible, this is a reasonable balance between quality and speed.
Update reference images for new sampler
Signed-off-by: Chris Kulla <ckulla@gmail.com>
* GPU string overhaul
Big string cleanup, yet again, for GPU. We still had this tricky
double indirection happening with strings for OptiX -- the string was
a pointer to a "global", which in turn held the hash. No, throw all
that out. Just directly store strings as the ustring hash on
OptiX. No extra indirection. Many fewer differences between how things
are handled on OptiX versus CPU (except for the fact itself that the
"string" holds the hash rather than the char*).
Some details worth noting:
* LLVM_Util loses "type_string()" and calls it "type_ustring()" to
emphasize it's the type we're using to represent a ustring, and
definitely not a C or C++std string. Also, LLVM_Util itself now lets
the caller tell it whether it prefers that the representation of a
ustring be the charptr or the hash, and then handles most of the
details internally. The BackendLLVM, then, makes that decision early
on, charptr for CPU and hash for GPU.
* Several more spots in LLVM_Util and BackendLLVM have been plumbed to
pass around llvm IR SSA variable names to try to make the resulting
LLVM IR more readable when I'm trying to debug it. (This was
strictly debugging scaffalding aid for me, but comes along for the
ride because it's useful to keep.) I also added a
BackendLLVM::llnamefmt to format strings, which only does the
formatting when the shading system knows it's going to save the IR
for inspection, and otherwise (that is, in production) it does not
do the string formatting or pass the names along, so it's not doing
all that string manipulation when no human is going to be looking at
the IR dumps.
* A lot of clunky OptiX-only code disappears.
* Some side effects of my reducing complexity of the IR as I debugged:
there were times we were generating redundant pointer casting in the
IR, such as grabbing a pointer, then casting it to a void*, then
casting back to some other kind of pointer. In some cases I reduced
that quite a bit. Again, this was mostly just a scramble to make the
IR more readable while I debugged, but it probably has a minor
performance gain as well since it reduced somewhat the sheer amount
of IR we generate and pass to the optimizer and JIT.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Remove register_string entirely from RendererServices
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Remove more register_string remnants
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* comment typo
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Set up a SECURITY.md explaining how to confidentially report
vulnerabilities.
* Create scorecards.yml workflow to run OSSF scorecard to check supply
chain security issues.
* Use exact SHA's for actions versions to guard against hijacked
actions repos that might try to substutite something else at the
version tag we use.
* Make sure actions we use don't get write perms on our repo.
* Tried to use harden-runner action to ensure no actions on the runner
overwrite source code or have unauthorized outgoing net connections
during the GHA run, but currently leaving it disabled because of too
many false positives -- it flags every dependency download as a case
of altering source code. Which it technically is, but it doesn't
matter for any jobs that aren't producing any artifacts that will be
consumed or relied on by users.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Most LLVM API IR calls that generate ops allow an optional name of the
llvm symbol that will hold the result of the operation. By default, it
will make one up, such as "%37", "%38", etc. Needless to say, these
are difficult to understand when reading volumes of IR dumped for
debugging purposes.
So in this patch, I'm starting to expose the ability for our own code
to pass names through our wrapper functions. I needed this for some
deep debugging.
I haven't instrumented the entire set, just trying it on for some of
the specific areas I needed to debug. But this is a useful first step.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This makes OptiX 7.0 the minimum version we accept, and greatly
simplifies a lot of code we were separately maintaining for 6 and 7.
This will go into main AFTER the release branch for 1.12 is split off.
The dropping of OptiX 6 will be a 1.13 (or whatever we call it) change.
Don't build the optix modules at all when not USE_OPTIX.
Totally remove stringtable.
Other simplifications stemming from these.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The implementation of simplex noise we've been using is public domain,
but looks like when we pasted the license notice into our repo, there
was one stray sentence (probably left over from an earlier license
scheme Stefan Gustavson used?) that makes reference to GPL. But the
software and its repo are NOT -- it's very clear that it's public
domain: https://github.com/stegu/perlin-noise/blob/master/LICENSE.md
So remove this confusing sentence.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We'd been waiting until the minimum OIIO requirement was high enough
to include support for the newer syntax.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
In a few of my big clang-format revisions, I was over-zealous, and did
not realize that I was making the command line option parsing unreadable.
Revert those secions.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Recent OIIO and fmt changes slightly changed this one place's output
with some version/platform combinations. Also make it match exactly
between the example-deformer and example-batch-deformer tests.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
fmt master seems to have just deprecated the default ability of
fmt::sprintf("%s") to printing anything that allows stream
output. This breaks use of Strutil::sprintf formatting of ustrings.
This isn't yet in any tagged release of fmtlib, but I'm trying to get
ahead of it.
The quick fix is to just change those handful of places that still use
sprintf convention with nonstandard types to use std::format style
calls, which we are slowly converting to anyway. (And maybe a couple
other spots while I was at it.)
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Banish the last of the sprintf calls (except the handful that need
to be there, because such formatting is still supported in shaders
themselves, so something needs to understand them).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Another semi-random collection of conversions. Bit by bit.
These are all internals, none of it involves public interfaces.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Use unity build to speed up OIIO build (for sufficiently new OIIO).
* Build OIIO with STOP_ON_WARNING=OFF, so that if a compiler upgrades
and makes new warnings for OIIO, that doesn't break OSL's CI build.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Add support for closures to the batched interface.
The renderer interface for closures, including registering
and processing them remains unchanged.
The Ci variable in BatchedShaderGlobals has moved from
the uniform portion to the varying portion, which may
require changes to any renderer code that was utilizing
the BatchedShaderGlobals.
To utilize closures with the batched interface, you register
the closures per usual, then once shading has finished,
the Ci in BatchedShaderGlobals will contain an Block of pointers
to the roots of a closure trees that can be processed
independently just like in the normal single-point interface.
This initial implementation is largely functional, however,
there may be more optimization gains available in the code
that copies the run-time SIMD values and scatters them
to the separate closure-data locations.
One remaining un-implemented feature is passing
_closures_ via get/setmessage() when using the batched
shading interface, and the locations where that would
need to be implemented are marked in the source.
Signed-off-by: Stephen Friedman <sfriedma@pixar.com>
* Address batched closures review comments:
* Remove unneded Block specializations for pointers.
* Rename null variable to null_value.
* remove a 0 default param to rop.llvm_void_ptr in batched_llvm_gen.cpp:llvm_gen_mul
* Optimizations to masking and loading in wide_opclosure.cpp
* Fix LLVM11+ builds by using llvm_vecor_type instead of llvm::VectorType::get
Signed-off-by: Stephen Friedman <sfriedma@pixar.com>
* Remove unneeded oslclosure.h in wide.h now that we aren't explicitly instantiating the templates.
Signed-off-by: Stephen Friedman <sfriedma@pixar.com>
The Intel OneAPI LLVM-based C++ compiler, as we download it from Intel
for our CI, bumped from 2022.0.0 to 2022.1.0.
New warning popped up -- newest Intel LLVM-based compiler warns about
overriding certain floating point math flags.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Fix warnings from stdint on windows by pre-including the standard header
* Make sure register_JIT_Global is a public symbol since it needs to be called by client code
* Fix how llvm-as, llvm-link and python are invoked by cmake so they can be properly found on Windows platforms
Signed-off-by: Chris Kulla <ckulla@gmail.com>
There's no rhyme or reason to this particular set of changes... I just
picked a few files and did the obvious conversion. Will do more in
subsequent PRs.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
A few minor changes were needed to address new warnings.
clang14 warns if you use bitwise `|` or `&` in an `if`, but there are
some places where we do that purposely to avoid branches.
Disable clang14 warnings about bitwise ops for all of liboslexec.
Also some batch shading related vectorized loops fail to vectorize
with clang14 -- though they worked fine with clang12 -- and so we
needed to mark those as "complex" SIMD loops. Is this a regression
with clang14?
Revise simd loop pragma for wide_transformc due to llvm14 regression
Signed-off-by: Larry Gritz <lg@larrygritz.com>
In order to move towards shader execution on devices, we need to replace
host side interactions with new concepts that can exist on the device (or host).
The current virtual function based RendererServices exemplifies this dependency
on host side objects.
To break the dependency we introduce free function based renderer services (rs_free_function.h)
To simplify its introduction, we only added the renderer service free functions required by opmatrix.cpp.
Goal is to have a single opmatrix that can be compiled for host or device (eventually).
Renderers can provide llvm bitcode representation of their renderer service
free functions to the ShadingSystem. During shader compilation/JIT the
renderer's bitcode is combined with opmatrix.cpp bitcode along with the JIT'd shader network.
The idea is the combined LLVM IR representation can then be lowered to the target
(be it on the host or device[SPIRV|PTX]). Thus a renderer's free function implementation can be shared.
opmatrix.cpp now only calls the free functions (not virtual RendererServices).
However liboslexec on the host contains an rs_fallback.cpp which forwards
calls through the existing virtual RendererServices for seamless compatibility
when a renderer does not provide bitcode for the renderer services.
Testshade has been modified to optionally to provide rs_bitcode,
utilizing a new Cmake macro, EMBED_LLVM_BITCODE_IN_CPP, to turn rs_simplerend.cpp (the free function implementation)
into bitcode.
Free functions can be enabled via --use-rs-bitcode flag in testshade.
Free functions need access to RenderState values because they lack a this
pointer like the RendererServices virtual methods have. Added a RenderState
containing minimal set of values required by free functions.
Testshade/SimpleRenderer is responsible for setting RenderState values.
Added ShadingSystem attribute rs_bitcode to hold renderer services free function bitcode (provided by the renderer)
Renderers using global variables (like ustring's) not in strdecls.h must register their global variables' addresses
using OSL::register_JIT_Global (const char *global_var_name, void *global_var_addr).
This is necessary for the JIT process to resolve global variables that are embedded in RendererServices bitcode.
A validation step occurs that emits an error with a list of unmapped global variables.
RendererServices can reference strings with the STRING_PARAM(varname) macro.
By default, STRING_PARAM(varname) works for any string in strdecls.h.
For any string NOT in strdecls.h, it will map to a global variable with a prefix of "RS_".
For example, STRING_PARAM(raster) maps to a global ustring RS_raster.
NOTE: C linkage with a "RS_" prefix is used to allow for unmangled
non-colliding global symbol names, so its easier to pass them to
OSL::register_JIT_Global(name, addr) for host execution
NOTE: the STRING_PARAMS macro adapts to OSL_HOST_RS_BITCODE macro
to utilize the RS_ prefix. RS_ prefixed versions of all OSL::Strings
intances have been created by rs_free_function.h, so the same STRING_PARAMS
macro can be used for renderer service or OSL strings.
NOTE: We expect the handling of strings to change in the future possibly being replaced by Handles entirely
It is the renderer's responsibility to instantiate any NEW ustring global variables with the proper RS_ prefix;
It is the renderer's free function implementation to import/extern any of these NEW global variables with the proper RS_ prefix;
For testshade, a rs_strdecls.h has been added and is utilized to instance the RS_ global variables and register their addresses with OSL in simplerend.cpp.
rs_strdecls.h is also used by rs_simplerend.cpp (free function implementation) to create the externs with the RS_ prefix.
NOTE: DO NOT instantiate global variables inside the free function renderer services, because we strip global constructors as part of the pruning process.
Added LLVM_Util::validate_global_mapping to identify unresolved globals.
Added utility macro __OSL_STRINGIFY(string_value) in oslversion.h.in so the
proper naming prefix is used and we don't have multiple implementations around.
Added liboslexec build step to produce bitcode containing OSL library functions that depend on free functions (here, opmatrix.cpp)
Added a new Cmake EMBED_LLVM_BITCODE_IN_CPP function in llvm_macros.cmake that replaces the LLVM_COMPILE macro
and provides for both linking and serializing multiple OSL cpp files.
Fixed cuda.macro to use new location of serialize-bc.py in src/build-scripts.
Fixed formatting issues.
Disabled erasing llvm.global_ctors from LLVM module because of issues on MacOS Apple Clang.
Left as future work.
Added better error checking for module_from_bitcode().
Signed-off-by: Steena Monteiro <steena.monteiro@intel.com>
New classes of warnings triggered when enabling option "opt_warnings":
* Creation of new strings mid-shader.
* Access to the characters of strings mid-shader.
We already warned about texture access by name that could not be
turned into a handle.
In all of these cases, we only warn *after* runtime optimization, so that shader
idioms in the source code that appear to do these forbidden things will not warn
as long as they can be reduced to safe operations in the course of the runtime
optimization step.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
checked_find_package:
* Optional SETVARIABLES gives a list of vars to set to TRUE if the package
is found.
* Optional DEBUG flag turns on extra debugging information about how the
package is found.
* Optional PREFER_CONFIG uses a dependency's exported config, if available,
rather than our Find module.
* Global ALWAYS_PREFER_CONFIG option can be set to ON to always prefer
config over Find modules for all dependencies.
fancy_add_executable:
* -DENABLE_INSTALL_<executable>=OFF lets you turn off the installation
of an executable.
other:
* Change the target name for doing the clang-format over the code to
"run-clang-format", because "clang-format" is an imported target name
of the libclang exported cmake (which we may wish to use soon).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Pass some extra headers to NVCC_COMPILE to make sure that the PTX files are regenerated whenever those headers are changed.
Signed-off-by: Tim Grant <tgrant@nvidia.com>
* Don't generate PTX for rend_lib.cu when using OptiX 6.
Signed-off-by: Tim Grant <tgrant@nvidia.com>
* Fix the callable program signature for the init and group functions in the OptiX wrappers.
Signed-off-by: Tim Grant <tgrant@nvidia.com>
* Fix issues with the LLVM arguments for osl_printf when targeting CUDA.
Signed-off-by: Tim Grant <tgrant@nvidia.com>
* Fix llvm_gen_construct_triple for OptiX. The 'to' argument was not handled correctly.
Signed-off-by: Tim Grant <tgrant@nvidia.com>
* Improve support for CUDA osl_get_matrix.
Signed-off-by: Tim Grant <tgrant@nvidia.com>
* Add basic support for named transforms to testshade/testrender.
Signed-off-by: Tim Grant <tgrant@nvidia.com>
API changes we had to take into account:
* TargetRegistry.h location
* No more DisableTailCalls field in PassManagerBuilder.
Needed to update the ref image for render-microfacet test, some sparklies changed.
Looks like the new LLVM probably JITs to ever so slightly different math code, tickling
some LSB differences that at 1 sample per pixel, results in some different sampling
directions leading to fireflies. We decided to just commit a new ref image and move on.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
In our generated OSLConfig.cmake, we inserted a find_dependency(Imath)
that accidentally demanded the OpenEXR_VERSION (should have been
Imath_VERSION). That works fine as long as Imath version is the same
(or higher) than the OpenEXR version. But if the installed OpenEXR
version is somehow ahead of the Imath version, it will fail.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
A small instalment on the long-term shift from printf-style formatting
to std::format style formatting.
Define preprocessor OSL_INTERNAL, which is defined when compiling OSL
itself, but will not be defined for downstream projects using the OSL
headers.
When building OSL, enable the symbol OIIO_ERRORHANDLER_HIDE_PRINTF,
which causes ErrorHandler methods using the old conventions to be
marked as deprecated and generate warnings.
Clean up all the calls revealed by those warnings. (And
opportunistically clean up some other things that happened to be in
the same files.)
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Also change the one exception mentioned to a std one, matching
Imath 3.x and insulating us from any of the old Iex custom exceptions
from Imath 2.x.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Changed a function signature in OIIO master that threw us off here
just a tad, as there was no direct type conversion the compiler could
figure out on its own. Easy adjustment gives it the right hint.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This eliminates a huge amount of redundancy in our CI descriptions --
instead of the entire CI logic replicated for each test in its own
universe, we group tests into (a) all the ones based on ASWF VFX
Platform containers, (b) non-VFXP-year oddballs run on Ubuntu, (c) Mac
tests, (d) clang-format verification. Each has the build logic
expressed once, and a short list of variables that change for each
individual test of that category.
This organization is much easier to read, maintain, modify tests, or
add new test variants, and cuts the size of the ci.yml file by over 50%.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Avoid many clang vectorization workarounds for OSL_INTEL_LLVM_COMPILER_VERSION
Added Makefile option VEC_REPORT to turn on generation of optimization reports
Reduce impact of REQUIRE_INF_NAN for icx by explicitly honoring inf, nan, and signed zeros vs. disabling fast-math
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
Add compiler support for Intel(r) oneAPI DPC++/C++ Compiler (based on LLVM technology).
Detect if CMAKE_CXX_COMPILER_ID matches "IntelLLVM" and set CMAKE_COMPILER_IS_INTELCLANG=1
NOTE: CMAKE_COMPILER_IS_INTEL will not be set, that is reserved for the classic Intel(r) C++ Compiler.
Define OSL_INTEL_LLVM_COMPILER to hold an encoded Intel(r) LLVM Compiler version
(e.g. 20220000), or 0 if not an Intel(r) LLVM Compiler.
Define OSL_INTEL_CLANG_VERSION to hold the encoded Clang version the
Intel(r) LLVM Compiler is based on (e.g. 140000),
or 0 if not an Intel(r) LLVM compiler.
Ensure OSL_CLANG_VERSION is when its a vanilla clang compiler, not apple or Intel(r) oneAPI DPC++/C++ Compiler.
Added OSL_ANY_CLANG which is set for any clang based compiler.
Removed OSL_NON_CLANG_COMPILER.
Updated uses of OSL_NON_CLANG_COMPILER with more specific and understandable combinations of OSL_ANY_CLANG, OSL_INTEL_CLANG_VERSION, and OSL_INTEL_COMPILER.
Make sure our math calculations are consistent,
especially division. Different compilers may have choose to use
reciprocal division loosing precision causing slightly different
results which can lead to aliasing differences when running the testsuite
Disabled reciprocal division for all projects.
if -ffast-math is enabled to to the underlying compiler, inf's and NaN's may not be handled properly (by design)
When building liboslexec, ensure any source files that require proper INF or NaN handling have additional command line flags passed if needed (as the default behavior maybe -ffast-math):
shadingsys.cpp
wide/wide_shadingsys
wide/wide_optest_float
Enable unitialized Matrix44, because all of its members get overwritten anyway, in impl_transform_normal_masked.
Added alternative test results where needed, wsing '.icx.' as shorthand for the Intel(r) oneAPI DPC++/C++ Compiler in alternative test result filenames.
Changed some tests to use the pretty(value) which clamps values close to 0 to be 0 to avoid lots of different alternative rest result files when using printf.
Added "--center" option to several regression tests to avoid aliasing in results between SIMD batched and scalar execution.
* Added icx-vp2022 to CI plan.
* To better clarify the classic vs. LLVM based Intel compilers and that they have version #'s not just 0 or 1,
renamed OSL_INTEL_COMPILER to OSL_INTEL_CLASSIC_COMPILER_VERSION
renamed OSL_INTEL_LLVM_COMPILER to OSL_INTEL_LLVM_COMPILER_VERSION
Removed unnecessary FMT_VERSION and OPENIMAGEIO_CMAKE_FLAGS from the icx-vp2022 worklow, added back USE_SIMD.
Fix spelling of losing.
Added OSL_OMP_COMPLEX_SIMD_LOOP(...) which is enabled or disabled in platform.h based on which compilers/versions can handle vectorizing complex nested control flow. This makes it cleaner to document which loops need to vectorization conditionally controlled and centralizes where it is managed.
* Removed extra fp-model=consistent for icx-vp2022 workflow
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
Slight reworking of -Werror logic to let the default behavior be
changed by setting a cmake variable. In particular, this allows us to
make a CI build disable -Werror when we need to.
Previously, having the "OSL_CI" env variable set trumped all else and
forced -Werror. Now it just contributes to the default behavior, but
setting `-DSTOP_ON_WARNING=OFF` and cmake config time can still
disable it.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Additionally, allow custom command line arguments to run after each
image is loaded, but before the comparison.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Turn on warnings about OIIO::Strutil::format and ustring::format
We should be using Strutil::sprintf / ustring::sprintf, or upgrading
entirely to Strutil::fmt::format / ustring::fmtformat.
Some day, the plain "format" will be deprecated, then disappear, then
reappear as compatible with std::format (as are fmt::format/fmtformat now).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The str() method was not part of the final C++17 std::string_view.
To ensure future interchangeability with std::string_view, change the
few places where we call str() in the OSL code base.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Track that the backfacing() shadeop requires N and I shader globals.
* Test ensuring that N and I are reported as needed shader globals when a backfacing() shadeop is used.
Signed-off-by: Stephen Friedman <sfriedma@pixar.com>
* Implement Batched pointcloud_search, pointcloud_get, and pointcloud_write.
Updated behavior of scalar pointcloud_search and pointcloud_get to not code gen branch with code gen runtime error for maxpoints/num_points larger than array size. Instead now it codgens a clamp of the maxpoints/num_points to the arraysize. This allows code to continue executing, and if accessed outside of the range, existing codgen range checks will still emit a runtime error.
Extracted class PointCloud defnition and some helper functions to pointcloud.h so it can be shared between batched and scalar implementation.
Moved PointCloud out of anonymous namespace to OSL::pvt so a single implementation (including static maps) could be shared between batched and scalar implementation.
Refactored implementation of
template<typename DataT, int WidthT> struct Masked;
template<typename DataT> struct Ref;
to flatten class hierarchy and fix bug where derivatives of unbounded arrays could not be constructed, IE:
MaskedDx<int[]> myDeriv(ptr, arraylength);
Added MaskedData::assign_val_lane_from_scalar
Renamed MaskedData::assign_val_from to MaskedData::assign_val_from_wide
Renamed MaskedData::assign_all_from to MaskedData::assign_all_from_scalar
Updated BatchedAnalysis to indicate that results of pointcloud_search, pointcloud_get, and pointcloud_write are always varying regardless of the symbols passed in.
Updated debugging in BatchedAnalysis to emit unmangled symbol names.
Properly implement (was broken previously) BatchedBackendLLVM::llvm_zero_derivs(const Symbol& sym, llvm::Value* count) including using code gen loop to reduce code bloat.
* Fixed warnings about BatchedRendererServices<WidthT>::pointcloud_get and BatchedRendererServices<WidthT>::pointcloud_write not returning a value.
* Fixed unused variable "ok" in batched pointcloud_write.
Fixed undefined type when building with USE_PARTIO=0
* Update pointcloud tests to avoid non-deterministic results for unsorted results and out of range max_points/num_points.
Added support for NOOPTIMIZE file tag for batched regression tests.
Made array-length-reg NOOPTIMIZE so its code gen would actually execute vs. constant fold.
Fixed bug for storing arraylength result in varying result symbol.
* Fix missing virtual function overrides of MyBatchedRendererServices in example-batched-deformer
* Change varying point cloud tests to use pixel centers in order to avoid situations where point cloud could provice multiple data points at equal distances causing non-determinism.
* More changes to avoid non-determinism in pointcloud test results
* Add alternate reference images for point cloud results that were coming back slightly different for essentially identical distances.
* Fixed bug in llvm_gen for raytype where the wrong type was passed as a parameter for a string.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
(many ported/replicated from various recent OIIO PRs)
* CI: bump to latest dependency releases
* Bugs fixed in build_opencolorio.bash, it botched some of the
directory choices when customizing where to place and build
it. Also, switch to the more standard BUILD_SHARED_LIBS cmake
variable to control whether to build dynamic or static libraries.
* Correctly handle the cmake variable VISIBILITY_MAP_FILE being set to
"" on the command line as a way to disable using the visibility map
file entirely.
* On Linux, also set link flags `--exclude-libs ALL` to hide symbols
from any embedded static library dependencies, to prevent clashes
with other versions of those libraries that may be compiled into a
DCC app.
* Some other minor touch-ups in compiler.cmake.
* Refresh build_openexr.bash with the latest changes from OIIO.
* CI workflow: revise failed build artifact storage to capture cmake
files.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
New OIIO guards the contents of its Imath.h with the
`OIIO_IMATH_H_INCLUDED` preprocessor symbol. This lets us in OSL
pre-define it to absolutely ensure that OSL headers are including
exactly the Imath headers they want, not accidentally getting the ones
that OIIO includes.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
OIIO's string_view::c_str() was cute, but it's a little dicey and it's
not present in the C++20 std::string_view. We didn't use it in a ton
of places, so replace with other idioms so that in the future we are
better able to interchange with or switch to std::string_view (when
our minimum supported C++ standard is high enough).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Here was the construct it was confused by:
struct Data {
float height;
};
Data foo(Data a, float b)
{
return Data; // <----- what's this madness?
}
Curiously, this revealed a source code error in one of our other tests,
oslc-err-struct-print, which had not been detected previously!
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Added Wide<const int, WidthT> wide_shadeindex, void* userdata_base_ptr, void* output_base_ptr to BatchedExecutor interface (and underlying implementation) that allows each data lane
of a batch to utilize a different shade index in combination with a uniform userdata and output base pointer.
Fixed bug in batched version of llvm_gen_compassign where index was being loaded as uniform value even when it was varying (really just caused assert/extra output in Release).
For scalar and batched, avoid generating code for init ops or default values when a sym_loc is found because unlike get_userdata, when supplied sym_loc can't fail.
Improved LLVM_Util::op_scatter to handle scalar destination data types, previously only wide destination were required but to handle scatter results from SOA wide data layout to AOS
output placement layout scalar destinations need to be supported.
Updated BatchedRendererServices to be less abstract replacing several pure virtual functions with default noop versions to better match behavior of RendererServices.
Changed testshade to utilize output placement by default in batched mode.
Added NOSCALAR option to testing framework to allow for batched only tests.
Added testsuite/example-batched-deformer to exercise userdata placement and simple example of utilizing batched interface.
Added alternate output for example-batched-deformer to account for some rounding differences.
Added comment to note that the batched implementation assumes that SymLocationDesc::offset is readable at point 0 (because we clamp masked off lanes to 0 vs. adding conditionals).
Renamed local variable wide_index_to_output to wide_index_for_userdata to more accurately describe how it is used.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
Implement batched versions of isnan, isinf, and isfinite.
Added batched regression testsuite ieee-fp-reg.
NOTE: its really difficult to test isnan(value) == true,
as its unclear to actually generate a NAN in OSL because
most functions are "safe" and check/avoid NAN's
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
We were passing along, in the exported cmake config files, a demand
for downstream clients to use a C++ standard version equal to that
which we used to build OSL itself. But that's not necessary. We may
choose to build with C++17 (for example), but all of our public APIs
are safe to use for C++14.
Split the build-time options to CMAKE_CXX_STANDARD for the standard we
build this project with, and DOWNSTREAM_CXX_STANDARD for the standard
we are declaring is the minimum to *use* the library.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Enable BATCHED execution of most testsuite entries that don't use closures or point cloud.
Added batched implementation of hash function.
Updated batched implementation to use the symbol.get_int(), symbol.get_float(), and symbol.get_string() methods vs. *(int*)symbol.data() approach.
Fixed bug in batched code gen where mangled symbol names were output as part of debug uninit warnings and range checking.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Fixed reparameter in batched execution.
Updated batched behavior for unlocked geom shader output parameters to function like scalar version.
Fixed batched bug when non-entry layers are executed and execution masks are compared producing a <i1x16> instead of i1 for use in conditional operation.
Added more reparameter test cases to cover arrays and arrays of aggregate types.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Enabled command line assignable user data for batched execution in testshade, which fixes testsuite/userdata-passthrough
Changed print_outputs for batched execution to not use << operator which includes parenthesis, but instead output each componenet with spaces to match the non-batched print_outputs.
renamed MaskedData<...>::assign_from(void* ptr_wide_data) to assign_val_from(const void* ptr_wide_data) to better reflect that only the value is assigned, and Dx and Dy are NOT assigned.
Added MaskedData<...>::assign_all_from(const void* ptr_data) to take a scalar DataT and broadcast it to all lanes of the MaskedData, masked off lanes are NOT overwritten, and value, Dx, Dy are assigned.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Apply clang-format to wide.h
Add some paranetheses to conditional to avoid warning.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Wrapped all printf arguments with pretty() in testsuite/geomath/test.osl to avoid small floating point differences from affecting printed results in test.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Add testsuite/geomath/ref/out-alt.txt to allow for minute floating point differences with some compiler/platform combinations.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
We used to build MaterialX shaders on our end. Now it's all done in
the MaterialX project itself, and along the way, we failed to keep in
sync with them. (And even if we had "kept up", what does that even mean?
We want OSL to be independent of any particular MX release, but the MX
shaders may change from release to release. Better to be entirely on
their side of the fence.)
We turned off building these shaders by default a long
time ago, now let's finally remove the last remnants and urge everyone
to rely on the MaterialX project for this material.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Added MaskedData::assign_from(void *ptr_wide_data) to perform a masked copy of wide data at some address.
Added enum class RenderServices::TraceOpt::LLVMMemberIndex to correlate C++ data members with LLVM type generated by BatchedBackendLLVM::llvm_type_batched_trace_options()
Fixed bug in BatchedBackendLLVM::llvm_load_arg where array types where only properly broadcasting uniform to varying arguments for the 0th element of the array.
For batched implemented trace options as a LLVM type with direct code gen to alloca, init default values, and set optional values vs. callbacks. Same code could be resused for non-batched impl.
Added BatchedMessageBuffer to context and BatchedMessageList to wide_opmessage.cpp to allow a shared buffer to be cleared by the ShadingSystem while allowing target specific compilation of SIMD import/export of message values into the list to compile in wide_opmessage.cpp.
Extended testshade's SimpleRenderer and BatchedSimpleRenderer to implement trace and getmessage to allow new tests to exercise some basic functionality.
Added new batched regression tests:
message-reg
trace-reg
Added new test: message-no-closure which is a simplied version of message with no closures
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Temporarily disable simd compilation of message import and export data to test if that is cause of crash in bleeding edge CI workflow.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Temporarily disable more simd compilation of batched message test if that is cause of crash in bleeding edge CI workflow.
Temporarily run only message-reg test for bleeding edge CI workflow
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Temporarily run only message-reg.regress.batched.opt for bleeding edge CI workflow
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Fix alignment of MessageBlock allocation so that reads/writes can be assumed aligned to Block<> data members
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Re-enable disabled simd statements from trouble shooting bleeding edge CI.
Re-neable all tests for bleeding edge workflow.
Applied clang-format to wide_opmessage.cpp
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* pybind11 2.9.0 release -- bump the vfx2022 and "latest" CI tests to
use it.
* Adjust other CI test pybind11 versions to get good coverage across all
the pybind11 releases, and to synchronize the vfx platfom tests with
the pybind11 release that was current for its respective year.
* Sync build_scripts/build_pybind11.bash with the latest in OIIO, including
bumping the default pybind11 version we download to 2.8.1.
* Opportunistic touch-up of warnings about llvm version to bring
into sync with our actual current requirements.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Among other cruft that existed to support versions 7 and 8, this lets
us completely get rid of PrePromoteLogicalOpsOnBitMasks custom pass,
which is only needed for LLVM < 8.
Also get rid of some cmake checks for LLVM versions that require
C++14 or higher -- not necessary now that C++14 is our minimum.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Add batched support for dict find, dict next, dict value.
Implementation takes advantage of uniform values and only does lookup once and broadcasts results if necessary.
When nodeid is varying, it falls back to simple loop over active data lanes performing lookups.
* Added new regression test xml-reg.
Enabled BATCHED execution of existing xml test
* Add new wide_opdictionary.cpp missed in original commit
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
Maybe somebody else is building this as a subproject or has a variable
called "CI"?
* For our CI runs, set an env variable "OSL_CI" (not just generic "CI").
* Default STOP_ON_WARNING to be off for supported releases, only on by
default for main branch where we're actively developing. Because you
want downstream users who are building supported releases to not get
tripped up when their compiler upgrades and there's a new warning.
* Make sure STOP_ON_WARNING is true for our CI runs, even for release
branches -- but now check OSL_CI, not CI, in case somebody else has
set that.
Fixes#1428
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Make PROJECT_VERSION_RELEASE_TYPE be a cache string, so individual
build configs or sites can override it.
* Make OSL_SUPPORTED_RELEASE be an overrideable option, and make
OSL_DEV_RELEASE be driven by it, rather than (possibly incorrectly)
independently set.
* Change OSL_IS_SUBPROJECT to PROJECT_IS_TOP_LEVEL (and with the opposite
sense), to match CMake 21+ behavior. And no need to set it ourselves if
using cmake 21+.
* All of the clang-format stuff should only apply if PROJECT_IS_TOP_LEVEL.
* Add a TIME_COMMANDS option to help debug performance of the build.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Add batched support for 2d texture calls, gettextureinfo.
To enable, added:
bool RenderServices::is_udim (TextureHandle *texture_handle)
TextureSystem::TextureHandle* BatchedRendererServices::resolve_udim_uniform(...,TextureSystem::TextureHandle*, float S, float T)
void BatchedRendererServices::resolve_udim(...,TextureSystem::TextureHandle*,Wide<const float> wS, Wide<const float> wT, Masked<TextureSystem::TextureHandle*> wresult)
which allows batched code gen to resolve_udim's for varying S & T and bin the resulting TextureHandle* and make calls to get_texture_info_uniform with a single resolved udim texture handle, then broadcast the results to data lanes with matching TextureHandle *'s.
Added extensive new regression tests:
gettextureinfo-reg
gettextureinfo-udim-reg
texture-opts-reg
Enabled BATCHED execution of existing tests:
gettextureinfo-udim
gettextureinfo
texture-alpha-derivs
texture-alpha
texture-blur
texture-connected-options
texture-derivs
texture-errormsg
texture-firstchannel
texture-interp
texture-missingalpha
texture-missingcolor
texture-simple
texture-smallderivs
texture-swirl
texture-udim
texture-width
texture-withderivs
texture-wrap
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Removed some commented out includes
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Add conditional compilation dependent on OIIO version to enable RendererServices::is_udim
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Fix clang-format issues
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Fix bug in testing.cmake where non-existant OPENIMAGEIO_VERSION was being checked, when it should have been OpenImageIO_VERSION.
Added quiet option to oiio-tool calls in regression tests to allow produced out.txt to match between baseline and batched runs.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Added optional silent=False argument to oiiotool in test suite, allowing us to avoid generating warnings for file already exists that was causing differences when comparing baseline/out.txt and out.txt in regression tests
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Added missing batched texture option rnd for the llvm code gen representation
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Fixed compilations issues with OIIO 2.4 by enabling "rnd" batched texture option.
Fixed some other compilation issues when OSL_DEV is defined.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Cleared up some comments around populating rnd.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Add support for batched texture3d.
Discovered mismatch between documentation, llvm_gen implementation, and oslc when passing user derivatives to texture3d.
oslc and documentation expect optional parameters "vector dpdx, vector dpdy, vector dpdz",
however llvm_gen and optexture implementation only accept "vector dpdx, vector dpdy"
and use 0 for dpdz when calling into the texture subsystem.
This commit includes changes to oslc to make the compiler match the implementation of accepting only "vector dpdx, vector dpdy".
Enable BATCHED execution of testsuite/texture3d
Add new regression test textur3d-opts-reg
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Revert oslc to make the compiler match the documentation of accepting only user inputs that include dpdz: "vector dpdx, vector dpdy, vector dpdz".
Fixed llvm_gen and optexture to properly extract user dpdz from texture3d options or initialize a Vec3(0) and pass through to texture subsystem.
Modified batched_llvm_gen and wide_optexture to properly extract user dpdz from texture3d options or initialize a Wide<Vec3>(0) and pass through to texture subsystem.
Updated texture3d-opts-reg test to pass dpdz for calls taking user derivatives.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Added batched support for environment.
Added BatchedRendererServices::environment virtual method
Added new test environment
Added new regression test environment-opt-reg
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* clang-format of batched_rendererservices.h
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Rename testsuite/environment-opts-reg to testsuite/texture-environment-opts-reg to line up with a different PR adding testsuite/texture-environment.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Removed some environment option test for "wrap" as that is an unsupported option for environment.
Added test and handling for "time" option to batched texture3d. Current choice is to just silently ignore "time" as currently supported 3d formats do not support time.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Removed redundant environment test as texture-environment covers it.
Enabled BATCHED execution of texture-environment
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Extended test timouts further for arithmetic-reg.regress.batched.opt and transform-reg.regress.batched.opt which were causing spurious CI timeouts.
Reduced some multipliers of user supplied derivs to environment tests to reduce differences in results.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
Co-authored-by: Larry Gritz <lg@larrygritz.com>
It was documnted for texture(), but not for the others, for which it
also has worked for many years.
Thanks for Alex Wells for pointing this out.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
I recently bumped the ctest per-test timeout to 300s because one of the
batched shading regression tests was occasionally taking longer than the
prior limit of 240s.
But there is a better way: Individual tests can have their timeouts
set individually via set_tests_properties(TIMEOUT), so I raised the
timeout to 300s for all tests whose names contain "-reg", also kept it
rather long (the old 240s) for the long render tests that are all named
"render-". Then for everything else, we can actually lower the default
to 60s, since no other tests should be long-running.
(For the curious, the reason these batched regression tests are so
expensive is that they test a bunch of combinations, and they first
run the test in the non-batch scalar mode, then again in batched mode,
and compare the results to ensure that the batch code never deviates
from scalar. This makes them take at least twice as long as they
ordinarily would. But also remember that these are only run for the
few CI test matrix entries where we are specifically testing batch
shading.)
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Increase testing timeout, some of the batch shading regression tests
seem to take longer than the prior 240s limit.
* Turn off OpenVDB when using the osl:2021 in combination with OpenEXR
3.x -- the version of OpenVDB in the 2021 container is too old to
understand OpenEXR 3.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Use LLVM 13 / clang 13 for the VFX Platform 2022 test.
* Add MacOS-11 test
* Poke around how to use arbitrary LLVM versions via Conan. (Works! But
I have not yet enabled it.)
Signed-off-by: Larry Gritz <lg@larrygritz.com>
OIIO master recently removed support for Field3D, which is rarely used
and the project is effectively unmaintained and isn't able to build
against the newer OpenEXR releases that we want to encourage people to
use.
So the primary purpose of this patch is to switch our unit test for
texture3d from using a field3d file (and thus would not run when using
newer OIIO) to a roughly equivalent test using openvdb (which is
supported, assuming OIIO is built in the presence of OpenVDB).
But, ha, I discovered two things, either one of which means that we
had actually not been unit testing texture3d at all lately: (1) To
discover whether the installed OIIO supported openvdb (and therefore
whether to bother running the test), we were relying on a totally
nonexistant OPENIMAGEIO_BIN cmake varaible. I don't know what that was
supposed to be (did I mean OIIOTOOL_BIN? but that also is not a
thing), but anyway, it caused it to always skip the test. (2) Oh boy,
the OIIO that we were relying on for may tests, provided by the ASWF
docker containers, was an OIIO that was not built in the presence of
OpenVDB (even though our OSL testing containers DO have OpenVDB), so
again it skipped the test because it needs that OIIO-side support for
texture3d() to know how to read openvdb files. Fix this by building
OIIO from scratch with openvdb support when doing the "VFX Platform
2022" test, rather than relying on the one in the container.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Because they can fail with some locales. Replace with OIIO::Strutil
parsing functions that are locale-independent.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Renaming the master branch to main is imminent. Here are the few
places where we refer directly to branch names that will no longer
exist.
While searching for "master/", I also noticed a couple files that
had the wrong project's copyright boilerplate on it (a lot of the
cmake scripts I write and submit to both projects simultaneously).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This is primarily for the sake of mingw, but also is the proper kind
of stream to pass to OIIO::Filesystem::open.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Use a cmake idiom so that you need an explicit -DCMAKE_INSTALL_PREFIX=
in order to write over /usr/local or any potentially sensitive area.
In the absence of this being set explicitly, it will configure the
cmake build so that an "install" ends up in ./dist (which is what the
Makefile wrapper did all along).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Added
template <int M00, int M01, int M02, int M03,
int M10, int M11, int M12, int M13,
int M20, int M21, int M22, int M23,
int M30, int M31, int M32, int M33, int DivisorT>
struct sfm::StaticMatrix44
to model a matrix whose elements are known at compile time. Internally it utilizes proxys instead of floats in order to optimize for special cases of 0, 1, -1 elements and not even generate math when possible. The StaticMatrix44 is used by wide_opspline.cpp to create specialized functions for each of the spline basis types. A dispatch function routes the compile time optimized spline or spline inverse routine.
Altered splineimpl.h to provide "int basis_type_of(StringParam basis_name)" to allow the batched implementation to map a string to spline enum value used as an index in the dispatch function to functions optimized for each type of spline basis (instead of generic 4x4 matrix implementation)
Added "T sfm::select_val(bool cond, const T left, const T right)" as alternative to the ternary operator which appeared to possibly take address of its arguments which prevented privatization of those variables when compiled inside a SIMD loop.
Added note to BatchedAnalysis implementation identifying that local variables scope is lost between .osl and .oso and when a function is inlined multiple times, its local variables could be treated as varying when they could have been uniform if somehow identified as a local variable by the Analyzer.
Enable BATCHED testing of:
spline-boundarybug
spline-derivbug
spline-nonuniformknots
spline
splineinverse-ident
splineinverse
Added new BATCHED_REGRESSION tests:
spline-reg
splineinverse-knots-ascend-reg
splineinverse-knots-descend-reg
Added sfm::Inverter to remove nested loop behavior of batched splineinverse as OIIO::invert has a loop which was called from inside the search interval loop, created a nested loop. Under SIMD/SIMT the nested loop could execute incoherently. Instead we choose to use the class sfm::Inverter to manage state between is_bracketed_by(min,max), meant to be called from the search interval loop, and bracketed_invert(min,max) meant to be called after exiting the search interval loop, avoiding a nested loop that could be executed at different interval's for each SIMD/SIMT lane/thread.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
Patrick Hodoul pointed out that we were forcing OpenColorIO to
initialize and read configs as soon as the ShadingSystem was
created. In this patch, we're changing to a lazy creation approach.
We remove the `OCIOColorSystem m_ocio_system` member and the
ocio_transform() method from ShadingSystem. Put them in the
ShadingContext instead. This allows lock-free access because each
context is used by only one thread at a time.
Within the OCIOColorSystem, make the `OIIO::ColorConfig` be a
shared_ptr rather than a member, so it can be created lazily. If the
Context's OCIOColorSystem needs a config and doesn't have its own yet,
it asks the ShadingSystem, which allocates one the first time (and
only that one time).
So, in short, SS allocates *one* shared ColorConfig, the first time
any context asks for one. The Context only asks the SS for this once
per context (i.e., once per thread). Individual shades just use the
ColorConfig of their context, not needing to lock for it (except for
that brief lock the first time any context asks for the very first
ColorConfig that's needed).
Some things were guarded by `OIIO_HAS_COLORPROCESSOR`, which is silly
because we no longer support OIIO versions so old that it's not
defined. But then OSL `undef`ed it for Cuda. I replaced these guards
by a more straightforward direct test of whether it was Cuda code.
I added a sample OCIO config to testsuite so we can use it for tests.
In particular, the new testsuite/ocio -- test using OCIO for color
transforms.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Fixes 1416
In OSL, function parameters must be marked as `output` or they are not
supposed to be writable. In issuing errors when a write is performed on
something not writable, we missed an interesting case:
```
void g(output float b) {
b = 1; // ok, since b is marked as output
}
void f(float a) {
g(a); // <--- should be disallowed, g is writing to f's non-output param
}
float x;
f(x);
```
This patch catches cases like these. The new error looks like this:
`test.osl:22: error: cannot write to non-output parameter "a", as passed to function g() as arg 1 (output parameter "b")`
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Add batched SIMD implementation for all color operations.
Added batched llvm code gen for: llvm_gen_blackbody, llvm_gen_luminance, llvm_gen_transformc (including code gen of loop to identify unique "from" and "to" space combinations to call library function with).
Made ColorSystem const correct.
Pulled much of ColorSystem's implementation into src/liboslexec/opcolor_impl.h so it could be inlined in both opcolor.cpp and wide_opcolor.cpp.
To enable SIMD fast path for ColorSystem::blackbody_rgb added new methods to expose underlying lookup table optimization.
bool can_lookup_blackbody(float T /*Kelvin*/) const;
Color3 lookup_blackbody_rgb (float T /*Kelvin*/) const;
Color3 compute_blackbody_rgb (float T /*Kelvin*/) const;
When can_lookup_blackbody() returns true
then lookup_blackbody_rgb can be safely called,
otherwise the expensive compute_blackbody_rgb is required.
Updated implementation of ColorSystem to succesfully vectorize on multiple compilers, optimized lookup table data types and indexed dereferences to generate prefered 32bit indexed gathers vs 64bit gathers which require multiple instructions and registers.
To improve code generation and less chances of aliasing issues, changed nested M[i][j] operator access to Matrix in ImathMatrix.h to directly access underlying 2d array M.x[i][j] also changed Vector operator V[i] to directly access V.x, V.y, V.z in OSL/IMathx/IMath*.h
Moved helper testIfAnyLaneIsNonZero from wide_opmatrix.cpp to be testIfAnyLaneIsNonZero in OSL/wide.h
Added sfm::min_val to provide an implementation of min which returns values not references to the original through a ternary (which can cause issues with vectorization). Added Clang specific sfm::min_val and sfm::max_val implementations to assist in vectorization.
Enabled BATCHED execution of existing testsuites: blackbody, color, transformc, wavelength_color
Added new testsuite BATCHED regression tests: blackbody-reg, color-reg, luminance-reg, transformc-reg, wavelength_color-reg
Added sfm::min_val to provide an implementation of min which returns values not references to the original through a ternary (which can cause issues with vectorization). Added Clang specific sfm::min_val and sfm::max_val implementations to assist in vectorization.
* Adopt OIIO calls to fast_cbrt and fast_expm1 for blackbody computation.
Use single precision for bb_spectrum object/operator and manually implement std::pow(wlm,-5.0f) to better enable vectorization.
Reduced masked code generation for rgb_to_hsl.
Removed duplicated code by implementing ColorSystem::blackbody_rgb(T) in terms of ColorSystem::can_lookup_blackbody(T), ColorSystem::lookup_blackbody_rgb(T), and ColorSystem::compute_blackbody_rgb(T). Removed some casting to uint16_t that was unnecessary.
Removed some test shaders that were not intended to be promoted.
* Implement batched llvm code generation for ops: andor, bitwise_binary_op, clamp, get_simple_SG_field, isconstant, select, unary_op, mix
Fixed bug in BatchedAnalysis where the complement operator was being treated as always having a boolean result. In reality the result of complement or other bitwise operation would only be boolean if its input parameters were forced to be boolean (left TODO note for future improvement).
Fixed bug in printf where integer who is forced_llvm_bool() is formatted as a float was not converted to an integer first.
Enabled BATCHED for execution for tests: and-or-not-synonyms, isconstant, logic, select.
Expanded testsuite/shaderglobals to exercise/access all shader global data members.
Add regression tests: andor-reg, bitwise-and-reg bitwise-or-reg bitwise-shl-reg
bitwise-shr-reg bitwise-xor-reg, complement-reg, mix-reg, select-reg
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
Remember that our current 'master' is really aiming for a release in
late 2021 at the earliest, and thus will be a version primarily
intended to be used widely in 2022. Release branches never drop
supported dependencies, so the 1.11 branch will continue to support
OpenEXR all the way back to 2.0.
VFX Platform guidance is to support the current year, plus up to 3 years
prior, so that means that for 2022-ish software, we should support back
to 2019 dependency ecosystem.
For OpenEXR, 2019 state of the art was 2.3 -- that's still dreadfully
old, and is no longer getting even critical or security bug fixes. But
it still means that we can comfortably upgrade our advertised minimum
OpenEXR from 2.0+ to 2.3+ and eliminate some build and dependency
logic that was dedicated to dealing with 2.0..2.2.
We're also raising the OpenEXR/Imath "recommended minimum" (build-time
warning, but not rejection) to 2.4. Even that is shaky, but the fact
that it's still in wide use and was officially (though IMHO unwisely)
the VFX Platform 2021 recommendation, means we can't yell at people
too much about it.
It is possible that by the time we make a supported OSL 1.12 release,
we will bump even further to OpenEXR 2.4 as the minimum. We'll make
that decision later. It would depend on our being confident that even
though 2.3 is technically within the 3 year window we try to support,
we cannot discern that anybody is actually using it (who would also
need the most recent OSL in the same app).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Added batched llvm code gen for: llvm_gen_blackbody, llvm_gen_luminance, llvm_gen_transformc (including code gen of loop to identify unique "from" and "to" space combinations to call library function with).
Made ColorSystem const correct.
Pulled much of ColorSystem's implementation into src/liboslexec/opcolor_impl.h so it could be inlined in both opcolor.cpp and wide_opcolor.cpp.
To enable SIMD fast path for ColorSystem::blackbody_rgb added new methods to expose underlying lookup table optimization.
bool can_lookup_blackbody(float T /*Kelvin*/) const;
Color3 lookup_blackbody_rgb (float T /*Kelvin*/) const;
Color3 compute_blackbody_rgb (float T /*Kelvin*/) const;
When can_lookup_blackbody() returns true
then lookup_blackbody_rgb can be safely called,
otherwise the expensive compute_blackbody_rgb is required.
Updated implementation of ColorSystem to succesfully vectorize on multiple compilers, optimized lookup table data types and indexed dereferences to generate prefered 32bit indexed gathers vs 64bit gathers which require multiple instructions and registers.
To improve code generation and less chances of aliasing issues, changed nested M[i][j] operator access to Matrix in ImathMatrix.h to directly access underlying 2d array M.x[i][j] also changed Vector operator V[i] to directly access V.x, V.y, V.z in OSL/IMathx/IMath*.h
Moved helper testIfAnyLaneIsNonZero from wide_opmatrix.cpp to be testIfAnyLaneIsNonZero in OSL/wide.h
Added sfm::min_val to provide an implementation of min which returns values not references to the original through a ternary (which can cause issues with vectorization). Added Clang specific sfm::min_val and sfm::max_val implementations to assist in vectorization.
Enabled BATCHED execution of existing testsuites: blackbody, color, transformc, wavelength_color
Added new testsuite BATCHED regression tests: blackbody-reg, color-reg, luminance-reg, transformc-reg, wavelength_color-reg
Added sfm::min_val to provide an implementation of min which returns values not references to the original through a ternary (which can cause issues with vectorization). Added Clang specific sfm::min_val and sfm::max_val implementations to assist in vectorization.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Adopt OIIO calls to fast_cbrt and fast_expm1 for blackbody computation.
Use single precision for bb_spectrum object/operator and manually implement std::pow(wlm,-5.0f) to better enable vectorization.
Reduced masked code generation for rgb_to_hsl.
Removed duplicated code by implementing ColorSystem::blackbody_rgb(T) in terms of ColorSystem::can_lookup_blackbody(T), ColorSystem::lookup_blackbody_rgb(T), and ColorSystem::compute_blackbody_rgb(T). Removed some casting to uint16_t that was unnecessary.
Removed some test shaders that were not intended to be promoted.
* Added additional test result for black body because the change from double to single precision caused a few results to differ slightly (max delta 0.0000534058)
* Incorporated changes from PR to "fix color conversion from RGB to xyY"
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
This change now requires renderers to manually link in the compiled rend_lib.cu
PTX code when using OptiX 7.
Signed-off-by: Clifford Stein <cstein@imageworks.com>
Changed the prototype of osl_get_attribute callback in liboslexec in
order it takes a copy of the attribute TypeDesc casted into a long long
instead of taking a pointer to it. This allows to make it work properly
on GPU else we have an invalid pointer.
Signed-off-by: nicolas <nicolas@isotropix.com>
Fix problem with unresolved symbols when multiple shaders call certain (#1401)functions on the GPU (ie. osl_printf()).
This fixes a problem I was seeingon the GPU: if I had two shaders that contained a printf(), then the second shader group would report osl_printf as being an unresolved function (the first shader group was fine). If I looked at the PTX for the second shader group, osl_printf was tagged as a "weak" function. Removing the "weak" attribute resolves the problem.
I'm not sure why osl_printf and rend_get_userdata should be different from the other functions in rend_lib.cu.
This is easily reproduced in testsuite/testoptix by adding a printf() to checkerboard.osl and microfacet_test.osl.
Signed-off-by: Clifford Stein <cstein@imageworks.com>
Added clang pragma and gcc attribute to disable Fused Multiply Add for nonAffineInverse to allow offline compilers to produce results closer to those of LLVM IR compilation (which has FMA disabled by default) which produced the reference images of the testsuite.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
Newly added functionality in TextureSystem (only in OIIO >= 2.3.7)
allows us to add a new flavor of gettextureinfo that takes texture
coordinates, and in the case of UDIM textures, will retrieve the info
for the particular UDIM texture that will be used for the given
position:
int gettextureinfo (string texturename, string paramname,
float s, float t, output TYPE destination)
NOTE: Use with caution, this will be more expensive than ordinary
calls to gettextureinfo that will tend to constant-fold effectively,
since the s,t coordinates are runtime-determined and won't be known
for runtime optimization.
If built against older versions of TextureSystem that lack support for
the new is_udim() and udim_resolve() methods, this new gettextureinfo
will simply use the old gettextureinfo() without coordinates, and thus
will not be able to retrieve position-specific texture info.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Most important: make sure we have a proper VFXPlatfom-2022 like test,
with Python 3.9 and OpenEXR/Imath 3.1.
Fix a few of the "name" entries for tests to match the set of versions
actually present on those images.
Remove apt-get install of libboost-regex-dev, since we use only
std::regex now that we're no longer using gcc4.8.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Similar to a recent change in OIIO... For historical reasons, our
wrapper makefiles assume that we build in `build/ARCH[.variant]`, so
maybe build/linux64, build/linux64.debug, etc.
Get rid of this extra bit of hierarchy, it no longer serves us and
it's contrary to modern cmake practice.
Also used this as an excuse to remove some SPI build script
customization that we no longer use.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This is analogous to the "output placement" that we added recently, but
for input of userdata.
Reminder: "userdata" means shader parameters marked as lockgeom=0 that
are expected to be supplied on a per-shade basis, such as interpolated
vertex data on geometric primitives. The primitive data binds by name to
matching shader parameters.
This patch concerns how the shader gets this per-point data.
Old school: in the middle of the shader, when userdata is needed, the
shader calls osl_bind_interpolated_param(), which makes a callback to
RendererServices::get_userdata(), which... does something magical and
renderer-specific to retrieve the data by name. This is expensive and
also challenging to implement on the GPU (which is why it remains
unimplemented).
This patch changes the methodology to the following, which is somewhat
analogous to how we did output placement:
0. It is assumed that there is an arena of memory where the userdata
is going to be, and the pointer to its start is passed down to the
shader via execute() (just like how the shaderglobals and output
arena pointers are passed), along with an index of the current
shade point.
1. The renderer pre-declares SymLocationDesc records for the userdata,
which specifies the offsets and strides within the arena where the
data can be found.
2. Prior to launching the shader execute(), the renderer is expected
fill in those values in the userdata arena for the points to be
shaded.
3. When JITing the part of the shader that initializes those userdata
parameter values, we simply JIT a simple data copy directly from
the computed position within the arena (offsetting the right amount
for this data, at this shade index) -- just a couple of
instructions per shade, no function calls, no RendererServices
callbacks, no name lookups!
4. There is no 4. That's it.
See the changes to osldeformer.cpp for an example of how this looks from
the renderer side. It's really straightforward.
Currently, if there is no SymLocationDesc for a lockgeom=0 parameter
that needs initialization, it will fall back to the old callback. We
may fully deprecate the old way, but not for a while, so this is back
compatible for now.
This seems to work, passes all tests, doesn't break anything, and works
for the osldeformer example!
There is follow-up work to do: I think we can pretty easily do the
same thing for ShaderGlobals ("make your own SG struct"), named marix
retrieval, and other attributes. Will do those in subsequent PRs if
this meets everybody's approval.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Well intentioned attempt to ensure that Cuda runtime library was in
the library path when tests were run, but it clobbered the existing
LD_LIBRARY_PATH.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Add batched code gen to generate a loop to calls to OSL library noise functions with binning of varying noisetypes and options. For example if half of a SIMD batch uses gabor option "bandwidth" of 10 and the other half default "bandwidth" of 1, then gabornoise will be called twice (once with bandwith option set to 10 and a mask with the 1st half of SIMD lanes enabled, then a 2nd time with bandwith option set to 1 and a mask with the 2nd half of SIMD lanes enabled.
Added batched OSL noise library functions which compile with outer loop explicit vectorization. NOTE: Perlin and some other noises have SIMD intrinsics that work on multiple data elements (block vectorization). When the the active number of lanes in a batch is below a tunable threshold, the legacy block vectorization version is called. This avoids the worst case scenarios of 1 active SIMD lane executing, and allows the 3-4 wide x,y,z data element version to run instead.
Add previosly omitted /Qopenmp-simd and -fopenmp compiler flags along with -DOSL_OPENMP_SIMD to cause OpenMP outer loop explicit SIMD vectorization of loops in batched OSL libraries.
Fix Mask::invoke_foreach to be OSL_NOINLINE, it was mistakenly set to OSL_FORCEINLINE
Update oslnoise.h to access target specific library sfm namespace
Improve compile time for 4d perlin noise with Dual arguments by using a 2 iteration loops to avoid repeating a large amount of code.
Changed ShadingSystemImpl::count_noise(void) to count_noise (int number=1) to allow batched version to update the call count with the number of active lanes in the current mask.
Fixed sfm::scalar_gabor3 and sfm::scalar_pgabor3 to not use the loop array index when storing partial results, when vectorizing an aliasing issue on the stack caused incorrect results.
Enabled BATCHED execution of all noise tests.
Added new tests: noise-generic, pnoise-generic
Added new BATCHED_REGRESSION tests: noise-gabor-reg, noise-reg, pnoise-reg
Avoid issue with constant folding in hashnoise with different compilers causing slightly different results from sin & cos to cause massively different results due to aliasing in hashnoise.
* Add openmp-simd and target compiler flags when CMAKE_COMPILER_IS_GNUCC for batched target specific liboslexec libraries
Move filter_gabor_kernel_2d from gabornoise.h to .cpp to avoid unused function warning.
* Revert to reference 4d perlin noise for Clang vectorization.
* Utilize reference filter_gabor_kernel_2d which uses its own gabor_mul_m22_v2(const Matrix22& m, const Vec2& v) instead of relying on Imath version which differs in naming between OSL/matrix22.h and <Imath/ImathMatrix.h>
Move filter_gabor_kernel_2d back from gabornoise.cpp to .h to share it between gabornoise.cpp and sfm_gabornoise.h.
Disable Clang vectorization of strlen because resulting binary causes crashes in testsuite (TODO: investigate)
* To avoid "may be used uninitialized in this function [-Werror=maybe-uninitialized]" messages add extra (unnecessary) initialization, as its a unlikely fallback case shouldn't affect performance much.
* Disabled GCC optimizations for nonAffineInverse to allow results to match LLVM-IR version for Matrix44 inversion.
Implemented better optimization for transform normal when matrix is uniform.
* Mark nonAffineInverse definition as OSL_HOSTDEVICE to match its forward declaration.
* Added environment variable TESTSHADE_LLVM_JIT_FMA to force FMA generation for JIT'd code, needed to produce reference i mages for tests.
Fixed warning message that was accessing a null pointer when testshade requests FMA without specifiying a target.
Added out_LLVM_JIT_FMA.tif image for hashnoise tests whose results vastly differ for small changes in input values, so we need a 2nd test image to compare against when FMA is enabled for LLVM JIT.
* Rearranged local stack variables inside of sfm::simplexnoise4 to instantiate Vec4 locals closer where they are used. This was necessary to get correct results with gcc (even with vectorization disabled), suspect issue with register overflow or stack stomping.
* Fix issue with impl_transform_normal_masked failing to vectorize when only the result is varying.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
OIIO 2.3 will rearrange some header files slightly. If you need to use
ImageCache, you must include imagecache.h (duh!). We were accidentally
relying on some leaked definitions in other header files, which will
soon be removed.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Implement batched versions algebraic, trigonometric, and transcendental OSL library functions.
Add extensive regression tests to exercise binary and ternary operations with combinations of uniform, varying, and constant arguments.
Added some reusable xmacros for shaders in the testsuite/common/shader's directory to create unary, binary, and ternary test shaders.
Increase timeout of CI tests from 180 seconds to 240 seconds to allow comprehensive regression tests to complete.
Alternative is to break regression tests into separate test directories so the individually finish faster.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
This feature (used, very possibly, only by me, for debugging) lets me
force OSL to write an archive that contains the oso files and a
serialized version of the shader network.
If no explicit name is given (by the "archive_filename" attribute), it
picks a name based on the shader group name. But that may not be a
safe string for a valid filename. We already eliminated anything
before the last slash, but it turns out that a colon (':') and pipe ('|')
can also really mess things up. So this small change protects against that.
For the actual commands sent to system(), escape special characters
in the strings and enclose any arguments that come from the user in
double quotes. That prevents shenanigans like asking for an archive
filename called "; rm -r *". By enclosing in double quotes, we'll
end up with a badly named filename produced by tar, rather than
possibly an arbitrary command being executed.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Background: OIIO now has errorf that are explicitly printf-like, and
errorfmt that are explicitly std::format-like, and also an old style
error() that is ambiguous. The ambiguous calls are (at this moment)
aliases to the printf ones, but will be deprecated at some point (and
may, later when it seems safe, get reborn as the std::format compliant
behavior).
This roots out and removes the last few calls to the old ambiguous
ones from the OSL codebase.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Update the "bleeding edge" CI test to gcc-11, and also LLVM 12,
which we hadn't tested before.
* A "latest releases" CI test uses gcc10 and LLVM 11.
* Turns out that nothing else on the OSL side needed to be fixed to
accommodate gcc 11 (though you need a very up-to-date OIIO if you
want to compile it with gcc11 unless you turn warnings off).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Implement batched matrix ops, including transform, determinant, transpose, and construction and getmatrix with spaces, multiplication, division, negation.
Moved helpers det2x2, det3x3, and det4x4 from opmatrix.cpp to Imathx.h so they could be shared with the wide_opmatrix.cpp.
Enabled BATCHED testing for matrix and transform testsuites
Added new BATCHED_REGRESSION testsuites array-reg, matrix-arithmetic-reg, matrix-compref-reg, matrix-reg, transform-reg to more fully exercise combinations of uniform and varying data types and indexing variables.
* Removed #if 0 code block of determinant match that was moved to Imathx.h
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
As discussed at the TSC meeting and on the mail list, I won't merge this
change until June 1. It will only be in master (future OSL 1.12), not
backported to any release branch.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Implement batched loop control flow (including, break, continue) along with early outs (return, exit). Also add batched support for printf and other string operations. Implement batched support for shadingsystem ops like bind_interpolated_param, get_userdata, rangecheck, uninitcheck_values, and nancheck. Implement batched support for negate, min/max, arraycopy, compassign, mxcompref, mxcompassign, .Added BATCHED_REGRESSION tests to the suite to exercise control flow and string ops and compare scaler and batched results. Enabled various tests to also execute in BATCHED mode. Enable unoptimized BATCHED mode testing, as constfolding was avoiding any work in optimized mode for some tests.
Fixed bug in testshaded batched mode that inverted population of x and y shader global values, but only noticed in order of side effects like printf in shaders.
Added buffered file support to ShadingSystem to allow batched fprint to write out to files in the correct order.
Fixed bug in BatchedAnalysis which was not handling unlocked geom correctly by requiring unlocked input parameters be varying inorder to accept Wide results from BatchedRendererServices::get_userdata.
LLVM_Util had previously added support managing masks, and its strategy should
be reviewed before trying to understand this commit's changes:
// Overview of masking strategy
//
// For if/then based on a conditional, the result of the conditional is
// conditionals written to a symbol that is a wide boolean. To handle
// nesting we will keep a stack of conditional masks. When a new
// conditional mask is pushed, it is combined with the previous mask on
// the top of the stack. It is the top of the mask stack that is used
// as the mask for any instructions requiring masking. When a
// conditional scope is exited, we just pop the top of the conditional
// mask stack.
//
// Loops are handled by pushing their conditional result onto the
// conditional mask stack, and popping it back off at the end of the
// loop.
//
// Handling of early outs (exit, return, break, continue) is done by
// storing a mask representing the lanes which are going to be
// deactivated (or continue executing) to an allocated location in the
// stack. We actually have to store these early out masks, so that we
// can apply it to the conditional mask stack. As that stack unwinds,
// any early out masks are applied to the top of the conditional mask
// stack. As there are different types of early outs that apply to
// different control flow scopes and multiple types can be present at
// the same time for different data lanes, we must store these early
// out masks at separate locations on the stack.
//
// To handle 'exit', we allocate a 'shader mask' that is initially
// populated with a startMaskValue representing the active lanes in a
// shader global batch. For example perhaps only 7 of 16 data lanes
// are populated, the startMaskValue would store that value and push it
// onto the conditional mask stack. We require a location on the stack
// for this 'shader mask' in order for an 'exit' statement to modify it
// and cause a subset of the lanes to be deactivated. If an exit was
// processed, the 'shader mask' is where it will be stored and pulled
// from when applying to the unwinding conditional mask stack.
//
// To handle 'return', we allocate a 'function mask' that is initially
// populated with a startMaskValue representing the active lanes when
// an inlined function was called. This startMaskValue should just be
// value at the top of the conditional mask stack. As function calls
// can be nested, we maintain a stack of function masks. We require a
// location on the stack for each 'function mask' in order for a
// 'return' statement to modify it and cause a subset of the lanes to
// be deactivated. If an return was processed, the 'return mask' is
// where it will be stored and pulled from when applying to the
// unwinding conditional mask stack. When a function scope is exited,
// the function mask is popped so any return's processed would apply to
// the outer function mask vs. the one just exited.
//
// To handle 'break', we use the condition symbol's location on the
// stack to directly disable data lanes for the loop execution. As we
// already had a location to store the conditional symbol's mask, no
// need to allocate or store a separate one. However code will need to
// reload the condition's value when a break 'could have been' called
// vs. relying on a conditional result already in a register.
//
// To handle 'continue', we allocate a 'continue mask' that is
// initially populated with a 0 representing the deactived lanes by
// 'continue' operations. We require a location on the stack for each
// 'continue mask' in order for a 'continue' statement to modify it and
// cause a subset of the lanes to be deactivated. If an continue was
// processed, the 'continue mask' is where it will be stored and pulled
// from when applying to the unwinding conditional mask stack. When a
// loop's scope is exited, the continue mask is popped so it would't be
// applied to an outer loop.
//
// The logic of when and how to actually apply these masks is handled
// by the caller, tracking/storing the different stacks and masks is
// handled here. And queries exist to see how many early outs
// operations have been processed, these counts can be used by the
// caller to determine which to generate code to apply each of them.
In our case BatchedBackendLLVM is the caller and during code gen will be calling:
push_shader_instance(llvm::Value* startMaskValue) and pop_shader_instance()
push_function_mask(llvm::Value* startMaskValue) and pop_function_mask()
push_masked_loop(llvm::Value* location_of_control_mask, llvm::Value* location_of_continue_mask) and pop_masked_loop()
And to handle if/then/else be calling
push_mask(llvm::Value *mask, bool negate, bool absolute) and pop_mask()
By checking masked_exit_count(), masked_return_count(), masked_break_count(), and masked_continue_count()
before and after code gen of a function call, then block, else block, loop body, etc.
we can detect if a exit, return, break or continue existed in the generated scope
and then can call apply_exit_to_mask_stack(), apply_return_to_mask_stack(),
apply_break_to_mask_stack(), or apply_continue_to_mask_stack() when needed.
Add utility function "foreach_unique" to provide scalar execution of a Functor
for each unique value in the a Wide data block along with a mask identify which
data lanes match the unique value.
Add utility function "unproxy(const T &elem)" to extract value type from a proxied
element. This is alternative to static_cast<ValueType>(elem), and better self documents.
Fixed compatibility bug where llvm 9+ asserted that bool was not a valid datatype for llvm::ConstantDataVector::getSplat.
Added LLVM_Util::test_mask_lane(llvm::Value *mask, llvm::Value * lane_index) to test a mask with a variable lane index.
* Updated OIIO::vformat calls to OIIO::sprintf.
Conditionally compile buffered_file_output_mutex to avoid unused variable warning when not building with USE_BATCHED.
Ran couple files through clang-format.
* Initialize local variable mask_slot to avoid maybe-uninitialized errors
* Fixed paths to additional oiiodiff that runs when a test fails to find a match to use the /baseline subdir when performing a BATCHED_REGRESSION test.
Improved control-flow-test to use 0.5 default value instead of 0 to tell difference between actual assignment and 0's on the heap.
* Changed some control-flow-reg tests to use non-zero default values to tell difference between uninitialized heap and setting of default values.
Changed order of parameters so that any initops that contain an "exit" are the final parameter so the exit doesn't skip assigning default values. Otherwise those parameters would be uninitialized data and cause intermitent CI failures.
* Deleted extraneious cmakefile settings for clang
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* "SOVERSION" option was set twice, oops!
* Stray "OIIO" in a name that should have been this project's name.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This PR adds closure declarations for the MaterialX PBS nodes. A discussion of their design has been ongoing in a thread at the MaterialX GitHub repo. The full history of this discussion can be found here: https://github.com/materialx/MaterialX/pull/614
Signed-off-by: Niklas Harrysson <niklas.harrysson@gmail.com>
This fixes CI in the Mac test. Something about the Homebrew upgrade to llvm 12.0.0_1 made this fix necessary to build.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Up until now, client apps declare which output variables are "renderer
outputs" before shader groups are compiled, and also after a shader is
executed, it is up to the app to extract the individual output values
and do with them what it needs to do. Those outputs live on the shader
heap (the temp storage used to run the shader), and information about
where it lives on the heap can be retrieved via find_symbol() and
symbol_address().
New paradigm:
Various inputs and outputs of shaders live in different memory arenas,
and for those inputs/outputs, the app declares (prior to shader group
compilation) which arena it lives in, its offset within the arena, and
the stride between entries (for different shades, such as different
rays in flight, different points on a mesh, pixels in an image, etc.).
The shader JITs into code that automatically pull inputs from their
sources and copy outputs to their destinations with no need for client
app code to call find_symbol or symbol_address or do any extra data
copies.
This symbol location data is communicated to the shading system through
new calls to add_symlocs() passing a new struct called a SymLocationDesc,
which has the aformentioned arena, offset, stride, etc.
Right now, the only supported arena is Outputs, and the actual call
signature to execute(), and to the shader JIT code itself, includes
extra parameters for the base pointer to the output arena (this could
conceivably change from shade to shade) and the integer shade index of
this shade.
I've tried to set this up in a general way, so that additional arenas
will be added later -- places where ShaderGlobals can be customized,
UserData, etc. Also there are placeholders for being able to talk about
absolute addreses and offsets into the heap, though I don't use them
at the moment. Some of the flexibility may be later removed as we learn
more about how we want to use this.
To show it in action, I have modified the "osldeformer" example to use
this new method to place the outputs directly where we want them.
It's just storing results into an array. The output arena base pointer
is the start of the array, the offset of the single output is 0, and
the stride is the size of each array element (one 3-float point).
As a much more complex example, testshade also uses the new technique
(unless you use --no-output-placement, but I'm not sure how long I'll
keep that). This is a good example of storing to images. Testshade thinks
of itself as shading pixels in an image, so there is an ImageBuf for every
renderer output (think of it like an AOV). The output arena base pointer
is the address of the (0,0) pixel of the ImageBuf for the first AOV; the
"offset" of each output variable is the address difference between that
AOV's first ImageBuf pixel and the arena base pointer, the stride is the
pixel data size for each output, and the shadeindex is the pixel index of
that shade.
It all works. Crazy!
OK, not quite all. Here's what hasn't been done yet:
* testrender does not yet use the new API (only testshade)
* GPU (have only tried CPU so far)
* Batch shading -- I don't want to mess with this while Alex is
still integrating all his work. We'll go back and catch the batch
shading up after he's all done.
These will come in subsequent PRs.
All the old calls should still work, so this ought not to be a source
compatibility break for renderers using OSL. Not at this point, anyway.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Left out some imath targets in liboslnoise
* Conform a couple libraries to conventions of the ${local_lib} notation
we use in the othe rlibraries.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This leaves the minimum C++ at 11, but bumps the default (if not set by
the user via CMAKE_CXX_STANDARD) from 11 to 14.
This is meant to start identifying any problems people have switching
to 14. We anticipate a hard switch to 14 minimum before the next OSL
major release. This change is in master only (i.e. future OSL), we would
never change minimum or default C++ standard in the supported release
branches (currently OSL 1.11.x).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Introduce cmake option USE_BATCHED to enabling building batched SIMD shader execution for a comma separated list of batch sizes, ISA's, and noFMA: (0, b8_AVX, b8_AVX2, b8_AVX2_noFMA, b8_AVX512, b8_AVX512_noFMA, b16_AVX512, b16_AVX512_noFMA)
When USE_BATCHED is set, -DOSL_USE_BATCHED=1 will be passed allowing for source files to conditionally compile in batched execution support.
When USE_BATCHED is set, cmake variable BUILD_BATCHED=True and projects can conditionally add batched sources to be built.
Updated OSL source with #ifdef OSL_USE_BATCHED/#endif were needed.
Improved/streamlined liboslexec/CMakeLists.txt to iterate over targets in USE_BATCHED vs. hardcoding all variants.
Updated _OSL_SUPPORTS_* defines to match USE_BATCHED values simplying creating compiler definitions in Cmake.
Added new BATCHED_REGRESSION testing mode, when a file named BATCHED_REGRESSION exists in a testsuite folder, the run.py will be executed and its output files copied to "./baseline". The same run.py is then run again but with environment variable TESTSHADE_BATCHED=1, which cause testshade to execute in batched mode. Then the results/image comparison happens between the batched output files and those in "./baseline". The "./ref" dir is never used in a regression test.
Presence of BATCHED_REGRESSION prevents non-regression tests from being created for a folder.
Converted arithmetic-cov to be a BATCHED_REGRESSION test.
Removed testsuite/arithmetic-cov/ref image files.
Renamed ShadingSystem::supports_batch_execution_at(int width) to ShadingSystem::configure_batch_execution_at(int width) to better identify its behavior: based on currently set attributes for llvm_jit_target and llvm_jit_fma, test if current machine is capable of supporting batched execution at the specified width. If no specific llvm_jit_target was requested, sets llvm_jit_target and llvm_jit_fma to the supported configuration for the requested width. Returns true if supported, false otherwise
Fix bug in batched Matrix*Matrix code generation where the order of arguments were being reversed (swap was intended to transform float*Matrix to a Matrix*float function call, but accidentally happened for Matrix*Matrix)
* Moved define of OSL_USE_BATCHED to oslconfig.h.in
Fixed order of includes in batched*.cpp so oslconfig.h is higher
* Renamed testsuite/arithmetic-cov to testsuite/arithmetic-reg to better represent it is a regression test.
Changed #ifdef OSL_USE_BATCHED to #if OSL_USE_BATCHED and rely on it being set to 0 or 1 by <OSL/oslconfig.h>
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
Generating code for prepare_closure and gen_closure was crashing.
The code now matches the implementation of the removed LLVM function.
Signed-off-by: Brecht Van Lommel <brecht@blender.org>
Ever since adding OpenEXR 3.0 support, we had some cases where
installations that built the 2.x IlmBase and OpenEXR portions
separately and installing to separate areas, had trouble being found
properly.
Remove the need to include OpenEXR/OpenEXRConfig.h
Also make some minor changes to ci.yml to make sure we are documenting
the right openexr versions used for each test.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
It seems that our liberal use of cmake argument `--config` should only
be used in conjunction with the `--build` flag, not for the initial
configuration step. Apparently this was just silently ignored for a
long time, but for CMake 3.20, it is an error and breaks a lot of our
CI cases and our build_blah.bash scripts.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Added building of ISA specific shared libraries lib_b16_AVX512_oslexec, lib_b8_AVX2_oslexec, lib_b8_AVX512_oslexec, lib_b8_AVX_oslexec to house precompiled OSL library functions that execute over batches of 8 or 16 in SIMD for the ISA. Compiler flags for OpenMP simd code gen and ISA targets has been added for Intel(r) C++ Compiler (ICC) and CLang (newer versions of GCC 6+ might be possible, but untested).
Implement batched llvm code gen for: generic function calls, useparam, compare ops, addition, subtraction, multiplication, division, modulus, assignment, component reference, construct triple, construct color, derivative extraction. Stubbed out all other code gen functions with TBD asserts.
Populate OpDescriptiors with valid wide version of llvm-generating routine
Added wide_opalgebraic.cpp which uses X-macros (instead of #define like llvm_ops.cpp) to define wide(batched) versions of OSL library functions: sqrt, inversesqrt, floor, ceil, trunc, round, sign, abs, fabs, fmod, and step.
The X-macro wrappers follow a pattern of manufacturing a target specific library function name with enough parameter types embeded in its name to uniquely identify it (vs. other versions). Then it declares local Wide<T> or Masked<T> wrappers that convert any void */char * parameters to references to Block<T,WidthT> data blocks of wide SOA data. Then an explicit OpenMP simd loop iterates over the data lanes and extract a local scalar values from the Wide|Masked wrappers, then the scalar implementation of the library function is then inlined using the local scalar values. Finally the result is writen back out to the data lane inside the Wide|Masked wrapper. This paradigm allows scalar implementations to be resused inside simd loops and avoid having to use intrinsics or assembly. It also allows the same implementation to be recompiled for different target ISA's and varius Widths (8|16). The build system will create copy of each wide_*.cpp to a target and batch size specific named b(8|16)_(AVX512|AVX2|AVX)_wide*.cpp and build it with different -D__OSL_TARGET_ISA and -D__OSL_WIDTH values which inturn will manufacture uniquie function names. Sometimes scalar algorithms/functions can be refactored to provide better performance when executing inside a SIMD loop. sfmath.h (SIMD friendly math) houses these alternative math functions, although many improvements have already been moved into OIIO as they benefit (or do no harm) to scalar code gen.
Made ShadingContext remember the ShaderGroup it just optimized. This allows symbol queries without actually jitting or executing a shader.
Improved TestShade to not actually execute the shader during setup_output_images, but to instead explicitly JIT scalar or batched version of the ShaderGroup (primarily to make sure JIT happens during the "setup" stage vs. lazily later).
Fix TestShade to explicitly set the number of OIIO worker threads to avoid overhead (and debugging confusion) of OIIO thread pools being created even when "-t 1" was requested.
Modified ShadingSystem to only perform group_post_jit_cleanup (delete operations of shader group) only if both scalar and wide JIT's have occured or if RendererServices doesn't support batching. Without this changed the operations were being deleted before a batched JIT could occur.
Extended testsuite framework to look for file named "BATCHED" which causes another run of the test with TESTSHADE_BATCHED=1
Added testsuite new tests with BATCHED enabled for passing tgh shaderglobal values, and increased coverage of arithmetic tests with reference images for float, color, point, vector, normal data types along with Dx Dy results.
Added utility macros __OSL_CONCAT, __OSL_CONCAT3, ..., __OSL_CONCAT10 to be able to easily manufacture function names.
Added macro __OSL_WIDE_PVT to give each target specific library its own namespace avoiding collisions should multiple libraries be loaded.
Added sfm::negate(const T &x) with optimized implementation.
Disabled some unreferenced functions warnings for ICC and removed some unused functions from batched_analysis.cpp
Updated BatchedBackendLLVM to match behavior of BackendLLVM by configuring its LLVM_Util based on ShadingSystem attributes.
Disable clang format for X macro based building of initializer arrays to prevent clang format from reordering the #include files.
Fix control flow in factory function TargetLibraryHelper::build to not trigger assert unnecessarily.
Limit list of OSL library functions in builtindecl_wide_xmacro to just those we have implemented so far because all functions listed must exist in the target specific library for it to successfully be loaded and resolved.
Added LLVM_Util::op_zero_if(llvm::Value *cond, llvm::Value *v) which allows its implementation to work around an LLVM issue where expensive instructions to produce the value (div, sqrt, etc) are duplicated (once with a mask, once without).
Fix bug in ShadingSystem::supports_batch_execution_at where jit_fma was being accidentally negated causing rest of logic to fail.
Implement ShadingSystem::BatchedExecutor<WidthT>::jit_group
* In Cmake, changed ISA specific target libraries from SHARED to MODULE so that on OSX they will be named .so instead of .dylib.
Moved new arithmetic tests into their own subdirectory arithmetic-cov so that the existing arithmetic test can run in non-batched mode as it uses printf which is still TBD for the batched version.
Fixed bug where shader ops weren't being cleaned up unless batched jit had occured.
Fixed compilation issue with newer LLVM versions and LLVMUtil::zero_if
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
Removed use of options.PrintMachineCode since this has been removed as
of llvm 12. Converted uses of DebugLoc::get to DILocation::get since
it has been deprecated and removed as of llvm 12.
PrintMachine code will only be used if llvm is lower than version 12.
TODO: Needs to be re-added in the new way for LLVM12. (LG just doesn't
have time to figure this out right now.)
Pass adjustment as various things have been deprecated or changed
names. (LG)
Signed-off-by: Euan Haahr <euanhaahrmail2@gmail.com>
Signed-off-by: Larry Gritz <lg@larrygritz.com>
In PR 1346, we overlooked the fact that OSLQuery::init() is defined
*differently* in liboslquery and the separate copy that used to be
in liboslexec.
In this patch, we remove the oslquery-side version (and in fact
coalesce it into the constructor, the only place it was used). So now
instead of being replicated differently in the two libraries, it only
exists in oslexec. That constructor, since it takes a ShaderGroup* as
an argument, only makes sense to call if you have a full ShadingSystem
nearby.
Add ShadingSystem::oslquery(), deprecate OSLQuery(ShaderGroup*).
This eliminates the weirdness of having this constructor undefined in
liboslquery, moves the functionality to the ShadingSystem.
Because of ABI compatibility issues, we have to merely deprecate (but
not remove) the OSLQuery functionality for now.
Avoid assertions for shaders with params that are arrays of indeterminite length
Signed-off-by: Larry Gritz <lg@larrygritz.com>
liboslquery and liboslnoise exist as separate libraries for apps that
only need to query shader parameters or use OSL's noise functions,
respectively, but dont want to pull in the rest of OSL and all its
heavy dependencies (including LLVM). But liboslexec also needs that
functionality and so separately put the individual modules inside.
This was causing some problems for a tricky use case, and I can't
think of a good reason for it to be done this way now. So in this
patch I remove the redundant modules and just make liboslquery and
liboslnoise into *link dependencies* of liboslexec. None of this
should affect downstream projects, I don't think; linking against
liboslexec ought to just automatically pull in those other libs.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* strings are no longer copied to the GPU and are instead passed around
as ustring hashes. This means we no longer need to copy string pointers
CUDA/PTX code. This should result in faster compilations because
strings are defined at "compile time" instead of "runtime".
* OSL's shaders' printf() output is now handled on the host/CPU
* testshade and testrender now use a separate optixLaunch() to set up
global variables
* removal of runtime compilation of CUDA (libnvrtc dependency eliminated!)
which was used to handle the setting of global variables in OptiX 7's
CUDA programs.
Signed-off-by: Clifford Stein <cstein@imageworks.com>
* Separate the CI stages into separate steps. (As was prototyped in OIIO)
* Combine the dependency-building scripts for CentOS and Ubuntu into a
single script.
* Get rid of cruft related to the CI systems we no longer use (Appveyor,
Travis, CircleCI).
* Get rid of some needless CI verbosity
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This speeds up CI builds by 50% or more, for PRs that are minor
changes by reusing the compiled results from previous runs. Most of
the remaining time cannot be squeezed down -- it's the instantiation
of docker containers and running the testsuite itself. But the build
(including of dependencies we build from scratch) is dramatically
faster.
At the same time, remove some commented-out test matrix cases. When/if
we re-enable those tests, we can add those back.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Add implementation for BatchedBackendLLVM. BatchedBackendLLVM implements the same set of functionality as BackendLLVM does, except it's implementation and many of it's methods handle wide data type (for symbols identified as varying (is_uniform()==false). Many methods exist for (re)use by batched_llvm_gen and batched_llvm_instance.cpp. A common theme is additional "bool is_uniform" parameter to identify if the operation we are generated code for is uniform or not. We cannot rely on the Symbol itself, as an operation will be varying if just 1 of its inputs is varying. This means that when uniform symbols are loaded as arguments, they will need to be widened/broadcast to partcipate in the varying operation. Additionally "bool index_is_uniform" is used to generate gather/scatters vs. a wide load when symbols are varying. Please remember the underlying "wide" data types are Structure of Arrays (SOA) and all address calculations have been modified inside BatchedBackendLLVM to work with SOA's.
Introduce helper class FuncSpec to encapsulate building function names that include a batch width and target ISA along with uniform and varying parameters.
Introduce TargetLibraryHelper interface and factory method to construct a Concrete TargetLibraryHelper
that provides a prefex string that all function calls will be prefixed with
and correctly initialize a function map for the shading system with
the functions from the target ISA library. Intent is batch size and target ISA specific shared libaries will be built and the correct on be loaded based on host machine.
Introduce TempScope. Any calls to getOrAllocateTemp during the lifetime of a TempScope
will be associated with the latest TempScope on the stack and
when that TempScope's lifetime ends, any temp allocations will
be marked unused and will be available for reuse by the next
call to getOrAllocateTemp.
Introduce liboslexec/wide/define_opname_macros.h to define macros to consistently build OSL library function names including batch width, target isa, 0 to 5 arguments, and optionally masking.
Added builtindecl_wide_xmacro.h which is the equivalent of builtindecl.h for identifying the function names and signatures of OSL library functions that exist in batch size and target ISA specific shared libraries.
* Added missing initialization of OpDescriptor::llvmgenwide
Added friend declaration for ConcreteTargetLibraryHelper to use private base class TargetLibraryHelper
* Replaced some older copyright notices with latest one.
Moved llvm::Value* arrind assignment in batched_backendllvm.cpp from innerloop.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
* Rename CLA files so they no longer say "draft."
* Change CONTRIBUTING.mv to correctly describe the new CLA process,
remove language related to the old SPI CLAs. Also some touchups to
README.md.
* Update the project_intake.md to check of various things that we've
completed.
* Change all references to the old imageworks GitHub URLs to the new
ASWF ones. (This touches comments at the start of basically every file.)
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* readme and documentation for windows build is added
* some fixes
* remove pythonpath
* add .vscode
* windows installation document is added to new directory
* add windows build script to new doc folder
* remove .vscode folder by adding it to .gitignore
* remove .vscode manually
* update document format
* change the name of the documentation to Readme.md
* add relative image path
* image now is ok
* remove copyright problems
* remove my website name
* add appropriate copyright notices to all the files
* a little fix
* Pixar copyright is added to the python file
* Modified Apache 2.0 License is added to THIRD_PARTY.md
* remove my website from the doc
* revert changes on THIRD-PARTY.md
* Modified Apache 2.0 License is reproduced
Introduce BatchedAnalysis which processes operations similar to code gen in order to build up relationships between operations and symbols controlling conditional control flow. Using these relationships and information on which shader globals are varing and which symbols renderer outputs, varying can be pushed down to just the symbols that will require it. Also analysis to determine which symbols can be forced in llvm to be boolean vs integers.
The idea is simple, but understanding & managing the relationships is complex, however it is entirely encapsulated in BatchedAnalysis.cpp. The main goal is to build up a multimap "m_symbols_dependent_upon" which maps a key symbol to all other symbols that due to operations, conditions, loops, function calls, early out control flow (break, continue, exit) would be forced to be varying if that key symbol where varying. We can use this map to recursively push varying of symbols down from shader globals, renderer outputs, and operations whose results are implicitly varying.
Several custom data/acceleration structures have been created to efficiently model these relations ships with their key feature being that their iterators are never invalidated as additional items are pushed and popped such that one can always iterate from a historical position back to the root:
DependencyTreeTracker, FunctionTreeTracker. Additonally LoopStack tracks loop nesting and aids in detecting cyclical reads.
Analysis is also performed to identify which symbols are read at a different conditional scope than they were written to and might require masking. Symbols written to and read from within the same (or deeper nesting) conditional scope do not require masking. A conditional scope is defined as the stack of of conditions evaluated to reach an operation and is managed by the DependencyTreeTracker.
When adding new ops, or when changing implementation or behavior of
existing ops, the following functions should be evaluated to see if
they need to be updated to handle to handle the new op or change:
bool are_op_results_always_implicitly_varying(ustring opname);
bool are_op_results_always_implicitly_uniform(ustring opname);
bool does_op_implementation_require_masking(ustring opname);
SymbolPtrVec * protected_shader_globals_op_implicitly_depends_on(ShaderInstance &inst, Opcode & opcode);
If there is a new or change in control flow op's are made,
a more detailed evaluation will be needed as their behavior
is hard coded vs. lookup based implementation of the
functions above.
RuntimeOptimizer uses BatchedAnalysis before it coalesces temporaries so that varying and uniform symbols do not get merged.
* Add ShadingSystem attribute "opt_batched_analysis"(defaults to false) which triggers additional analysis during runtime optimization to set Symbol::is_uniform, Symbol::forced_llvm_bool(), and Opcode::requires_masking. Only coalesce_temps for symbols whose is_uniform() and forced_llvm_bool() values match. Update testshade to set opt_batched_analysis when batched execution is requested.
* Initialize opt_batched_analysis to true when RendererServices provides a BatchedRendererServices for WidthOf<16> or WidthOf<8>.
Changed testshade from explicitly enabling opt_batched_analysis to explicitly disabling it when not in batched execution mode.
Removed test for opt_batched_analysis from coalesce symbols pass and just rely on a Symbol's is_uniform() and forced_llvm_bool() to be set correctly.
OIIO removed "RB-2.1" branch marker because it's not expecting any
further development in the obsolete 2.1 branch. We still test against
it, though, so replace its designation in our CI with the specific
branch tag of the last OIIO 2.1 release.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Add storage to Symbol for analysis result bool forced_llvm_bool().
Results of a compare_op and other ops with logically boolean
results under certain conditions could be forced to be represented
in llvm as a boolean <i1> vs. an integer <i32>. This simplifies
code generation, and under batched execution is a requirement
to make efficient use of hardware masking registers by allowing a
vector of bools <16 x i1> vs. integers <16 x i32>. However the
underlying OIIO::TypeDesc as well as OSL does not support bools,
therefore they need to be promoted to integers when interacting with
other integer op's.
The value of forced_llvm_bool() is currently only respected during
batched execution. Forced bools should not be coalesced with regular
ints, only other forced bools.
* Nicolas Guiard is a new TSC member representing Isotropix
* Chris Kulla has had a new affiliation for a while now
* Clean up a bit of language left over from when this document was
just a proposal.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Add ShadingSystem attribute "searchpath:library" (similar to searchpath:shader) that is used to identify directories where target ISA optimized OSL shared libraries may exist.
Remove testshade command line parameters of "path" and "librarypath" as they were never used and can still be specified with "options".
During batched analysis we will be figuring out which symbols need to be varying and which can be uniform during batched execution, we will also be figuring out the minimum number of operations which require masking as well as storing/caching some other operation specific flags used later in batched code generation. We choose to store these analysis results inside Symbol and Opcode directly so that they are efficiently accessible and stable under optimization passes than might invidate OpcodeVec.
Modeled after recent similar changes in OIIO.
* Detect if OSL is being built as a subproject, and if so, set
`OSL_IS_SUBPROJECT` (a.k.a. `${PROJECT_NAME}_IS_SUBPROJECT`).
* Move all the testing logic out of the main CMakeLists.txt (and a few
testing-related macros from oiio_macros.cmake) and into a new
src/cmake/testing.cmake.
* Move all of the packaging logic out of the main CMakeLists.txt and
into a new src/cmake/packaging.cmake.
* Make all the testing and packaging inactive if this is being built
as a subproject. Also disable the clang-format target for subprojects.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Upon reading cmake docs mre carefully, it's clear that when you pass a
version to find_package (which does much of the heavy lifting inside
checked_find_package), it DOESN'T MEAN you want that version or
later. It means you want to find a version that is "compatible" with
the named version. The meaning of "compatible" is up to the package.
Some packages define it as being >= the requested version. Others
define it as being the same major release, or same major AND minor
release, or exact.
But generally speaking, we only care about the minimum version (e.g.,
"I want FooBar 8.0 or later"). And it's not guaranteed that is what
the package's compatibility will do. In fact, FooBar may be using
compatibility mode "SameMajorVersion", which will make our find of
FooBar succeed for 8.0, 8.1, 8.2, but if a user encounters FooBar 9.0,
our build will reject it, even though it's probably fine.
So I augmented our checked_find_package wrapper to take optional
VERSION_MIN to give the real minimum version, when that's the semantic
we really want (in which case we would omit the ordinary version
number used for the compatibility check and let VERSION_MIN do all the
work). There's also an optional (and usually omitted) VERSION_MAX that
you can use if you are sure that we must be less than a certain
version (the max is exclusive, you must be <, not <=, whereas the min
is inclusive >=).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Some minor refactoring of the recently introduced get_vec3 introduced
a bug into the constant folding of + and - for mixed floats and
triples: the constant produced was of the type of the first operand,
instead of the result. This fixes.
Added handy helper to make vector constant (and disambiguate it from
the one that makes a color constant).
* Add Symbol::wide_dataoffset() to track offset in heap to wide version of symbol's data
Add flag to Context to track if last execution was batched
Add template<int WidthT> ShadingContext::Batched interface to group together batched versions of execute_init, execute_layer, and execute methods.
Instantiate ShadingCOntext::Batched<8> and ShadingContext::Batched<16>.
Refactor std::vector<std::pair<ErrorHandler::ErrCode, std::string> m_buffered_errors to use proper ErrorItem class with optional Mask to identify which data lanes the Error message applies to.
Added struct ErrorBatch and std::vector<ErrorBatch> m_buffered_error_batches to track which entries in m_buffered_errors came from batched execution as those ranges need to processed multiple times, once per active data lane.
Refactored ShadingContext::process_errors into a templated helper to correlate batched and not batched errors effectively serializing the message output up to the ShadingSy
stem.
Modify ShadingContext::symbol_data to use the symbol's wide_dataoffset() when last execution was batched.
Add RunLLVMGroupFuncWide to model a a function pointer to a batched verson of a JIT'd shader function that accepts batched shader globals and an integer representing the mask (active data lanes) to execute on.
Added ShadingSystemImpl::Batched<int WidthT> interface to group together batched methods jit_group, and jit_all_groups
Instantiate ShadingSystemImpl::Batched<8> and ShadingSystemImpl::Batched<16>
Added ShaderGroup::batch_jitted to track if a group had been jitted for batching (separately from scalar jit)
Added ShaderGroup::llvm_groupdata_wide_size to track amount of heap space needed for a wide version of the group data structure.
To hold onto wide versions of llvm functions side by side with scalar, added llvm_compiled_wide_version, llvm_compiled_wide_init, llvm_compiled_wide_layer.
Add plumbing through ShadyingSystem into Context to support Batched Execution.
* Removed ErrorBatch as it was deemed unnecessary (unreachable usage scenario), which simplifies processing buffered errors.
Changed bool m_execution_is_batched to int batch_size_executed, so we know exactly how many data lanes to process errors for.
Fixed some clang-format issues in osl_pvt.h
After talking this over with Luke Emrose it seems unwise to allow
merging of layers that produce renderer outputs.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Most failures in this function returned false, but if it got as far as
parsing but failed that stage, it ended up returning true (success),
which is clearly wrong.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
As OpenEXR works toward their 3.0 release, which includes splitting
Imath (the vector, matrix, math, and 'half' stuff) out into a separate
repo, their top of tree master finally breaks our "bleeding edge"
build.
This is a little tricky, because now our support of this dependency is
attempting to span 3 eras: 3.0 with the full split of openexr from
imath, recent openexr+ilmbase 2.x that supplies reliable exported
cmake configs, and older 2.x that predates cmake configs. Eventually
we'll phase out older versions and clean up.
Fixes in this patch generally include:
* Add a new CI test of VFX2021 + openexr/imath master. (This will go away
when 3.0 is released and that's what we use for the main 2021 case.)
* For new enough versions of OpenEXR, use the exported config file and
targets, rather than the old style OPENEXR_LIBRARIES style variables.
We have kind of a mixed idiom use because we're still supporting older
versions that predate use of the config files.
* Deal with the fact that the Imath headers now live inside
`<Imath/ImathBlah.h>` rather than `<OpenEXR/ImathBlah.h>`. But of
course, you need to start including things just to find out the
version. It's tricky. The logic is to look at
`<OpenEXR/OpenEXRConfig.h>` to determine the version we're dealing
with, then include Imath stuff from one or the other depending on
what we found. I rearranged quite a few includes to ensure that all
this logic only happens in a couple spots and the rest gets included
transitively. As it turns out, quite a few of our `#include`s of
Imath headers were redundant.
* OSL's exported config needs to look for OpenEXR.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Two new preprocessor directives that oslc will respond to:
#pragma error "my message"
#pragma warning "my message"
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Added 64bit specific popcount call for GCC/Clang in Mask<int WidthT>::count()
* Implement BatchedSimpleRenderer for TestShade
* Introduce template<int WidthT> struct BatchedShaderGlobals
to store upto WidthT instances of ShaderGlobal data. The data for a batch is broken into underlying uniform and varying data structures. Uniform contains a fixed set of shader global values that are shared between all instances. Varying has Block<DataT, WidthT> members so each instance can hold a different value. The Block data structure is specialized to hold the underlying data type in a SIMD friendly Structure of Arrays (SOA) data layout but provides an array subscript interface to a proxy that can import/export the DataT in its orignal data layout.
Introduce template<int WidthT> class ShadingSystem::BatchedExecutor as an interface to execute batches of shader globals through a shader network. BatchedExecutor provides similar execute methods that accept a BatchedShaderGlobals and batchSize instead of individual ShaderGlobal(s). This is just the interface, batched backend implemention to follow in subsequent pull requests.
Introduce template<typename DataT, int WidthT> Wide to hold a reference to Block<DataT, WidthT> and provide access to the underlying SOA data layout. Wide can handle array types and const correctness. The Wide adapter is lightweight and can be copied by value, used as function parameter.
Once batched execution finishes, the resulting symbol address can be passed to a Wide<DataT, Width> to access the resulting data in a type safe manner.
Updated testshade with command line option "--batched" causing batched execution of 16 or 8 instances through the shader network and to extract the results. A later pull request will add batched interface to renderer services.
* Add new file src/include/OSL/batched_shaderglobals.h
* Disambiguate fabs call to fabs(double) so it doesn't try OSL::fabs which has no overload for double
* clang-format of batched_shaderglobals.h
* Add BatchedRenderServices, Mask<int WidthT> to represent active lanes during SIMD execution, BatchedTextureOptions, Masked<typename DataT, int WidthT>
* Removed unnecessary .c_str(), changed call to ::fabs to std::fabs
* Removed BatchedRendererServicesBases in favor of simple virtual functions is_overriden_*().
Improved some comments for BatchedRendererServices.h
Workaround some QT macro issues.
* Add BatchedRendererServices implimentation which mainly asserts that its methods are never directly called.
Fixed up issues with OSL_USING_DATA_WIDTH(WIDTH) macro.
* Removed Dz Structure Of Arrays wrappers as OSL doesn't currently support Dz.
* clang-format of mask.h and wide.h
* Added GCC and MSVC specific intrinsics for Mask<int WidthT>::count() and ::first_on()
Moved documentation for Wide, Masked, and other classes in wide.h from top overview to be near class definition.
Fixed some typos.
* [cmake] fixed a compilation error when generating bitcode for Cuda files.
Basically the problem is that Boost::thread does some compile time checks that rely (when compiling with MSVC) on the runtime lib being set (/MD switch for instance) And the command that generates the bitcode doesn't specify this, and is not compatible with the switch (clang++ doesn't support MSVC switches, clang-cl is the one that does, but clang-cl doesn't support -O3 and other switches ...)
The easiest solution is to just disable the compile checks from Boost::threads by defined the macro used to guard the checks header.
* [cmake] silenced a ton of warnings about dllimport attributes being ignored on Windows, when generating bitcodes
* [cmake] fixed MSVC's command line warning about overwriting a /W argument.
* better fix for the dllimport attribute not being supported when compiling for Cuda
- the OSL export macros now check for __CUDACC__
- added comments in the CMake to remember to restore the warnings when the 3rd parties' export macros are also fixed.
* clang-format fix in export.h
* Bug fix: optimizer interaction output param that also is userdata
There's a "pass-through parameter" idiom, where an output param gets
its initial value from an input, it gets modified by the shader, but
it's also an output.
If the modification is optimized away -- say, if it was originally
`theparam *= scale` and scale is 1 -- then it may be that you end up
with no ops in the shader that read from the parameter. If the
parameter was hooked up to userdata, then get_userdata will never be
called to retrieve the value (because it's usually lazy, only doing
the callback the first time it's needed), and so the output value of
parameter will be its parameter default value in the shader, NOT the
value of the interpolatd variable, which was never retrieved and
copied into it, and also that a shader consisting of *only* that
particular flavor of useparam is not marked as "does nothing" (thus
skipping any code generation at all).
This patch adjusts the logic somewhat so that if that output parameter
is connected downstream or is a renderer output, then we definitely
initialize it with a useparam op, even if it is otherwise apparently
not read from within the shader.
Added a new test for this specific case.
Also necessitated update to reference output of debug_uninit test,
only because the extra useparam op caused some of the instruction
numbering to change.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* testshade: allow arbitrary command line-specificed userdata
To ease testing, add `--userdata` command line option to testshade,
which constructs a ParamValueList that will be used to look up
"userdata" from the shader.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The version of Mask::first_on turned out not to have all the necessary
clauses fleshed out, breaking on Windows. So I refactored to get it
right.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
The specific combination of Mac + Python 3.8/3.9 + the lastest
pybind11 release causes a crash in the testsuite/python-oslquery
test. It works on other platforms and with older versions of pybind11.
I've tried to understand exactly what's going wrong, and have come up
empty and can't justify spending more time on it. It's suspiciously
similar to several recently "fixed" pybind11+python interaction bugs,
I suspect there are a couple edge cases they missed and that it will
be fixed eventually in a future pybind11 release (or possibly a future
python patch, there does seem to be a problem that pybind11 uncovered
in the cpython code).
In the mean time, a slight rewrite of the loop avoids the problem for
the purposes of our CI tests.
Also a few adjustments to switch back to python 3.9 after some recent
Homebrew fixes that had made us revert back to 3.8 for a while.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Added 64bit specific popcount call for GCC/Clang in Mask<int WidthT>::count()
* Introduce template<int WidthT> struct BatchedShaderGlobals
to store upto WidthT instances of ShaderGlobal data. The data for a batch is broken into underlying uniform and varying data structures. Uniform contains a fixed set of shader global values that are shared between all instances. Varying has Block<DataT, WidthT> members so each instance can hold a different value. The Block data structure is specialized to hold the underlying data type in a SIMD friendly Structure of Arrays (SOA) data layout but provides an array subscript interface to a proxy that can import/export the DataT in its orignal data layout.
Introduce template<int WidthT> class ShadingSystem::BatchedExecutor as an interface to execute batches of shader globals through a shader network. BatchedExecutor provides similar execute methods that accept a BatchedShaderGlobals and batchSize instead of individual ShaderGlobal(s). This is just the interface, batched backend implemention to follow in subsequent pull requests.
Introduce template<typename DataT, int WidthT> Wide to hold a reference to Block<DataT, WidthT> and provide access to the underlying SOA data layout. Wide can handle array types and const correctness. The Wide adapter is lightweight and can be copied by value, used as function parameter.
Once batched execution finishes, the resulting symbol address can be passed to a Wide<DataT, Width> to access the resulting data in a type safe manner.
Updated testshade with command line option "--batched" causing batched execution of 16 or 8 instances through the shader network and to extract the results. A later pull request will add batched interface to renderer services.
* Add new file src/include/OSL/batched_shaderglobals.h
* Disambiguate fabs call to fabs(double) so it doesn't try OSL::fabs which has no overload for double
* clang-format of batched_shaderglobals.h
* Add BatchedRenderServices, Mask<int WidthT> to represent active lanes during SIMD execution, BatchedTextureOptions, Masked<typename DataT, int WidthT>
* Removed unnecessary .c_str(), changed call to ::fabs to std::fabs
* Removed BatchedRendererServicesBases in favor of simple virtual functions is_overriden_*().
Improved some comments for BatchedRendererServices.h
Workaround some QT macro issues.
* Add BatchedRendererServices implimentation which mainly asserts that its methods are never directly called.
Fixed up issues with OSL_USING_DATA_WIDTH(WIDTH) macro.
* Removed Dz Structure Of Arrays wrappers as OSL doesn't currently support Dz.
* clang-format of mask.h and wide.h
* Added GCC and MSVC specific intrinsics for Mask<int WidthT>::count() and ::first_on()
Moved documentation for Wide, Masked, and other classes in wide.h from top overview to be near class definition.
Fixed some typos.
Bad cut and paste made this all wrong. Fix.
Also rename the template's AlignmentT -> Alignment to help reduce
confusion because we usually use the T end to indicate a template
parameter that's a type.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
It can't run the tests, because there is no GPU on the GHA
instance. But at least we can tell if a PR will breake the build
itself. Later we will find a way to execute these via ASWF's AWS
instances that are for this purpose.
Includes the following changes:
* Download optix headers when needed. The ASWF docker container we're
running on already has Cuda.
* Use CUDA_TARGET_ARCH CMake variable instead of hard coding in
llvm_util.cpp.
* Bump cuda target default to sm_60.
* Make CI failure artifacts include CMakeCache.txt for inspection.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Change bleeding edge CI test to use LLVM 11.
A variety of changes were necessary to the code in LLVM_Util:
* `#if` guards for differing calls related to VectorType::get(),
CreateShuffleVector.
* constant-generating functions now return llvm::Constant* instead of
llvm::Value*.
* New wide_constant(width,value) functions for when you want a width other
than the default vector_width.
* Renamed llvmVectorGet -> llvm_vector_type to be more symmetric with
other methds we have.
* Simplified places that could have made calls to constant(),
constant_bool(), wide_constant(), llvm_vector_type(), and
call_function(), but instead did more complex llvm calls by hand.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
With just a few minor adjustments, we're able to eliminate these two
headers (and everything else they pull in) from every client of
oslexec.h. For the purposes of oslexec.h's declarations, all uses are
either pointers or references, so we only need the names
forward-declared, but we don't need the implementations revealed to
everybody including oslexec.h.
In particular, this hides the RendererServices and BatchedShaderGlobals
from a lot of translation units, should cut down on compile time and
reduce exposure of certain semi-internals to only the modules that
need them.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Introduce template<int WidthT> struct BatchedShaderGlobals
to store up to WidthT instances of ShaderGlobal data. The data for a batch is broken into underlying uniform and varying data structures. Uniform contains a fixed set of shader global values that are shared between all instances. Varying has Block<DataT, WidthT> members so each instance can hold a different value. The Block data structure is specialized to hold the underlying data type in a SIMD friendly Structure of Arrays (SOA) data layout but provides an array subscript interface to a proxy that can import/export the DataT in its orignal data layout.
Introduce template<int WidthT> class ShadingSystem::BatchedExecutor as an interface to execute batches of shader globals through a shader network. BatchedExecutor provides similar execute methods that accept a BatchedShaderGlobals and batchSize instead of individual ShaderGlobal(s). This is just the interface, batched backend implemention to follow in subsequent pull requests.
Introduce template<typename DataT, int WidthT> Wide to hold a reference to Block<DataT, WidthT> and provide access to the underlying SOA data layout. Wide can handle array types and const correctness. The Wide adapter is lightweight and can be copied by value, used as function parameter.
Once batched execution finishes, the resulting symbol address can be passed to a Wide<DataT, Width> to access the resulting data in a type safe manner.
Updated testshade with command line option "--batched" causing batched execution of 16 or 8 instances through the shader network and to extract the results. A later pull request will add batched interface to renderer services.
* Add new file src/include/OSL/batched_shaderglobals.h
* Disambiguate fabs call to fabs(double) so it doesn't try OSL::fabs which has no overload for double
* clang-format of batched_shaderglobals.h
* Removed unnecessary .c_str(), changed call to ::fabs to std::fabs
* Simplify implementation of TypeSpec::is_triple, more efficient.
* Add TypeSpec::is_triple_based(), analogous to all the other
is_X/is_X_based.
* Bug fix: Symbol::get_string()'s assertion was checking whether it was
a string, but should have checked if it was string_based.
* Add Symbol::get_vec3(), analogous to get_int/float/string.
* Add Symbol::coerce_float and coerce_vec3, like get but with some type
coercions.
* Add Symbol::dataptr() as synonym for data(). This is for "forward
compatibility", a future PR will make more widespread use of this.
* Lots and lots of places were error-prone and verbose casts are replaced
with the new methods, and some logic simplified.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Repeat of https://github.com/OpenImageIO/oiio/pull/2741
One of our CI matrix entries recently started failing due to changes
in somethign on GHCI's Ubuntu image. Fix by switching this matrix
to use the appropriate ASWF docker image.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Bring back the idea of a Findpugixml.cmake file because it turns out
that (for pugixml <= 1.10) the exported config files are not reliable.
* Fix CI scripts to build PugiXML (with a few different versions). We
previously were relying on the embedded version for all CI tests, which
masked problems associated with using an external PugiXML.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Added OSLEXECPUBLIC attributes to inner classes of LLVM_Util to fix
linker errors on MSVC
* do not include cxxabi.h on non GNUC platforms
* fixed windows linker issues
- Link with Version.lib (system lib) to prevent missing symbols in
ClangDriver.lib
- Do not define OSL_EXPORTS for oslcomp/oslquery as they should import
(not export)
Co-authored-by: Dieter De Baets <dieter.baets@unity3d.com>
* OSO re-entrant parser.
* OSL re-entrant parser.
* Formatting
* Use homebrew flex & bison for OSX
* Remove old oslparse()/osoparse() extern declarations.
* Set minimum versions of Flex and Bison
Add logic to flexbison.cmake to prefer the Homebrew version if it's
found (and if no BISON_ROOT is set) because the system install in
/usr/bin is too old to properly build with the new reentrant parser
directives.
Co-authored-by: rzulak <rzulak@imageworks.com>
This is good for optimization, allowing "de-virtualization" of function
calls in cases when the compiler can be sure that there can be no further
subclassing.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This patch drops support for OIIO 2.0, making 2.1 the minimum as we
move forward on the way to OSL 1.12 (or 2.0?). The current public
supported release of OIIO is 2.2, so this means that we are still
supporting both the current and the previous year's OIIO major
releases, as is our usual policy (that is, for an OSL that won't
be an official release until some time in 2021).
This patch does not change the supported OIIO releases of prior OSL
releases. In particular, the current release family, OSL 1.11, will
continue to support OIIO 2.0. We never change minimum dependencies
during the course of a release series.
One way this becomes a big simplification is that OIIO 2.1 started
exporting cmake configurations directly, and thus we no longer need
our copy of FindOpenImageIO.cmake, and can jettison some old
conventions that it used for older OIIO releases.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Move things around -- add new SS::getattribute queries for the
dependencies, SIMD build flags and runtime capabilities, and
Cuda/OptiX versions. Then use these queries as the implementation for
reporting them in the statistics output.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
testshade --help or --runstats now print additional info about the
dependency versions, hardware capabilities enabled at build time, and
hardware capabilities detected at runtime. This should help debugging
of user experiences as an easy way to find out what OSL thinks it's
working with in terms of its environment.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Extra char prevented the "release type" from being amended onto the
version string.
Also, change the OSL_INTRO_STRING to use the preferred convention
of "Open Shading Language" rather than "OpenShadingLanguage".
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Work by Alex Wells.
When generating the code to copy one layer's outputs to a downstream
layer's inputs, we can skip the copy entirely if we already know that
the downstream layer will not be run.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
Work of Alex Wells.
Like we do already for "for" loops, when generating extra ops to test for
uninitialized values, "do-while" loops also need to skip testing of their
condition variable (which we know has not been written yet, under most
circumstances).
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
Work of Alex Wells, Intel.
Added custom optimization passes to improve AVX/AVX2 code generation
which are automatically added when targeting the host:
* PrePromoteLogicalOpsOnBitMasks
* PreventBitMasksFromBeingLiveinsToBasicBlocks
Added support for experimental llvm_optimize: 10,11,12,13. Optlevels
10, 11, 12, 13 explicitly create optimization passes. They are
stripped down versions of clang's -O0, -O1, -O2, -O3. They try to
provide similar results with improved optimization time by removing
some expensive passes that were repeated many times and omitting other
passes that are not applicable or not profitable. Useful for
debugging, optlevel 10 adds next to no additional passes.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Switch from old cmake macros that were marked obsolete in cmake 3.12 to
the new FindPython.cmake module.
Instead of defaulting to python 2.7, default to whatever is found.
But user can still set PYTHON_VERSION variable to request a specific
one in case there are multiple.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Luke Emrose noticed that for texture calls that retrieved alpha, if
derivatives were later taken *only* of the alpha but not the main
color result of the texture call, those derivs would always end up 0,
the texture call not noticing that it needed to call the version of
texture that returns derivatives.
This patch provides a fix. It was a simple matter of it testing only
whether main result derivs were needed, when determining which variety
of texture must be called.
Curiously, this was already done correctly for texture3d!
Added testsuite/texture-alpha-derivs to test this specific situation.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Work by Alex Wells.
This will be used extensively by the SIMD batched shading.
* Added support for masked operations by managing a stack of masks
that were created by conditionals or control flow. Detailed
explanation of the Masking Strategy in llvm_util.h.
* Added support for logically masked functions (handling of masked
return from inside OSL functions)
* Added support for masked loops (handling of masked break and or
masked continue).
* Added support for masked exit (handling of masked exit from entire
shader).
* Added support for management of when masking is required with a
ScopedMasking object to allow nesting during code gen.
* op_store (and other store variations) respect the current masking
requirement and current top of the mask stack (not just the top
entry, but a combined view of the entire mask stack).
* Added support for results of conditional operations to remain
boolean. Most masking operations are expecting vector of bool
values, not a vector of integers., e.g. when vector width is 16, for
scalar or <16 x i1> for wide. When a conditional result is used by
another operation a numerical value it is automatically converted to
an integer.
* Added `llvm::Type *type_native_mask() const;` because different
ISA's may have different representations of a mask from llvm's
vector of bits that comparison operations emit. And we need varying
boolean symbols to have the correct data type (size) on the stack
and in data structures.
* Added gather and scatter operations.
* Added helpers to convert from bool<->int, native_mask<->llvm_mask
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Periodic comparison of analogous build scripts in OIIO & OSL and
synchronize to ensure that my current most favored idioms are in both.
Split checked_find_package into separate file for easier
synchronization between projects.
Bumped OpenEXR versions to latest patches.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Rename LICENSE -> LICENSE.md for uniformity with other projects
and our own documentation.
Rename LICENSE-THIRD-PARTY.md -> THIRD-PARTY.md because it was
confusing GitHub's autodetection of the license (this makes many
tools, including third part such as the ASWF's landscape fail to
correctly categorize this project as BSD-3-Clause).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
There's a bug in the current Homebrew llvm 10.0.1 where
it fails to link because of missing libxml2.tbd. It's fixed
upstream in LLVM, but not yet fixed in Homebrew. Hopefully
a subsequent 10.0.x or 11.x will fix it, but for now, just
fall back to llvm@9.
Also issue a warning in the build scripts if this condition
is detected, so if a user encounters it on their system, they
have a hint about how to address it.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This new file contains the hashes of commits whose only function was
to reformat large amounts of code.
Ordinarily, this would make for confusing 'git blame' output, by
making old code written by author A appear to have a new modification
date and authorship by B, when in fact we know that B just did some
reformatting. This can make it especially hard to discern code history
after after commits that establish the use of "clang-format".
A new git feature (requiring git >= 2.23) allows pointing to a file
containing hashes of commits to be ignored by git blame:
git blame --ignore-revs-file .git-blame-ignore-revs
To avoid having to remember this all the time, consider adding this to
your git config:
git config blame.ignoreRevsFile .git-blame-ignore-revs
Note that git supports this feature but does not dictate the name of
the file. I'm just following the convention I'm seeing pop up in other
projects of calling it ".git-blame-ignore-revs".
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We used to disable fmt and fall back to tinyformat, when using Cuda.
The new OIIO 2.2 removes tinyformat and the OIIO_USE_FMT preprocessor
symbol. So make sure that we also tell fmt to avoid using the int128
type, not supported by Cuda. That seems to be enough to let it use
the rest of fmt without trouble.
Work by Alex Wells.
Separate OSL optimization from JIT by adding optional parameter
`bool do_jit = true` to all optimize_group* methods. This will allow
optimization without JIT'ing for the host, or optimizing once but
JIT'ing for multiple back ends. It will be necessary if one will be
JIT'ing only for batched execution, we would not want to pay the cost
of JIT'ing a scalar version as well (but still could if we wanted both
scalar and batched versions simultaneously).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Remove badges for Travis and Appveyor, since we are relying on GHA now.
Rename the ci workflow definition file from workflow.yml to ci.yml, to
support other workflows later.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
To be less reliant on a version embedded in OIIO, provide a handy
build_pugixml.bash script. And change to relying on PugiXML's exported
cmake configuration files, no more need for our FindPugiXML.cmake.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Work from Alex Wells:
Add support for generating debug information to track function call
stack, source file, and line number inside generated LLVM module.
When enabled, can debug with source and line numbers. Also Intel(r)
VTune(tm) Profiler associated assembly to shader source and line
number.
The big upshot here is that when this debugging is enabled,
* gdb/lldb can debug a renderer app, all the way down to inside JITed
shaders.
* Stack dumps upon a crash will show you the full stack trace
including where you were in a shader.
* Profilers such as vtune can be used to accurately profile shaders.
The profiling only works if your LLVM was built with the option
`-DLLVM_USE_INTEL_JITEVENTS=ON`.
Due to some ABI changes on the LLVM side, the debugging symbols only
work properly if you are using LLVM >= 7.1.
Some other specifics that got rolled into this:
New ShadingSystem attributes to control this: `llvm_debugging_symbols`
and `llvm_profiling_events`. Both default to 0 ordinarily, a renderer
needs to enable them, which we only recommend doing if the renderer
itself is compiled for debugging or profiling. The testshade and
testrender program automatically enable these options.
Debugging features to dump struct layouts and validate LLVM struct
representation based off expected offsets populated from host versions
of the data structures:
void dump_struct_data_layout(llvm::Type *Ty);
void validate_struct_data_layout(llvm::Type *Ty,
const vector & expected_offset_by_index);
Added op_functioncall_nr to as a noop with the name of the function
that was inlined. Previously constfolding would eliminate a
op_functioncall if it had no return value. Now we transmute into a
op_functioncall_nr. There should be no real shader execution cost to
this, but now can debug through nested OSL shader level function
calls. We needed the op_functioncall_nr to trigger the creation of
debug info for an inlined function call.
When not debugging, don't waste by naming basic blocks.
Note that for now, we disable debugging symbols for PTX mode. At some
point we will have to do some refactoring to re-enable this, if/when
there is a use for it for PTX debugging.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Avoid emitting any alloca instructions inside loops and reuse
temporaries across the body of a function, which means we should not
emit them in conditional branches either. So always place alloca at
the very beginning of a function. To do that, we save the current
insertion point, change it to the beginning of the function, emit the
alloca, then restore the insertion point to where it was previously.
Work by Alex Wells.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Work by Alex Wells.
Leaks are prevented now, though at the cost of a bit more housekeeping
on the part of the user of LLVM_Util. Each thread must have its own
LLVM_Util::PerThreadInfo, which must outlive any LLVM_Util (and which
must be passed a reference of the PerThreadInfo). Also, there must
somewhere in the app exist a ScopedJitMemoryUser that must outlive all
of the LLVM_Utils.
Expose LLVM_Util::PerThreadInfo (separate from its implementation so
it can be embedded in OSL::PerThreadInfo vs. managed separately.
Improve memory cleanup for JIT'd code by adding
LLVM_Util::ScopedJitMemoryUser. JIT'd code needs to exist with a
longer lifetime than the LLVM_Util object. To enable better cleanup
at shutdown, the lifetime of all JIT'd code is controlled by the the
existence of ScopedJitMemoryUser objects. When the last
ScopedJitMemoryUser goes out of scope or is deleted, then the
underlying memory managers will be deleted. Each ShadyingSystem
instance has a ScopedJitMemoryUser.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Fix template casting reference to work properly with the latest
pybind11 master.
* Mac CI: use Python 3.8.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
This fully enables the mechanism and reformat the first batch of files.
For now, we will continue to exclude the llvm-related code and
OptiX-related code from the reformatting, since changing those at this
time will make merging difficult for work that is in progress. We'll
add those after they have fully settled down. But in the mean time, we
can start by reformatting the more stable parts of the code base.
The reformatting will be staged in reasonable batches over a series of
PRs.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Add ISA-specific targeting and additional JIT control options
ISA means "instruction set architecture", by the way.
Heavy lifting by Alex Wells, some refactoring and adding options by LG.
A number of additional controls over the JIT process are added to the
ShadingSystem:
- "llvm_jit_target" Can be set to the name of an architecture ("none",
"x64", "SSE4.2", "AVX2", "AVX2_noFMA", "AVX512", "AVX512_noFMA") to
request that the JIT choose instructions for a specific machine ISA.
Also "host" means to select whatever the host natively supports.
It checks if the host CPU supports the requested architecture and
will automatically restrict it to a choice your machine supports.
The default is "none" meaning not to generate code for anything beyond
the default Intel target. There seems to be little benefit to using
this at this time for single-point shading, but we anticipate that
it will be important to the forthcoming batch shading mode.
- "vector_width" Can set the vector width for SIMD instructions,
independently of the JIT target (for example, you could JIT for
AVX512 but still only make 8-wide SIMD instructions). The current
default is 4, meaning it will not generate 8-wide or 16-wide
vector ops, even when targeting and running on AVX512. We have found
significant performance penalties when running single point shading
mode on AVX512 hardware, unless we use this restriction (but no such
slowdown on AVX2 hardware).
- "llvm_jit_fma" Enables use of fused multiply-add instructions on
architectures that support them. When enabled, resulting code is
slightly faster and improves rounding accuracy for some
computations. But the default is 0, because enabling it can make
results differ slightly on machines with different architetures (bad
for testing, and maybe leading to problems if you have machines with
differing architectures working on different frames of the same
shot).
- "llvm_jit_aggressive" Enables the "aggressive" JIT option, whatever
that means. Currently defaulted to 0.
- "llvm_dumpasm" a debugging mode that lets developers see the actual
JIT code as assembly language.
The bottom line here is that the defaults are carefully chosen to be
performance-neutral on all architectures. We do not currently advise
adjusting these settings for single-point shading mode. When we add
batch-shading mode, we will probably have different defaults or
guidance to offer when using that mode (and the specific advice may
depend on the specific architecture you are running on).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Fixes from review suggestions
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Co-authored-by: Alex M. Wells <alex.m.wells@intel.com>
We had disabled RTTI throughout the OSL code base to avoid linkage
problems against LLVM, which builds with RTTI disabled by default.
But it's actually ok to mix code modules with differing RTTI status --
that's fine. It's just that you can't mix RTTI status within a class
inheritance hierarchy.
We observe that
* The only classes we inherit from RTTI-free LLVM are solely defined
and used within llvm_util.cpp.
* The only classes exposed in llvm_util.h have no virtual functions
and aren't designed to be subclassed.
So in fact, it appears that we can get by with only using no-rtti when
compiling that one module, and don't need to mess with rtti anywhere
else (including downstream users). This is an important step to
supporting python bindings, since pybind11 requires RTTI.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
We don't use this quite yet, but are staging it for easy review.
Work by Alex Wells. Some refactoring and minor edits by LG.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Spell out the exact requirements when building against LLVM 10,
which option to set to bump to C++14, which compiler versions are
necessary.
Closes#1205
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Shader params declared with lockgeom=0
surface foo (float bar = 0 [[ int lockgeom=0 ]]) { }
will generate code making underlying calls to RendererServices->get_userdata
to automatically bind userdata (a.k.a. interpolated "primitive variables")
that share the same name.
But, oddly, this was not true for output parameters:
surface foo (output float bar = 0 [[ int lockgeom=0 ]]) { }
This was tripping some people up, and in retrospect seems strange.
This patch removes this restriction, allowing both input and output
parameters tagged with lockgeom=0 to attempt to retrieve userdata to get
their initial values.
Add test of userdata retrieval to make sure this works.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
I tried to keep the code simple rather than optimize it heavily, so it runs slower than the previous code. On the plus side, its shorter and doesn't have any dependency on OIIO hashes.
Signed-off-by: Christopher Kulla <ckulla@gmail.com>
By default, shader group layers execute lazily, meaning the node will
not execute until/unless one of its outputs is needed by a downstream
layer, and if in the course of optimization all of its outgoing
connections are eliminated, then the entire node will be pruned from
the shader graph. But if the node contained error() calls (say, to do
sanity check on parameters or other conditions), those error calls
will never be issued, possibly masking a problem.
This patch introduces a new ShadingSystem option:
"lazyerror" (default: 1) -- when set to 0, it means that the presence
in a shader layer of an error() call that *can't* be constant-folded
away will make the layer will execute unconditionally, thereby making
the error() call run even if the layer's outputs are never "pulled" by
any downstream layer.
Furthermore, if lazyerror=0, if you have shaders that after
optimization still have error calls (and thus will run the layer
unditionally), then if option "opt_warnings" is nonzero, you will get
a warning about the presence of a not-optimized-away error call (you
may want to know about this for performance reasons, if your intent
was that all error calls should be resolveable by the optimizer).
Along the way I also added a few more optimization-debugging logging
messages, including understanding exactly when and why unused connections
are pruned.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Refine O7 string handling. Avoid initializing string variables in the module PTX.
* Replace '.' with '_' when constructing PTX variables from symbol names.
* Remove the now-unnecessary DummyFunction from rend_lib.cu.
* Change the dimension check for issuing a 'missing output' warning in testrender/testshade.
Co-authored-by: Tim Grant <tgrant@nvidia.com>
Many times, people can ask how they can use OSL to make more general
computational engines (not a classic renderer), and one use case that
comes up frequently is to use it to control deformations. This patch
includes a new testsuite example that does exactly this, for a test
case -- setting up a small shader network that operates on a series
of points.
It's a completely self-contained example of how to do this. There is a
lot of boilerplate for such a simple task, but I was really trying to
make sure to provide examples of all the little pieces you'd need to
do a more complex use use.
To make sure it builds, I added it to the testsuite itself, as
testsuite/example-deformer.
Interestingly, it is our first testsuite entry that needs compilation
and linking with OSL -- and not using OSL's own build system! So not
only does it test the code functionality itself, but it also serves as
a test for our exported CMake config.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
People linking against oslexec needed to know to use -fno-rtti when
building their apps. Very brittle. This correctly adds the flag as
an interface compile flag in our exported CMake config files.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
For attribute and getattribute, we had some specializations that make
it extra easy to do the common cases of a single float, int, string.
This is just adding the same kind of convenience functions for
Parameter() and ReParameter().
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Use the new ASWF VFX Platform 2021 docker image for gcc9/py37/llvm10/C++17
* Alter the "bleeding edge" CI test to use gcc10
* Fix some new warnings that pop up with gcc9 and gcc10
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Because they are UTF-8 safe and work properly on MinGW.
Simplified a bit more as I went -- some reading idioms could be
replaced by a single call to Filesystem::read_text_file, some close()
calls were redundant because streams close upon destruction / scope
exit, some open flags unnecessary because the default write mode
already will truncate an existing file.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
There was an oslc bug a long time ago (prior to OSL 1.10) in which the
3-argument versions of regex_search and regex_match failed to mark
their 'results' parameter as write-only in the oso.
In addition to fixing the bug in oslc so that subsequently-produced
oso files are correct, in order to properly interpret these
historically-compiled oso's, we also look for this particular case
upon reading the oso, in loadshader.cpp, and mark it correctly. This
was PR #922.
Except...
1) We neglected to distinguish the 2-arg version from the 3-arg version,
only the latter of which should be so marked.
2) We tested for regex_search twice (!) instead of testing for
regex_search and also regex_match.
Apologies! Here is the fix.
Thanks to Alex Wells for identifying the problem.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
It's not officially part of the public API, but making the symbols
public is necessary for some upcoming work where Alex has separate DLLs
for different CPU/SIMD implementation modules and the symbols will need
to be visible to the contents of those DLLs.
Signed-off-by: Alex M. Wells <alex.m.wells@intel.com>
We aren't saving object files like an offline C compiler; we are
JITing and immediately executing on a specific machine, so we may as
well have LLVM optimize for that very chip. This patch from Alex Wells
adds optimization passes that do so.
LG amended it to make it a shading system option "llvm_target_host"
(defaulting to true). And we force it to off when compiling for
OptiX, since obviously the GPU is not the host architecture.
Production benchmarks do not show a noticeable speed gain in scalar
code. (Nor does it slow down.) There is no harm in targeting the
specific host CPU, and it may be more helpful when we are dealing with
vectorized code down the road.
Alex also added a developer only macro OSL_DEV_ONLY(STUFF) to leave
developer debugging/tracing code around and only have it active if
compiled with -DOSL_DEV
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Co-authored-by: Alex M. Wells <alex.m.wells@intel.com>
I'm not sure how this happened, but TypeSpec had `is_float_based` and
`is_floatbased` methods, and they were subtly different (the latter is
false for float-based arrays). This has led to at least one bug.
A careful examination reveals that the vast majority of uses of
`is_floatbased` really intended `is_float_based`, so I changed all
references to this.
For the few where excluding arrays is the right logic, I added a
separate test for arrayness.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
In the compiler, symbol.name() is the human-readable name and
symbol.mangled() is a mangled symbol that keeps identical names in
different scopes from clashing. We use mangled() in the oso file but
name() in error messages.
However... at runtime, it reads the pre-mangled name from the oso, and
doesn't know the original name, so name() and mangled() are identical
and are both the mangled version. That's fine for almost everything,
but there are just a few runtime errors that involve the symbol name,
and we would prefer that these be the unmangled version, for the sake
of clarity when communicating errors to the user.
So we add a Symbol::unmangled() that trims the mangling from the front
of a mangled symbol, and in runtime error messages we use this to strip
the mangling.
I also modified Symbol::mangled() slightly, to make sure NVRO is used.
The observed behavior doesn't change, only the idiom.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Previously, we had `LLVM_Util::internalize_module_functions` that
reduced the LLVM optimize + JIT work by finding function symbols in an
IR module that started with "osl_" and were not our known entry points
and marking them with "internal" visibility. This lets LLVM recognize
them, if they are not called internally to the module, as dead code
and discard them before spending any time optimizing or jitting those
functions.
This patch from Alex Wells adds an additional method,
`prune_and_internalize_module`, that is even more aggressive, pruning
all symbols that we don't think can be reached with our generated code
(are not "materialized", in LLVM parlance), and then marking
everything that is not a known entry point as "internal".
N.B. Some care is needed not to mark as internal a few symbols that
Cuda adds and which if removed will cause the OptiX shader to
fail. These seem to all contain the substring "rti_internal_".
This is indeed slightly better, and in extensive benchmarks of
production shaders, we're seeing about a 10% average reduction in LLVM
opt+jit time. This is a tiny portion of a fully rendered scene, but
for interactive renders where you care about "time to first pixel", it
is more significant to reduce this compilation overhead as much as
possible.
I (LG) am a little hesitant to deprecate the old method right away, in
case any as-yet undiscovered flaw is found or just to aid
benchmarking. So I added a ShadingSystem string attribute
"llvm_prune_ir_strategy" that can take on any of the values "prune"
(default, the new way, full prune and aggressively internalize),
"internalize" (old way, just internalize function symbols that aren't
entry points), and "none" (don't do either -- just for benchmarking or
debugging).
I also benchmarked the "none" strategy, and it spends an average of over
2x longer in LLVM opt+jit than our old "internalize." So either strategy
helps substantially cut down on opt+jit time, though "prune" seems the
fastest, and is thus the new default. As we gain more confidence in it
in production, we may eventually deprecate and remove the "internalize"
strategy, leaving only "prune."
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Co-authored-by: Alex M. Wells <alex.m.wells@intel.com>
These aren't used anywhere in OSL's codebase yet. But they will be.
By Alex Wells.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Co-authored-by: Alex M. Wells <alex.m.wells@intel.com>
Both were plagued by checking the entirety of the matrix, when in fact
only one element needs to be checked. This is wasteful because it
checks elements that aren't read or written by the op we are checking,
and it can also lead to "false positives", errors about elements that
are not touched by the present op.
Refinement of the debug_nan and debug_uninit cases was by Alex Wells.
LG added comments to clarify the implementation of these functions.
LG also amended the testsuite/debug-uninit to specifically test
arrays, triples, and matrices (checking for true positives and check
against false positives). In the process, he noticed another bug (!)
where float-based arrays were not being correctly initialized with
their uninitialized value sentinels, and thus were not detecting use
of uninitialized values therein, because instead of calling
is_float_based, we called is_floatbased, which excludes arrays.
(The similar names are yet ANOTHER bug, or at least unintended to be
so confusing, and this will be addressed in a separate PR).
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Co-authored-by: Alex M. Wells <alex.m.wells@intel.com>
Point/color and array accesses were range checked at runtime EXCEPT
when the indices were constant and known to be within legal range.
But matrix component access (m[r][c] = x, x = m[r][c]) were always
range checked.
This patch from Alex Wells (and then enhanced by LG) skips the range
checking on the row or column indices, or both, when they are constant
and >= 0 and < 4, which is probably a lot of the time, thus speeding
up matrix component access a little by skipping the range checking
when it's safe to do so.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
Co-authored-by: Alex M. Wells <alex.m.wells@intel.com>
Since 9.0 LLVM defaults to building a single clang-cpp library for
non-developer builds.
As this is just the default, and some distributions enable it only with
LLVM 10 (e.g. Fedora), make the clang-cpp search dependent on the build
mode.
Modify the CI test matrix to include several tests using the ASWF docker
images to test exactly the VFX Platform configurations for 2019 and 2020.
Additionally:
* Some other minor juggling of GitHub CI configuration and dependency
versions used for each test.
* Have the "2021-ish" test bump OpenEXR dependency to the newly
released 2.5.
* Alter teh build_openimageio.bash helper script to use strictly the
cmake build system, not the Makefile wrapper.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
I had noticed the comment didn't match the code and updated the code not realizing it was the comment that was wrong ...
Signed-off-by: Christopher Kulla <ckulla@gmail.com>
On OSX, with LLVM 10, it seems that there can be a double linkage of
the libc++ that is built with llvm and the one in /usr/lib.
This leads to a very subtle problem where oslc's use of libclang for
preprocessing is unable to find included files when there are multiple
include search paths (!), unless you have the llvm lib directory first
in your DYLD_LIBRARY_PATH. That is obviously too brittle a fix.
Solve by forcing use of statically linked llvm/clang libraries when
on OSX and using LLVM+clang 10+, even if the user did not explicitly
request that.
Signed-off-by: Larry Gritz <lg@larrygritz.com>
* Fix newly broken Mac CI -- GHA's default homebrew now has llvm 10, which
requires C++14.
* Schedule a nightly cron job for CI.
* Do not launch full builds and tests for pushes that only change
documentation (md, rst, and tex files).
We previously did not support indexed strings (only int, float, vector).
Also some SPI adjustments to partio finding, the rest of you can
ignore that part.
* Technical Charter, governance documents, Code of Conduct (approved by TSC)
* Draft copies of the proposed new CLAs.
These are as identical as possible to the corresponding documents of
OCIO (differing only for name changes and the like, and in just one or
two places, substituting an equivalent but slightly more clearly
worded version of some piece from OpenEXR). None of the important
legal stuff like CLAs or licenses have been altered from the OCIO and
ASWF standard template as they were approved (by my organization) for
that project.
The license remains the 3-clause BSD we have always used, so there is
no reason to contact prior authors for re-licensing. But when we get
around to moving the repo from the Imageworks GitHub account to the
ASWF one, the new CLAs will take effect and subsequent contributors
will need to sign them (even if they had previously signed the Sony
ones).
This brings us into conformance with ASWF standards for license text
and how the copyright notices read. (And was cleared by Sony and Linux
Foundation.) This is the same set of changes made to all the other
projects coming into the foundation, including OpenColorIO.
Please note that none of the licenses are changing -- it's the same
3-clause BSD license we have always used. Also, this doesn't change
any of the copyright holders -- it has always been the case that the
authors of each patch retain the copyright to their specific additions
or changes. OSL has never required copyright assignment for
contributions.
* Add a LICENSE-THIRD-PARTY.md file that documents imported software.
* Update documentation license to CC-BY-4.0 (from 3.0).
* Fully reproduce the Qt license in the two files derived from Qt
example code.
* Change copyright notice to the new wording of 'Contributors to the
Open Shading Language project', per ASWF convention.
* Replace lengthy per-file copyright boilerplate with SPDX short form.
(This is more readable and is easily scanned by automated software to
detect which licenses are used.)
* For any files lacking a copyright notice, license text, or SPDX
identifier, add them.
* Fix test output for many unit tests where line numbers changed after
license info was added to .osl files in the testsuite.
Some of the internal matrix ops didn't use the usual naming
convention. Fix. Also remove at least one that is never used because
it's a combination that oslc will never output.
This is much like how oslversion.h is generated from oslversion.h.in,
which allows things to be "baked in" from cmake control (version
numbers and namespaces, for example, in the case of oslversion.h).
Now with this patch, oslconfig.h is generated in a similar manner. I
don't actually have any build-time changes as part of this patch, but
some are coming in future planned patches, so I wanted this in place
first.
Avoid potential aliasing issues by avoiding use of Vec3::operator[]
which does a reinterpret_cast and instead explicitly use data member
names .x, .y, .z.
This doesn't matter much for scalar code, but when loops are
explicitly or automatically vectorized, using myvec[0] can confuse the
compiler enough to inhibit vectorization, whereas myvec.x does not.
* I haven't been diligent about keeping these shaders in perfect sync with
the MX spec.
* Because the MX shader spec changes with each MX version, it seems
inherently confusing and error prone to have a version inside OSL,
which is not versioned the same way or at the same times.
* Modern MX has all the reference OSL code and builds the shaders on its
end, so soon (if not now), it will be completely redundant to have this
here.
* Building the many hundreds of MX shaders is a significant portion of
OSL build time (when using ccache at least), but most people don't need
them.
So in light of these, we will eventually phase out and remove the MX
shader generation and compilation from the OSL package. Since I'm not
sure who that would break or inconvenience now, we'll slate the total
removal for some future major release (AFTER we are sure from the MX
folks that it serves no more value here at all).
But in the mean time, I think we should at least change the default to
not build it. Users who rely on it can always build with
`OSL_BUILD_MATERIALX=1` if they want.
* build_foo scripts fail immediately upon error.
* Don't auto-download pybind11 (but do provide a script to do dl/build it)
* Developer snapshot builds from master should have so name include
the patch number to help avoid linking incompatible shared libs.
* REQUIRED_DEPS and OPTIONAL_DEPS let you more easily turn off dependencies
by name (beware unintended consequences!)
* Make the build area layout more closely resemble the dist area layout.
There was a huge amount of duplication between sfmath.h and OIIO's
fmath.h, with the ones here having only very minor modifications. With
almost no exceptions, I was able to port the fixes here back to
fmath. This now eliminates the redundant copies on the OSL side.
* Remove sfm::ifloor in favor of OIIO::ifloor
* Remove sfm::clamp in favor of improved OIIO::clamp
* Remove sfm::madd in favor of the improved OIIO::madd
* Remove sfm::lerp, bilerp, trilerp in favor of the fixed OIIO fmath version.
* Remove sfm::bit-cast in favor of the improved OIIO fmath version
* Remove sfm::negate in favor of OIIO fast_neg, add Dual fst_neg
* Remove sfm:log functions in favor of improved OIIO fmath.
* Remove unnecessary sfm::absi
* Eliminate most svm transcendentals, use improved OIIO ones instead
* Rearrange safe_fmod
* Remove sfm::absf which did nothing but call std::abs
* Keep sfm::isinf, but only for Intel
The requisite changes should be in both OIIO master ("2.2.2-dev" and later)
and the current release branch 2.1.12 and later. It should also build against
earlier versions of OIIO, but they won't have all the performance additions
for the vectorized loops.
* Fix broken exported cmake configs that omitted public include dirs
* fix more config includes
* Fix CMake errors about exported deps being in the source tree
A little black magic to ensure that a true cmake config file with
imported targets is preferred over my home grown partio finding module.
Rename FindPartio.cmake -> Findpartio.cmake to match the project naming.
Modify Findpartio.cmake to set up partio::partio target, use it properly.
This all paves the way for an eventual future in which Partio correctly
exports its targets.
* Make sure that LLVM ops are marked as always_inline
This allows LLVM to truly inline the functions when the optimization level is set to 1 or higher
* Small improvements to llvm_output_bitcode feature
This patches uses underscores instead of dots in the filename to avoid the file from being "hidden" if it started with a "/". Also handle the ":" character which shows up in some material naming conventions and would likely be problematic on windows.
Also add a message to indicate when the bitcode files are written, and make sure the message shows up immediately instead of being deferred.
* Change default LLVM JIT optimization level to 1
The default set of optimization passes produces better code than our previous hand-picked set of passes when combined with the always_inline annotation. Surprisingly, it even reduces the total JIT time as well.
Also fix an alignment issue in llvm_gen_print on the GPU revealed by moving to the more complete set of LLVM optimization passes
* Rename them also, take away the `_cuda` suffix because they contain more than
just Cuda decorations, restore their original names.
* Copy a bit of updated material from the openexr versions (as of 2.4)
back into ours, to make a someday reconciliation an easier merge.
* This is all more clearly indicating the semantics of what we intend here:
these are Imath functions that we had to modify (for whatever reason) but that
we intend to push back to OpenEXR and then eventually they can disappear
here once our minimum OpenEXR version requirements are known to include the
fixes.
The tricky bit was that we have variants of tests: mytest, mytest.opt,
and when using OptiX, also mytest.optix and mytest.optix.opt.
Those previously all ran from the same working directory, which was
fine if run in serial. When they run at the same time, they will
clobber each other's outputs. So the testsuite logic had to be changed
so that each of the variants runs in its own special test directory.
More substantial overhaul/simplification of testsuite setup.
Including explaining to ctest which tests need to allocate a few cores.
On my workstation at work, this makes running the whole testsuite
(including all the optix tests) go from 5 mins to 30 seconds. Woo!
* Fix `<<` stream output that broke its pretty formatting in a recent
overhaul.
* Supply missing Dual/float comparison operators: for example, we had
Dual<float, but not float<Dual.
* Add equalAll that is like equalVal, but also compares whether the
derivs are equal (when both items being compared have derivatives).
For some reason, the namespace declaration was attached to the end of
OSL_NAMESPACE_EXIT. Which would make an error in any source module that
had multiple OSL_NAMESPACE_ENTER/EXIT blocks.
* Move OSL_NODISCARD by porting OIIO's OIIO_DISCARD.
* Also port OIIO_FALLTHROUGH.
* Move osl_assume_aligned to the end so it can be inside the usual
namespace. Also rename it assume_aligned to match C++20 and use a
`using` so that for C++20 it just aliases to the std one.
* Introduce sfm (SIMD Friend Math) namespace with versions of math functions friendly to use inside SIMD loops.
Introduce IntHashBasedNoise to serve as base for HashNoise & CellNoise.
Reimplemented PeriodicHashNoise and PeriodicCellNose to use an adapter to wrap their input values and reuse existing HashNoise & CellNoise implementations.
Introduce code generation policies for Perlin based based functions and Noise objects in order to resuse the same perlin noise functions with or without SIMD intrinsics. Using a "scalar"(non-intrinsic) version of the algorithm is required when calling from inside a SIMD loop.
Defined Scalar versions of any Noise objects that used SIMD intrinsics internally.
Provide optimized inthash, sfm_simplexnoise, sfm_gabornoise implementations suitable for use inside SIMD loops.
Introduce wide.h with Block<T, int Width> to store data as structure of arrays internally, more Wide and Masked wrappers to Block<T> be added in future.
Added osl::conjunction and osl::enable_if_type for compatibility with older C++ versions that do not have their equivalents defined in std::
Added OSL_NON_INTEL_CLANG = [0 | 1] to identify when compiler is a truely clang, as OSX still has __clang__ defined when using Intel compiler.
Added OSL_NODISCARD
Added OSL_OMP_PRAGMA(unquoted_text) that will only be emitted when -DOSL_OPENMP_SIMD and -DNDEBUG (disabled in debug builds)
Added OSL_FORCEINLINE_BLOCK on supported compilers will recursively inline the following statement
Split common portions of gabornoise.cpp into gabornoise.h that can be shared with sfm_gabornoise.h (the SIMD friendly version of gabornoise).
* Fixed warning error: suggest parentheses around comparison in operand of & [-Werror=parentheses] in sfmath.h
* Fixed typo bug when using C++17
* Added missing OSL_HOSTDEVICE to many noise and sfm functions to enable successful nvcc builds
We now recognize the metadata annotation "int range_checking=0" on a shader to mean that the shader author is certain that range checks can be safely omitted for this particular shader. This can help speedup particularly expensive shaders without needing to globally disable the checks in other shaders.
This caching layer was added a long time ago when the renderer at SPI took a long time to find
out that an attribute query failed. This is no longer the case, and our usage of attribute lookups
has changed enough as to make the caching no longer helpful (and sometimes hurtful, such as
when a shader performs many successful queries). Renderers that need the caching behavior
can re-implement it internally to their RendererServices class.
Completing the recent dependency generation feature, -MT lets you
customize the name of the dependency target.
Also, to match the C/C++ compilers, we now accept both
-MF<filename> # all as one arg
-MT<target>
-MF <filename> # two args
-MT <target>
oslconfig.h was getting a little jumbled. Split off all the compiler,
language, and platform detection and macros that need to vary based on
those, put in a new platform.h.
Both in platform.h and what remains in oslconfig.h, reorder for clarity
and to group similar things together. Almost no actual code has changed.
One of the few changes is that I beefed up the pragma macros (now in
platform.h) to inclulde a few more items.
You see, we are taking a file path, which on Windows might look like
"c:\path\to\new\osl" (the Windows design mistake we will all live with
forever), and then injecting that into the osl source parse string as
a double quoted string, it will be unescaped and turn into
"c:\path<tab>o<newline>ew\osl", which of course is wrong.
This leads to hilarious bugs on Windows if OSL is installed someplace
where one of the directories in the sequence starts with an "n", as
above.
The string concatenation op was going through sprintf which forced at least one
memory allocation to happen, in addition to the expense of going through the parsing logic, etc ... This was roughly doubling the cost of this (already expensive) operation. This commit implements a more direct concatenation via memcpy, and uses a small stack buffer in most cases.
Not sure why, but something changed recently on GitHub CI that is
causing our CI tests to fail... but not on the OIIO built itself, only
on the OpenEXR dependency build, and only when using OpenEXR 2.2.
I don't have time for this nonsense, so just adjust all the GH CI
tests to use OpenEXR >= 2.3. Adjust some of the TravisCI tests to use
OpenEXR 2.2 so we still have some proper test coverage of it.
libsquish saves a little OIIO build time by not having to build the
embedded version.
qt5 lets the CI build osltoy (doesn't run it, but makes sure there's a
clean build).. and sure enough, there wasn't a clean build on gcc8, so
I have a fix for a picky warning.
* Math and aliasing improvements:
Move inclusion of Imath*cuda.h from Imathx.h to oslconfig.h
Avoid aliasing in helper math functions by not using Vec3::operator [] and instead accessing .x, .y, .z data members directly. Likewise for Vec2.
Avoid aliasing in helper math functions by not calling Matrix44::operator [] which returns a float *. Instead use the x[4][4] datamember directly which keeps it type safe and avoids casting down to a float *.
Made math helper functions static OSL_FORCELINE where possible to avoid any collisions with other translation units who might be compiled with different compiler flags (like target ISA).
To make some of the aliasing avoidance changes, needed to introduce some free functions to be called instead of existing base class methods (which exist in a different open source project).
Partitioned general purpose inverse of Matrix44 into:
test_if_affine
affineInverse - fast path that is SIMD friendly
nonAffineInverse - slow path to be used outside SIMD loop to
handle any non-affine matrices
Changed math helpers declared in OSL namespace to work on OSL::Matrix44, OSL::Vec3, etc. instead of Imath::Matrix44<T> as the helpers would not have function should OSL::Matrix44 be defined as anything other than Imath::Matrix44<T>.
Updated Matrix22 to avoid taking addresses with memset and memcpy that could prevent compiler Scalar Replacement of Aggregates optimizations.
Avoid aliasing issues in Matrix22 implementation by removing Matrix22::operator[] and using .x[2][2] directly and avoiding Vec2::operator[].
Removed default parameter to Matrix22::inverse(bool singExc) and added Matrix22::inverse() const noexcept, to allow for improved execution within SIMD loops.
Unrolled small loops in math helpers to avoid introducing control flow and dependencies during execution.
Updated opmatrix.cpp to call free function versions of transpose and multDirMatrix to utilized the versions added to Imathx.h
Added OSL_PRAGMA(unquoted) as a cross platform means of declaring a pragma
Added OSL_INTEL_PRAGMA to emit pragmas that are specific to ICC
Added OSL_CLANG_PRAGMA to emit pragmas that are specific to Clang
Added OSL_CLANG_ATTRIBUTE to emit attributes that are specific to Clang
* Fixed compiler warning error: suggest parentheses around comparison in operand of | [-Werror=parentheses]
* Fix error: OSL_v1_11::Matrix44 OSL_v1_11::nonAffineInverse(const Matrix44&) defined but not used [-Werror=unused-function
* Port a bunch of compiler identification and manipulation macros from
OIIO's platform.h to OSL's oslconfig.h. (Renaming OIIO_blah -> OSL_blah
obviously.)
* Rename the recently-added OSL_INLINE -> OSL_FORCEINLINE to match the
similar macro from OIIO and use OIIO's definition which is is more
robust to a wider range of compilers. Rename many places where we
previously used OIIO_FORCEINLINE to the OSL name.
* Rename the recently-added OSL_EXPECT_TRUE and OSL_EXPECT_FALSE ->
OSL_LIKELY and OSL_UNLIKELY, porting and matching the similar macro
that we already used extensively from OIIO. Rename all the places
where we used it with the OIIO name.
* Change bare references to `__cplusplus >= 201402L` into
`OSL_CPLUSPUS_VERSION >= 14`.
* Change stray OIIO_CONSTEXPR to OSL_CONSTEXPR which we have had for
a while now.
Something changed about homebrew versons, I guess, and the combination
of python + cmake on both GitHub CI and TravisCI broke OSX. Modifying
install_homebrew_deps.bash to be sure it upgraded to the latest
version of both packages seems to solve the problem.
Ported from OIIO, recently fixed here.
* Refactor implementation of template<class T, int PARTIALS> dual to not use array based data members and avoid any control flow loops for operations on it. The change better enables compilers to keep values in registers via SROA (Scalar Replacement of Aggregates) optimizations as well as enable more efficient SIMD vectorization of loops using dual types.
Update opcolor implementation to avoid subscript operators [0],[1],[2] and use named data members .x, .y. z to avoid pointer aliasing issues.
Update opcolor implementation to avoid mat[?][?] operators that loose multidimensional C array data type and access the underlying c array directly mat.x[?][?]
* To address internal compiler error issue with gcc8 & c++14 changed static loop unrolling strategy for __GNUC__.
Removed use of a local variable partial_count that was erronously being marked as unused by a compiler.
* When compiled with USE_OPTIX=1, there were some missing OSL_HOSTDEVICE decorations on the constructors for the Dual specializations
The old macros were from OIIO, namespace polluting, and ASSERT always
aborted the process. The new macros won't conflict with other apps,
and OSL_ASSERT prints error always but only aborts for debug mode,
which discussion concluded was the preferred behavior.
I also policed up the whole set of assertions, and there were many
instances of:
* ASSERT that I deemed downgradable to DASSERT.
* ASSERT/DASSERT that I deemed entirely unnecessary and deleted.
* A few asserts that should always have been OIIO_CHECK_* unit test
macros.
This patch adds support for `-M`, `-MM`, `-MD`, `-MMD`, `-MF` flags,
akin to how they are used in gcc or clang, to generate dependency
files that can be used by make, ninja, etc., to express header
dependencies of osl files. For example, output might look like:
$ oslc -MMD test.osl
Compiled test.osl -> test.oso
$ cat test.d
test.oso: test.osl \
dist/macosx/share/OSL/shaders/color2.h \
dist/macosx/share/OSL/shaders/stdosl.h
`-MD` and `-MMD` output a dependency file (all headers for `-MD`, just
user headers for `-MMD`), by default to `<inputbasename>.d`, but
overrideable by `-MF<filename>`.
`-M` and `-MM` are versions of `-MD` and `-MMD` but that print the
resulting dependency list to stdout.
We also adjust the printing of "Compiled foo.osl -> foo.oso" so that
it does not happen in the case of `-E`, `-M`, and `-MM` when other
specific output is expected to comprise the whole of stdout.
Also, I *think* that we now no longer use Boost Wave at all on any
platforms, so I put in an `#error` compiler directive to verify. If
nobody complains, I'll soon excise all of the Boost Wave specific
code.
LLVM 9 is about to branch... supporting 6,7,8,9 is more than enough, we don't
need to support 5 versions back.
Also, LLVM 6 is the minimum that can be used with the new OptiX work.
Found a couple more internals to change the names to be extra sure we
keep straight which ones are "printf-style" formatting ("%6.2f"), versus
future C++ converstion of std::format (Python-like: "{:6.2}"). Currently
we use exclusively printf-style, but some day there may be a mix, so I'm
removing ambiguity in the names. This is strictly internal, does not
change any public APIs.
This can be useful in debugging situations, since the oso file would
completly contain a readable (though preprocessed) and recompilable
version of the source.
In the oso file, this is signified by the hint `%preprocessed_source`,
and following that the remainder of the file is assumed to be source
code or comments and will be ignored by the oso parser.
New macros:
* OSL_ASSERT / OSL_ASSERT_MSG -- always test, always print error if the
test fails, abort only in debug build.
* OSL_DASSERT / OSL_DASSERT_MSG -- debug mode only test, print, abort.
Does nothing for release build.
Eschew the old OIIO macros ASSERT/DASSERT, since it is perferable to not
purposely abort when in release mode.
Ported over from OIIO copies of a few other related macros.
This patch just adds definitions, we will change the existing macros
in subsequent PRs.
OIIO master/2.1 on OSX changed the extension used for imageio plugins.
Make us do the right thing here. OSX problem only, and only when
straddling the OIIO 2.0/2.1 boundary.
With OIIO 2.0 as our minimum, change all the internal error reporting
functions taking printf-style arguments to use the errorf() convention,
thus reserving the name error() for an eventual future where we use
C++20 format conventions.
Also make sure that we are consistently using the right OIIO functions
themselves -- sprintf, not format, for example.
This doesn't affect any of OSL's public APIs, this is all strictly
internal and does not break public API compatibility.
* Allow testshade to override global locale for testing. Either with --locale command line arg or TESTSHADE_LOCALE env variable.
* Ensure locale independence: I found several additional places where we were not being locale independent, so if run in a locale with "," as the decimal separators, output would change or bad things would happen.
"Unescape" all string literals in .osl and .oso as they are parsed,
then "escape" them properly for output. We weren't being careful and
consistent before. This led to at least two subtle behavioral errors:
1. oslinfo would appear to add an extra set of escape characters to
metadata values that contained escape codes.
2. If s = "\n", then strlen(s) was 1 if s was a local variable, but was
incorrectly 2 (wowza!) if s was a parameter!
* Add fixed version of parse_string (https://github.com/OpenImageIO/oiio/pull/2386)
I prototyped all this stuff in OIIO over the last month, and now am
finally porting the methodology to OSL. Major changes:
* Bump CMake minimum to 3.12
* Massive refactor of cmake files to move to modern idioms and best
practices. Beware some changed conventions for controlling the build:
- CMAKE_CXX_STANDARD instead of USE_CPP
- CXX_VISIBILITY_PRESET instead of HIDESYMBOLS
- BUILD_SHARED_LIBS instead of BUILDSTATIC
- Package_ROOT to hint where to find dependencies, instead of
some random selection of PACKAGEHOME, PACKAGE_ROOT_DIR, etc.
* Start using GitHub CI.
* Make following DeviceString variables to the underlying char* conditional.
* Update the string example code for the microfacet ID in the CUDA wrapper.
* Add an OptiX test for string closure params using the microfacet distribution.
* Overhaul how OptiX variables are created.
* Use the sym TypeDesc instead of c_str() when mapping to an OptiX type.
* Prefer TypeDesc over std::string for communicating types when creating CUDA variables. Remove redundant call to createOptixMetadata().
* Extend ShadingSystem::getattribute of type pointer's to also return the size of the pointer.
Add "colorsystem:colorspace" to ShadingSystem::getattribute.
Remove private include of opcolor.h.
* Be a little less hackey about the whole thing
* Have gettatrr return size and number of strings for a more general pattern.
Includes slight reformatting of the license text to make it correctly
be auto-recognized by GitHub and have the license displayed. The content
hasn't changed, it's still the 3-clause BSD, but it's *exactly* copied
from what GitHub wants to match against.
Stale assignment elision was broken in the presence of `exit`.
In general
a = 10;
... stuff not involving `a`...
a = 20;
sees two assignments within the same basic block, and because `a` is not used between the two assignments, the first one can be eliminated.
But this optimization ("stale assignment elision") was wrong for this modified code:
a = 10;
return; // note: `return` in the main shader body is the same as `exit()`
a = 20;
because the `return` changes things -- the second assignment won't be executed, and therefore should not retroactively eliminate the earlier assignment. The correct final value of a is indeed 10.
The problem boiled down to a bug in OSOProcessorBase::find_basic_blocks, where the `exit` op was simply not recognized as terminating its basic block (since it is a jump).
Spatial geometry triples (point, vector, normal) now support A.x, A.y,
A.z as synonyms for A[0], A[1], A[2], respectively. It's not the
notation I prefer (obviously, or I would have done it 25 years ago),
but other people really dig it, so fine by me.
Colors now support A.r, A.g, A.b as synonyms for A[0], A[1], A[2],
respectively. To be honest, this worries me a little because I fear
we will regret it if we ever shift to supporting spectral rendering
(and the components will not necessarily represent red, green, and
blue, and may have arbitrary number). But again, people really seem to
like this, so fine, but don't come complaining to me in a few years
when your shaders all need rewriting for your new spectral renderer.
In both cases, it's for direct reference to lvalues (variables, including
references passed as arguments to functions):
val = P.x; // yes
P.y = val; // yes
function_with_float_param (P.z); // yes
But you can't use this notations on arbitrary expressions:
val = function_returning_color().r; // no
val = (P + N).y; // no, double ick
We are also NOT supporting "swizzling."
Note that this is strictly an oslc-side change, underneath C.r compiles
to the same ops as C[0] always did, so newly compiled shader using this
new syntax can be executed with old renderers/shading engines.
Keep warnings about marked-as-deprecated methods from breaking the OSL build.
On OSX, Homebrew just upgraded to a new enough Qt to see this breakage.
Also fix install_homebrew_deps to make sure Travis OSX test can find
Qt so it builds osltoy. (It wasn't before, so this build break would
not have been found by Travis-CI.)
Refactor call_function methods to use a span instead of raw pointers to unify disparate idioms
Simplify llvm_call_function to use a short vector as it is never used on very long arrays
Remove get_texture_perthread from RendererServices to reduce confusion
If a texture_handle is not provided to the texture related calls, attempt to get one and then always use the texture_handle version of the call since it also accepts a texture_thread_info
Always fetch a texture_handle before trying to make a call into the OIIO texture system so that texture_thread_info can be consistently used
Constant fold gettextureinfo even if the call would fail (and generate an error message to maintain runtime behavior)
Constant fold gettextureinfo even for arrays (did not require extra code)
The current implementation of the OptiX run time defines its own closure structures that replicate that of the native OSL closure structures. This is likely just legacy code as it was easier to use OptiX types like float3 than add support the native OSL types. Removing this duplication and using the native OSL closures improves the maintainability of the code as you will no longer need duplicate logic if the structures ever change.
* Add support for native OSL closures in OptiX
* Ensure Imathx.h is included before OpenImageIo headers in device code
* Add more intelligent memory alignment logic
* Remove unnecessary ustring.h include
Allow shader params with empty brace inits:
type myarray[] = {},
mystruct S = {}
Un-lengthed arrays, if they aren't connected or given a concrete
instance value, still default to length 1 (we don't allow 0-length
arrays, that's a whole other can of worms), but this is now treated as
equivalent to `= { default_value }` (0 for int or float, "" for string).
And it works for structs, too, just means to initialize all the elements
to is obvious (0, "") default values.
It's syntactic sugar, but it removes some clutter for people who want to
have shader parameters that are arrays of undetermined length (and thus
are probably expected to have a runtime-assigned instance value or be
conntected to an output of an earlier layer within the shader group),
removes the requirement for spelling out an initializing value that
won't be used anyway.
When using the variety of ShaderGroupBegin that takes a serialized
description of the group, errors in the individual 'shader', 'param',
or 'connect' commands were not bubbled up into a proper error message
for the ShaderGroupBegin call itself, nor caused it to return null
as the group.
Be more diligent about printing the group name with all group setup
errors. This includes in testshade, no longer accidentally clearing the
group name.
Recently, OIIO sysutil.h added a way to set up a stack trace dump in
case of crash. Use those in OSL binary commands (it does not
automatically happen from the library). This only work with a new
enough version of OIIO, and only if that OIIO was compiled with a new
enough boost to contain boost's stack trace library (>= 1.65).
This is primarily to help developers figure out what's going on with
CI tests fail on remote machines -- you can't catch it in the
debugger, but maybe the stack trace will give you the clue you need as
to what's going on. I definitely don't expect it to be very helpful if
the crash happens in the JITed shader. But if the crash is in our code,
maybe it will help the detective work.
To that end, we also change the testsuite runtest.py so that when a
command fails entirely, print its console output.
And bump appveyor boost to 1.69, so that it's new enough to support
the stack trace library.
Add 'OPTIX' label files to a whole bunch of testsuite tests to force
them to run in OptiX mode also.
Make sure optix tests run both optimized and unoptimized.
Moany tests still ran 'testsuite -g 2 2' to run at 4 points as if on a
grid. This was an artifact from the early days when we had a "batched
shader bytecode interpreter" that handled "varying" specially and
needed to run on multiple points at once to ensure that they were
shading entire grids properly. But once we switched to LLVM JIT and a
"single point shading model", that was no longer necessary. But never
had a good reason to change them before.
Lots of tests "failed" OptiX mode just because printf wasn't working
correctly for multiple points. That's silly, we can test printf interleaving
separately and with one test. No need to make passing dozens or hundreds
of others contingent on that one problem, rather than hinging on whether
OptiX path can really implement the feature that the test was designed
to demonstrate.
So for a bunch of tests, I changed them from running 'testsuite -g 2 2' to
just running on one point, and enabled OptiX mode to get a pass.
All in all now, I believe we pass 37 tests in OptiX mode (both optimized
and not optimized). We will try to grow this week by week.
* Allow OSL to build with USE_LLVM_BITCODE enabled on windows
* Explicitly call python script from cmake python
* Added NOMINMAX define required before including optix_world.h on windows
* Send bitcode as an array: avoids a temporary allocation and passing a std::vector.
* Add OSL_CONSTANT_DATA decorator, and remove the CPU->GPU transfer of pvt::Spline::gBasisSet.
* Add BackendLLVM::llvm_load_string
* Remove unused optix calls.
* Warn about missed optimizations
* Properly track gettextureinfo handle use in the stats.
* New ShadingSystem option: "opt_warnings" enables warnings about
things that couldn't be optimized and may be performance issues.
* New ShadingSystem option: "gpu_opt_error" enables full error status
of the subset of those warnings that are also hard no-go's when
executing on GPUs.
As an example, the first such warning/error is for texture calls where
the name of the texture cannot be fully resolved to a known string
constant during the course of runtime optimization. We will add other
warnings/errors over time.
The warnings/errors are sent back to the renderer's ErrorHandler,
and as an example they look like this:
Optimization warning:
group: bumpy_material_14
layer: disp_texture_2
source: shaders/tex.osl:58
issue: texture(): texture name cannot be reduced to a constant.
* Optix: support array and va_args printing
* Adresss issues raised.
* Remove thread-testing from osl_printf for Optix (this should be handled by the test-runner).
* Fix bug from PR 1001, default value for missing OSL_SOURCE_DIR env
variable should not have included the testsuite directory itself.
* Be sure to correctly redirect all stderr to the same saved file as
stdout. This makes it easier to have testsuite entries that test
warning or error output.
* Add additional more Imath_cuda types and overrides, but try to consolidate their inclusion (or maybe use the pragma in Imathx.h).
* Remove unused include.
* Fix an issue with OSL::ShadingSystemImpl and it's colorspace.
* Add colorspace strings to Device.
We previously had each test run twice: once with -O0 and again with
-O2. Because one tests the actual literal code specified in the shader
(without the possibility of being optimized away) and the other tests
the optimization to make sure it still matches.
This patch extends the logic to allow the possibility of every test
running in optix mode as well, if the code base is built with
USE_OPTIX=1.
For now, it only runs optix mode if the individual test has "optix"
in its name, or if the test directory has a file called "OPTIX" present.
Also, if environment variable TESTSUITE_OPTIX=1, it will try to run
*all* tests with optix mode (that's just for me... most of them fail
at the moment).
The eventual goal is for these conditions to change, so that in a
USE_OPTIX=1 build, *all* tests run optix flavors unless not marked
with a NOOPTIX file in their directory.
At the moment, *6* tests pass for me in OptiX mode. Now we can start
attacking the features to get more and more tests to pass identically
on CPU and GPU.
I realized that there were many tests that are only trying to test
that oslc issues the right error or warning for various malformed
code, or that are only compiling the shader to test oslinfo, or a few
other such cases that run oslc only and never try to execute their
shaders. So I also liberally sprinkled NOOPTIX as well as NOOPTIMIZE
marker files in those test directories, to skip optix tests, and also
skip a second "optimized" run of the same test (which was pointless,
because it wasn't executing at all).
We had a weird chicken-and-egg problem after a recent testshade
refactor, where getargs parsing made the individual shadingsystem
calls to set up the group, but also set other options we wished had
been established before this parsing starts.
The solution is change the main getargs to NOT fully set up the shader
group, but instead to just stash away the group setup args for later
processing. Then after the renderer and SS are fully set up
considering all the other options, we re-parse the stashed options and
set up the shader group.
This fixes some embarrassing reliance on the TESTSHADE_OPTIX environment
variable during tests.
* Allow testsuite to run (and hopefully pass) when built outside the source tree.
* Search for PTX files next to binary before PTX_PATH (installed path).
Fixes an issue when rebuilding with changes to PTX, that build would have to be installed to be tested.
* Add OSL::optix::Exception and catch the real optix::Exceptions in testshade.
* Make OptixGridRenderer::load_ptx_file without Optix a compile error
* Use OIIO to get the program directory and change OSL_USE_OPTIX guard
* Make optix_compat a private include & not installed.
Explicitly include stdlib.h for malloc and free.
The implementation of #984 inadvertently lost one of the old
variants of the SS::Parameter() call. There was a 3-param version
and a 4-param version separately declared, but somehow in the code
that was merged, only the 4-param version has an implementation.
Fix by merging the declarations so that it's a 4-param method with
a default value for the last argument (which I think was my intent
all along).
Works for simple cases. Use with caution, there is a LOT that doesn't
work in the OptiX path. Embarrassingly so. But this refactor makes it so
that the last of the old "testoptix" oriented tests now work using
"testshade --optix".
It also requires that environment variable TESTSHADE_OPTIX=1 must be
set. This is for weird reasons that have to do with argument orders,
and will go away in the next refactor of testshade, so I don't even
want to bother explaining. It's about to get fixed.
There is a lot that doesn't work properly. Frankly, I would encourage
onlookers to wait for another couple of checkins from me before wasting
any time on this. The most embarrassing parts will be fixed soon.
* Revise shader install path and stdosl.h search
Issue #955 points out that we install the example shaders (and stdosl.h)
in PREFIX/shaders. That may be fine for a totally custom directory, but
it's nonstandard and rightly frowned upon if the prefix is a system-wide
place like /usr. Just like docs go in PREFIX/share/doc, probably the
shaders ought to go in PREFIX/share/shaders.
So this patch makes that change. Does anybody think that will be
extremely disruptive?
Modify oslc/liboslcomp to search in the new location (as well as
the old), and while we're at it, I was able to significantly clean up
and compress find_stdoslpath(). Also adjusted what in retrospect was
a logical error in how we search for stdosl: I now think that the
user-supplied include paths (for example, `oslc -Ifoo -Ibar`)
probably ought to be the first places searched, not the last (as it was
before).
Fixes#955
* Move shaders to .../shared/OSL/shaders
* Allow option to control shader install dir
Consider int=float to be a warning, not an error.
This is partly because C allows it and we ran accross some
examples where it seems more strict than necessary.
(Of course it works without warning if you use a cast, showing
that any trucation is intentional.)
Turns out that some of the various type checking we do for simple
assignment,
x = y
was not being checked at all for variable declaration with initialization,
Type x = y
So this patch adds the necessary checks. And also goes back and improves
the error messages for regular variable assignment, changing from messages
like
Cannot assign 'float' to 'int'
to the more clear (I think, because it names the variable, and looks
more like the grammatical construct being complained about)
Cannot assign int i = float
This extensive refactor fully merges all remaining testoptix
functionality into testrender, and deletes the remnants of testoptix.
There's still lots of refactoring I intend to do moving forward,
especially as I made an optix mode to 'testshade' as well. But this
is a good checkpoint to merge.
This attempts to merge them. In the process, it's a pretty big overhaul
of the internal organization of testrender, a lot of stuff moves around
between files and classes are refactored significantly.
N.B. render-veachmis changes slighty, only on osx, and I don't know why.
Return to investigate later.
Revise to bring in line with Roman's changes
I made a bunch of changes to testoptix, not altering any functionality
but just mashing things around to make it more symmetric to testrender
and make diffing between them easier.
Several file have been removed from testoptix, which now gets them
from the copies in testrender (so at least there's only one copy now):
all the .cu files, testoptix stringtable.{cpp,h},
shading.{cpp,h}, raytracer.h.
All that's left now in testoptix is testoptix.cpp, optixrend.h,
optirend.cpp. In a subsequent PR I will complete this process by
replicating all remaining testoptix functionality in testrender and
then getting rid of the remainder of testoptix.
This attempts to merge them. In the process, it's a pretty big overhaul
of the internal organization of testrender, a lot of stuff moves around
between files and classes are refactored significantly.
N.B. render-veachmis changes slighty, only on osx, and I don't know why.
Return to investigate later.
Revise to bring in line with Roman's changes.
I made a bunch of changes to testoptix, not altering any functionality
but just mashing things around to make it more symmetric to testrender
and make diffing between them easier.
Several file have been removed from testoptix, which now gets them
from the copies in testrender (so at least there's only one copy now):
all the .cu files, testoptix stringtable.{cpp,h},
shading.{cpp,h}, raytracer.h.
All that's left now in testoptix is testoptix.cpp, optixrend.h,
optirend.cpp. In a subsequent PR I will complete this process by
replicating all remaining testoptix functionality in testrender and
then getting rid of the remainder of testoptix.
The ShaderGroupBegin / ShaderGroupEnd (with interspersed Parameter,
Shader, and ConnectShaders calls between them) relied on hidden state
indicating which shader group was the single one undergoing
specification. This made the shader declaration process unable to
occur concurrenty for different shaders by different threads.
This patch introduces new versions of Parameter(), Shader(), and
ConnectShaders() that take an explicit `ShaderGroup&` parameter and
don't depend on any saved state. Remember that ShaderGroupBegin
returns a ShaderGroupRef, it's this dereferenced object that is passed
to the other functions that help specify the contents of the group.
This allows multithreaded apps to be building the scene, including
shader specification, from many threads concurrently.
The old stateful versions remain, and are fine to continue using for
any apps that are sure that all shader declaration will be done
serially (though we will perhaps mull over whether to eventually
deprecate it entirely). Though internally, the amount of saved state
is now reduced to merely the one shared pointer that designates which
is the single active group being specified, the other former bits of
state either being entirely redundant or moving to be local to the
group itself.
It's easiest to explain the situation by example:
void receive_array_3 (float array[3]) { }
void receive_array_any (float array[]) { }
receive_array_3 ({1.0, 2.0, 3.0});
receive_array_any ({1.0, 2.0, 3.0});
Both of these would fail/crash by hitting an assertion in oslc. The
general pattern is when passing an initializer-list (like just saying
`{ a, b, c }`) to a user function formal parameter that's an array.
They failed for somewhat different reasons if the formal parameter
was a concretely-sized array or if it was an unsized array.
A little bit of typecheck-fu handles this case.
* Change -r to -res to match testrender (and for that matter, make -res
work for testshade, not -g). (In both cases, the old oddballs are
silent synonyms.)
* Respond to several additional OSL options, and set them the same
was as testrender
* Allow --iters (much like it works in testshade) to make it easy to
run all the rays many times to make accurate timing easier.
Also do a little bit of synchronizing of near-identical code between
testshade, testrender, and testoptix. And synchronize some of the
analogous command line arguments (e.g., -res for everybody, not -res
for testrender and -r for testoptix and -g for testsahde).
We previously had warnings that attempted to catch mistakes in using
the "comma operator" when the user intended an "aggregate type
constructor", for example:
color x = (a, b, c); // comma operator! equivalent to: x = c;
color x = color(a,b,c); // this is what they probably meant
However, the way we implemented the warning was specific to assignment:
x = (a, b, c);
and variable declaration with initialization:
color x = (a, b, c);
But it missed similar bugs in other constructs, for example
color x = Cdiffuse * (a, b, c);
Here the user surely intends to multiply Cdiffuse channel-by-channel
with color(a,b,c), but what they will actually get is Cfiffuse scaled
by float c.
This patch changes the strategy for this warning, no longer tied to
variable declaration or assignment, but more generally any time the
comma operator is used inside expression parentheses, it will give a
warning.
Rely on CMake's own finding of flex/bison, simplifying our code to
just the helpful shortcut macro. Also make finding these REQUIRED,
which makes an easier-to-interpret error message than simply not
defining the macro.
* Factor out the CUDA-to-LLVM compilation shared between liboslexec and testoptix.
* Add some decorators and typedefs to allow math headers to be used for device code.
* Add device support for Gabor noise and generic noise. Add NoiseOptCUDA and a faux ShadingContext to use in device code.
* Rejigger the OSL DeviceStrings namespace.
* Add a test for Gabor and generic noise in testoptix-noise.
* Update DeviceString example code in wrapper.cu.
* Destroy the OptiX Context before exiting testoptix.
* Refine DevieString handling. Improve handling of userdata params when running in OptiX.
* Bring testoptix up to date.
* Use a more consistent namespace for DeivceStrings.
* Use ustring::format instead of sprintf when constructing CUDA variable names.
* Use a std::string instead of a ustring when constructing the CUDA variable name. Guard the ustring::sprintf use when constructing the userdata param name.
* Replace hash noise with simplex in the OptiX noise test (hash is unstable if ray hit points change ever so slightly as OptiX evolves).
This is not useful in practice and we've seen a case of exponential DFA
state blowup in issue #971 because of its accidental use. This does not
make LPEs totally robust to ReDoS but prevents this particular mistake.
This patch allows an optional suffix to be specified that will let you
produce something like liboslexec_spi.so or whatever. Use with caution,
or preferably not at all, if you don't absolutely need it).
Example of how it's used: Suppose you're at a facility that needs (don't
ask why) multiple versions of OSL to be built. And namespacing is not
enough, because if you have two copies of liboslexec.so, you can't
separately rpath them.
There were a variety of places that took a Context* that defaulted
to null, where passing null means to let the called funtion allocate
a context.
This relies on a boost::thread_specific_pointer which could be easily
mishandled and lead to problems.
So with this patch, we move toward requiring the app to allocate and
own the contexts and PerThreadInfo objects.
* No more default null parameter value on contexts or threadinfo.
* Explicitly passing nullptr will still work (use with caution!) on
2.0 with a warning, but will be a full error in 2.1 as that functionality
is totally removed.
* execute() takes a ShadingContext& instead of ShadingConext*, to emphasize
that it must not be null. (For 2.0, the pointer one will still exist
and work, but is slated for deprecation.)
* Fixed comments that document "it's advised o pass a contexts, but simple
apps can choose not to" to now imply that it's required.
* A bit of reorganization of SS::getattribute -- some group attrib
queries required the group to already be optimized, and there were many
calls to optimze_group that all had the implicit null context. These
are all coalesced now, and it allocates a temporary context.
Fixes#691
Fixes#654.
There's a problem with the lock we use to protect the global
`oslcompiler` pointer -- it's locked for the duration `osl_parse_buffer`
(where it's assigned), but it is used and also cleared to NULL after the
lock is released (in particular, the clearing that happens in compile()
and compile_buffer().
The solution is to move the lock itself to compile/compile_buffer, and
do all of the assignment, use, and clear within the scope of that lock.
I think this went unnoticed for so long because most of us highly
multithread the shader execution (and runtime optimization), but not the
initial compilation -- except for Issue #654, and my sincere apologies for
letting that slip through the cracks.
I guess everbody who builds with NAMESPACE=... to customize OSL's namespace
must (like we do) pick a name that include the substring "OSL".
Because if they don't... a bunch of public API symbols will be improperly
hidden!
Oops. Seems like the most foolproof solution is that if a custom namespace
is supplied that doesn't include the substring "OSL", have the build
script prepend "OSL_". Then the symbol hiding regex will work.
(You may object: "But then you're not using exactly what they asked for!"
Correct, but remember we are already overriding their literal choice by
automatically appending the OSL version number as well.)
While there, also take care to hide symbols containing "pvt", which I
think should not be exposed, despite the fact that they're in the main
OSL namespace.
I have no idea how this ever worked. In fact, it must never have, because
our checked-in reference output was wrong. But it kept matching the same
wrong until it didn't, and became a different kind of wrong on at least
one machine.
When doing transformc, where the space names necessitated using OCIO for
the transformation math, and only for the derivative case (the non-deriv
was fine all along), we were simply passing the wrong stride parameters
to the colorproc->apply() call.
Also, I noticed that we were trying to save a per-context "last
transformation" to avoid the lookup when doing repeats of needing the
same colorproc (basically, a 1-item LRU cached lookup of the
colorproc). But we never saved the space name, so the cache was not
operating.
We were clipping the range incorrectly for bezier, where it needs
to step by 3 CVs for each span. Worked ok for the other spline
types, which step by 1.
Add a test that verify that a spline designed to give a f(x)=x response
does so, and also that its inverse is perfect.
Fixes#778
The problem was specific to using an integer variable directly as the
loop test, rather than a comparison. So for example, this would work
fine:
i = 3;
while (i > 0) {
--i;
}
But this would loop forever:
i = 3;
while (i) {
--i;
}
Because of a hilarious chain of events, this would eventually lead to
the optimizer thinking that since the value of i was nonzero at the
beginning of the loop, it must be equivalent to while(1). There was a
subtle assumption that the scope of the loop control variable would be
entirey inside the condition code. That was true of the the temporary
created to hold the result of "i > 0", but not true of the direct use of
i itself. The solution is to force it to create a bool comparison
(which turns into the same machine code later anyway, so there is no
efficiency issue here).
Fixes#776
An empty conditional clause of a `for` statement was leading to bad
things. It's understood that `for (;;)` means `for (;1;)`, so just
make sure that happens when parsing.
Fixes#688
I didn't notice this in Tim's code before today, but he made the
retrieval of these group attributes more complex than they needed to be.
Instead of messing with pointers (which we avoid), you can just directly
getattribute(name,string&). So now we use that. Also eliminate group_name
entirely, since it's redundant with the existing groupname.
It was never supposed to be allowed. It used to be ignored in most
cases, error neither detected nor issued. PR #878 "fixed" it to detect
the error. But there were too many shaders that did it, users are
confused and it breaks things.
So this patch downgrades it from an error to a warning. If you ignore
the warning, you get what you deserve (which is probably not
bad... people lived with it for years). Anybody who wants it to be fully
illegal can always compile with -Werror to turn warnings in to full
errors.
I also "uniformized" two separate error messages on the same general
topic, so they look the same.
To make debugging easier and be more human-readable, oso files are
littered with comment lines reproducing the source code, so that you can
see which source corresponds to which "oso assembly".
The workhorse here is the retrieve_source(filename,line) function. But
it was unspeakably inefficient, reading the entire source file every
time an op needed a different source file than the previous op, and
scanning the file to find the n-th line *every* time. This seems like
the lookup is O(n^2) and also that we're rereading source files
potentially thousands of times.
Theory: If we cache all the files we read, so we never read each source
file more than once, and also remember which line we used last (we tend
to go in increasing order in a source file) and resume the search from
there, backtracking only when the line is less than the last one needed,
it ought to be blazingly fast!
Practice: It barely makes any difference. Compiling all 600 MaterialX
source osl files takes 30 seconds on my laptop, and this fix saves maybe
a second or two at most.
Oh well. But I already made the change, so may as well keep the more
efficient algorithm. Perhaps there are other cases where it gives a
bigger advantage.
The `OSLCompiler::compile_buffer()` method, which is used to compile
osl source code from a memory buffer, had unhelpful error messages
such as:
<buffer>:11: error: Unknown struct name: x
because of course, there was no filename, just a memory buffer. This
is ugly and confusing to users.
This patch adds an optional parameter to compile_buffer() that lets you
supply a "filename" to use for error reporting of the mem-buffer-compiled
shader. This makes pretty error messages if you supply a good name:
zap.osl:11: error: Unknown struct name: x
An application of this is shown with the changes in osltoy.
For my own debugging purposes, I also implemented a `-buffer` option to
oslc, which just forces a file read, compile-from-buffer, and then file
write, so that it's easier for me to exercise the buffer code path.
One side was trying to make use of dresultdz while the llvm codegen was always providing nullptr. Since the shading system only supports Dual2 at the moment for derivative lookups, it seems more consistent to forget about trying to come up with a meaningful Z derivative for the time being.
If you have a function that is polymorphic based on return type --
like cellnoise, which could return a float or a color -- you will get
a warning about the ambiguity in certain situations which you can resolve
with an explicit type cast:
(float) cellnoise (u, v) // the cast disambiguates which you want
But Thiago found that the similar idiom using a type constructor (rather
than a cast) like this:
float (cellnoise (u, v)) // fails to disambiguate cellnoise?
fails to disambiguate and gives a warning about ambiguous types (it will
choose the float-returning kind, but will warn about the ambiguity, which
will be an error if you compile with `-Wall`).
This patch fixes the problem by making ASTtype_constructor::typecheck()
aware that float(expr) ought to expect expr to be a float if possible,
and also triple(expr,expr,expr) should expect all three of the channel
expressions to disambiguate to float. It previously made no such inference
in those cases. Note, however, that triple(expr) did previously know that
the expression ought to prefer to disambiguate to the same triple type.
While investigating, I also saw that the oso generated by that same
constructor sometimes did an extra unnecessary temporary assignment, so
while I was ast it I also fixed that with the changes you see to
ASTtype_constructor::codegen (in fact, these changes altered more code
than the type checking part).
Fixes#930
OIIO's ustring has an "operator int()" that is possibly ill-advised, as
claimed by Thiago Ize and John Haddon from an oiio-dev thread from back
in 2015 (I'm doing some old thread necromancy). They claimed that an
operator int (or for that matter bool) could result in subtle bugs from
type conversions you don't expect to happen. The fact that there's no
equivalent operator int/bool for std::string or std::string_view
bolsters their argument.
I started off just trying to eliminate all OSL uses that assume this
operator would work (the proper idiom is simply `!str.empty()`). Along the
way, I found *TWO* bugs related to this -- one in osltoy and one
when constant folding "wrap" arguments to texture functions! So Thiago
and John were surely right, and I am appropriately chastened and
belatedly trying to find a way to get rid of the troublesome operator.
* Use the "official" FindOpenImageIO.cmake from OIIO (recently
added). This necessitates some renaming, for example standard use of
OPENIMAGEIO_ROOT_DIR.
* Change inconsistent naming of OSLHOME (and OSL_HOME) to the more
standard OSL_ROOT_DIR.
* Copy updated FindOpenEXR.cmake from recent OIIO (and related changes).
* Eliminate all of testsuite/MaterialX, which was not used, never fully
worked, and was certainly out of date with our MX support. If we want
to add unit tests for MX shaders, I think I will take a completely
different approach. So for now, just eliminate the clutter.
* Eliminate site/pixar/Makefile-bits, which they say they no longer use
(some of those customizations were the very last uses of some weird
custom stuff in our older FindOpenEXR.cmake, now eliminated).
Recent changes used a new ImageBufAlgo convention only present in current
OIIO master, not older 1.8 that we still promise compatibility with.
Didn't catch this before because the Travis tests don't install qt and
therefore don't build osltoy at all. Add qt to the Mac travis setup so
that it at least builds to prevent this from happening.
New rules:
* If NO "searchpath:shader" attribute is specified to the ShadingSystem,
the current working directory (".") will be searched.
* If a searchpath is specified, it will look ONLY in those directories,
in order. The current working directory will only be searched if
it is listed among the paths.
Zap is seeing sporadic (but frustraingly impossible to reproduce) crash
reports from inside this function, specifically in the part that parses
%argrw hints.
There was a longstanding comment: // FIXME -- use string_view ops here
So, sure, rewrite the whole thing using string_view and Strutil::parse
functions, and remove the assertions, and put extra guards to make sure
we aren't acessing a symbol or op (via blah.back()) if there aren't any
in the vector yet, and pretty much just let it ignore or be robust to any
malformed hints.
I've never seen a repro case, so I don't know that this will fix it, but
it's programmed a lot more defensively now, so hopefully there will be
no further crashes (after I backport this to whatever version Zap is
using). I even put in an error message for a certain way to malform
%argrw, in the hopes that it might give us a clue, but even that is just
a guess.
For these functions, oslc failed to mark the results output param as
write only.
Fix the bug in oslc, and also put in a provision in loadshader that will
correct the old already-compiled incorrect files.
Thanks to Steena Monteiro from Intel for pointing this out!
The changes in oslcomp.cpp seem only to be necessary with particular
combinations of clang7 and older llvm. Which we don't recommend! Better
to use the same generation of clang and llvm. But in any case, may as
well make set those options explicitly.
Also switch SPI to build with clang7/llvm7 by default.
A suspicious line in testrender seems to depend on undefined behavior
regarding the specific order of evaluation of quantities in a math expression.
It has been there forever, but only seems to fail on certain builds of gcc 5
and possibly clang 7.
It's just a one-line fix to remove the order-of-evaluation ambiguity.
Thanks, Tim Grant, for pointing this out!
'Pause' (which changes its label to 'continue' when paused) stops the
clock and continued execution of the shader, until 'continue' is hit.
'Reset' sets the time counter back to 0.0 again.
It starts in a paused state, and starts the clock the first time you
successfully compile a shader.
WARNING: possible visible change in procedural patterns
It's expected that cellnoise(x) == cellnoise(floor(x)), but we found an
ugly bug wherein cellnoise internally used a helper function
quick_floor, which is supposed to take the floor of a float argument and
return its integer equivalent. But unfortunately it had a really odd
error: when x was an exact negative integer, it gave the value for the
next smaller integer.
quick_floor( 1.00) == 1
quick_floor( 0.01) == 0
quick_floor( 0.00) == 0
quick_floor(-0.01) == -1
quick_floor(-0.99) == -1
quick_floor(-1.00) == -2 // WRONG!!! Should be -1
quick_floor(-1.01) == -2
This in turn caused cellnoise lookups of exact negative integer values
to be incorrect lookups of the next lower-numbered cell, i.e.,
cellnoise(i) having the same value as cellnoise(i-epsilon), but again,
only for i<0 and i==floor(i) exact integer values.
Now, if your shader made a visual representation of callnoise(x,y), it
would look just fine, unless a particular x,y fell *exactly* on integer
coordinates, in which case it would be ever so slightly off (by being
the color of the square immedately to the left instead of to the right,
as it should be, say). Nobody would ever notice that.
Also, if your shader took the floor before passing to cellnoise, like:
cellnoise(floor(x),floor(y)), it would also *look* correct, though the
negative values would be wrong, since cellnoise(-1,-1) would have
returned the value that should have been cellnoise(-2,-2), etc., but
you never would have noticed that.
But... there were some possible uses of cellnoise where it was behaving
very strangely, and the main place you would notice this is certain
types of "voronoi" noises where you are lookup up cellnoise values while
looping over adjacent lattice points. (Thanks, Zap, for noticing this,
and so sorry that there was a legit bug here!)
It was a very slight possible problem for perlin style noise as well,
but since those noises are "continuous" and defined by gradients, you
really, really, would not have known that the occasional pixel had noise
results that were ever so slightly off in its derivatives only at those
isolated integer lattice points. I think no visible artifacts were
discernable under real-world conditions.
The solution is to bite the bullet and replace the clever (but extremely
minor) speedup for computing floor, with a legit floor call. It's ever
so slightly more expensive if you isolate the cost of just raw cellnoise
calls, but in the context of a real render I don't think you'll be able
to measure it.
WARNING: This may change the appearance of certain "Voronoi" patterns,
or possibly other uses of cellnoise as random number sequences. It also
changes the appearance of "Gabor" noise, which internally also used the
faulty quick_floor in such a way that it affects the whole pattern.
Note that this should NOT change the appearance of any of the
perlin-style noises, and you may not notice any change for most ordinary
uses of cellnoise, either.
I changed the oslnoise unit tests to check that this is working
properly, and updated all of the noise unit tests that generate images
to "center" the patterns so that they visibly display the negative
regions of the noise domain (they had all shown only positive domain,
which is probably inviting trouble if there was ever a noise bug that
only affects what happens for negative values or right on the axes).
This caused a lot of the reference images to change, even for the tests
of noise patterns that were unaffected. Also, I took the chance to merge
a few of the signed/unsigned tests for less repetition.
I'm very sorry about this; changing the appearance of noise functions is
not something we take lightly, we do it only as a last resort. It was
clearly wrong before and had some very unexpected failure modes. But I'm
only making this fix in master, which is destined to be "OSL 2.0", so I
figure it's as good a time as any to make a change that could possibly
make visible changes in some shaders.
If you see this as some kind of major betrayal that will screw up your
productions, let me know and if pressed I can perhaps make some kind of
emulation of the old buggy behavior for people who need to preserve it.
But I'd rather not do that if it's not absolutely necessary.
* Modernize the way we generate memset and memcpy ops -- we still had
some artifacts of LLVM 3.x; since LLVM 4+, there has been a more
direct way to generate the equivalent of memset/memcpy (and be
careful, the API changes slightly for LLVM 7.0).
* Some header file changes with llvm 7.
* Minor ref-vs-ptr miscellany for WriteBitcodeToFile and CloneModule.
* Slight change to addPassesToEmitFile call signature.
* Radically simplify build_materialX_osl.py -- It previously had a whole
lot of unused complexity by being able to build multiple template
expansions at once, and to do ALL template expansions without the cmake
system. But we never used it that way. So simplify it to handle a single
shader and type at a time, which is how it's called from our cmake
scripts. This makes it easier to maintain and removes a lot of redundant
logic that was duplicated (possibly incorrectly) between cmake and
python.
* Modify MaterialX build process to handle "OTHERTYPE" so we can get a
templated cross-product of two type lists (though by default, OTHERTYPE
is "none", so it's just a linear list of types we expand to). OTHERTYPE
is also allowed to be "same" which means it's the same as the first TYPE.
This makes it easy to make a templated node where for a bunch of types T,
it has a T,T version, and also a T,float version.
* Implement MX "convert" node, which utilizes the ability to have two
templated types.
* Clean-up of lots of nodes where we special cased the vecN/colN versus
itself, versus vecN/colN versus float. Now that we can template on two
types, these all simplify to one template for the "same" case and the
"float" case.
* File/node naming convention: for all nodes that can be templated on
a single type, we continue to call the OSL shader mx_thefunction_thetype.osl
(and .oso). But for all nodes that can be templated on two types,
(add, smoothstep, etc.), they will be named mx_thefunction_type1_type2.osl.
* MaterialX: fix definition of ramp4 top vs bottom
In recent times, it's been decided that "top" always refers to v=1 and
"bottom" refers to v=0.
* MaterialX: various clean ups to texture mapping nodes
* image node: wasn't honoring addressmode or filtertype params.
* Fixes to triplanarprojection: wasn't properly normalizing by the
combined weight of the three views, wasn't honoring filtertype param,
wasn't forcing periodic wrap.
* Implement the new "tiledimage" node.
* MaterialX: extend add/sub/mul/div to include matrices per 1.36 spec
* Add a variant of rotate() that just takes an axis, rotate about the origin.
* MaterialX: add rotate node (new with MaterialX 1.36)
* rotate2d note is kept for back-compatibility, but considered deprecated, since it was dropped from 1.36.
* MaterialX: implement new transformpoint, transformvector, transformnormal nodes
* MaterialX: add new nodes rgbtohsv, hsvtorgb
* MaterialX: add new 1.36 node 'hsvadjust', deprecate 'hueshift'
* MatrialX: Add new 1.36 nodes: extract, separate
* Add a device_string type for representing static strings on CUDA devices. Update LLVM codegen to generate and handle device_strings. Add mechanisms to register and query integer IDs associated with device_strings.
* Move the DEVICE_STRING_PARAM macro.
* Remove some redundant calls in addGlobalVariable.
* Remove OptiX exceptions, since they are not well supported.
* Unify the OptiX and non-OptiX path for passing the noise name.
* Re-add osl_printf to rend_lib.cu.
* Add a very basic print test for testoptix.
* Make the warmup launch in testoptix optional. Fix minor indentation problem in rend_lib.cu.
* De-unify the OptiX and non-OptiX path for passing the noise name. Fix some formatting issues with rend_lib.cu.
* Simplify test_print.osl.
* Add ASSERTs to certain OptiX-only functions as a sanity check.
* Add detailed comment to device_string.h.
* Slightly rework device_string example in wrapper.cu.
* Add some macros to make registering device_strings less painful.
* Pulled more of the 'standard' strings into strdecls.h. Added DeviceStrings namespace. Using the STRDECLs in setup_string_tags.
* Update the comment in device_string.h with usage notes for the STRDECLs.
* Changes to compile and link device_strings.cpp separately.
* Working on a device memory string table.
G# ot string assignment seemingly working.
Cleanup.
* Simplify StringTable a little.
* Pack the string length and hash value into the string table, and add strlen and hash functions to rend_lib.cu.
* Replace vector with map in StringTable.
* Handle OptiX printf with wrong args by printing warning and ignoring print call.
* Add rudimentary check for collisions between global variable names.
Fix a bug in the offset alignment logic for the string table.
Add getchar. Add StrRep type for casting raw DeviceString pointers to make accessing hash and len more convenient.
Modified ustring to use cudaMallocManaged. Modified StringTable to use the ustring pointers instead of a separate table. Not working with varying strings.
* Use regular device memory instead of Unified Memory for string table. Add assert to limit active devices to 1. Miscellaenous comments and refactoring.
* Added better re-casting macro for DeviceString. Added comments. Added test for string ops. Misc. small fixes.
* Update OptiX version-finding logic in CMake.
* Rename the CUDA-specific GlobalVariable functions. Use string format when constructing the mangled variable names.
* Add TODO comments about string arrays in CUDA/OptiX.
* Add license to stringtable.cpp.
If you have something like
float table[65536] = { 0.1, 0.35, -0.72, ... }
This might take TENS of seconds for oslc to parse, because the mechanics
of how that long list of values is parsed -- it involves a singly-linked
list of ASTNodes, and with each new one, we are appending to the far end
of the linked list. This leads to an O(n^2) parsing time, where n is the
length of the list.
The straightforward (but wrong!) way to fix it is to change the Bison
grammar for init_expression_list from "left-recursive" to "right-recursive".
That makes it natural to prepend items (O(1)) to the accumulating list,
but it's very inefficient for Bison and can overrun Bison's fixed-size
parsing stack. Bison docs discurage use of right-recursion in grammars,
since left-recursion grammar can parse arbitrarily deep recursive
structures in a fixed memory size.
OK, so how do we fix this? Trickery! Keep left recursion, but accumulate
in the reverse order (i.e., prepend, not append), and then do a single
reverse of the backwards list at the very end. That makes the whole
parsing affair O(n) instead of O(n^2).
Sounds good, but one final wrinkle is that the list we need to reverse
is is a raw pointer to the head, followed by ref-counted pointers to
each subsequent element. In order to keep from accidentally freeing
things if the ref counts reach zero, we have our fancy reverse()
function rely on a brand spanking new feature of OIIO's intrusive_ptr<>
to add a release() method (https://github.com/OpenImageIO/oiio/pull/1986).
But since that doesn't exist in older OIIO, to avoid a compatibility
break, we only do our special left recursion + backwards list + reversal
trick if the OIIO version we build against has that new feature.
Otherwise, it's #ifdef'ed out and we just do it the old slow way.
This fixes OSL issue #723, and for the pathological test case presented
in that issue, it improves compile time (on my laptop) from ~26 seconds
to 0.4 seconds.
I also checked that this funny business with the reversal doesn't slow
down the simple cases for ordinary shader. Compiling the whole
collection of 500+ MaterialX shaders took about 20.5 seconds, and with
the new parsing code is exactly the same, or at least, too close to
accurately discern in the time trials.
Deprecate the use of shader "types" (surface, displacement, etc.)
designated in OSL source code. Those words will still be accepted for
backward-compatibility, but they are no longer required or encouraged.
Generic "shader" type is the only one we will talk about as we move
forward.
Excise the ShaderUse enum. Renderers (or non-renderers using OSL for
programmability) are allowed to determine and document whatever type of
shader "uses" they want to support. The only enforcement from the
liboslexec side is that all the shader nodes within one group must be
tagged with the same "use" string.
The "use" parameter to ShadingSystem::Shader() is intended to be removed
entirely, eventually. However, that's the first of three (optional)
parameters to Shader(), so it's a subtle change that can break renderers
in annoying and confusing ways if we do it all at once, so we'll do it
in stages: For this major release, we will make all three parameters to
Shader() be required (no optional params), which even if it generates a
compile-time error for a renderer, is a very obvious one to fix. Then in
a later major release (2.1?), after we know that all renderer code is
making 3-argument calls to Shader(), we will introduce a 2-argument
version that lacks a "use" parameter (can't break renderer code) and
deprecate the 3-argument call that has the useless "use" parameter, and
then in a later release, we can remove the 3-argument version once and
for all.
Add new queries for shader groups: "globals_read" and "globals_write",
which return a bit field of which shaderglobals fields will be read or
written, respectively, by the shader group after it's optimized.
The bitfield meanings are given by the new enum class SGBits, defined
in shaderglobals.h. This is easier to query (as an `int`) than to sort
through the list of strings you'd get with "globals_needed", and allows
renderers (if they so desire) to (a) verify that shaders don't use or
modify globals they aren't supposed to, for the type of shader use they
are being used for; (b) potentially to avoid any expensive computations
that would need to be performed to prepare certain shaderglobals fields,
for shader groups that won't end up using them.
Also make the `I` global variable be writeable and clarify that if a
global is writeable, it's writeable from any shader type, and we don't
intend to do differing checks based on shader type.
In dual.h, we static_assert involving std::initializer_list<T>::size(),
which is constexpr only for C++14 and on. This seemed to work for clang
and gcc, but MSVC 15 doesn't like it. Guard it so that it only applies
for C++ >= 14, in which the constexpr makes it more certainly legal for
static_assert to reference initializer_list::size().
Fixes#897
Allowing the base class to be used as a trivial RendererServices, can
be helpful for certain applications where not much renderer functionality
is needed.
* Changes to compile OSL noise functions for CUDA: Add CUDA-compatible variant of ImathMatrix.h; add HOSTDEVICE macro to noise functions; modify CMake build for liboslexec to compile and link multiple shadeops sources when compiling for CUDA; change Module link order when compiling to PTX; bump minimum CUDA version to 8.0; set CUDA path for clang according to CUDA version; add CUDA_TARGET_ARCH CMake variable.
* Add testoptix-noise to testsuite.
* Fix up the CMake build flags in liboslexec. Remove the HOST_DEVICE macro from some hash functions in opnoise.cpp. Clean up a few unneeded preprocessor macros.
* Change a few hash function signatures to take void* instead of float*, and cast as appropriate.
Previously, exactly identical error and warning messages (including from
shaders themselves) printed the first time and suppressed the duplicates.
The rationale is that an error in a shader could easily turn into thousands
of error messages if duplicates weren't suppressed.
There wasn't a way to turn this behavior off.
Now it's controlled by ShadingSystem attribute "error_repeats", an int,
which has default value 0 (do not issue repeated errors). If changed
to 1, repeated errors and warnings will be allowed to pass to the error
handler, without suppression in the ShadingSystem.
Furthermore, the act of setting the attribute (to either 0 or 1) has the
side effect of clearing the list of remembered errors. This means that
even if you want error suppression, poking it in this way will reset the
counters, which might be a handy thing to do at the start of each frame
(if you are using a single ShadingSystem to render multiple frames) so
that you start the counts fresh for each render, rather than for each
ShadingSystem.
It has been a longstanding problem that oslc did not properly flag
attempts to write to function paramseters that were not marked as
'output'.
I've rigged this so that in the current master (and any past releases,
assuming we backport this to release branches), it will just be a
warning. In future versions, it will be a full error. That should give
people some advanced warning (no pun intended) to fix any problems that
this catches that previously went unnoticed.
Testshade set certain command line options as a by-product of certain
other ones, and that introduced a weird order dependency to the arguments
that was confusing. In particular,
testshade -O0 myshader
worked as expected, but
testshade myshader -O0
set the options (including optimization level) when parsing myshader,
before it got to the -O0.
So this patch simply makes sure that the shader options are
re-established after all arguments are parsed.
In this change, we specify the VERSION and SOVERSION properties of library
shared objects so that the SONAME in the shared object contains the OSL version
number suffix (major.minor). This also results in the build creating versioned
symlinks back to the installed shared objects. Code that links against the
libraries can then link against a specific version of OSL such that the correct
version can be dynamically linked at runtime.
This follows a similar pattern to the one used in OpenImageIO.
* Apply initial round of changes to liboslexec to support PTX generation. Add function decorators to shadeops in llvm_ops.cpp and to functions in dual.h to support device compilation. Depends on modified version of fmath.h from OIIO, and includes a local version of ImathVec.h from OpenEXR that has been modified to support CUDA compilation.
* Update testoptix to reflect changes to liboslexec. testoptix now loads scenes from the testsuite, compiles the ShaderGroups to PTX, and attempts to render the scene using flat shading. The implementation is limited and very experimental.
* Update the testsuite to reflect the current state of testoptix.
* Get rid of the 'native_target' flag and initialize all LLVM targets in all cases.
* Use ustring::format to construct OptiX variable names, instead of std::string concatenation.
* Don't delete m_const_map entries in destructor, since they are owned by the LLVM Module.
* Cache whether or not the renderer supports OptiX.
* Use the serialization script from liboslexec for testoptix.
* Fix wrong header include.
* Add OPTIX_EXTRA_LIBS CMake variable. Add include path for ILMBase. Specify CUDA path when compiling with clang.
* Refactor ptx_compile_group: replace deprecated default CodeModel option, simplify target/layout option, reword some assert messages.
* Add virtual destructor to Primitive struct in testoptix to appease pedantic compilers.
* Add check for NVPTX target to CMake build.
* Add guard against redefinition of HOSTDEVICE macro. Rename HOSTDEVICE IMATH_HOSTDEVICE. Add decorators to Imath::Vec4 types. General reformatting of ImathVec_cuda.h.
* Move OSL_HOSTDEVICE definition into oslconfig.h, and update includes accordingly.
For many grammatical constructs, the "source line" recorded in the AST
node is where the grammatical case ENDS. As an example,
1: shader foo (int holdout
2: =
3: 1)
4: { }
There is a warning because 'holdout' masks another symbol at the same
scope. You would think the warning should be reported as occurring on
line 1, but it is reported for line 3, because that's where the clause
ends.
This will tend to happen for grammatical constructs that look like
ANCHOR potentially-long-expression-that-may-span-lines
where any sensible person would want the warning associated this to be
on the line of the anchor, not wherever the long expression happens to
end.
In this patch, I carefully fix a whole lot of these grammatical
constructs to patch the 'sourceline' of the AST node they create with
the line that a particular anchor symbol is on.
There's a bit of extra work for ASTvariable_declaration, because there
are some errors (for duplicate symbols, etc.) that happen in the
constructor itself, so it's too late to return from the constructor and
then reset the source line; the misleading error has already been
issued. So in that case, I added a line override parameter to the ctr
itself, mimicking the solution I implemented some time back for a
similar situation with function declarations.
First checkin of effort to make OSL for OptiX/Cuda.
* Add (optional) OptiX and CUDA dependencies to the build system. None of this comes into play unless you build with the optional USE_OPTIX=1
* Add testoptix project to verify basic OptiX functionality. This is similar in spirit to the existing "testrender", but using OptiX to do all the heavy lifting. It does not currently shade with OSL... just laying the groundwork for a test harness.
* Add testoptix to the testsuite.
* Use NVCC for offline PTX generation rather than using NVRTC. Use CUDA_TOOLKIT_ROOT_DIR instead of CUDAHOME in CMake build.
The most important change here is to modify the practice making
multi-line warnings/errors by issuing a warning/error for the first line
followed by message() calls for the subsequent line. Instead, we
generate those line-by-line messages as strings, and at the very end
issue a single warning or error call.
Also, directly call m_compiler->warning or error, don't bypass the
compiler by calling the app's error handler.
The reason both of those are important is that I have a separate change
I am ready to submit that is about giving a `#pragma nowarn` that can be
used to suppress warnings on the following line. This is totally borked
by warnings that span multiple calls and where most of their output is
in the form of message() and thus aren't easily recognized as part of
the warning.
This includes some general stylistic cleanup also in a few spots, some
preferences I have but but did not feel was worth nit picking line by
line in the original PRs.
This lets you have constructs like
// pass a {} list as a triple, matrix, or struct
void func (point p);
func ({x, y, z});
// Assign {} to a struct if the types match up
struct vec2 { float x, y; };
vec2 v = {a,b};
// Compact 'return' notation, it knows to construct the return type
vec2 func (float a, float b)
{
return {a, b};
}
Along the way, a number of steps:
* Initialize struct array elements without going through a temporary.
* Refactor codegen of aassign into single method.
* Allow float <=> int conversions using "aassign" without going through a temporary.
* Allow initializer lists as return types.
* Allow initializer lists as function/constructor arguments.
* C++11 style initializer lists.
* Complete ASTcompound_initializer type check/binding.
* Remove ASTNode typecheck_initlist & typecheck_struct_initializers.
Everything now flows properly through ASTcompound_initializer.
* Adapt to recent function-overload changes.
Hide some implementation details that don't need to be public.
* Error on ambiguity of arguments using init-lists.
* Make warnings & errors involving initializer list arguments more obvious.
Adds color closures, structs, and void to the defined precedence order of resolving via return type.
Errors when disambiguating via return type and more than one user struct is a match.
For some time, the current master had advertised that it requires OIIO
>= 1.8. This patch simplify code by removing #if clauses related to OIIO
versions older than that.
If a shader input parameter has this metadata present and set to
nonzero, no incoming connections via ConnectShaders() will be accepted,
resulting in a runtime error if it is attempted.
* Error on redefinition of functions in same scope with matching signatures.
* Move function redefinition in same scope to back to a warning.
* Call ASTNode::warning with a fully formed string instead of piecemealing formatting.
* Swap order of scope and argument comparison to avoid unnecessary string comparisons.
For function overloading in ambiguous cases, we were (a) doing some weird things; and (b) had the resolution dependent on the order the functions were declared, which seems bad.
This set of changes introduces a new polymorphic ranking system that is better documented, consistent, and not declaration-order-dependent.
For now, if the OSL_LEGACY_FUNCTION_RESOLUTION env variable is set, it should revert to old behavior.
We also fix a bug where type casts (like `(vector)foo()`) would properly pass the desired type hint down to the type checking and selection of foo, but type constructors (like `vector(foo())`) would not. This was an error, and is now also fixed. This may change behavior in some cases.
Better ambiguity error messages overall.
* Appveyor: Fix FreeType discovery for building OIIO and testing OSL.
* Appveyor: Fix passing CMake wrong directory to OpenImageIO.
Always create the download directory and skip Appveyor’s test discovery step.
Connecting only the r,g, or b channels into another color is common in a lot shader networks.
* Support connections between Symbol components.
* Add component connection tests.
* Debug aid: human readable Connection
* Avoid errors for component connections when optimizing.
* Remove init ops for any fully connected destination param.
* Turn off Turn off whitespace insertion when using boost::wave.
* Fix un-initialized members of ASTfunction_call.
Move all initialization into constructor so some compilers can error on regression.
Previously, if you for some reason defined two identical functions in
the same scope (including redefining a "built-in" function exactly in
the outermost scope), it would just silently accept it, and then you
wouldn't be sure which one would actually end up getting called.
For now it's a warning. After people have had a chance to scrub their
code bases, perhaps we will upgrade this to a full error in a future
release.
Previously the order of overload declarations would affect which was chosen.
This attempts to define how overloads are chosen in a more concrete way by scoring:
1. all arguments match without conversion
2. prefer promotion of int -> float over float -> int
3. use return type to resolve any possible ambiguities
Why write documentation when you could make yet another documentation
generator?
I like the *idea* of Doxygen -- generate documentation by extracting the
comments in code, particularly those explaining the individual functions
or methods that are part of an API. But Doxygen, ick. Its markup
conventions make the code comments themselves hard to read, and the
output HTML is... not very aesthetically pleasing (to me, at
least). And it's really a kitchen sink package, a lot of baggage to
carry along for the 2% of it that I really want.
On the other hand, I just love Morgan McGuire's awesome Markdeep
package (https://casual-effects.com/markdeep/), which is an extension
of Markdown but adds a crap ton of neato features that are great for
code, math, and diagrams.
So I tried my hand at doing something really small and "good enough" to
let you mark up your code comments -- but MINIMALLY, so subtly that the
casual observer might not even notice, and using markdeep for anything
fancy -- and then extract that material to generate Markdeep-based
documentation pages that look beautiful.
So here is the prototype. The entire implementation is just one Python
script, src/build-scripts/docdeep. In addition to documenting itself,
I also used oslquery.h to test the idea, so you can see both how the
docdeep markup looks in code, and as well, now we finally have OSLQuery
class documentation, which we never had before. Ha! Those two pieces
of documentation are generated automatically as part of the OSL build, so
you can see what we did on the CMake side to make that happen.
To see how you use it in source, look at oslquery.h.
(Better to just look at the new oslquery in toto, rather than trying to
understand just the diffs of this patch.)
To see what the resulting docs look like, build OSL and look at the
resulting share/doc/OSL/OSLQuery.md.html in your browser.
This is still experimental. I expect it to undergo a lot of revision.
When it settles down, if this looks useful to people and there is
demand, I might split this off into a separate open source project (just
the docdeep script) so that people can use it for other projects without
the overhead of downloading and building OSL. Let's see how it goes.
Feedback appreciated.
* Better guaranteed unique naming of missing names.
* When dumping ll/bc files for debugging to filenames derived from the
group name, beware group names that have slashes in them! Also,
sheesh, watch out for group names that are so long that they exceed
the longest filename allowed (256 chars for most modern filesystems).
* Decided only the ll files are useful for debugging, I don't think we
need the binary bc files.
* Simplify ShaderGroup ctr by using C++11 data member defaults and
delegating constructors.
* Add a new shadingsys option "llvm_optput_bitcode" that will dump the
bitcode for each group, even if the other debug options aren't turned all
the way on.
* bitcode now dumps in text as well as binary. (It saves a step of running
"llvm-dis" if your goal is just ot see the text in the first place, and also
it helps the llvm bug where sometimes it refuses to work because of mismatched
version strings.)
* Make sure that both in the bitcode output and other debug symbols,
we are more careful about being sure the shader group name is included,
not just the layer name, and also if a layer name is not specified to
ss->Shader(), make one up based on the shader master name and the
layer number.
osltoy is a GUI app that lets you interactively edit a pattern shader
and adjust its parameters, seeing the updated rendered results in
real time.
It's the OSL equivalent https://www.shadertoy.com/
The GUI uses Qt5. If that's not on your system (or if you purposely
disable it with the build-time option USE_QT=0), osltoy will not be
built (but the rest of OSL will).
A few other changes that come along with the ride:
* The `shade_image` utility function has changed its API slightly: the
last (optional) parameter that used to be a simple thread count is now
a `parallel_image_options` struct. But this struct can be implicitly
constructed from an int (which is, in fact, the thread count), so any
source code that calls shade_image should not need to be altered.
* New mandelbrot.osl shader has been added. It computes Julia sets, too.
Enjoy!
Compiler error in AppleClang (Xcode8) is
/Users/mark/dev/Thirdparty/OSL-1.9/src/liboslcomp/ast.cpp:76:33: error: call to implicitly-deleted copy constructor of 'std::atomic<int>'
i, node_counts[i], node_counts_peak[i]);
^~~~~~~~~~~~~~
Compiler error in Gcc (4.8) is
/home/mark/dev/Thirdparty/OSL-1.9/src/liboslcomp/ast.cpp:76:68: error: use of deleted function ‘std::atomic<int>::atomic(const std::atomic<int>&)’
i, node_counts[i], node_counts_peak[i]);
^
Believe this is a problem because printf accepts arguments by value, whereas OIIO 1.8's Strutil::printf takes by reference, and you can't copy a std::atomic.
* VisualStudio 2015 x64 build fix
include/OSL/dual.h(64): error C2476: 'constexpr' constructor does not
initialize all members
According to http://en.cppreference.com/w/cpp/language/constexpr, this
is an expected error for C++11, but not for C++14 and beyond.
* VisualStudio 2015 x64 build fix
Using SimplePool in the member variables in ShadingContext is causing an
'attempting to reference a deleted function' error for the
std::unique_ptr<char>s.
Applying the rule of 5, to disallow copying or moving a SimplePool.
Default is 0, for old behavior in which it's an error for
LoadMemoryCompiledShader to try to load the same named shader more than
once.
If set to nonzero, this is allowed, replacing the old shader master.
This does not change any already declared shader groups, which
will continue to use the original definition at the time of their
declaration. But subsequent groups that reference the master will get
the latest one.
Functions that take optional argument lists in token/value form, like
`texture` or `pointcloud_search` needed to mark just those value
references as "writeable" that are (for example, the one following
"alpha" for texture calls, but all of the optional value slots for
pointcloud_search).
The logic on the oslc side here was broken -- it worked when the token
was a string literal, but when it was a variable whose value wouldn't be
known until runtime, it was failing to mark it as writeable, whereas the
correct thing to do is to conservatively conclude that it *might* be
written to (we don't know yet) and mark it as such.
The cascade of errors could lead to incorrect runtime optimization, if
the only used output of the function call was one of these parameters
that failed to be marked as an output, so so the whole function call
might be subject to the "useless op elision" optimization wherein ops
whose writes are never subsequently read can be eliminated entirely.
This cleans up a lot of #if cases where we were trying to support a very
wide range of LLVM releases in the face of changing APIs. With 3.5 gone,
so too is the "old" JIT (it's all MCJIT now).
By the time we get around to the next major release, we may further
reduce the complexity by dropping support for 3.9 (i.e., minimum LLVM >=
4.0), but I'll leave that for a later review.
Also tweak the Travis settings to assure we test against all the supported
LLVM releases.
1. Instead of installing libboost1.55-all-dev, which truly is everything
boost, just install the boost headers and the library components for the
very small number of non-header things we want.
2. Move the Linux package installation to be per-matrix-entry, this
allows us to customize by only installing the specific g++ version we
need, and exclude some boost components (using gcc >= 4.9 removes the
need for boost regex, for example).
This shaves several MINUTES of time from each test matrix entry, which
had been needlessly downloading and installing unneeded packages.
By default, look for a system install. If that's not found, check
the OIIO distribution, where another header-only copy can also usually
be found. Eliminate the USE_EXTERNAL_PUGIXML, I don't think anybody
depends on that as long as the default is to check system libs first.
The only thing this doesn't do well is to have a system PugiXML but
*force* us to use the one from OIIO (whose only possible advantage is that
it's namespaced). If this is a problem in practice, we can return to make
that possible.
We can't fix the code -- it's not "our" code, it it's the stuff
generated as part of the build process by flex itself. The expedient
thing is to just suppress the warning in the flex-generated module.
Please ask questions ("how do I...?" "why doesn't this work?") on the [osl-dev mail list](http://groups.google.com/group/osl-dev), not by filing issues.
Please ask questions ("how do I...?" "why doesn't this work?") on the [osl-dev mail list](https://lists.aswf.io/g/osl-dev), not by filing issues.
Make concrete suggestions for fixing the code ("here is a patch") by filing
This is a checklist for TSC's to review as part of the intake process. The TSC should review this entire list during the kickoff meeting. For anything outstanding, create an [issue](../issues) to track and link to it in the list
- Existing Project Governance
- [X] Project License exists ( [LICENSE.md](../LICENSE.md) ) and is OSI-approved
- [X] Any third-party components/dependencies included are listed along with their licenses ( [THIRD_PARTY.md](../THIRD_PARTY.md) )
- [X] Governance defined, outlining community roles and how decsions are made ( [GOVERNANCE.md](../GOVERNANCE.md] )
- Closed by https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/pull/1137
- [X] Contribution Policy defined ( [CONTRIBUTING.md](../CONTRIBUTING.md) )
- Closed by https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/pull/1137
- [X] Code of Conduct defined ( existing or pull from [ASWF Sample Project](https://github.com/AcademySoftwareFoundation/aswf-sample-project/blob/master/CODE_OF_CONDUCT.md) )
- Closed by https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/pull/1137
- [ ] Release methodology defined ( [release.md](process/release.md) )
- New Project Governance
- [X] TSC members identified
- [X] First TSC meeting held
- [X] TSC meeting cadence set and added to project calendar
- [X] CLA Approved ( if used ) ( [CCLA](CLA-corporate.md) and [ICLA](CLA-individual.md) )
- Project charter ( [charter.md](charter.md) )
- [X] Approved by TSC
- Closed by https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/pull/1137
[](https://www.youtube.com/watch?v=nmwMz1YnGPA "Open Shading Language Reel 2020")
[Read or subscribe to the OSL development mail list](http://groups.google.com/group/osl-dev)
[Read or subscribe to the OSL development mail list](https://lists.aswf.io/g/osl-dev)
Email the lead architect: lg AT imageworks DOT com
[Most recent PDF of the OSL language specification](https://github.com/imageworks/OpenShadingLanguage/blob/master/src/doc/osl-languagespec.pdf
[Most recent PDF of the OSL language specification](https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/blob/main/src/doc/osl-languagespec.pdf
)
[OSL home page at SPI](http://opensource.imageworks.com/?p=osl)
[OSL home page](http://openshadinglanguage.org)
[Sony Pictures Imageworks main open source page](http://opensource.imageworks.com)
If you want to contribute code back to the project, you'll need to
| main | :white_check_mark: :construction: ALL fixes immediately, but this is a branch under development with a frequently unstable ABI and occasionally unstable API. |
| 1.12.x | :white_check_mark: All fixes that can be backported without breaking ABI compatibility. New tagged releases monthly. |
| 1.11.x | :warning: Only the most critical fixes, only if they can be easily backported. |
| <= 1.10.x | :x: No longer receiving patches of any kind. |
## Reporting a Vulnerability
If you think you've found a potential vulnerability in OSL, please report it
by emailing the project administrators at
[security@openshadinglanguage.org](security@openshadinglanguage.org). Only the
project administrators have access to these messages. Include detailed steps to
reproduce the issue, and any other information that could aid an
investigation. Our policy is to respond to vulnerability reports within 14
days.
Our policy is to address critical security vulnerabilities rapidly and post
The OSL Documentation Project is a work in progress.
The plan is to adopt the OCIO Documentation Project's methodology for maintaining and building documentation of various kinds as it becomes the blueprint for the ASWF.
There is much work to be done here.
Of particular interest are these topics. Please consider adding any information you might have about any of these to the repo:
* Any material at all for people learning to write OSL shaders (beyond the formal language spec, which is not intended to be a tutorial or cookbook for writing shaders).
* Example .osl code and/or .h files that are:
1. Generally useful, but do not warrant inclusion into the OSL source itself.
2. Tutorial in nature.
* Robust installation or build instructions for Windows and macOS.
* Guide for how to integrate OSL into an application such as a renderer.
If you have some relevant content and you'd like to discuss its inclusion, please use the OSL Documentation discussion:
## **Building, Using and Debugging OSL in Windows**
---
#### Copyright Contributors to the Open Shading Language project.
#### SPDX-License-Identifier: CC-BY-4.0
---
### **Contents**
1. Introduction
- Experimental
- My Setup
- Inspiration
2. Preparing Environment
- Cleaning Up the PATH
- Set-Up needed Environment Variables
- Set Flags based on your needs
3. Building (Release, Debug)
- Running the code based on your config
- Be Patient
- Errors
4. Running
5. Development in Windows
---
### **1. Introduction**
This document is prepared for those who have problem in building OSL in Windows OS, this is completely custom setup. And it is developing to make it compatible with all situations.
(`Disclaimer:` This Process is completely Experimental and it worked on my system, maybe you get error with changing some configs, during this process).
At a first glance it seems to be complicated, but it is very easy to do, bare with me.
So it would be great to share our experiences to reach to best way of building in Windows (and of course other Operating Systems) to save our time and energy to focus on OSL development and OSL Shader writing.
### **1.1 Experimental**
After a lot of efforts on manual building OSL on Windows, Finally I found a clean way to do that, I explained every steps I did, I hope it works for you too. This method is stricktly related to Environment Variables you create and Flags you set to the command and need patient and hard disk.
I've built OSL with minimal needed packages, for extra packages, they should be set as flag (**script is under development**)
### **1.2 My Setup**
This is my setup I have on my PC for building OSL, so you need to install some software before start.
**OpenShadingLanguage:** cloned from latest master branch
**Boost:** 1.70
**LLVM:** 11.0.0 C++ is set on C++14
**CLANG:** 11.0.0 C++ is set on C++14
**WinFlexBison:** 2.5.22
**Zlib:** 1.2.11
**JPEG:** 2.0.5
**TIFF:** 4.1.0
**PNG:** 1.6.35
**PyBind11:** 2.6.0
**LibRaw:** 0.20.2
**OpenImageIO:** 2.1.20
**OpenColorIO:** 1.1.0
**OpenEXR/**IlmBase: 2.5.3
**GLUT:** 3.2.1 freeglut on windows (this is needed for partio) (unfortunately Partio didn't build, it has a problem) (I’ve created a PR for it in Disney github repo)
**Disney Partio:** latest from master (zip file) (Skip this for now) (I didn't build OSL with Partio)
**PugiXML:** 1.10
---
### **1.3 Inspiration**
After lots of manual building dependencies one by one and fix errors and things I got inspired by `PIXAR USD INSTALLATION METHOD` and borrowed its structure to build OSL.
so I studied its python build script and found we can use same logic for installing OSL, fortunately after changing and adding OSL needed packages and some custom setup it worked fine
It still needs some other works to make it complete for all dependencies (optional).
We will keep Pixar name on it as inspiration code (or we can keep their copyright)
You can Run the installer from: `yourpath/OpenShadingLanguage/doc/build_install/windows/build_osl.bat`
---
### **21. Preparing Environment**
There are bunch od prerequisites for building OSL that I explained in following sections.
### **2.1 Cleaning Up the PATH**
Be sure you don't have any custom item added to PATH in your global Environment Variables of Windows
You can do it:
`Start Menu > Edit the system environmment variables > Environment Variables > User Variables/System Variables`
so you should not have any path that refer to previous osl installation folders
Recommendation: remove added path for cmake, git, python, Qt, llvm
it should be clean now.
### **2.2 Set-Up needed Environment Variables**
For running batch scripts without any problem I recommend to create some base environment variables.
**Needed Environment Variables: (Key/Value based on my installation folder)**
Open `doc/build_install/windows/osl_env_vars_setup.bat` in your editor and edit paths' values based on your installation folders.
### **2.3 Set Flags based on your needs**
The main build launcher is here `doc/build_install/windows/build_osl.bat`
For now we can not assign pre-installed llvm or boost so we have to build all together at once and put all of them in one directory.
I am working on it to see how we can use external llvm and boost installation without error.
**Table of flags we can assign to the command**
| Flag | State | Can Use |
| ------------- | --------------- | ------- |
| --osl | Implemented | Yes |
| --python | Implemented | Yes |
| --zlib | Implemented | Yes |
| --boost | Implemented | Yes |
| --llvm | Implemented | Yes |
| --clang | Implemented | Yes |
| --pugixml | Implemented | Yes |
| --openexr | Implemented | Yes |
| --tiff | Implemented | Yes |
| --jpeg | Implemented | Yes |
| --png | Implemented | Yes |
| --flex | Implemented | Yes |
| --bison | Implemented | Yes |
| --opencolorio | Implemented | Yes |
| --openimageio | Implemented | Yes |
| --libraw | Implemented | Yes |
| --pybind11 | Implemented | Yes |
| --Ptex | Implemented | No |
| --openvdb | Implemented | No |
| --partio | Implemented | No |
| --ffmpeg | Not-Implemented | No |
| --field3d | Not-Implemented | No |
| --opencv | Not-Implemented | No |
| --gif | Not-Implemented | No |
| --heif | Not-Implemented | No |
| --squish | Not-Implemented | No |
| --dcmtk | Not-Implemented | No |
| --webp | Not-Implemented | No |
---
### **3. Building (Release, Debug)**
We Prepared some batch files to do it for you automatically.
Based on Environment Variables you set in previous sections you can just call them and wait for it to be built and installed.
**My Method:**
- Open Powershell
- Go to: yourPath/OpenShadingLanguage/doc/build_install/windows
- Run build_osl.bat release
### **3.1 Running the code based on your config**
Based on the config (`Release` or `Debug`) that you send for installer it will create directory in `BASE_LOCATION` for you
`BASE_LOCATION/osl_release` for Release mode
`BASE_LOCATION/osl_debug` for Debug mode
**IMPORTANT:** `Good to know you can not have Release and Debug files in one directory at the same time, it makes conflict and give you weired errors. So keep them separate.`
### **3.2 Be Patient**
You should be patient because the whole process (based on your PC config) takes time between 30 minutes to 1 hour
And it takes:
**Release:** `~8Gb` of your hard disk (after installation) (Download size is `~350Mb`)
**Debug:** `~90Gb` of your hard disk (after installation) (Download size is `~400Mb`)
so please set a location to `BASE_LOCATION` that have enough space.
### **3.4 Errors**
If you got any error in any step of building process, you can remove the related package's build directory from `BASE_LOCATION/build` and run the installer again.
If you got any error please report to the forum of OSL.
---
### **4. Running**
We Prepared some batch files to do it for you automatically. Based on Environment Variables you set in previous sections you can just call it.
**Recommendation:** use `Powershell`
**TIP:** If you add your osl installation folders (both release and debug) in global Windows Environment Variable, you will get error.
So you can add release path or debug path in your global environment variables. Or create and use batch scripts.
For adding path you should add these two paths to your global env variables:
• PATH : `<osl install dir>\bin`;
• PATH : `<osl install dir>\lib`;
For example, you can open osltoy in releae mode with running `osltoy_release_launcher.bat`
if you have question, feel free to ask.
---
### **5. Development in Windows**
After editing your code, when you want to see your changes you should build osl again. But this time, it is so faster than before, (if you keep build folders) So just need to rebuild it in release mode.
**Release mode**:
- open powershell
- cd OpenShadingLanguage repo
- cd build_script
- run build_osl.bat release
- run osl_release_sln.bat
- example: set osltoy as default project
- run it in release x64 mode
**Debug mode**:
- open powershell
- cd OpenShadingLanguage repo
- cd build_script
- run build_osl.bat debug
- run osl_debug_sln.bat
- example: set osltoy as default project
- run it in debug x64 mode
If you open Visual Studio solution (sln) from release or debug, you can build it from inside it after your code editing
I recommend using: (in **powershell**)
- **osl_release_sln.bat** or **osl_debug_sln.bat**
now you can use Visual Studio for debugging.
---
**TIP about OSLToy:**
If you load a shader in `osltoy` and when hit Recompile, it is crashed and closed, probabely your output color attribute name is not `Cout`
It is mandatory to have at least one `Cout` output Variable in any shader, Otherwise osltoy will crash.
That's it
TIP: this process is getting better, I will update the document soon
**Conclusion:**
For easy installation you just need to setup your Environment Variables in `yourPath/OpenShadingLanguage/doc/build_install/windows/osl_env_vars_setup.bat` based on your paths
It is a custom setup, for example all dependencies are built before osl, because I had lots of issues on using external installation (like llvm)
and I tried this method on llvm 7, 8 and 9, all with error, so my current build setup is on llvm 11 and C++14, it can be changed in code. (I am trying to make it easy to use for changing configs with arguments)
Please take a look at docs and let me know my mistakes.
This method worked fine for me for release, debug and debugging in visual studio.
I am trying to build OSL on `Linux` with same method. If it worked fine I will Update the build system.
set(ILMBASE_INCLUDE_DIR${ILMBASE_INCLUDE_PATH}CACHESTRING"The include paths needed to use IlmBase")
set(OPENEXR_INCLUDE_DIR${OPENEXR_INCLUDE_PATH}CACHESTRING"The include paths needed to use OpenEXR")
set(ILMBASE_LIBRARIES${OPENEXR_IMATH_LIBRARY}${OPENEXR_IEX_LIBRARY}${OPENEXR_HALF_LIBRARY}${OPENEXR_ILMTHREAD_LIBRARY}${ILMBASE_PTHREADS}CACHESTRING"The libraries needed to use IlmBase")
set(OPENEXR_LIBRARIES${OPENEXR_ILMIMF_LIBRARY}${ILMBASE_LIBRARIES}${ZLIB_LIBRARIES}CACHESTRING"The libraries needed to use OpenEXR")
set(ILMBASE_LIBRARIES${OPENEXR_IMATH_LIBRARY}${OPENEXR_IEX_LIBRARY}${OPENEXR_HALF_LIBRARY}${OPENEXR_ILMTHREAD_LIBRARY}${ILMBASE_PTHREADS}CACHESTRING"The libraries needed to use IlmBase")
set(OPENEXR_LIBRARIES${OPENEXR_ILMIMF_LIBRARY}${ILMBASE_LIBRARIES}${ZLIB_LIBRARIES}CACHESTRING"The libraries needed to use OpenEXR")
set(CPACK_PACKAGE_DESCRIPTION_SUMMARY"Open Shading Language is the de facto standard for shading in modern path tracers used in film VFX and animation")
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Larry Gritz
