BRL-CAD/OpenShadingLanguage
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

rs_free_function (#1494)

Browse Source 
In order to move towards shader execution on devices, we need to replace
host side interactions with new concepts that can exist on the device (or host).
The current virtual function based RendererServices exemplifies this dependency
on host side objects.

To break the dependency we introduce free function based renderer services (rs_free_function.h)
To simplify its introduction, we only added the renderer service free functions required by opmatrix.cpp.
Goal is to have a single opmatrix that can be compiled for host or device (eventually).
Renderers can provide llvm bitcode representation of their renderer service
free functions to the ShadingSystem.  During shader compilation/JIT the
renderer's bitcode is combined with opmatrix.cpp bitcode along with the JIT'd shader network.
The idea is the combined LLVM IR representation can then be lowered to the target
(be it on the host or device[SPIRV|PTX]).  Thus a renderer's free function implementation can be shared.

opmatrix.cpp now only calls the free functions (not virtual RendererServices).
However liboslexec on the host contains an rs_fallback.cpp which forwards
calls through the existing virtual RendererServices for seamless compatibility
when a renderer does not provide bitcode for the renderer services.

Testshade has been modified to optionally to provide rs_bitcode,
utilizing a new Cmake macro, EMBED_LLVM_BITCODE_IN_CPP, to turn rs_simplerend.cpp (the free function implementation)
into bitcode.

Free functions can be enabled via --use-rs-bitcode flag in testshade.

Free functions need access to RenderState values because they lack a this
pointer like the RendererServices virtual methods have. Added a RenderState
containing minimal set of values required by free functions.
Testshade/SimpleRenderer is responsible for setting RenderState values.

Added ShadingSystem attribute rs_bitcode to hold renderer services free function bitcode (provided by the renderer)

Renderers using global variables (like ustring's) not in strdecls.h must register their global variables' addresses
using OSL::register_JIT_Global (const char *global_var_name, void *global_var_addr).
This is necessary for the JIT process to resolve global variables that are embedded in RendererServices bitcode.
A validation step occurs that emits an error with a list of unmapped global variables.

RendererServices can reference strings with the STRING_PARAM(varname) macro.
By default, STRING_PARAM(varname) works for any string in strdecls.h.
For any string NOT in strdecls.h, it will map to a global variable with a prefix of "RS_".
For example, STRING_PARAM(raster) maps to a global ustring RS_raster.
NOTE:  C linkage with a "RS_" prefix is used to allow for unmangled
non-colliding global symbol names, so its easier to pass them to
OSL::register_JIT_Global(name, addr) for host execution
NOTE:  the STRING_PARAMS macro adapts to OSL_HOST_RS_BITCODE macro
to utilize the RS_ prefix.  RS_ prefixed versions of all OSL::Strings
intances have been created by rs_free_function.h, so the same STRING_PARAMS
macro can be used for renderer service or OSL strings.
NOTE:  We expect the handling of strings to change in the future possibly being replaced by Handles entirely

It is the renderer's responsibility to instantiate any NEW ustring global variables with the proper RS_ prefix;
It is the renderer's free function implementation to import/extern any of these NEW global variables with the proper RS_ prefix;
For testshade, a rs_strdecls.h has been added and is utilized to instance the RS_ global variables and register their addresses with OSL in simplerend.cpp.
rs_strdecls.h is also used by rs_simplerend.cpp (free function implementation) to create the externs with the RS_ prefix.

NOTE: DO NOT instantiate global variables inside the free function renderer services, because we strip global constructors as part of the pruning process.

Added LLVM_Util::validate_global_mapping to identify unresolved globals.

Added utility macro __OSL_STRINGIFY(string_value) in oslversion.h.in so the
proper naming prefix is used and we don't have multiple implementations around.

Added liboslexec build step to produce bitcode containing OSL library functions that depend on free functions (here, opmatrix.cpp)

Added a new Cmake EMBED_LLVM_BITCODE_IN_CPP function in llvm_macros.cmake that replaces the LLVM_COMPILE macro
and provides for both linking and serializing multiple OSL cpp files.

Fixed cuda.macro to use new location of serialize-bc.py in src/build-scripts.
Fixed formatting issues.

Disabled erasing llvm.global_ctors from LLVM module because of issues on MacOS Apple Clang.
Left as future work.

Added better error checking for module_from_bitcode().

Signed-off-by: Steena Monteiro <steena.monteiro@intel.com>
This commit is contained in:
Steena Monteiro
2022-05-03 21:22:02 -07:00
committed by GitHub
parent 880b44620a
commit e5dd190874
34 changed files with 884 additions and 128 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CMakeLists.txt
+1
View File
@@ -161,6 +161,7 @@ include (pythonutils)
include (externalpackages)
include (flexbison)
include (cuda_macros)
include (llvm_macros)
# Include all our testing apparatus and utils, but not if it's a subproject
if (${PROJECT_NAME}_BUILD_TESTS AND PROJECT_IS_TOP_LEVEL)
src/liboslexec/serialize-bc.py → src/build-scripts/serialize-bc.py
View File
src/cmake/compiler.cmake
+3
View File
@@ -468,6 +468,7 @@ if (PROJECT_IS_TOP_LEVEL)
"src/liboslcomp/*.cpp"
"src/liboslcomp/*.h"
"src/liboslexec/batched_rendservices.cpp"
"src/liboslexec/rs_fallback.cpp"
"src/liboslquery/*.cpp"
"src/liboslquery/*.h"
"src/oslinfo/*.cpp"
@@ -475,6 +476,8 @@ if (PROJECT_IS_TOP_LEVEL)
"src/osl.imageio/*.cpp"
"src/osltoy/*.cpp"
"src/osltoy/*.h"
"src/testshade/render_state.h"
"src/testshade/rs_simplerend.cpp"
"testsuite/*.cpp"
CACHE STRING "Glob patterns to include for clang-format")
set (CLANG_FORMAT_EXCLUDES
src/cmake/cuda_macros.cmake
+2 -2
View File
@@ -130,8 +130,8 @@ function ( LLVM_COMPILE_CUDA llvm_src headers prefix llvm_bc_cpp_generated extra
MAKE_CUDA_BITCODE (${llvm_src} "" llvm_bc "${extra_clang_args}")
add_custom_command (OUTPUT ${llvm_bc_cpp}
COMMAND python "${CMAKE_SOURCE_DIR}/src/liboslexec/serialize-bc.py" ${llvm_bc} ${llvm_bc_cpp} ${prefix}
COMMAND python "${CMAKE_SOURCE_DIR}/src/build-scripts/serialize-bc.py" ${llvm_bc} ${llvm_bc_cpp} ${prefix}
MAIN_DEPENDENCY ${llvm_src}
DEPENDS "${CMAKE_SOURCE_DIR}/src/liboslexec/serialize-bc.py" ${llvm_src} ${headers} ${llvm_bc}
DEPENDS "${CMAKE_SOURCE_DIR}/src/build-scripts/serialize-bc.py" ${llvm_src} ${headers} ${llvm_bc}
WORKING_DIRECTORY "${CMAKE_CURRENT_SOURCE_DIR}" )
endfunction ()
src/cmake/llvm_macros.cmake
+128
View File
@@ -0,0 +1,128 @@
# Copyright Contributors to the Open Shading Language project.
# SPDX-License-Identifier: BSD-3-Clause
# https://github.com/AcademySoftwareFoundation/OpenShadingLanguage
function ( EMBED_LLVM_BITCODE_IN_CPP src_list suffix output_name list_to_append_cpp extra_clang_args)
if (VERBOSE)
message (STATUS "EMBED_LLVM_BITCODE_IN_CPP src_list=${src_list}")
endif ()
foreach ( src ${src_list} )
get_filename_component ( src_we ${src} NAME_WE )
set ( src_asm "${CMAKE_CURRENT_BINARY_DIR}/${src_we}${suffix}.s" )
set ( src_bc "${CMAKE_CURRENT_BINARY_DIR}/${src_we}${suffix}.bc" )
if (VERBOSE)
message (STATUS "EMBED_LLVM_BITCODE_IN_CPP in=${src}")
message (STATUS "EMBED_LLVM_BITCODE_IN_CPP asm=${src_asm}")
message (STATUS "EMBED_LLVM_BITCODE_IN_CPP bc=${src_bc}")
endif ()
list ( APPEND src_bc_list ${src_bc} )
get_property (CURRENT_DEFINITIONS DIRECTORY PROPERTY COMPILE_DEFINITIONS)
if (VERBOSE)
message (STATUS "Current #defines are ${CURRENT_DEFINITIONS}")
endif ()
foreach (def ${CURRENT_DEFINITIONS})
set (LLVM_COMPILE_FLAGS ${LLVM_COMPILE_FLAGS} "-D${def}")
endforeach()
set (LLVM_COMPILE_FLAGS ${LLVM_COMPILE_FLAGS} ${SIMD_COMPILE_FLAGS} ${CSTD_FLAGS} ${TOOLCHAIN_FLAGS})
# Avoid generating __dso_handle external global
set (LLVM_COMPILE_FLAGS ${LLVM_COMPILE_FLAGS} "-fno-use-cxa-atexit")
# Figure out what program we will use to make the bitcode.
if (NOT LLVM_BC_GENERATOR)
find_program (LLVM_BC_GENERATOR NAMES "clang++"
PATHS "${LLVM_DIRECTORY}/bin"
NO_CMAKE_PATH NO_DEFAULT_PATH NO_CMAKE_SYSTEM_PATH
NO_SYSTEM_ENVIRONMENT_PATH NO_CMAKE_ENVIRONMENT_PATH)
endif ()
# If that didn't work, look anywhere
if (NOT LLVM_BC_GENERATOR)
# Wasn't in their build, look anywhere
find_program (LLVM_BC_GENERATOR NAMES clang++ llvm-g++)
endif ()
if (NOT LLVM_BC_GENERATOR)
message (FATAL_ERROR "You must have a valid llvm bitcode generator (clang++) somewhere.")
endif ()
if (VERBOSE)
message (STATUS "Using LLVM_BC_GENERATOR ${LLVM_BC_GENERATOR} to generate bitcode.")
endif()
# Fix specific problem I had on new Apple systems (e.g. Mavericks) with
# LLVM/libc++ installed -- for some reason, LLVM 3.4 wasn't finding it,
# so in that specific case, append another -I to point it in the right
# direction.
#if (APPLE AND ${LLVM_BC_GENERATOR} MATCHES ".*clang.*")
# exec_program ( "${LLVM_BC_GENERATOR}" ARGS --version OUTPUT_VARIABLE MY_CLANG_VERSION )
# string (REGEX REPLACE "clang version ([0-9][.][0-9]+).*" "\\1" MY_CLANG_VERSION "${MY_CLANG_VERSION}")
# if ((${MY_CLANG_VERSION} VERSION_GREATER "3.3")
# AND (EXISTS "/usr/lib/libc++.dylib")
# AND (EXISTS "/Library/Developer/CommandLineTools/usr/lib/c++/v1"))
# set (LLVM_COMPILE_FLAGS ${LLVM_COMPILE_FLAGS} "-I/Library/Developer/CommandLineTools/usr/lib/c++/v1")
# endif ()
#endif ()
list (TRANSFORM IMATH_INCLUDES PREPEND -I
OUTPUT_VARIABLE ALL_IMATH_INCLUDES)
list (TRANSFORM OPENEXR_INCLUDES PREPEND -I
OUTPUT_VARIABLE ALL_OPENEXR_INCLUDES)
list (TRANSFORM OpenImageIO_INCLUDES PREPEND -I
OUTPUT_VARIABLE ALL_OpenImageIO_INCLUDES)
# Command to turn the .cpp file into LLVM assembly language .s, into
# LLVM bitcode .bc, then back into a C++ file with the bc embedded!
add_custom_command ( OUTPUT ${src_bc}
COMMAND ${LLVM_BC_GENERATOR}
${LLVM_COMPILE_FLAGS}
"-I${CMAKE_CURRENT_SOURCE_DIR}"
"-I${CMAKE_SOURCE_DIR}/src/include"
"-I${CMAKE_BINARY_DIR}/include"
${ALL_OpenImageIO_INCLUDES}
${ALL_IMATH_INCLUDES}
#"-isystem ${Boost_INCLUDE_DIRS}" #Does not pick up usr installed boost/thread/tss.hpp for oslexec_pvt.h
"-I${Boost_INCLUDE_DIRS}"
-DOSL_COMPILING_TO_BITCODE=1
-Wno-deprecated-register
# the following 2 warnings can be restored when all 3rd parties have fixed their export macros
# (dllimport attribute is not supported when compiling for Cuda and triggers a ton of warnings)
-Wno-ignored-attributes -Wno-unknown-attributes
-O3 -fno-math-errno -S -emit-llvm ${extra_clang_args}
-o ${src_asm} ${src}
COMMAND "${LLVM_DIRECTORY}/bin/llvm-as" -f -o ${src_bc} ${src_asm}
# Do NOT setup a MAIN_DEPENDENCY because only 1 may exist
# and we may have the several outputs dependant on the same source
DEPENDS ${src} ${exec_headers} ${PROJECT_PUBLIC_HEADERS}
WORKING_DIRECTORY "${CMAKE_CURRENT_SOURCE_DIR}" )
endforeach ()
if (VERBOSE)
message ( STATUS "^^^^^^^^^^^^^^^^^^^^^^^^^^" )
message ( STATUS "src_bc_list: ${src_bc_list} ")
message ( STATUS "^^^^^^^^^^^^^^^^^^^^^^^^^^" )
endif()
# Link all of the individual LLVM bitcode files
set ( linked_src_bc "${CMAKE_CURRENT_BINARY_DIR}/${output_name}.bc" )
add_custom_command ( OUTPUT ${linked_src_bc}
COMMAND "${LLVM_DIRECTORY}/bin/llvm-link" -internalize ${src_bc_list} -o ${linked_src_bc}
DEPENDS ${src_bc_list} ${exec_headers} ${PROJECT_PUBLIC_HEADERS} ${src_list}
WORKING_DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}" )
# Serialize the linked bitcode into a CPP file
set ( src_bc_cpp "${CMAKE_CURRENT_BINARY_DIR}/${output_name}.bc.cpp" )
add_custom_command ( OUTPUT ${src_bc_cpp}
COMMAND python "${CMAKE_SOURCE_DIR}/src/build-scripts/serialize-bc.py"
${linked_src_bc} ${src_bc_cpp} ${output_name}
DEPENDS "${CMAKE_SOURCE_DIR}/src/build-scripts/serialize-bc.py" ${linked_src_bc}
${exec_headers} ${PROJECT_PUBLIC_HEADERS}
WORKING_DIRECTORY "${CMAKE_CURRENT_SOURCE_DIR}" )
# add generated .cpp with embedded bitcode to the list of soures
set ( ${list_to_append_cpp} ${${list_to_append_cpp}} ${src_bc_cpp} PARENT_SCOPE )
endfunction ( )
src/cmake/testing.cmake
+21
View File
@@ -90,6 +90,7 @@ macro ( TESTSUITE )
endif ()
set (test_all_optix $ENV{TESTSUITE_OPTIX})
set (test_all_batched $ENV{TESTSUITE_BATCHED})
set (test_all_rs_bitcode $ENV{TESTSUITE_RS_BITCODE})
# Add the tests if all is well.
set (ALL_TEST_LIST "")
set (_testsuite "${CMAKE_SOURCE_DIR}/testsuite")
@@ -180,6 +181,25 @@ macro ( TESTSUITE )
ENV TESTSHADE_OPT=0 OSL_REGRESSION_TEST=BATCHED )
endif ()
endif ()
# if there is an RS_BITCODE marker file in the directory.
if ((EXISTS "${_testsrcdir}/RS_BITCODE" OR test_all_rs_bitcode)
AND NOT EXISTS "${_testsrcdir}/BATCHED_REGRESSION"
AND NOT EXISTS "${_testsrcdir}/RS_BITCODE_REGRESSION"
AND NOT EXISTS "${_testsrcdir}/NOOPTIMIZE")
add_one_testsuite ("${_testname}.rsbitcode.opt" "${_testsrcdir}"
ENV TESTSHADE_OPT=2 TESTSHADE_RS_BITCODE=1 )
endif ()
# if there is an RS_BITCODE_REGRESSION marker file in the directory.
if (EXISTS "${_testsrcdir}/RS_BITCODE_REGRESSION" OR (
test_all_rs_bitcode AND EXISTS "${_testsrcdir}/BATCHED_REGRESSION")
AND NOT EXISTS "${_testsrcdir}/NOOPTIMIZE")
# optimized for right now
add_one_testsuite ("${_testname}.regress.rsbitcode.opt" "${_testsrcdir}"
ENV TESTSHADE_OPT=2 OSL_REGRESSION_TEST=RS_BITCODE )
endif ()
endforeach ()
if (VERBOSE)
message (STATUS "Added tests: ${ALL_TEST_LIST}")
@@ -348,5 +368,6 @@ macro (osl_add_all_tests)
set_tests_properties (arithmetic-reg.regress.batched.opt PROPERTIES TIMEOUT 800)
set_tests_properties (transform-reg.regress.batched.opt PROPERTIES TIMEOUT 800)
endif ()
set_tests_properties (matrix-reg.regress.rsbitcode.opt PROPERTIES TIMEOUT 800)
endmacro()
src/include/OSL/device_string.h
+7 -1
View File
@@ -132,7 +132,11 @@ namespace DeviceStrings {
#undef STRDECL
}
#else
# define STRING_PARAMS(x) StringParams::x
# ifdef OSL_HOST_RS_BITCODE
# define STRING_PARAMS(x) RS_##x
# else
# define STRING_PARAMS(x) StringParams::x
# endif
#endif
@@ -146,3 +150,5 @@ namespace StringParams = OSL_NAMESPACE::DeviceStrings;
#else
namespace StringParams = OSL_NAMESPACE::Strings;
#endif
src/include/OSL/llvm_util.h
+14
View File
@@ -228,6 +228,15 @@ public:
// Name for this TargetISA enum.
static const char* target_isa_name(TargetISA isa);
/// Add a global mapping of a variable to its address
/// explicitly instead of relying on dlsym.
static void add_global_mapping (const char *global_var_name, void *global_var_addr);
/// Check all external global variables to ensure they have
/// been mapped to an address, a vector is populated with
/// the names that are unmapped.
void validate_global_mappings (std::vector<std::string> &names_of_unmapped_globals);
// For CPU compilation, inventory the host CPU capabilities. You can
// optionally request a specific ISA by name. If `no_fma` is true,
// specifically pretend there is no FMA capability, even if the hardware
@@ -908,6 +917,11 @@ public:
/// file. If err is not NULL, errors will be deposited there.
void write_bitcode_file (const char *filename, std::string *err=NULL);
/// Insert bitcode from another module into the current Module
/// returns true if successful, false otherwise.
/// Ownership of the other_module is transferred.
bool absorb_module (std::unique_ptr<llvm::Module> other_module );
/// Generate PTX for the current Module and return it as a string
bool ptx_compile_group (llvm::Module* lib_module, const std::string& name,
std::string& out);
src/include/OSL/oslexec.h
+6 -3
View File
@@ -29,6 +29,11 @@ template<typename DataT, int WidthT> struct Wide;
#endif
/// RendererServices free function bitcode may reference global variables that
/// only exist in the renderer's program space and won't automatically be
/// found when referenced in JIT'd code inside OSL. A renderer may register
/// the addresses of these global variables to allow valid JIT to occur.
void register_JIT_Global(const char* global_var_name, void* global_var_addr);
/// Opaque pointer to whatever the renderer uses to represent a
/// (potentially motion-blurred) coordinate transformation.
@@ -45,9 +50,7 @@ class ShadingSystemImpl;
}
#if defined(__CUDA_ARCH__) && OPTIX_VERSION >= 70000
# define STRINGIFY(x) XSTR(x)
# define XSTR(x) #x
# define STRING_PARAMS(x) UStringHash::Hash(STRINGIFY(x))
# define STRING_PARAMS(x) UStringHash::Hash(__OSL_STRINGIFY(x))
#else
# define STRING_PARAMS(x) StringParams::x
#endif
src/include/OSL/oslversion.h.in
+3
View File
@@ -103,4 +103,7 @@ namespace @PROJ_NAME@ = @PROJ_NAMESPACE_V@;
#define __OSL_WIDE_PVT __OSL_CONCAT5(b,__OSL_WIDTH,_,__OSL_TARGET_ISA,_pvt)
#endif
#define __OSL_INDIRECT_STRINGIFY(x) #x
#define __OSL_STRINGIFY(x) __OSL_INDIRECT_STRINGIFY(x)
#endif /* OSLVERSION_H */
src/include/OSL/rendererservices.h
+1
View File
@@ -42,6 +42,7 @@ OSL_GCC_PRAGMA(GCC diagnostic ignored "-Wunused-parameter")
/// RendererServices defines an abstract interface through which a
/// renderer may provide callback to the ShadingSystem.
class OSLEXECPUBLIC RendererServices {
// Keep interface in sync with rs_free_function.h
public:
typedef TextureSystem::TextureHandle TextureHandle;
typedef TextureSystem::Perthread TexturePerthread;
src/include/OSL/rs_free_function.h
+128
View File
@@ -0,0 +1,128 @@
// Copyright Contributors to the Open Shading Language project.
// SPDX-License-Identifier: BSD-3-Clause
// https://github.com/AcademySoftwareFoundation/OpenShadingLanguage
#include <OSL/oslconfig.h>
// Make all the strings defined in OSL
// available to the renderer services free functions
OSL_NAMESPACE_ENTER
namespace Strings {
#define STRDECL(str, var_name) OSLEXECPUBLIC extern const ustring var_name;
#include <OSL/strdecls.h>
#undef STRDECL
}; // namespace Strings
OSL_NAMESPACE_EXIT
#ifdef OSL_HOST_RS_BITCODE
extern "C" {
# define STRDECL(str, var_name) \
const OSL::ustring& RS_##var_name = OSL::Strings::var_name;
# include <OSL/strdecls.h>
# undef STRDECL
}
#endif
#include <OSL/device_string.h>
#include <OSL/shaderglobals.h>
// Prefix for OSL shade op declarations.
// "C" linkage (no C++ name mangling) and local visibility
#define OSL_RSOP extern "C"
// We are choosing to use unique names encoding parameters directly
// as opposed to using overloaded functions.
// Keep free functions in sync with virtual function based RendererServices.
/// Get the 4x4 matrix that transforms by the specified
/// transformation at the given time. Return true if ok, false
/// on error.
OSL_RSOP bool
rs_get_matrix_xform_time(OSL::ShaderGlobals* sg, OSL::Matrix44& result,
OSL::TransformationPtr from, float time);
/// Get the 4x4 matrix that transforms by the specified
/// transformation at the given time. Return true if ok, false on
/// error. Suggested implementation is to use rs_get_matrix_xform_time and
/// invert it, but a particular renderer may have a better technique.
OSL_RSOP bool
rs_get_inverse_matrix_xform_time(OSL::ShaderGlobals* sg, OSL::Matrix44& result,
OSL::TransformationPtr xform, float time);
/// Get the 4x4 matrix that transforms points from the named
/// 'from' coordinate system to "common" space at the given time.
/// Returns true if ok, false if the named matrix is not known.
OSL_RSOP bool
rs_get_matrix_space_time(OSL::ShaderGlobals* sg, OSL::Matrix44& result,
OSL::StringParam from, float time);
/// Get the 4x4 matrix that transforms points from "common" space to
/// the named 'to' coordinate system to at the given time. Suggested
/// implementation is to use rs_get_matrix_space_time and invert it, but a
/// particular renderer may have a better technique.
OSL_RSOP bool
rs_get_inverse_matrix_space_time(OSL::ShaderGlobals* sg, OSL::Matrix44& result,
OSL::StringParam to, float time);
/// Get the 4x4 matrix that transforms by the specified
/// transformation. Return true if ok, false on error. Since no
/// time value is given, also return false if the transformation may
/// be time-varying.
OSL_RSOP bool
rs_get_matrix_xform(OSL::ShaderGlobals* sg, OSL::Matrix44& result,
OSL::TransformationPtr xform);
/// Get the 4x4 matrix that transforms by the specified
/// transformation. Return true if ok, false on error. Since no
/// time value is given, also return false if the transformation may
/// be time-varying. Suggested implementation is to use
/// rs_get_matrix_xform and invert it, but a particular renderer may have a
/// better technique.
OSL_RSOP bool
rs_get_inverse_matrix_xform(OSL::ShaderGlobals* sg, OSL::Matrix44& result,
OSL::TransformationPtr xform);
/// Get the 4x4 matrix that transforms 'from' to "common" space.
/// Since there is no time value passed, return false if the
/// transformation may be time-varying (as well as if it's not found
/// at all).
OSL_RSOP bool
rs_get_matrix_space(OSL::ShaderGlobals* sg, OSL::Matrix44& result,
OSL::StringParam from);
/// Get the 4x4 matrix that transforms points from "common" space to
/// the named 'to' coordinate system. Since there is no time value
/// passed, return false if the transformation may be time-varying
/// (as well as if it's not found at all). Suggested
/// implementation is to use rs_get_matrix_space and invert it, but a
/// particular renderer may have a better technique.
OSL_RSOP bool
rs_get_inverse_matrix_space(OSL::ShaderGlobals* sg, OSL::Matrix44& result,
OSL::StringParam to);
/// Transform points Pin[0..npoints-1] in named coordinate system
/// 'from' into 'to' coordinates, storing the result in Pout[] using
/// the specified vector semantic (POINT, VECTOR, NORMAL). The
/// function returns true if the renderer correctly transformed the
/// points, false if it failed (for example, because it did not know
/// the name of one of the coordinate systems). Suggested implementation
/// is simply to make appropriate calls to rs_get_matrix_space and
/// get_inverse_matrix_space. The existence of this function is to allow
/// some renderers to provide transformations that cannot be
/// expressed by a 4x4 matrix.
///
/// Note, the virtual function based RendererServices::transform_points
/// provides additional functionality/modes that are only used during
/// code generation to detect if it required to even call rs_transform_points
/// during evaluation of the shader. There is no need for the free
/// function version to handle these additional modes.
///
/// Note to implementations: just return 'false'
/// if there isn't a special nonlinear transformation between the
/// two spaces.
OSL_RSOP bool
rs_transform_points(OSL::ShaderGlobals* sg, OSL::StringParam from,
OSL::StringParam to, float time, const OSL::Vec3* Pin,
OSL::Vec3* Pout, int npoints,
OSL::TypeDesc::VECSEMANTICS vectype);
src/liboslexec/CMakeLists.txt
+14 -91
View File
@@ -120,6 +120,7 @@ set (lib_src
shadeimage.cpp
backendllvm.cpp
llvm_gen.cpp llvm_instance.cpp llvm_util.cpp
rs_fallback.cpp
)
if (BUILD_BATCHED)
list(APPEND lib_src
@@ -179,94 +180,6 @@ foreach(src ${lib_src})
endif()
endforeach(src)
macro ( LLVM_COMPILE llvm_src srclist )
get_filename_component ( llvmsrc_we ${llvm_src} NAME_WE )
set ( llvm_asm "${CMAKE_CURRENT_BINARY_DIR}/${llvmsrc_we}.s" )
set ( llvm_bc "${CMAKE_CURRENT_BINARY_DIR}/${llvmsrc_we}.bc" )
set ( llvm_bc_cpp "${CMAKE_CURRENT_BINARY_DIR}/${llvmsrc_we}.bc.cpp" )
if (VERBOSE)
message (STATUS "LLVM_COMPILE in=${llvm_src}")
message (STATUS "LLVM_COMPILE asm=${llvm_asm}")
message (STATUS "LLVM_COMPILE bc=${llvm_bc}")
message (STATUS "LLVM_COMPILE cpp=${llvm_bc_cpp}")
endif ()
set ( ${srclist} ${${srclist}} ${llvm_bc_cpp} )
get_property (CURRENT_DEFINITIONS DIRECTORY PROPERTY COMPILE_DEFINITIONS)
if (VERBOSE)
message (STATUS "Current #defines are ${CURRENT_DEFINITIONS}")
endif ()
foreach (def ${CURRENT_DEFINITIONS})
set (LLVM_COMPILE_FLAGS ${LLVM_COMPILE_FLAGS} "-D${def}")
endforeach()
set (LLVM_COMPILE_FLAGS ${LLVM_COMPILE_FLAGS} ${SIMD_COMPILE_FLAGS} ${CSTD_FLAGS} ${TOOLCHAIN_FLAGS})
# Figure out what program we will use to make the bitcode.
if (NOT LLVM_BC_GENERATOR)
find_program (LLVM_BC_GENERATOR NAMES "clang++"
PATHS "${LLVM_DIRECTORY}/bin"
NO_CMAKE_PATH NO_DEFAULT_PATH NO_CMAKE_SYSTEM_PATH
NO_SYSTEM_ENVIRONMENT_PATH NO_CMAKE_ENVIRONMENT_PATH)
endif ()
# If that didn't work, look anywhere
if (NOT LLVM_BC_GENERATOR)
# Wasn't in their build, look anywhere
find_program (LLVM_BC_GENERATOR NAMES clang++ llvm-g++)
endif ()
if (NOT LLVM_BC_GENERATOR)
message (FATAL_ERROR "You must have a valid llvm bitcode generator (clang++) somewhere.")
endif ()
if (VERBOSE)
message (STATUS "Using LLVM_BC_GENERATOR ${LLVM_BC_GENERATOR} to generate bitcode.")
endif()
# Fix specific problem I had on new Apple systems (e.g. Mavericks) with
# LLVM/libc++ installed -- for some reason, LLVM 3.4 wasn't finding it,
# so in that specific case, append another -I to point it in the right
# direction.
#if (APPLE AND ${LLVM_BC_GENERATOR} MATCHES ".*clang.*")
# exec_program ( "${LLVM_BC_GENERATOR}" ARGS --version OUTPUT_VARIABLE MY_CLANG_VERSION )
# string (REGEX REPLACE "clang version ([0-9][.][0-9]+).*" "\\1" MY_CLANG_VERSION "${MY_CLANG_VERSION}")
# if ((${MY_CLANG_VERSION} VERSION_GREATER "3.3")
# AND (EXISTS "/usr/lib/libc++.dylib")
# AND (EXISTS "/Library/Developer/CommandLineTools/usr/lib/c++/v1"))
# set (LLVM_COMPILE_FLAGS ${LLVM_COMPILE_FLAGS} "-I/Library/Developer/CommandLineTools/usr/lib/c++/v1")
# endif ()
#endif ()
list (TRANSFORM IMATH_INCLUDES PREPEND -I
OUTPUT_VARIABLE ALL_IMATH_INCLUDES)
list (TRANSFORM OPENEXR_INCLUDES PREPEND -I
OUTPUT_VARIABLE ALL_OPENEXR_INCLUDES)
list (TRANSFORM OpenImageIO_INCLUDES PREPEND -I
OUTPUT_VARIABLE ALL_OpenImageIO_INCLUDES)
# Command to turn the .cpp file into LLVM assembly language .s, into
# LLVM bitcode .bc, then back into a C++ file with the bc embedded!
add_custom_command ( OUTPUT ${llvm_bc_cpp}
COMMAND ${LLVM_BC_GENERATOR}
${LLVM_COMPILE_FLAGS}
"-I${CMAKE_CURRENT_SOURCE_DIR}"
"-I${CMAKE_SOURCE_DIR}/src/include"
"-I${CMAKE_BINARY_DIR}/include"
${ALL_OpenImageIO_INCLUDES}
${ALL_IMATH_INCLUDES}
"-isystem ${Boost_INCLUDE_DIRS}"
-DOSL_COMPILING_TO_BITCODE=1
-Wno-deprecated-register
# the following 2 warnings can be restored when all 3rd parties have fixed their export macros
# (dllimport attribute is not supported when compiling for Cuda and triggers a ton of warnings)
-Wno-ignored-attributes -Wno-unknown-attributes
-O3 -fno-math-errno -S -emit-llvm -o ${llvm_asm} ${llvm_src}
COMMAND "${LLVM_DIRECTORY}/bin/llvm-as" -f -o ${llvm_bc} ${llvm_asm}
COMMAND python "${CMAKE_CURRENT_SOURCE_DIR}/serialize-bc.py" ${llvm_bc} ${llvm_bc_cpp} "osl_llvm_compiled_ops"
MAIN_DEPENDENCY ${llvm_src}
DEPENDS "${CMAKE_CURRENT_SOURCE_DIR}/serialize-bc.py"
${exec_headers} ${PROJECT_PUBLIC_HEADERS}
WORKING_DIRECTORY "${CMAKE_CURRENT_SOURCE_DIR}" )
endmacro ( )
macro ( CUDA_SHADEOPS_COMPILE srclist )
# Add all of the "shadeops" sources that need to be compiled to LLVM bitcode for CUDA
set ( shadeops_srcs
@@ -297,15 +210,25 @@ macro ( CUDA_SHADEOPS_COMPILE srclist )
# Serialize the linked bitcode into a CPP file and add it to the list of liboslexec soures
add_custom_command ( OUTPUT ${shadeops_bc_cuda_cpp}
COMMAND python "${CMAKE_CURRENT_SOURCE_DIR}/serialize-bc.py"
COMMAND python "${CMAKE_SOURCE_DIR}/src/build-scripts/serialize-bc.py"
${linked_shadeops_bc} ${shadeops_bc_cuda_cpp} "osl_llvm_compiled_ops_cuda"
DEPENDS "${CMAKE_CURRENT_SOURCE_DIR}/serialize-bc.py" ${linked_shadeops_bc}
DEPENDS "${CMAKE_SOURCE_DIR}/src/build-scripts/serialize-bc.py" ${linked_shadeops_bc}
${exec_headers} ${PROJECT_PUBLIC_HEADERS}
WORKING_DIRECTORY "${CMAKE_CURRENT_SOURCE_DIR}" )
endmacro ( )
if (USE_LLVM_BITCODE)
LLVM_COMPILE ( llvm_ops.cpp lib_src )
set (llvm_ops_srcs llvm_ops.cpp)
EMBED_LLVM_BITCODE_IN_CPP ( "${llvm_ops_srcs}" "_host" "osl_llvm_compiled_ops" lib_src "")
set (rs_dependant_ops_srcs
opmatrix.cpp
)
# Achieve the effect of absorbing osl_llvm_compiled_ops by adding its
# sources to rs_dependant_ops_srcs which avoids having to do it at runtime.
list (APPEND rs_dependant_ops_srcs ${llvm_ops_srcs})
EMBED_LLVM_BITCODE_IN_CPP ( "${rs_dependant_ops_srcs}" "_host_rs" "osl_llvm_compiled_rs_dependant_ops" lib_src "")
# Optionally repeat the bitcode compilation with CUDA-specific options
if (CUDA_FOUND)
src/liboslexec/backendllvm.cpp
+1
View File
@@ -78,6 +78,7 @@ BackendLLVM::BackendLLVM (ShadingSystemImpl &shadingsys,
check_cwd (shadingsys);
#endif
m_use_optix = shadingsys.renderer()->supports ("OptiX");
m_use_rs_bitcode = !shadingsys.m_rs_bitcode.empty();
ll.dumpasm(shadingsys.m_llvm_dumpasm);
ll.jit_fma(shadingsys.m_llvm_jit_fma);
ll.jit_aggressive(shadingsys.m_llvm_jit_aggressive);
src/liboslexec/backendllvm.h
+5
View File
@@ -446,6 +446,9 @@ public:
/// Return whether or not we are compiling for an OptiX-based renderer.
bool use_optix() { return m_use_optix; }
/// Return if we should compile against free function versions of Renderer Service.
bool use_rs_bitcode() {return m_use_rs_bitcode; }
/// Return the userdata index for the given Symbol. Return -1 if the Symbol
/// is not an input parameter or is constant and therefore doesn't have an
/// entry in the groupdata struct.
@@ -490,6 +493,8 @@ private:
bool m_use_optix; ///< Compile for OptiX?
bool m_use_rs_bitcode; /// To use free function versions of Renderer Service functions.
friend class ShadingSystemImpl;
};
src/liboslexec/llvm_instance.cpp
+56 -5
View File
@@ -112,6 +112,9 @@ extern unsigned char osl_llvm_compiled_ops_block[];
extern int osl_llvm_compiled_ops_cuda_size;
extern unsigned char osl_llvm_compiled_ops_cuda_block[];
extern int osl_llvm_compiled_rs_dependant_ops_size;
extern unsigned char osl_llvm_compiled_rs_dependant_ops_block[];
using namespace OSL::pvt;
OSL_NAMESPACE_ENTER
@@ -1324,23 +1327,49 @@ BackendLLVM::run ()
#endif
#ifdef OSL_LLVM_NO_BITCODE
OSL_ASSERT(!use_rs_bitcode());
ll.module (ll.new_module ("llvm_ops"));
#else
if (! use_optix()) {
ll.module (ll.module_from_bitcode ((char*)osl_llvm_compiled_ops_block,
osl_llvm_compiled_ops_size,
"llvm_ops", &err));
if(use_rs_bitcode()){
ll.module (ll.module_from_bitcode ((char*)osl_llvm_compiled_rs_dependant_ops_block,
osl_llvm_compiled_rs_dependant_ops_size,
"llvm_rs_dependant_ops", &err));
if (err.length())
shadingcontext()->errorf("llvm::parseBitcodeFile returned '%s' for llvm_rs_dependant_ops\n", err);
std::vector<char>& rs_free_function_bitcode = shadingsys().m_rs_bitcode;
OSL_ASSERT (rs_free_function_bitcode.size() && "Free Function bitcode is empty");
llvm::Module* rs_free_functions_module =
ll.module_from_bitcode (static_cast<const char*>(rs_free_function_bitcode.data()),
rs_free_function_bitcode.size(), "rs_free_functions", &err);
if (err.length())
shadingcontext()->errorf("llvm::parseBitcodeFile returned '%s' for rs_free_functions\n", err);
std::unique_ptr<llvm::Module> rs_free_functions_module_ptr (rs_free_functions_module);
bool success = ll.absorb_module(std::move(rs_free_functions_module_ptr));
if (!success)
shadingcontext()->errorf("LLVM_Util::absorb_module failed'\n");
} else {
ll.module (ll.module_from_bitcode ((char*)osl_llvm_compiled_ops_block,
osl_llvm_compiled_ops_size,
"llvm_ops", &err));
if (err.length())
shadingcontext()->errorf("llvm::parseBitcodeFile returned '%s' for llvm_ops\n", err);
}
} else {
#ifdef OSL_LLVM_CUDA_BITCODE
ll.module (ll.module_from_bitcode ((char*)osl_llvm_compiled_ops_cuda_block,
osl_llvm_compiled_ops_cuda_size,
"llvm_ops", &err));
if (err.length())
shadingcontext()->errorf("llvm::parseBitcodeFile returned '%s' for cuda llvm_ops\n", err);
#else
OSL_ASSERT (0 && "Must generate LLVM CUDA bitcode for OptiX");
#endif
}
if (err.length())
shadingcontext()->errorf("ParseBitcodeFile returned '%s'\n", err);
OSL_ASSERT (ll.module());
#endif
@@ -1475,6 +1504,17 @@ BackendLLVM::run ()
}
}
if(use_rs_bitcode()) {
std::vector<std::string> names_of_unmapped_globals;
ll.validate_global_mappings(names_of_unmapped_globals);
if (!names_of_unmapped_globals.empty()) {
shadingsys().errorfmt("Renderers should call osl::register_JIT_Global(const char* global_var_name, void* global_var_addr) for each global variable used by its free function renderer services bitcode");
for(const auto &unmapped_name:names_of_unmapped_globals) {
shadingsys().errorfmt(">>>>External global variable {} was not mapped to an address!", unmapped_name);
}
}
}
// Optimize the LLVM IR unless it's a do-nothing group.
if (! group().does_nothing())
ll.do_optimize();
@@ -1482,10 +1522,21 @@ BackendLLVM::run ()
m_stat_llvm_opt_time += timer.lap();
if (llvm_debug()) {
#if 1
// Feel it is more useful to get a dump of the entire optimized module
// vs. individual layer functions. Especially now because we have pruned all
// unused function declarations and functions out of the the module.
// Big benefit is that the module output can be cut and pasted into
// https://godbolt.org/ compiler explorer as LLVM IR with a LLC target
// and -mcpu= options to see what machine code will be generated by
// different LLC versions and cpu targets
std::cout << "module after opt = \n" << ll.module_string() << "\n";
#else
for (auto&& f : funcs)
if (f)
shadingsys().infofmt("func after opt = {}\n",
ll.bitcode_string(f));
#endif
}
// Debug code to dump the post-optimized bitcode to a file
src/liboslexec/llvm_util.cpp
+52 -1
View File
@@ -80,6 +80,8 @@
#include <llvm/Transforms/Scalar/GVN.h>
#include <llvm/Transforms/Utils.h>
#include <llvm/Support/DynamicLibrary.h>
#if OSL_LLVM_VERSION >= 120
#include <llvm/CodeGen/Passes.h>
#endif
@@ -1620,6 +1622,11 @@ LLVM_Util::getPointerToFunction (llvm::Function *func)
}
void
LLVM_Util::add_global_mapping (const char *global_var_name, void *global_var_addr)
{
llvm::sys::DynamicLibrary::AddSymbol(global_var_name, global_var_addr);
}
void
LLVM_Util::InstallLazyFunctionCreator (void* (*P)(const std::string &))
@@ -1974,6 +1981,32 @@ void
LLVM_Util::prune_and_internalize_module (std::unordered_set<llvm::Function*> external_functions,
Linkage default_linkage, std::string *out_err)
{
// WIP: Investigating stripping global constructors and destructors
// Avoid calling global variable constructors
// as it is generally creates order of execution issues
// and difficult to support on devices.
// When pruning, lots of extraneous code is materialized
// and kept alive by global constructors that are
// NEVER ACTUALLY CALLED!
// Should plumbing to execute the global constructors
// be created, then this should be revisitted
// llvm::GlobalVariable *ctors_gv = m_llvm_module->getGlobalVariable("llvm.global_ctors");
// if(ctors_gv!=nullptr)
// {
// ctors_gv->eraseFromParent();
// } else {
// // TODO: investigate if this symbol exists on Mac OS
// OSL_ASSERT("llvm.global_ctors not found");
// }
// llvm::GlobalVariable *dtors_gv = m_llvm_module->getGlobalVariable("llvm.global_dtors");
// if(dtors_gv!=nullptr)
// {
// dtors_gv->eraseFromParent();
// } else {
// // TODO: investigate if this symbol exists on Mac OS
// OSL_ASSERT("llvm.global_dtors not found");
// }
// Turn tracing for pruning on locally
#if defined(OSL_DEV)
#define __OSL_PRUNE_ONLY(...) __VA_ARGS__
@@ -2201,6 +2234,19 @@ LLVM_Util::prune_and_internalize_module (std::unordered_set<llvm::Function*> ext
m_ModuleIsPruned = true;
}
void
LLVM_Util::validate_global_mappings(std::vector<std::string> &names_of_unmapped_globals)
{
for (llvm::GlobalVariable& global : m_llvm_module->globals()) {
if (global.hasExternalLinkage()) {
void *global_addr = llvm::sys::DynamicLibrary::SearchForAddressOfSymbol(global.getName().data());
if (global_addr == nullptr) {
names_of_unmapped_globals.push_back(global.getName().str());
}
}
}
}
// Soon to be deprecated
@@ -5520,7 +5566,12 @@ LLVM_Util::write_bitcode_file (const char *filename, std::string *err)
}
}
bool
LLVM_Util::absorb_module (std::unique_ptr<llvm::Module> other_module_ptr /*We claim ownership of this module */)
{
bool failed = llvm::Linker::linkModules (*module(), std::move (other_module_ptr));
return !failed;
}
bool
LLVM_Util::ptx_compile_group (llvm::Module* lib_module, const std::string& name,
src/liboslexec/opmatrix.cpp
+18 -17
View File
@@ -14,6 +14,7 @@
#include <OSL/dual.h>
#include <OSL/dual_vec.h>
#include <OSL/device_string.h>
#include <OSL/rs_free_function.h>
#include <OpenImageIO/fmath.h>
#include <OpenImageIO/simd.h>
@@ -133,20 +134,20 @@ osl_get_matrix (void *sg_, void *r, const char *from)
{
ShaderGlobals *sg = (ShaderGlobals *)sg_;
ShadingContext *ctx = (ShadingContext *)sg->context;
if (USTR(from) == Strings::common ||
USTR(from) == ctx->shadingsys().commonspace_synonym()) {
if (HDSTR(from) == STRING_PARAMS(common) ||
HDSTR(from) == ctx->shadingsys().commonspace_synonym()) {
MAT(r).makeIdentity ();
return true;
}
if (USTR(from) == Strings::shader) {
ctx->renderer()->get_matrix (sg, MAT(r), sg->shader2common, sg->time);
if (HDSTR(from) == STRING_PARAMS(shader)) {
rs_get_matrix_xform_time(sg, MAT(r), sg->shader2common, sg->time);
return true;
}
if (USTR(from) == Strings::object) {
ctx->renderer()->get_matrix (sg, MAT(r), sg->object2common, sg->time);
if (HDSTR(from) == STRING_PARAMS(object)) {
rs_get_matrix_xform_time(sg, MAT(r), sg->object2common, sg->time);
return true;
}
int ok = ctx->renderer()->get_matrix (sg, MAT(r), USTR(from), sg->time);
int ok = rs_get_matrix_space_time(sg, MAT(r), HDSTR(from), sg->time);
if (! ok) {
MAT(r).makeIdentity();
ShadingContext *ctx = (ShadingContext *)((ShaderGlobals *)sg)->context;
@@ -163,20 +164,20 @@ osl_get_inverse_matrix (void *sg_, void *r, const char *to)
{
ShaderGlobals *sg = (ShaderGlobals *)sg_;
ShadingContext *ctx = (ShadingContext *)sg->context;
if (USTR(to) == Strings::common ||
USTR(to) == ctx->shadingsys().commonspace_synonym()) {
if (HDSTR(to) == STRING_PARAMS(common) ||
HDSTR(to) == ctx->shadingsys().commonspace_synonym()) {
MAT(r).makeIdentity ();
return true;
}
if (USTR(to) == Strings::shader) {
ctx->renderer()->get_inverse_matrix (sg, MAT(r), sg->shader2common, sg->time);
if (HDSTR(to) == STRING_PARAMS(shader)) {
rs_get_inverse_matrix_xform_time(sg, MAT(r), sg->shader2common, sg->time);
return true;
}
if (USTR(to) == Strings::object) {
ctx->renderer()->get_inverse_matrix (sg, MAT(r), sg->object2common, sg->time);
if (HDSTR(to) == STRING_PARAMS(object)) {
rs_get_inverse_matrix_xform_time(sg, MAT(r), sg->object2common, sg->time);
return true;
}
int ok = ctx->renderer()->get_inverse_matrix (sg, MAT(r), USTR(to), sg->time);
int ok = rs_get_inverse_matrix_space_time(sg, MAT(r), HDSTR(to), sg->time);
if (! ok) {
MAT(r).makeIdentity ();
ShadingContext *ctx = (ShadingContext *)((ShaderGlobals *)sg)->context;
@@ -290,15 +291,15 @@ osl_transform_triple_nonlinear (void *sg_, void *Pin, int Pin_derivs,
{
ShaderGlobals *sg = (ShaderGlobals *)sg_;
#ifndef __CUDACC__
RendererServices *rend = sg->renderer;
if (rend->transform_points (sg, USTR(from), USTR(to), sg->time,
if (rs_transform_points (sg, HDSTR(from), HDSTR(to), sg->time,
(const Vec3 *)Pin, (Vec3 *)Pout, 1,
(TypeDesc::VECSEMANTICS)vectype)) {
// Renderer had a direct way to transform the points between the
// two spaces.
if (Pout_derivs) {
if (Pin_derivs) {
rend->transform_points (sg, USTR(from), USTR(to), sg->time,
rs_transform_points (sg, HDSTR(from), HDSTR(to), sg->time,
(const Vec3 *)Pin+1,
(Vec3 *)Pout+1, 2, TypeDesc::VECTOR);
} else {
src/liboslexec/oslexec_pvt.h
+2
View File
@@ -758,6 +758,8 @@ private:
// Pre-compiled support library
std::vector<char> m_lib_bitcode; ///> Container for the pre-compiled library bitcode
std::vector<char> m_rs_bitcode; ///> Container for the pre-compiled renderer services free function bitcode
// Options
int m_statslevel; ///< Statistics level
bool m_lazylayers; ///< Evaluate layers on demand?
src/liboslexec/rs_fallback.cpp
+76
View File
@@ -0,0 +1,76 @@
// Copyright Contributors to the Open Shading Language project.
// SPDX-License-Identifier: BSD-3-Clause
// https://github.com/AcademySoftwareFoundation/OpenShadingLanguage
#include <OSL/rs_free_function.h>
#include <OSL/rendererservices.h>
// Host only fallback implementation of free function renderer services that
// simply forward the calls to the existing virtual RendererServices methods
OSL_RSOP bool
rs_get_matrix_xform_time(OSL::ShaderGlobals* sg, OSL::Matrix44& result,
OSL::TransformationPtr from, float time)
{
return sg->renderer->get_matrix(sg, result, from, time);
}
OSL_RSOP bool
rs_get_inverse_matrix_xform_time(OSL::ShaderGlobals* sg, OSL::Matrix44& result,
OSL::TransformationPtr xform, float time)
{
return sg->renderer->get_inverse_matrix(sg, result, xform, time);
}
OSL_RSOP bool
rs_get_matrix_space_time(OSL::ShaderGlobals* sg, OSL::Matrix44& result,
OSL::StringParam from, float time)
{
return sg->renderer->get_matrix(sg, result, from, time);
}
OSL_RSOP bool
rs_get_inverse_matrix_space_time(OSL::ShaderGlobals* sg, OSL::Matrix44& result,
OSL::StringParam to, float time)
{
return sg->renderer->get_inverse_matrix(sg, result, to, time);
}
OSL_RSOP bool
rs_get_matrix_xform(OSL::ShaderGlobals* sg, OSL::Matrix44& result,
OSL::TransformationPtr xform)
{
return sg->renderer->get_matrix(sg, result, xform);
}
OSL_RSOP bool
rs_get_inverse_matrix_xform(OSL::ShaderGlobals* sg, OSL::Matrix44& result,
OSL::TransformationPtr xform)
{
return sg->renderer->get_inverse_matrix(sg, result, xform);
}
OSL_RSOP bool
rs_get_matrix_space(OSL::ShaderGlobals* sg, OSL::Matrix44& result,
OSL::StringParam from)
{
return sg->renderer->get_matrix(sg, result, from);
}
OSL_RSOP bool
rs_get_inverse_matrix_space(OSL::ShaderGlobals* sg, OSL::Matrix44& result,
OSL::StringParam to)
{
return sg->renderer->get_inverse_matrix(sg, result, to);
}
OSL_RSOP bool
rs_transform_points(OSL::ShaderGlobals* sg, OSL::StringParam from,
OSL::StringParam to, float time, const OSL::Vec3* Pin,
OSL::Vec3* Pout, int npoints,
OSL::TypeDesc::VECSEMANTICS vectype)
{
return sg->renderer->transform_points(sg, from, to, time, Pin, Pout,
npoints, vectype);
}
src/liboslexec/shadingsys.cpp
+19 -1
View File
@@ -930,7 +930,11 @@ ShadingSystem::convert_value (void *dst, TypeDesc dsttype,
return false; // Unsupported conversion
}
void
register_JIT_Global (const char *global_var_name, void *global_var_addr)
{
LLVM_Util::add_global_mapping(global_var_name, global_var_addr);
}
PerThreadInfo::PerThreadInfo ()
{
@@ -1576,6 +1580,20 @@ ShadingSystemImpl::attribute (string_view name, TypeDesc type,
}
return true;
}
if (name == "rs_bitcode" && type.basetype == TypeDesc::UINT8) {
if (type.arraylen < 0) {
errorfmt("Invalid bitcode size: {}", type.arraylen);
return false;
}
m_rs_bitcode.clear();
if (type.arraylen) {
const char* bytes = static_cast<const char*>(val);
std::copy(bytes, bytes + type.arraylen,
back_inserter(m_rs_bitcode));
}
return true;
}
if (name == "error_repeats") {
// Special case: setting error_repeats also clears the "previously
// seen" error and warning lists.
src/testrender/cuda/rend_lib.h
+1 -3
View File
@@ -21,9 +21,7 @@ namespace DeviceStrings {
rtDeclareVariable(OSL_NAMESPACE::DeviceString, var_name, , );
# define STRING_PARAMS(x) StringParams::x
#else
# define STRINGIFY(x) XSTR(x)
# define XSTR(x) #x
# define STRING_PARAMS(x) UStringHash::Hash(STRINGIFY(x))
# define STRING_PARAMS(x) UStringHash::Hash(__OSL_STRINGIFY(x))
// Don't declare anything
# define STRDECL(str,var_name)
src/testshade/CMakeLists.txt
+4
View File
@@ -60,6 +60,10 @@ if (USE_OPTIX)
DESTINATION ${OSL_PTX_INSTALL_DIR})
endif()
set ( rs_srcs
rs_simplerend.cpp )
EMBED_LLVM_BITCODE_IN_CPP ( "${rs_srcs}" "_host" "testshade_llvm_compiled_rs" testshade_srcs "-DOSL_HOST_RS_BITCODE=1")
add_executable ( testshade ${testshade_srcs} testshademain.cpp )
target_link_libraries (testshade
src/testshade/render_state.h
+22
View File
@@ -0,0 +1,22 @@
// Copyright Contributors to the Open Shading Language project.
// SPDX-License-Identifier: BSD-3-Clause
// https://github.com/AcademySoftwareFoundation/OpenShadingLanguage
#pragma once
#include <OSL/oslconfig.h>
#include <OSL/device_string.h> // for StringParam
// All the the state free functions in rs_simplerend.cpp will need to do their job
// NOTE: Additional data is here that will be used by rs_simplerend.cpp in future PR's
// procedurally generating ShaderGlobals.
struct RenderState {
int xres;
int yres;
OSL::Matrix44 world_to_camera;
OSL::StringParam projection;
float fov;
float hither;
float yon;
};
src/testshade/rs_simplerend.cpp
+186
View File
@@ -0,0 +1,186 @@
// Copyright Contributors to the Open Shading Language project.
// SPDX-License-Identifier: BSD-3-Clause
// https://github.com/AcademySoftwareFoundation/OpenShadingLanguage
#ifndef OSL_HOST_RS_BITCODE
# error OSL_HOST_RS_BITCODE must be defined by your build system.
#endif
#include <OSL/rs_free_function.h>
#include "render_state.h"
// Extern all global string variables used by free function renderer services.
// NOTE: C linkage with a "RS_" prefix is used to allow for unmangled
// non-colliding global symbol names, so its easier to pass them to
// OSL::register_JIT_Global(name, addr) for host execution
// NOTE: the STRING_PARAMS macro adapts to OSL_HOST_RS_BITCODE
// to utilize the RS_ prefix. RS_ prefixed versions of all OSL::Strings
// intances have been created by rs_free_function.h, so the same STRING_PARAMS
// macro can be used for renderer service or OSL strings.
#define RS_STRDECL(str, var_name) extern "C" OSL::ustring RS_##var_name;
#include "rs_strdecls.h"
#undef RS_STRDECL
// Keep free functions in sync with virtual function based SimpleRenderer.
OSL_RSOP bool
rs_get_matrix_xform_time(OSL::ShaderGlobals* /*sg*/, OSL::Matrix44& result,
OSL::TransformationPtr xform, float /*time*/)
{
// SimpleRenderer doesn't understand motion blur and transformations
// are just simple 4x4 matrices.
auto ptr = reinterpret_cast<OSL::Matrix44 const*>(xform);
result = *ptr;
return true;
}
OSL_RSOP bool
rs_get_inverse_matrix_xform_time(OSL::ShaderGlobals* sg, OSL::Matrix44& result,
OSL::TransformationPtr xform, float time)
{
bool ok = rs_get_matrix_xform_time(sg, result, xform, time);
if (ok) {
result.invert();
}
return ok;
}
OSL_RSOP bool
rs_get_matrix_space_time(OSL::ShaderGlobals* /*sg*/, OSL::Matrix44& result,
OSL::StringParam from, float /*time*/)
{
if (from == STRING_PARAMS(myspace)) {
OSL::Matrix44 Mmyspace;
Mmyspace.scale(OSL::Vec3(1.0, 2.0, 1.0));
result = Mmyspace;
return true;
} else {
return false;
}
}
OSL_RSOP bool
rs_get_inverse_matrix_space_time(OSL::ShaderGlobals* sg, OSL::Matrix44& result,
OSL::StringParam to, float time)
{
using OSL::Matrix44;
RenderState* rs = reinterpret_cast<RenderState*>(sg->renderstate);
if (to == STRING_PARAMS(camera) || to == STRING_PARAMS(screen)
|| to == STRING_PARAMS(NDC) || to == STRING_PARAMS(raster)) {
Matrix44 M { rs->world_to_camera };
if (to == STRING_PARAMS(screen) || to == STRING_PARAMS(NDC)
|| to == STRING_PARAMS(raster)) {
float depthrange = (double)rs->yon - (double)rs->hither;
//OSL::StringParam proj{rs->projection.m_chars};
const auto& proj = rs->projection;
if (proj == STRING_PARAMS(perspective)) {
float tanhalffov = OIIO::fast_tan(0.5f * rs->fov * M_PI
/ 180.0);
// clang-format off
Matrix44 camera_to_screen (1/tanhalffov, 0, 0, 0,
0, 1/tanhalffov, 0, 0,
0, 0, rs->yon/depthrange, 1,
0, 0, -(rs->yon*rs->hither)/depthrange, 0);
// clang-format on
M = M * camera_to_screen;
} else {
// clang-format off
Matrix44 camera_to_screen (1, 0, 0, 0,
0, 1, 0, 0,
0, 0, 1/depthrange, 0,
0, 0, -(rs->hither)/depthrange, 1);
// clang-format on
M = M * camera_to_screen;
}
if (to == STRING_PARAMS(NDC) || to == STRING_PARAMS(raster)) {
float screenleft = -1.0, screenwidth = 2.0;
float screenbottom = -1.0, screenheight = 2.0;
// clang-format off
Matrix44 screen_to_ndc (1/screenwidth, 0, 0, 0,
0, 1/screenheight, 0, 0,
0, 0, 1, 0,
-screenleft/screenwidth, -screenbottom/screenheight, 0, 1);
// clang-format on
M = M * screen_to_ndc;
if (to == STRING_PARAMS(raster)) {
// clang-format off
Matrix44 ndc_to_raster (rs->xres, 0, 0, 0,
0, rs->yres, 0, 0,
0, 0, 1, 0,
0, 0, 0, 1);
M = M * ndc_to_raster;
// clang-format on
}
}
}
result = M;
return true;
} else {
bool ok = rs_get_matrix_space_time(sg, result, to, time);
if (ok) {
result.invert();
}
return ok;
}
}
OSL_RSOP bool
rs_get_matrix_xform(OSL::ShaderGlobals* /*sg*/, OSL::Matrix44& result,
OSL::TransformationPtr xform)
{
// SimpleRenderer doesn't understand motion blur and transformations
// are just simple 4x4 matrices.
auto ptr = reinterpret_cast<OSL::Matrix44 const*>(xform);
result = *ptr;
return true;
}
OSL_RSOP bool
rs_get_inverse_matrix_xform(OSL::ShaderGlobals* sg, OSL::Matrix44& result,
OSL::TransformationPtr xform)
{
bool ok = rs_get_matrix_xform(sg, result, xform);
if (ok) {
result.invert();
}
return ok;
}
OSL_RSOP bool
rs_get_matrix_space(OSL::ShaderGlobals* /*sg*/, OSL::Matrix44& /*result*/,
OSL::StringParam from)
{
if (from == STRING_PARAMS(myspace)) {
return true;
} else {
return false;
}
}
OSL_RSOP bool
rs_get_inverse_matrix_space(OSL::ShaderGlobals* sg, OSL::Matrix44& result,
OSL::StringParam to)
{
bool ok = rs_get_matrix_space(sg, result, to);
if (ok) {
result.invert();
}
return ok;
}
OSL_RSOP bool
rs_transform_points(OSL::ShaderGlobals* /*sg*/, OSL::StringParam /*from*/,
OSL::StringParam /*to*/, float /*time*/,
const OSL::Vec3* /*Pin*/, OSL::Vec3* /*Pout*/,
int /*npoints*/, OSL::TypeDesc::VECSEMANTICS /*vectype*/)
{
return false;
}
src/testshade/rs_strdecls.h
+19
View File
@@ -0,0 +1,19 @@
// SPDX-License-Identifier: BSD-3-Clause
// https://github.com/AcademySoftwareFoundation/OpenShadingLanguage
// This file contains "declarations" for all the strings that might get used in
// JITed shader code or in renderer code. But the declaration itself is
// dependent on the RS_STRDECL macro, which should be declared by the outer file
// prior to including this file. Thus, this list may be repurposed and included
// multiple times, with different RS_STRDECL definitions.
#ifndef RS_STRDECL
# error Do not include this file unless RS_STRDECL is defined
#endif
RS_STRDECL("perspective", perspective)
RS_STRDECL("raster", raster)
RS_STRDECL("myspace", myspace)
src/testshade/simplerend.cpp
+41
View File
@@ -8,9 +8,24 @@
#include <OSL/oslexec.h>
#include <OSL/genclosure.h>
#include "simplerend.h"
// Instance all global string variables used by SimpleRend.cpp or
// rs_simplerend.cpp (free functions).
// NOTE: C linkage with a "RS_" prefix is used to allow for unmangled
// non-colliding global symbol names, so its easier to pass them to
// OSL::register_JIT_Global(name, addr) for host execution
extern "C" {
#define RS_STRDECL(str, var_name) OSL::ustring RS_##var_name{str};
#include "rs_strdecls.h"
#undef RS_STRDECL
}
using namespace OSL;
// anonymous namespace
namespace {
@@ -190,6 +205,13 @@ SimpleRenderer::attribute (string_view name, TypeDesc type, const void *value)
f->init(name, type, 1, value);
}
void
SimpleRenderer::register_JIT_Global_Variables() //callable from testshade
{
#define RS_STRDECL(str, var_name) OSL::register_JIT_Global(__OSL_STRINGIFY(RS_##var_name), &RS_##var_name);
#include "rs_strdecls.h"
#undef RS_STRDECL
}
void
@@ -666,5 +688,24 @@ SimpleRenderer::add_output (string_view varname, string_view filename,
}
void
SimpleRenderer::export_state(RenderState &state) const
{
state.xres = m_xres;
state.yres = m_yres;
state.fov = m_fov;
state.hither = m_hither;
state.yon = m_yon;
state.world_to_camera = OSL::Matrix44(1.0, 0.0, 0.0, 0.0,
0.0, 1.0, 0.0, 0.0,
0.0, 0.0, 1.0, 0.0,
0.0, 0.0, 0.0, 1.0);
//perspective is not a member of StringParams (i.e not in strdecls.h)
state.projection = u_perspective;
}
OSL_NAMESPACE_EXIT
src/testshade/simplerend.h
+5
View File
@@ -18,6 +18,7 @@
# include "batched_simplerend.h"
#endif
#include "render_state.h"
OSL_NAMESPACE_ENTER
@@ -28,6 +29,7 @@ void register_closures(OSL::ShadingSystem* shadingsys);
class SimpleRenderer : public RendererServices
{
// Keep implementation in sync with rs_simplerend.cpp
template<int>
friend class BatchedSimpleRenderer;
public:
@@ -117,6 +119,7 @@ public:
virtual void init_shadingsys (ShadingSystem *ss) {
shadingsys = ss;
}
virtual void export_state(RenderState &) const;
virtual void prepare_render () { }
virtual void warmup () { }
virtual void render (int /*xres*/, int /*yres*/) { }
@@ -126,6 +129,8 @@ public:
// they aren't already.
virtual void finalize_pixel_buffer () { }
static void register_JIT_Global_Variables();
// ShaderGroupRef storage
std::vector<ShaderGroupRef>& shaders() { return m_shaders; }
src/testshade/testshade.cpp
+42 -4
View File
@@ -37,8 +37,14 @@
# include <OSL/batched_shaderglobals.h>
#endif
#include "optixgridrender.h"
#include "render_state.h"
#include "simplerend.h"
extern int testshade_llvm_compiled_rs_size;
extern unsigned char testshade_llvm_compiled_rs_block[];
using namespace OSL;
using OIIO::ParamValue;
using OIIO::ParamValueList;
@@ -110,6 +116,7 @@ static std::string localename = OIIO::Sysutil::getenv("TESTSHADE_LOCALE");
static OIIO::ParamValueList userdata;
static char* userdata_base_ptr = nullptr;
static char* output_base_ptr = nullptr;
static bool use_rs_bitcode = false; // use free function bitcode version of renderer services
@@ -160,6 +167,16 @@ set_shadingsys_options ()
llvm_opt = atoi(llvm_opt_env);
shadingsys->attribute ("llvm_optimize", llvm_opt);
if (const char *use_rs_bitcode_env = getenv ("TESTSHADE_RS_BITCODE")) {
use_rs_bitcode = atoi(use_rs_bitcode_env);
}
if(use_rs_bitcode)
{
shadingsys->attribute("rs_bitcode", {OSL::TypeDesc::UINT8, testshade_llvm_compiled_rs_size},
testshade_llvm_compiled_rs_block);
}
shadingsys->attribute ("profile", int(profile));
shadingsys->attribute ("lockgeom", 1);
shadingsys->attribute ("debug_nan", debugnan);
@@ -682,6 +699,7 @@ getargs (int argc, const char *argv[])
"Add userdata (args: name value) (options: type=%s)",
"--userdata_isconnected", &userdata_isconnected, "Consider lockgeom=0 to be isconnected()",
"--locale %s", &localename, "Set a different locale",
"--use_rs_bitcode", &use_rs_bitcode, "Use free function bitcode Renderer services",
NULL);
if (ap.parse(argc, argv) < 0 /*|| (shadernames.empty() && groupspec.empty())*/) {
std::cerr << ap.geterror() << std::endl;
@@ -764,6 +782,9 @@ setup_transformations (SimpleRenderer &rend, OSL::Matrix44 &Mshad,
rend.name_transform ("myspace", Mmyspace);
}
// NOTE: each host thread could end up with its own RenderState.
// Starting simple with a single instance for now
static RenderState theRenderState;
// Set up the ShaderGlobals fields for pixel (x,y).
@@ -774,9 +795,17 @@ setup_shaderglobals (ShaderGlobals &sg, ShadingSystem *shadingsys,
// Just zero the whole thing out to start
memset ((char *)&sg, 0, sizeof(ShaderGlobals));
// In our SimpleRenderer, the "renderstate" itself just a pointer to
// the ShaderGlobals.
sg.renderstate = &sg;
if (use_rs_bitcode) {
// When using free function bitcode to implement renderer services,
// any state data they need will need to be passed here
// the ShaderGlobals.
sg.renderstate = &theRenderState;
} else {
// In our SimpleRenderer, the "renderstate" itself just a pointer to
// the ShaderGlobals.
sg.renderstate = &sg;
}
// Set "shader" space to be Mshad. In a real renderer, this may be
// different for each shader group.
@@ -1723,7 +1752,7 @@ test_shade (int argc, const char *argv[])
else
#endif
rend = new SimpleRenderer;
// Other renderer and global options
if (debug1 || verbose)
rend->errhandler().verbosity (ErrorHandler::VERBOSE);
@@ -1818,6 +1847,11 @@ test_shade (int argc, const char *argv[])
// options change their values.
set_shadingsys_options ();
if(use_rs_bitcode)
{
SimpleRenderer::register_JIT_Global_Variables();
}
if (groupname.size())
shadingsys->attribute (shadergroup.get(), "groupname", groupname);
@@ -1917,6 +1951,10 @@ test_shade (int argc, const char *argv[])
synchio();
rend->prepare_render ();
if (use_rs_bitcode) {
// SimpleRend to supply the required state for render service free functions
rend->export_state(theRenderState);
}
double setuptime = timer.lap ();
testsuite/matrix-arithmetic-reg/RS_BITCODE_REGRESSION
View File
testsuite/matrix-compref-reg/RS_BITCODE_REGRESSION
View File
testsuite/matrix-reg/RS_BITCODE_REGRESSION
View File
testsuite/matrix/RS_BITCODE
View File
testsuite/runtest.py
+7
View File
@@ -297,6 +297,13 @@ def runtest (command, outputs, failureok=0, failthresh=0, failpercent=0, regress
if test_environ == None :
test_environ = os.environ
test_environ["TESTSHADE_BATCHED"] = "1"
if regression == "RS_BITCODE" :
if test_environ == None :
test_environ = os.environ
test_environ["TESTSHADE_RS_BITCODE"] = "1"
print ("command = ", command)
for sub_command in command.split(splitsymbol):