Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 tbbdev
|
6f0562a18b | ||
|
tbbdev
|
4a86cd6d48 |
@@ -1,18 +1,17 @@
|
||||
# Intel(R) Threading Building Blocks 2017 Update 5
|
||||
[] (https://github.com/01org/tbb/releases/tag/2017_U5)
|
||||
[](https://github.com/01org/tbb/releases/tag/2017_U5)
|
||||
[](LICENSE)
|
||||
|
||||
Intel(R) Threading Building Blocks (Intel(R) TBB) lets you easily write parallel C++ programs that take
|
||||
full advantage of multicore performance, that are portable, composable and have future-proof scalability.
|
||||
|
||||
## Release Information
|
||||
Here are the latest [Changes] (CHANGES) and [Release Notes]
|
||||
(doc/Release_Notes.txt) (contains system requirements and known issues).
|
||||
Here are the latest [Changes](CHANGES) and [Release Notes](doc/Release_Notes.txt) (contains system requirements and known issues).
|
||||
|
||||
## Documentation
|
||||
* Intel(R) TBB [tutorial] (https://software.intel.com/en-us/tbb-tutorial)
|
||||
* Intel(R) TBB general documentation: [stable] (https://software.intel.com/en-us/tbb-documentation)
|
||||
and [latest] (https://www.threadingbuildingblocks.org/docs/help/index.htm)
|
||||
* Intel(R) TBB [tutorial](https://software.intel.com/en-us/tbb-tutorial)
|
||||
* Intel(R) TBB general documentation: [stable](https://software.intel.com/en-us/tbb-documentation)
|
||||
and [latest](https://www.threadingbuildingblocks.org/docs/help/index.htm)
|
||||
|
||||
## Support
|
||||
Please report issues and suggestions via
|
||||
@@ -20,10 +19,10 @@ Please report issues and suggestions via
|
||||
[Intel(R) TBB forum](http://software.intel.com/en-us/forums/intel-threading-building-blocks/).
|
||||
|
||||
## How to Contribute
|
||||
Please, read the instructions on the official [Intel(R) TBB open source site] (https://www.threadingbuildingblocks.org/submit-contribution).
|
||||
Please, read the instructions on the official [Intel(R) TBB open source site](https://www.threadingbuildingblocks.org/submit-contribution).
|
||||
|
||||
## Engineering team contacts
|
||||
* [E-mail us.] (mailto:inteltbbdevelopers@intel.com)
|
||||
* [E-mail us.](mailto:inteltbbdevelopers@intel.com)
|
||||
|
||||
------------------------------------------------------------------------
|
||||
Intel and the Intel logo are trademarks of Intel Corporation or its subsidiaries in the U.S. and/or other countries.
|
||||
|
||||
+5
-1
@@ -19,14 +19,18 @@
|
||||
all: release test
|
||||
|
||||
clean:
|
||||
$(MAKE) -rf rml/Makefile tbb_root=.. clean
|
||||
python setup.py clean
|
||||
-rm -rf build/ tbb_wrap.* _TBB.* *.pyc TBB.py*
|
||||
|
||||
release: TBB.py
|
||||
release: rml_release TBB.py
|
||||
|
||||
TBB.py: tbb.i tbb.src.py setup.py
|
||||
python setup.py build_ext -f --inplace
|
||||
|
||||
ipc-rml:
|
||||
$(MAKE) -rf rml/Makefile tbb_root=.. rml_dll cfg=release
|
||||
|
||||
test: TBB.py
|
||||
python TBB.py test
|
||||
|
||||
|
||||
@@ -0,0 +1,24 @@
|
||||
#!/usr/bin/env python
|
||||
#
|
||||
# Copyright (c) 2005-2017 Intel Corporation
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
|
||||
from tbb import *
|
||||
from tbb import __all__, __doc__
|
||||
|
||||
if __name__ == "__main__":
|
||||
from tbb import _main
|
||||
import sys
|
||||
sys.exit(_main())
|
||||
+28
-11
@@ -4,10 +4,10 @@
|
||||
</H2>
|
||||
|
||||
<H2>Overview</H2>
|
||||
It is a preview Python* module which unlocks opportunities for additional performance in multi-threaded Python programs by enabling threading composability
|
||||
It is a preview Python* module which unlocks opportunities for additional performance in multi-threaded and multiprocess Python programs by enabling threading composability
|
||||
between two or more thread-enabled libraries like Numpy, Scipy, Sklearn, Dask, Joblib, and etc.
|
||||
<p></p>
|
||||
The biggest improvement can be achieved when a task pool like the ThreadPool from the Python standard library or libraries like Dask or Joblib (used in multi-threading mode)
|
||||
The biggest improvement can be achieved when a task pool like the ThreadPool or Pool from the Python standard library or libraries like Dask or Joblib (used either in multi-threading or multi-processing mode)
|
||||
execute tasks calling compute-intensive functions of Numpy/Scipy/Sklearn/PyDAAL which in turn are parallelized using Intel® Math Kernel Library or/and Intel® TBB.
|
||||
<p></p>
|
||||
The module implements Pool class with the standard interface using Intel® TBB which can be used to replace Python's ThreadPool.
|
||||
@@ -15,20 +15,32 @@ Thanks to the monkey-patching technique implemented in class Monkey, no source c
|
||||
<p></p>
|
||||
For more information and examples, please refer to <A HREF="http://software.intel.com/en-us/blogs/2016/04/04/unleash-parallel-performance-of-python-programs">online blog</A>.
|
||||
|
||||
<H2>Directories</H2>
|
||||
<DL>
|
||||
<DT><A HREF="rml">rml</A>
|
||||
<DD>The folder contains needed sources to build the plugin with cross-process dynamic thread scheduler implementation.
|
||||
<DT><A HREF="tbb">tbb</A>
|
||||
<DD>The folder contains Python module sources.
|
||||
</DL>
|
||||
|
||||
<H2>Files</H2>
|
||||
<DL>
|
||||
<DT><A HREF="setup.py">setup.py</A>
|
||||
<DD>Standard Python setup script.
|
||||
<DT><A HREF="Makefile">Makefile</A>
|
||||
<DD>Makefile for building, installing, and testing. See below.
|
||||
<DT><A HREF="tbb.i">tbb.i</A>
|
||||
<DD>SWIG interface description file.
|
||||
<DT><A HREF="tbb.src.py">tbb.src.py</A>
|
||||
<DD>Python part of module implementation.
|
||||
<DT><A HREF="TBB.py">TBB.py</A>
|
||||
<DD>Alternative entry point for Python module.
|
||||
</DL>
|
||||
|
||||
<A NAME=build><H2>Build and install</H2></A>
|
||||
Prior to building it, please set up the environment using corresponding tbbvars script, e.g. `source tbbvars.sh intel64`
|
||||
Prior to building it, please do the following first:
|
||||
<UL>
|
||||
<LI>Download this source package;</LI>
|
||||
<LI>Build Intel® TBB (for more informantion, please refer to <A HREF="../index.html">documentation</A>);</LI>
|
||||
<LI>Set up the environment using generated tbbvars.sh script that can be found in $TBBROOT/build/*_release/ or $TBBROOT/build/*_debug/ folders.</LI>
|
||||
|
||||
</UL>
|
||||
<DL>
|
||||
<DT><TT>make</TT>
|
||||
<DD>Default build and run. Equivalent to 'make release test'.
|
||||
@@ -48,10 +60,15 @@ Please note, CXX=icl works on Windows only with Intel® Distribution for Pyth
|
||||
|
||||
<H2>Command-line interface</H2>
|
||||
<DL>
|
||||
<DT><TT>pydoc TBB</TT>
|
||||
<DD>Read built-in documentation for Python interfaces.
|
||||
<DT><TT>python -m TBB your_script.py</TT>
|
||||
<DD>Run your_script.py in context of `with TBB.Monkey():` when Intel TBB is enabled.
|
||||
<DT><TT>python -m tbb -h</TT>
|
||||
<DD>Print documentation on command-line interface</DD>
|
||||
<DT><TT>pydoc tbb</TT>
|
||||
<DD>Read built-in documentation for Python interfaces.</DD>
|
||||
<DT><TT>python-tbb your_script.py</TT>
|
||||
<DT><TT>python -m tbb your_script.py</TT>
|
||||
<DD>Run your_script.py in context of `with tbb.Monkey():` when Intel® TBB is enabled. By default only multi-threading will be covered.</DD>
|
||||
<DT><TT>python -m tbb --ipc your_script.py</TT>
|
||||
<DD>Run your_script.py in context of `with tbb.Monkey():` when Intel® TBB enabled in both multi-threading and multi-processing modes.</DD>
|
||||
</DL>
|
||||
|
||||
<H2>System Requirements</H2>
|
||||
|
||||
@@ -0,0 +1,151 @@
|
||||
# Copyright (c) 2005-2017 Intel Corporation
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
#
|
||||
#
|
||||
#
|
||||
|
||||
|
||||
.NOTPARALLEL:
|
||||
|
||||
tbb_root ?= ../..
|
||||
BUILDING_PHASE=0
|
||||
TEST_RESOURCE = $(RML.RES)
|
||||
include $(tbb_root)/build/common.inc
|
||||
DEBUG_SUFFIX=$(findstring _debug,_$(cfg))
|
||||
|
||||
ifeq (android,$(target))
|
||||
$(error "RML is not supported on Android")
|
||||
endif
|
||||
|
||||
.PHONY: default_rml rml rml_test clean
|
||||
|
||||
# default target
|
||||
default_rml: rml rml_test
|
||||
|
||||
RML_ROOT ?= $(tbb_root)/src/rml
|
||||
RML_SERVER_ROOT = $(RML_ROOT)/server
|
||||
# TODO: new API needs to be added for this server, exposing everything
|
||||
RML.DEF =
|
||||
|
||||
VPATH = $(tbb_root)/src/tbb $(tbb_root)/src/tbb/$(ASSEMBLY_SOURCE)
|
||||
VPATH += $(tbb_root)/python/rml $(RML_ROOT)/test $(tbb_root)/src/test
|
||||
|
||||
include $(tbb_root)/build/common_rules.inc
|
||||
|
||||
#--------------------------------------------------------------------------
|
||||
# Define rules for making the RML server shared library and client objects.
|
||||
#--------------------------------------------------------------------------
|
||||
|
||||
# Object files that make up RML server
|
||||
RML_SERVER.OBJ = ipc_server.$(OBJ)
|
||||
|
||||
# Object files that RML clients need
|
||||
RML_TBB_CLIENT.OBJ ?= ipc_utils.$(OBJ)
|
||||
RML.OBJ = $(RML_SERVER.OBJ) $(RML_TBB_CLIENT.OBJ)
|
||||
ifeq (windows,$(tbb_os))
|
||||
RML_ASM.OBJ = $(if $(findstring intel64,$(arch)),$(TBB_ASM.OBJ))
|
||||
endif
|
||||
ifeq (linux,$(tbb_os))
|
||||
RML_ASM.OBJ = $(if $(findstring ia64,$(arch)),$(TBB_ASM.OBJ))
|
||||
endif
|
||||
|
||||
RML_TBB_DEP= cache_aligned_allocator_rml.$(OBJ) dynamic_link_rml.$(OBJ) tbb_misc_rml.$(OBJ) tbb_misc_ex_rml.$(OBJ)
|
||||
TBB_DEP_NON_RML_TEST?= cache_aligned_allocator_rml.$(OBJ) dynamic_link_rml.$(OBJ) $(RML_ASM.OBJ) tbb_misc_rml.$(OBJ) tbb_misc_ex_rml.$(OBJ)
|
||||
ifeq ($(cfg),debug)
|
||||
RML_TBB_DEP+= spin_mutex_rml.$(OBJ)
|
||||
TBB_DEP_RML_TEST?= $(RML_ASM.OBJ) tbb_misc_rml.$(OBJ)
|
||||
else
|
||||
TBB_DEP_RML_TEST?= $(RML_ASM.OBJ)
|
||||
endif
|
||||
LIBS += $(LIBDL)
|
||||
|
||||
INCLUDES += $(INCLUDE_KEY)$(RML_ROOT)/include $(INCLUDE_KEY).
|
||||
T_INCLUDES = $(INCLUDES) $(INCLUDE_KEY)$(tbb_root)/src/test $(INCLUDE_KEY)$(RML_SERVER_ROOT)
|
||||
|
||||
# Suppress superfluous warnings for RML compilation
|
||||
R_CPLUS_FLAGS = $(subst DO_ITT_NOTIFY,DO_ITT_NOTIFY=0,$(CPLUS_FLAGS)) $(WARNING_SUPPRESS) \
|
||||
$(DEFINE_KEY)TBB_USE_THREADING_TOOLS=0 $(DEFINE_KEY)__TBB_RML_STATIC=1 $(DEFINE_KEY)__TBB_NO_IMPLICIT_LINKAGE=1
|
||||
|
||||
%.$(OBJ): %.cpp
|
||||
$(CPLUS) $(COMPILE_ONLY) $(R_CPLUS_FLAGS) $(PIC_KEY) $(DSE_KEY) $(INCLUDES) $<
|
||||
|
||||
tbb_misc_rml.$(OBJ) $(RML_SERVER.OBJ): version_string.ver
|
||||
|
||||
RML_TEST.OBJ = test_job_automaton.$(OBJ) test_thread_monitor.$(OBJ) test_rml_tbb.$(OBJ)
|
||||
|
||||
$(RML_TBB_DEP): %_rml.$(OBJ): %.cpp
|
||||
$(CPLUS) $(COMPILE_ONLY) $(OUTPUTOBJ_KEY)$@ $(R_CPLUS_FLAGS) $(PIC_KEY) $(DSE_KEY) $(INCLUDES) $<
|
||||
|
||||
$(RML_TEST.OBJ): %.$(OBJ): %.cpp
|
||||
$(CPLUS) $(COMPILE_ONLY) $(R_CPLUS_FLAGS) $(PIC_KEY) $(T_INCLUDES) $<
|
||||
|
||||
ifneq (,$(RML.DEF))
|
||||
rml.def: $(RML.DEF)
|
||||
$(CPLUS) $(PREPROC_ONLY) $< $(CPLUS_FLAGS) $(INCLUDES) > $@
|
||||
|
||||
LIB_LINK_FLAGS += $(EXPORT_KEY)rml.def
|
||||
$(RML.DLL): rml.def
|
||||
endif
|
||||
|
||||
$(RML.DLL): CPLUS_FLAGS += $(SDL_FLAGS)
|
||||
$(RML.DLL): BUILDING_LIBRARY = $(RML.DLL)
|
||||
$(RML.DLL): $(RML_TBB_DEP) $(RML.OBJ) $(RML.RES) $(RML_NO_VERSION.DLL) $(RML_ASM.OBJ)
|
||||
$(LIB_LINK_CMD) $(LIB_OUTPUT_KEY)$(RML.DLL) $(RML.OBJ) $(RML_TBB_DEP) $(RML_ASM.OBJ) $(RML.RES) $(LIB_LINK_LIBS) $(LIB_LINK_FLAGS)
|
||||
|
||||
ifneq (,$(RML_NO_VERSION.DLL))
|
||||
$(RML_NO_VERSION.DLL):
|
||||
echo "INPUT ($(RML.DLL))" > $(RML_NO_VERSION.DLL)
|
||||
endif
|
||||
|
||||
rml: rml_dll
|
||||
rml_dll: $(RML.DLL)
|
||||
|
||||
#------------------------------------------------------
|
||||
# End of rules for making the RML server shared library
|
||||
#------------------------------------------------------
|
||||
|
||||
#------------------------------------------------------
|
||||
# Define rules for making the RML unit tests
|
||||
#------------------------------------------------------
|
||||
|
||||
add_debug=$(basename $(1))_debug$(suffix $(1))
|
||||
cross_suffix=$(if $(crosstest),$(if $(DEBUG_SUFFIX),$(subst _debug,,$(1)),$(call add_debug,$(1))),$(1))
|
||||
|
||||
RML_TESTS = test_job_automaton.$(TEST_EXT) test_thread_monitor.$(TEST_EXT)
|
||||
RML_CUSTOM_TESTS = test_rml_tbb.$(TEST_EXT)
|
||||
|
||||
test_rml_tbb.$(TEST_EXT): test_rml_tbb.$(OBJ) $(RML_TBB_CLIENT.OBJ) $(TBB_DEP_RML_TEST)
|
||||
$(CPLUS) $(OUTPUT_KEY)$@ $(CPLUS_FLAGS) test_rml_tbb.$(OBJ) $(RML_TBB_CLIENT.OBJ) $(TBB_DEP_RML_TEST) $(LIBS) $(LINK_FLAGS)
|
||||
|
||||
$(RML_TESTS): %.$(TEST_EXT): %.$(OBJ) $(TBB_DEP_NON_RML_TEST)
|
||||
$(CPLUS) $(OUTPUT_KEY)$@ $(CPLUS_FLAGS) $< $(TBB_DEP_NON_RML_TEST) $(LIBS) $(LINK_FLAGS)
|
||||
|
||||
### run_cmd is usually empty
|
||||
rml_test: $(call cross_suffix,$(RML.DLL)) $(TEST_PREREQUISITE) $(RML_TESTS) $(RML_CUSTOM_TESTS)
|
||||
$(run_cmd) ./test_job_automaton.$(TEST_EXT) $(args)
|
||||
$(run_cmd) ./test_thread_monitor.$(TEST_EXT) $(args)
|
||||
$(run_cmd) ./test_rml_tbb.$(TEST_EXT) $(args)
|
||||
|
||||
#------------------------------------------------------
|
||||
# End of rules for making the TBBMalloc unit tests
|
||||
#------------------------------------------------------
|
||||
|
||||
# Include automatically generated dependencies
|
||||
-include *.d
|
||||
|
||||
clean:
|
||||
-rm -rf *.o *.so* *.d *.def version_string.ver
|
||||
-rm -rf $(work_dir)_release/libirml*
|
||||
-rm -rf $(work_dir)_debug/libirml*
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,144 @@
|
||||
/*
|
||||
Copyright (c) 2017 Intel Corporation
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
|
||||
|
||||
|
||||
|
||||
*/
|
||||
|
||||
#include "ipc_utils.h"
|
||||
|
||||
#include <stdlib.h>
|
||||
#include <stdio.h>
|
||||
#include <limits.h>
|
||||
#include <string.h>
|
||||
#include <unistd.h>
|
||||
|
||||
namespace tbb {
|
||||
namespace internal {
|
||||
namespace rml {
|
||||
|
||||
#define MAX_STR_LEN 255
|
||||
#define STARTTIME_ITEM_ID 21
|
||||
|
||||
static char* get_stat_item(char* line, int item_id) {
|
||||
int id = 0, i = 0;
|
||||
|
||||
while( id!=item_id ) {
|
||||
while( line[i]!='(' && line[i]!=' ' && line[i]!='\0' ) {
|
||||
++i;
|
||||
}
|
||||
if( line[i]==' ' ) {
|
||||
++id;
|
||||
++i;
|
||||
} else if( line[i]=='(' ) {
|
||||
while( line[i]!=')' && line[i]!='\0' ) {
|
||||
++i;
|
||||
}
|
||||
if( line[i]==')' ) {
|
||||
++i;
|
||||
} else {
|
||||
return NULL;
|
||||
}
|
||||
} else {
|
||||
return NULL;
|
||||
}
|
||||
}
|
||||
|
||||
return line + i;
|
||||
}
|
||||
|
||||
unsigned long long get_start_time(int pid) {
|
||||
const char* stat_file_path_template = "/proc/%d/stat";
|
||||
char stat_file_path[MAX_STR_LEN + 1];
|
||||
sprintf( stat_file_path, stat_file_path_template, pid );
|
||||
|
||||
FILE* stat_file = fopen( stat_file_path, "rt" );
|
||||
if( stat_file==NULL ) {
|
||||
return 0;
|
||||
}
|
||||
|
||||
char stat_line[MAX_STR_LEN + 1];
|
||||
char* line = fgets( stat_line, MAX_STR_LEN, stat_file );
|
||||
if( line==NULL ) {
|
||||
return 0;
|
||||
}
|
||||
|
||||
char* starttime_str = get_stat_item( stat_line, STARTTIME_ITEM_ID );
|
||||
if( starttime_str==NULL ) {
|
||||
return 0;
|
||||
}
|
||||
|
||||
unsigned long long starttime = strtoull( starttime_str, NULL, 10 );
|
||||
if( starttime==ULLONG_MAX ) {
|
||||
return 0;
|
||||
}
|
||||
|
||||
return starttime;
|
||||
}
|
||||
|
||||
char* get_shared_name(const char* prefix, int pid, unsigned long long time) {
|
||||
const char* name_template = "%s_%d_%llu";
|
||||
const int digits_in_int = 10;
|
||||
const int digits_in_long = 20;
|
||||
|
||||
int len = strlen( name_template ) + strlen( prefix ) + digits_in_int + digits_in_long + 1;
|
||||
char* name = new char[len];
|
||||
sprintf( name, name_template, prefix, pid, time );
|
||||
|
||||
return name;
|
||||
}
|
||||
|
||||
char* get_shared_name(const char* prefix) {
|
||||
int pid = getpgrp();
|
||||
unsigned long long time = get_start_time( pid );
|
||||
return get_shared_name( prefix, pid, time );
|
||||
}
|
||||
|
||||
int get_num_threads(const char* env_var) {
|
||||
if( env_var==NULL ) {
|
||||
return 0;
|
||||
}
|
||||
|
||||
char* value = getenv( env_var );
|
||||
if( value==NULL ) {
|
||||
return 0;
|
||||
}
|
||||
|
||||
int num_threads = (int)strtol( value, NULL, 10 );
|
||||
return num_threads;
|
||||
}
|
||||
|
||||
bool get_enable_flag(const char* env_var) {
|
||||
if( env_var==NULL ) {
|
||||
return false;
|
||||
}
|
||||
|
||||
char* value = getenv( env_var );
|
||||
if( value==NULL ) {
|
||||
return false;
|
||||
}
|
||||
|
||||
if( strcmp( value, "0" ) == 0 ||
|
||||
strcmp( value, "false" ) == 0 ||
|
||||
strcmp( value, "False" ) == 0 ||
|
||||
strcmp( value, "FALSE" ) == 0 ) {
|
||||
return false;
|
||||
}
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
}}} //tbb::internal::rml
|
||||
@@ -0,0 +1,34 @@
|
||||
/*
|
||||
Copyright (c) 2017 Intel Corporation
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
|
||||
|
||||
|
||||
|
||||
*/
|
||||
|
||||
#ifndef __IPC_UTILS_H
|
||||
#define __IPC_UTILS_H
|
||||
|
||||
namespace tbb {
|
||||
namespace internal {
|
||||
namespace rml {
|
||||
|
||||
char* get_shared_name(const char* prefix);
|
||||
int get_num_threads(const char* env_var);
|
||||
bool get_enable_flag(const char* env_var);
|
||||
|
||||
}}} //tbb::internal::rml
|
||||
|
||||
#endif
|
||||
+6
-6
@@ -58,13 +58,14 @@ else:
|
||||
tbb_flag = ['-tbb'] if use_compiler_tbb else []
|
||||
compile_flags = ['-std=c++11', '-Wno-unused-variable']
|
||||
|
||||
_tbb = Extension("_TBB", ["tbb.i"],
|
||||
_tbb = Extension("tbb._api", ["tbb/api.i"],
|
||||
include_dirs=[os.path.join(tbb_root, 'include')] if not use_compiler_tbb else [],
|
||||
swig_opts =['-c++', '-O', '-threads'] + ( # add '-builtin' later
|
||||
['-I' + os.path.join(tbb_root, 'include')] if not use_compiler_tbb else []),
|
||||
extra_compile_args=compile_flags + tbb_flag,
|
||||
extra_link_args=tbb_flag,
|
||||
libraries =['tbb'] if not use_compiler_tbb else [],
|
||||
libraries =(['tbb'] if not use_compiler_tbb else []) +
|
||||
(['irml'] if platform.system() == "Linux" else []), # TODO: why do we need this?
|
||||
library_dirs=[os.path.join(tbb_root, 'lib', 'intel64', 'gcc4.4'), # for Linux
|
||||
os.path.join(tbb_root, 'lib'), # for MacOS
|
||||
os.path.join(tbb_root, 'lib', 'intel64', 'vc_mt'), # for Windows
|
||||
@@ -98,11 +99,9 @@ setup( name ="TBB",
|
||||
'Intended Audience :: Other Audience',
|
||||
'Intended Audience :: Science/Research',
|
||||
'License :: OSI Approved :: Apache Software License',
|
||||
'License :: Other/Proprietary License',
|
||||
'Operating System :: MacOS :: MacOS X',
|
||||
'Operating System :: Microsoft :: Windows',
|
||||
'Operating System :: POSIX',
|
||||
'Operating System :: Unix',
|
||||
'Operating System :: POSIX :: Linux',
|
||||
'Programming Language :: Python',
|
||||
'Programming Language :: Python :: 2',
|
||||
'Programming Language :: Python :: 3',
|
||||
@@ -110,8 +109,9 @@ setup( name ="TBB",
|
||||
'Topic :: System :: Hardware :: Symmetric Multi-processing',
|
||||
'Topic :: Software Development :: Libraries',
|
||||
],
|
||||
keywords='tbb multiprocessing multithreading composable parallelism',
|
||||
keywords='TBB multiprocessing multithreading composable parallelism',
|
||||
ext_modules=[_tbb],
|
||||
packages=['tbb'],
|
||||
py_modules=['TBB'],
|
||||
cmdclass={'build': TBBBuild}
|
||||
)
|
||||
|
||||
@@ -0,0 +1,264 @@
|
||||
#
|
||||
# Copyright (c) 2005-2017 Intel Corporation
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
from __future__ import print_function
|
||||
|
||||
import multiprocessing.pool
|
||||
import ctypes
|
||||
import atexit
|
||||
import sys
|
||||
import os
|
||||
|
||||
from .api import *
|
||||
from .api import __all__ as api__all
|
||||
from .pool import *
|
||||
from .pool import __all__ as pool__all
|
||||
|
||||
__all__ = ["Monkey"] + api__all + pool__all
|
||||
|
||||
__doc__ = """
|
||||
Python API for Intel(R) Threading Building Blocks library (Intel(R) TBB)
|
||||
extended with standard Python's pools implementation and monkey-patching.
|
||||
|
||||
Command-line interface example:
|
||||
$ python -m tbb $your_script.py
|
||||
Runs your_script.py in context of tbb.Monkey
|
||||
"""
|
||||
|
||||
|
||||
ipc_enabled = False
|
||||
libirml = "libirml.so.1"
|
||||
|
||||
|
||||
def _test(arg=None):
|
||||
"""Some tests"""
|
||||
import platform
|
||||
if platform.system() == "Linux":
|
||||
ctypes.CDLL(libirml)
|
||||
from .test import test
|
||||
test(arg)
|
||||
print("done")
|
||||
|
||||
|
||||
def tbb_process_pool_worker27(inqueue, outqueue, initializer=None, initargs=(),
|
||||
maxtasks=None):
|
||||
from multiprocessing.pool import worker
|
||||
worker(inqueue, outqueue, initializer, initargs, maxtasks)
|
||||
if ipc_enabled:
|
||||
try:
|
||||
librml = ctypes.CDLL(libirml)
|
||||
librml.release_resources()
|
||||
except:
|
||||
print("Warning: Can not load ", libirml, file=sys.stderr)
|
||||
|
||||
|
||||
class TBBProcessPool27(multiprocessing.pool.Pool):
|
||||
def _repopulate_pool(self):
|
||||
"""Bring the number of pool processes up to the specified number,
|
||||
for use after reaping workers which have exited.
|
||||
"""
|
||||
from multiprocessing.util import debug
|
||||
|
||||
for i in range(self._processes - len(self._pool)):
|
||||
w = self.Process(target=tbb_process_pool_worker27,
|
||||
args=(self._inqueue, self._outqueue,
|
||||
self._initializer,
|
||||
self._initargs, self._maxtasksperchild)
|
||||
)
|
||||
self._pool.append(w)
|
||||
w.name = w.name.replace('Process', 'PoolWorker')
|
||||
w.daemon = True
|
||||
w.start()
|
||||
debug('added worker')
|
||||
|
||||
def __del__(self):
|
||||
self.close()
|
||||
for p in self._pool:
|
||||
p.join()
|
||||
|
||||
def __exit__(self):
|
||||
self.close()
|
||||
for p in self._pool:
|
||||
p.join()
|
||||
|
||||
|
||||
def tbb_process_pool_worker35(inqueue, outqueue, initializer=None, initargs=(),
|
||||
maxtasks=None, wrap_exception=False):
|
||||
from multiprocessing.pool import worker
|
||||
worker(inqueue, outqueue, initializer, initargs, maxtasks, wrap_exception)
|
||||
if ipc_enabled:
|
||||
try:
|
||||
librml = ctypes.CDLL(libirml)
|
||||
librml.release_resources()
|
||||
except:
|
||||
print("Warning: Can not load ", libirml, file=sys.stderr)
|
||||
|
||||
|
||||
class TBBProcessPool35(multiprocessing.pool.Pool):
|
||||
def _repopulate_pool(self):
|
||||
"""Bring the number of pool processes up to the specified number,
|
||||
for use after reaping workers which have exited.
|
||||
"""
|
||||
from multiprocessing.util import debug
|
||||
|
||||
for i in range(self._processes - len(self._pool)):
|
||||
w = self.Process(target=tbb_process_pool_worker35,
|
||||
args=(self._inqueue, self._outqueue,
|
||||
self._initializer,
|
||||
self._initargs, self._maxtasksperchild,
|
||||
self._wrap_exception)
|
||||
)
|
||||
self._pool.append(w)
|
||||
w.name = w.name.replace('Process', 'PoolWorker')
|
||||
w.daemon = True
|
||||
w.start()
|
||||
debug('added worker')
|
||||
|
||||
def __del__(self):
|
||||
self.close()
|
||||
for p in self._pool:
|
||||
p.join()
|
||||
|
||||
def __exit__(self):
|
||||
self.close()
|
||||
for p in self._pool:
|
||||
p.join()
|
||||
|
||||
|
||||
class Monkey:
|
||||
"""
|
||||
Context manager which replaces standard multiprocessing.pool
|
||||
implementations with tbb.pool using monkey-patching. It also enables TBB
|
||||
threading for Intel(R) Math Kernel Library (Intel(R) MKL). For example:
|
||||
|
||||
with tbb.Monkey():
|
||||
run_my_numpy_code()
|
||||
|
||||
It allows multiple parallel tasks to be executed on the same thread pool
|
||||
and coordinate number of threads across multiple processes thus avoiding
|
||||
overheads from oversubscription.
|
||||
"""
|
||||
_items = {'ThreadPool' : None,
|
||||
'Pool' : None}
|
||||
_modules = {'ThreadPool' : None,
|
||||
'Pool' : None}
|
||||
|
||||
def __init__(self, max_num_threads=None):
|
||||
"""
|
||||
Create context manager for running under TBB scheduler.
|
||||
:param max_num_threads: if specified, limits maximal number of threads
|
||||
"""
|
||||
if max_num_threads:
|
||||
self.ctl = global_control(global_control.max_allowed_parallelism, int(max_num_threads))
|
||||
|
||||
def _patch(self, class_name, module_name, object):
|
||||
self._modules[class_name] = __import__(module_name, globals(),
|
||||
locals(), [class_name])
|
||||
if self._modules[class_name] == None:
|
||||
return
|
||||
oldattr = getattr(self._modules[class_name], class_name, None)
|
||||
if oldattr == None:
|
||||
self._modules[class_name] = None
|
||||
return
|
||||
self._items[class_name] = oldattr
|
||||
setattr(self._modules[class_name], class_name, object)
|
||||
|
||||
def __enter__(self):
|
||||
self.env = os.getenv('MKL_THREADING_LAYER')
|
||||
os.environ['MKL_THREADING_LAYER'] = 'TBB'
|
||||
if sys.version_info.major == 2 and sys.version_info.minor >= 7:
|
||||
self._patch("Pool", "multiprocessing.pool", TBBProcessPool27)
|
||||
elif sys.version_info.major == 3 and sys.version_info.minor >= 5:
|
||||
self._patch("Pool", "multiprocessing.pool", TBBProcessPool35)
|
||||
self._patch("ThreadPool", "multiprocessing.pool", Pool)
|
||||
return self
|
||||
|
||||
def __exit__(self, exc_type, exc_value, traceback):
|
||||
if self.env is None:
|
||||
del os.environ['MKL_THREADING_LAYER']
|
||||
else:
|
||||
os.environ['MKL_THREADING_LAYER'] = self.env
|
||||
for name in self._items.keys():
|
||||
setattr(self._modules[name], name, self._items[name])
|
||||
|
||||
|
||||
def init_sem_name():
|
||||
try:
|
||||
librml = ctypes.CDLL(libirml)
|
||||
librml.set_active_sem_name()
|
||||
librml.set_stop_sem_name()
|
||||
except Exception as e:
|
||||
print("Warning: Can not initialize name of shared semaphores:", e,
|
||||
file=sys.stderr)
|
||||
|
||||
|
||||
def tbb_atexit():
|
||||
if ipc_enabled:
|
||||
try:
|
||||
librml = ctypes.CDLL(libirml)
|
||||
librml.release_semaphores()
|
||||
except:
|
||||
print("Warning: Can not release shared semaphores",
|
||||
file=sys.stderr)
|
||||
|
||||
|
||||
def _main():
|
||||
# Run the module specified as the next command line argument
|
||||
# python -m TBB user_app.py
|
||||
global ipc_enabled
|
||||
|
||||
import platform
|
||||
import argparse
|
||||
parser = argparse.ArgumentParser(prog="python -m tbb", description="""
|
||||
Run your Python script in context of tbb.Monkey, which
|
||||
replaces standard Python pools and threading layer of
|
||||
Intel(R) Math Kernel Library by implementation based on
|
||||
Intel(R) Threading Building Blocks. It enables multiple parallel
|
||||
tasks to be executed on the same thread pool and coordinate
|
||||
number of threads across multiple processes thus avoiding
|
||||
overheads from oversubscription.
|
||||
""", formatter_class=argparse.ArgumentDefaultsHelpFormatter)
|
||||
if platform.system() == "Linux":
|
||||
parser.add_argument('--ipc', action='store_true',
|
||||
help="Enable inter-precess (IPC) synchronization")
|
||||
parser.add_argument('-p', '--max-num-threads', default=default_num_threads(), type=int,
|
||||
help="Initialize TBB with P max number of threads per process", metavar='P')
|
||||
#parser.add_argument('-a', '--replace-allocator', action='store_true',
|
||||
# help="Enable TBB malloc")
|
||||
#parser.add_argument('--huge-pages', action='store_true',
|
||||
# help="Enable huge pages for TBB malloc")
|
||||
parser.add_argument('-m', action='store_true', dest='module',
|
||||
help="Executes following as a module")
|
||||
parser.add_argument('name', help="Script or module name")
|
||||
parser.add_argument('args', nargs=argparse.REMAINDER,
|
||||
help="Command line arguments")
|
||||
args = parser.parse_args()
|
||||
sys.argv = [args.name] + args.args
|
||||
|
||||
ipc_enabled = platform.system() == "Linux" and args.ipc
|
||||
os.environ["IPC_ENABLE"] = "1" if ipc_enabled else "0"
|
||||
if ipc_enabled:
|
||||
atexit.register(tbb_atexit)
|
||||
init_sem_name()
|
||||
|
||||
if not os.environ.get("KMP_BLOCKTIME"): # TODO move
|
||||
os.environ["KMP_BLOCKTIME"] = "0"
|
||||
if '_' + args.name in globals():
|
||||
return globals()['_' + args.name](*args.args)
|
||||
else:
|
||||
import runpy
|
||||
runf = runpy.run_module if args.module else runpy.run_path
|
||||
with Monkey(max_num_threads=args.max_num_threads):
|
||||
runf(args.name, run_name='__main__')
|
||||
@@ -0,0 +1,19 @@
|
||||
#
|
||||
# Copyright (c) 2005-2017 Intel Corporation
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
|
||||
from . import _main
|
||||
from sys import exit
|
||||
exit(_main())
|
||||
@@ -18,50 +18,19 @@
|
||||
#
|
||||
#
|
||||
|
||||
|
||||
# Based on the software developed by:
|
||||
# Copyright (c) 2008,2016 david decotigny (Pool of threads)
|
||||
# Copyright (c) 2006-2008, R Oudkerk (multiprocessing.Pool)
|
||||
# All rights reserved.
|
||||
#
|
||||
# Redistribution and use in source and binary forms, with or without
|
||||
# modification, are permitted provided that the following conditions
|
||||
# are met:
|
||||
#
|
||||
# 1. Redistributions of source code must retain the above copyright
|
||||
# notice, this list of conditions and the following disclaimer.
|
||||
# 2. Redistributions in binary form must reproduce the above copyright
|
||||
# notice, this list of conditions and the following disclaimer in the
|
||||
# documentation and/or other materials provided with the distribution.
|
||||
# 3. Neither the name of author nor the names of any contributors may be
|
||||
# used to endorse or promote products derived from this software
|
||||
# without specific prior written permission.
|
||||
#
|
||||
# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS "AS IS" AND
|
||||
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
|
||||
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
|
||||
# ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
|
||||
# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
|
||||
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
|
||||
# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
|
||||
# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
|
||||
# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
|
||||
# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
|
||||
# SUCH DAMAGE.
|
||||
#
|
||||
|
||||
from __future__ import print_function
|
||||
__all__ = ["task_arena", "task_group", "task_scheduler_init", "global_control", "default_num_threads"]
|
||||
%}
|
||||
%begin %{
|
||||
/* Defines Python wrappers for Intel(R) Threading Building Blocks (Intel TBB).*/
|
||||
%}
|
||||
%module TBB
|
||||
%module api
|
||||
|
||||
#if SWIG_VERSION < 0x030001
|
||||
#error SWIG version 3.0.6 or newer is required for correct functioning
|
||||
#endif
|
||||
|
||||
%{
|
||||
#define TBB_PREVIEW_GLOBAL_CONTROL 1
|
||||
#include <tbb/tbb.h>
|
||||
using namespace tbb;
|
||||
|
||||
@@ -140,7 +109,21 @@ namespace tbb {
|
||||
};
|
||||
};
|
||||
|
||||
}
|
||||
class global_control {
|
||||
public:
|
||||
enum parameter {
|
||||
max_allowed_parallelism,
|
||||
thread_stack_size,
|
||||
parameter_max // insert new parameters above this point
|
||||
};
|
||||
global_control(parameter param, size_t value);
|
||||
~global_control();
|
||||
static size_t active_value(parameter param);
|
||||
};
|
||||
|
||||
// Python part of the module
|
||||
%pythoncode "tbb.src.py"
|
||||
} // tbb
|
||||
|
||||
// Additional definitions for Python part of the module
|
||||
%pythoncode %{
|
||||
default_num_threads = task_scheduler_init_default_num_threads
|
||||
%}
|
||||
@@ -1,4 +1,3 @@
|
||||
|
||||
#
|
||||
# Copyright (c) 2005-2017 Intel Corporation
|
||||
#
|
||||
@@ -73,20 +72,14 @@
|
||||
import sys
|
||||
import threading
|
||||
import traceback
|
||||
from .api import *
|
||||
|
||||
__all__ = ["Pool", "Monkey", "task_arena", "task_group", "task_scheduler_init"]
|
||||
__all__ = ["Pool", "TimeoutError"]
|
||||
__doc__ = """
|
||||
Python API to Intel(R) Threading Building Blocks library (Intel TBB)
|
||||
extended with standard Pool implementation and monkey-patching.
|
||||
|
||||
Command-line interface:
|
||||
$ python -m TBB $your_script.py
|
||||
|
||||
Runs your_script.py in context of `with Monkey():`
|
||||
Standard Python Pool implementation based on Python API
|
||||
for Intel(R) Threading Building Blocks library (Intel(R) TBB)
|
||||
"""
|
||||
|
||||
default_num_threads = task_scheduler_init_default_num_threads
|
||||
|
||||
|
||||
class TimeoutError(Exception):
|
||||
"""Raised when a result is not available within the given timeout"""
|
||||
@@ -96,7 +89,7 @@ class TimeoutError(Exception):
|
||||
class Pool(object):
|
||||
"""
|
||||
The Pool class provides standard multiprocessing.Pool interface
|
||||
which is mapped onto Intel TBB tasks executing in its thread pool
|
||||
which is mapped onto Intel(R) TBB tasks executing in its thread pool
|
||||
"""
|
||||
|
||||
def __init__(self, nworkers=0, name="Pool"):
|
||||
@@ -639,202 +632,3 @@ class OrderedResultCollector(AbstractResultCollector):
|
||||
self._to_notify._set_exception()
|
||||
else:
|
||||
self._to_notify._set_value(lst)
|
||||
|
||||
|
||||
def _test(arg=None):
|
||||
"""Some tests"""
|
||||
if arg == "-v":
|
||||
def say(*x):
|
||||
print(*x)
|
||||
else:
|
||||
def say(*x):
|
||||
pass
|
||||
say("Start Pool testing")
|
||||
import time
|
||||
|
||||
get_tid = lambda: threading.current_thread().ident
|
||||
|
||||
def return42():
|
||||
return 42
|
||||
|
||||
def f(x):
|
||||
return x * x
|
||||
|
||||
def work(mseconds):
|
||||
res = str(mseconds)
|
||||
if mseconds < 0:
|
||||
mseconds = -mseconds
|
||||
say("[%d] Start to work for %fms..." % (get_tid(), mseconds*10))
|
||||
time.sleep(mseconds/100.)
|
||||
say("[%d] Work done (%fms)." % (get_tid(), mseconds*10))
|
||||
return res
|
||||
|
||||
### Test copy/pasted from multiprocessing
|
||||
pool = Pool(4) # start worker threads
|
||||
|
||||
# edge cases
|
||||
assert pool.map(return42, []) == []
|
||||
assert pool.apply_async(return42, []).get() == 42
|
||||
assert pool.apply(return42, []) == 42
|
||||
assert list(pool.imap(return42, iter([]))) == []
|
||||
assert list(pool.imap_unordered(return42, iter([]))) == []
|
||||
assert pool.map_async(return42, []).get() == []
|
||||
assert list(pool.imap_async(return42, iter([])).get()) == []
|
||||
assert list(pool.imap_unordered_async(return42, iter([])).get()) == []
|
||||
|
||||
# basic tests
|
||||
result = pool.apply_async(f, (10,)) # evaluate "f(10)" asynchronously
|
||||
assert result.get(timeout=1) == 100 # ... unless slow computer
|
||||
assert list(pool.map(f, range(10))) == list(map(f, range(10)))
|
||||
it = pool.imap(f, range(10))
|
||||
assert next(it) == 0
|
||||
assert next(it) == 1
|
||||
assert next(it) == 4
|
||||
|
||||
# Test apply_sync exceptions
|
||||
result = pool.apply_async(time.sleep, (3,))
|
||||
try:
|
||||
say(result.get(timeout=1)) # raises `TimeoutError`
|
||||
except TimeoutError:
|
||||
say("Good. Got expected timeout exception.")
|
||||
else:
|
||||
assert False, "Expected exception !"
|
||||
assert result.get() is None # sleep() returns None
|
||||
|
||||
def cb(s):
|
||||
say("Result ready: %s" % s)
|
||||
|
||||
# Test imap()
|
||||
assert list(pool.imap(work, range(10, 3, -1), chunksize=4)) == list(map(
|
||||
str, range(10, 3, -1)))
|
||||
|
||||
# Test imap_unordered()
|
||||
assert sorted(pool.imap_unordered(work, range(10, 3, -1))) == sorted(map(
|
||||
str, range(10, 3, -1)))
|
||||
|
||||
# Test map_async()
|
||||
result = pool.map_async(work, range(10), callback=cb)
|
||||
try:
|
||||
result.get(timeout=0.01) # raises `TimeoutError`
|
||||
except TimeoutError:
|
||||
say("Good. Got expected timeout exception.")
|
||||
else:
|
||||
assert False, "Expected exception !"
|
||||
say(result.get())
|
||||
|
||||
# Test imap_async()
|
||||
result = pool.imap_async(work, range(3, 10), callback=cb)
|
||||
try:
|
||||
result.get(timeout=0.01) # raises `TimeoutError`
|
||||
except TimeoutError:
|
||||
say("Good. Got expected timeout exception.")
|
||||
else:
|
||||
assert False, "Expected exception !"
|
||||
for i in result.get():
|
||||
say("Item:", i)
|
||||
say("### Loop again:")
|
||||
for i in result.get():
|
||||
say("Item2:", i)
|
||||
|
||||
# Test imap_unordered_async()
|
||||
result = pool.imap_unordered_async(work, range(10, 3, -1), callback=cb)
|
||||
try:
|
||||
say(result.get(timeout=0.01)) # raises `TimeoutError`
|
||||
except TimeoutError:
|
||||
say("Good. Got expected timeout exception.")
|
||||
else:
|
||||
assert False, "Expected exception !"
|
||||
for i in result.get():
|
||||
say("Item1:", i)
|
||||
for i in result.get():
|
||||
say("Item2:", i)
|
||||
r = result.get()
|
||||
for i in r:
|
||||
say("Item3:", i)
|
||||
for i in r:
|
||||
say("Item4:", i)
|
||||
for i in r:
|
||||
say("Item5:", i)
|
||||
|
||||
#
|
||||
# The case for the exceptions
|
||||
#
|
||||
|
||||
# Exceptions in imap_unordered_async()
|
||||
result = pool.imap_unordered_async(work, range(2, -10, -1), callback=cb)
|
||||
time.sleep(3)
|
||||
try:
|
||||
for i in result.get():
|
||||
say("Got item:", i)
|
||||
except (IOError, ValueError):
|
||||
say("Good. Got expected exception")
|
||||
|
||||
# Exceptions in imap_async()
|
||||
result = pool.imap_async(work, range(2, -10, -1), callback=cb)
|
||||
time.sleep(3)
|
||||
try:
|
||||
for i in result.get():
|
||||
say("Got item:", i)
|
||||
except (IOError, ValueError):
|
||||
say("Good. Got expected exception")
|
||||
|
||||
# Stop the test: need to stop the pool !!!
|
||||
pool.terminate()
|
||||
pool.join()
|
||||
print("done")
|
||||
|
||||
|
||||
# End of david's derived file content
|
||||
|
||||
class Monkey:
|
||||
"""
|
||||
Context manager which replaces standard multiprocessing.pool.ThreadPool
|
||||
implementation with TBB.Pool using monkey-patching. It also enables TBB
|
||||
threading for Intel MKL. For example:
|
||||
|
||||
with TBB.Monkey():
|
||||
run_my_numpy_code()
|
||||
|
||||
"""
|
||||
_items = {'ThreadPool': None}
|
||||
|
||||
def __init__(self):
|
||||
pass
|
||||
|
||||
def __enter__(self):
|
||||
import os
|
||||
self.env = os.getenv('MKL_THREADING_LAYER')
|
||||
os.environ['MKL_THREADING_LAYER'] = 'TBB'
|
||||
self.module = __import__('multiprocessing.pool', globals(), locals(), self._items.keys())
|
||||
for name in self._items.keys():
|
||||
oldattr = getattr(self.module, name)
|
||||
self._items[name] = oldattr
|
||||
setattr(self.module, name, Pool)
|
||||
return self
|
||||
|
||||
def __exit__(self, exc_type, exc_value, traceback):
|
||||
import os
|
||||
if self.env is None:
|
||||
del os.environ['MKL_THREADING_LAYER']
|
||||
else:
|
||||
os.environ['MKL_THREADING_LAYER'] = self.env
|
||||
for name in self._items.keys():
|
||||
setattr(self.module, name, self._items[name])
|
||||
|
||||
|
||||
def _main():
|
||||
# Run the module specified as the next command line argument
|
||||
# python -m TBB user_app.py
|
||||
del sys.argv[0] # shift arguments
|
||||
if len(sys.argv) < 1:
|
||||
print("No file name specified for execution", file=sys.stderr)
|
||||
elif '_' + sys.argv[0] in globals():
|
||||
globals()['_' + sys.argv[0]](*sys.argv[1:])
|
||||
else:
|
||||
import runpy
|
||||
with Monkey():
|
||||
runpy.run_path(sys.argv[0], run_name='__main__')
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(_main())
|
||||
@@ -0,0 +1,198 @@
|
||||
#
|
||||
# Copyright (c) 2005-2017 Intel Corporation
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
#
|
||||
#
|
||||
#
|
||||
|
||||
# Based on the software developed by:
|
||||
# Copyright (c) 2008,2016 david decotigny (Pool of threads)
|
||||
# Copyright (c) 2006-2008, R Oudkerk (multiprocessing.Pool)
|
||||
# All rights reserved.
|
||||
#
|
||||
# Redistribution and use in source and binary forms, with or without
|
||||
# modification, are permitted provided that the following conditions
|
||||
# are met:
|
||||
#
|
||||
# 1. Redistributions of source code must retain the above copyright
|
||||
# notice, this list of conditions and the following disclaimer.
|
||||
# 2. Redistributions in binary form must reproduce the above copyright
|
||||
# notice, this list of conditions and the following disclaimer in the
|
||||
# documentation and/or other materials provided with the distribution.
|
||||
# 3. Neither the name of author nor the names of any contributors may be
|
||||
# used to endorse or promote products derived from this software
|
||||
# without specific prior written permission.
|
||||
#
|
||||
# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS "AS IS" AND
|
||||
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
|
||||
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
|
||||
# ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
|
||||
# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
|
||||
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
|
||||
# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
|
||||
# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
|
||||
# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
|
||||
# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
|
||||
# SUCH DAMAGE.
|
||||
#
|
||||
|
||||
from __future__ import print_function
|
||||
import time
|
||||
import threading
|
||||
|
||||
from .api import *
|
||||
from .pool import *
|
||||
|
||||
|
||||
def test(arg=None):
|
||||
if arg == "-v":
|
||||
def say(*x):
|
||||
print(*x)
|
||||
else:
|
||||
def say(*x):
|
||||
pass
|
||||
say("Start Pool testing")
|
||||
|
||||
get_tid = lambda: threading.current_thread().ident
|
||||
|
||||
def return42():
|
||||
return 42
|
||||
|
||||
def f(x):
|
||||
return x * x
|
||||
|
||||
def work(mseconds):
|
||||
res = str(mseconds)
|
||||
if mseconds < 0:
|
||||
mseconds = -mseconds
|
||||
say("[%d] Start to work for %fms..." % (get_tid(), mseconds*10))
|
||||
time.sleep(mseconds/100.)
|
||||
say("[%d] Work done (%fms)." % (get_tid(), mseconds*10))
|
||||
return res
|
||||
|
||||
### Test copy/pasted from multiprocessing
|
||||
pool = Pool(4) # start worker threads
|
||||
|
||||
# edge cases
|
||||
assert pool.map(return42, []) == []
|
||||
assert pool.apply_async(return42, []).get() == 42
|
||||
assert pool.apply(return42, []) == 42
|
||||
assert list(pool.imap(return42, iter([]))) == []
|
||||
assert list(pool.imap_unordered(return42, iter([]))) == []
|
||||
assert pool.map_async(return42, []).get() == []
|
||||
assert list(pool.imap_async(return42, iter([])).get()) == []
|
||||
assert list(pool.imap_unordered_async(return42, iter([])).get()) == []
|
||||
|
||||
# basic tests
|
||||
result = pool.apply_async(f, (10,)) # evaluate "f(10)" asynchronously
|
||||
assert result.get(timeout=1) == 100 # ... unless slow computer
|
||||
assert list(pool.map(f, range(10))) == list(map(f, range(10)))
|
||||
it = pool.imap(f, range(10))
|
||||
assert next(it) == 0
|
||||
assert next(it) == 1
|
||||
assert next(it) == 4
|
||||
|
||||
# Test apply_sync exceptions
|
||||
result = pool.apply_async(time.sleep, (3,))
|
||||
try:
|
||||
say(result.get(timeout=1)) # raises `TimeoutError`
|
||||
except TimeoutError:
|
||||
say("Good. Got expected timeout exception.")
|
||||
else:
|
||||
assert False, "Expected exception !"
|
||||
assert result.get() is None # sleep() returns None
|
||||
|
||||
def cb(s):
|
||||
say("Result ready: %s" % s)
|
||||
|
||||
# Test imap()
|
||||
assert list(pool.imap(work, range(10, 3, -1), chunksize=4)) == list(map(
|
||||
str, range(10, 3, -1)))
|
||||
|
||||
# Test imap_unordered()
|
||||
assert sorted(pool.imap_unordered(work, range(10, 3, -1))) == sorted(map(
|
||||
str, range(10, 3, -1)))
|
||||
|
||||
# Test map_async()
|
||||
result = pool.map_async(work, range(10), callback=cb)
|
||||
try:
|
||||
result.get(timeout=0.01) # raises `TimeoutError`
|
||||
except TimeoutError:
|
||||
say("Good. Got expected timeout exception.")
|
||||
else:
|
||||
assert False, "Expected exception !"
|
||||
say(result.get())
|
||||
|
||||
# Test imap_async()
|
||||
result = pool.imap_async(work, range(3, 10), callback=cb)
|
||||
try:
|
||||
result.get(timeout=0.01) # raises `TimeoutError`
|
||||
except TimeoutError:
|
||||
say("Good. Got expected timeout exception.")
|
||||
else:
|
||||
assert False, "Expected exception !"
|
||||
for i in result.get():
|
||||
say("Item:", i)
|
||||
say("### Loop again:")
|
||||
for i in result.get():
|
||||
say("Item2:", i)
|
||||
|
||||
# Test imap_unordered_async()
|
||||
result = pool.imap_unordered_async(work, range(10, 3, -1), callback=cb)
|
||||
try:
|
||||
say(result.get(timeout=0.01)) # raises `TimeoutError`
|
||||
except TimeoutError:
|
||||
say("Good. Got expected timeout exception.")
|
||||
else:
|
||||
assert False, "Expected exception !"
|
||||
for i in result.get():
|
||||
say("Item1:", i)
|
||||
for i in result.get():
|
||||
say("Item2:", i)
|
||||
r = result.get()
|
||||
for i in r:
|
||||
say("Item3:", i)
|
||||
for i in r:
|
||||
say("Item4:", i)
|
||||
for i in r:
|
||||
say("Item5:", i)
|
||||
|
||||
#
|
||||
# The case for the exceptions
|
||||
#
|
||||
|
||||
# Exceptions in imap_unordered_async()
|
||||
result = pool.imap_unordered_async(work, range(2, -10, -1), callback=cb)
|
||||
time.sleep(3)
|
||||
try:
|
||||
for i in result.get():
|
||||
say("Got item:", i)
|
||||
except (IOError, ValueError):
|
||||
say("Good. Got expected exception")
|
||||
|
||||
# Exceptions in imap_async()
|
||||
result = pool.imap_async(work, range(2, -10, -1), callback=cb)
|
||||
time.sleep(3)
|
||||
try:
|
||||
for i in result.get():
|
||||
say("Got item:", i)
|
||||
except (IOError, ValueError):
|
||||
say("Good. Got expected exception")
|
||||
|
||||
# Stop the test: need to stop the pool !!!
|
||||
pool.terminate()
|
||||
pool.join()
|
||||
|
||||
|
||||
Reference in New Issue
Block a user