Multiprocessing with nested loops and some numpy function callsCalling a function of a module by using its name (a string)What's the best way to break from nested (for) loops?Is there a NumPy function to return the first index of something in an array?Breaking out of nested loopsDynamic processes in PythonHow do I break out of nested loops in Java?Calling remove in foreach loop in Javapython multiprocessing vs threading for cpu bound work on windows and linuxUse numpy array in shared memory for multiprocessingWhat is the difference between flatten and ravel functions in numpy?
Output visual diagram of picture
Why didn’t Eve recognize the little cockroach as a living organism?
Derivative of an interpolated function
Capacitor electron flow
Do native speakers use "ultima" and "proxima" frequently in spoken English?
Why doesn't Gödel's incompleteness theorem apply to false statements?
Why do Radio Buttons not fill the entire outer circle?
Did I make a mistake by ccing email to boss to others?
Why is indicated airspeed rather than ground speed used during the takeoff roll?
Showing mass murder in a kid's book
Can a Knock spell open the door to Mordenkainen's Magnificent Mansion?
Center page as a whole without centering each element individually
Turning a hard to access nut?
Non-Borel set in arbitrary metric space
Would a primitive species be able to learn English from reading books alone?
Asserting that Atheism and Theism are both faith based positions
How to split IPA spelling into syllables
How do you justify more code being written by following clean code practices?
Mortal danger in mid-grade literature
What is the tangent at a sharp point on a curve?
Should I be concerned about student access to a test bank?
Trouble reading roman numeral notation with flats
What is the purpose of using a decision tree?
What should be the ideal length of sentences in a blog post for ease of reading?
Multiprocessing with nested loops and some numpy function calls
Calling a function of a module by using its name (a string)What's the best way to break from nested (for) loops?Is there a NumPy function to return the first index of something in an array?Breaking out of nested loopsDynamic processes in PythonHow do I break out of nested loops in Java?Calling remove in foreach loop in Javapython multiprocessing vs threading for cpu bound work on windows and linuxUse numpy array in shared memory for multiprocessingWhat is the difference between flatten and ravel functions in numpy?
I have read some coding examples about multiprocessing and am stil quite confused about it. Here is my contrived example:
import numpy as np
def data_processing(x,y,z): return np.array([x,y])*(z**0.5)
def foo(n1,n2):
final_result =
for i in range(n1):
result = np.zeros([n2,n2])
for j1 in range(n2):
for j2 in range(j1):
temp= data_processing(j1,j2,i)
result[j1,j2] = np.prod(temp)
final_result[str(i)] = result
return final_result
if __name__ == '__main__':
X = foo(9,9)
If I want to run this piece of code while utilizing all of the cpu cores, what should I change? Thank you in advance
python loops numpy multiprocessing nested-loops
add a comment |
I have read some coding examples about multiprocessing and am stil quite confused about it. Here is my contrived example:
import numpy as np
def data_processing(x,y,z): return np.array([x,y])*(z**0.5)
def foo(n1,n2):
final_result =
for i in range(n1):
result = np.zeros([n2,n2])
for j1 in range(n2):
for j2 in range(j1):
temp= data_processing(j1,j2,i)
result[j1,j2] = np.prod(temp)
final_result[str(i)] = result
return final_result
if __name__ == '__main__':
X = foo(9,9)
If I want to run this piece of code while utilizing all of the cpu cores, what should I change? Thank you in advance
python loops numpy multiprocessing nested-loops
add a comment |
I have read some coding examples about multiprocessing and am stil quite confused about it. Here is my contrived example:
import numpy as np
def data_processing(x,y,z): return np.array([x,y])*(z**0.5)
def foo(n1,n2):
final_result =
for i in range(n1):
result = np.zeros([n2,n2])
for j1 in range(n2):
for j2 in range(j1):
temp= data_processing(j1,j2,i)
result[j1,j2] = np.prod(temp)
final_result[str(i)] = result
return final_result
if __name__ == '__main__':
X = foo(9,9)
If I want to run this piece of code while utilizing all of the cpu cores, what should I change? Thank you in advance
python loops numpy multiprocessing nested-loops
I have read some coding examples about multiprocessing and am stil quite confused about it. Here is my contrived example:
import numpy as np
def data_processing(x,y,z): return np.array([x,y])*(z**0.5)
def foo(n1,n2):
final_result =
for i in range(n1):
result = np.zeros([n2,n2])
for j1 in range(n2):
for j2 in range(j1):
temp= data_processing(j1,j2,i)
result[j1,j2] = np.prod(temp)
final_result[str(i)] = result
return final_result
if __name__ == '__main__':
X = foo(9,9)
If I want to run this piece of code while utilizing all of the cpu cores, what should I change? Thank you in advance
python loops numpy multiprocessing nested-loops
python loops numpy multiprocessing nested-loops
asked Mar 7 at 20:33
mathguymathguy
1197
1197
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
Maby this can help.
import multiprocessing
import numpy as np
import time
import multiprocessing
def data_processing(x, y, z): return np.array([x, y]) * (z ** 0.5)
def foo(n1, n2, id=0, return_dict=[None]):
final_result =
for i in range(n1):
result = np.zeros([n2, n2])
for j1 in range(n2):
for j2 in range(j1):
temp = data_processing(j1, j2, i)
result[j1, j2] = np.prod(temp)
final_result[str(i)] = result
return_dict[id] = final_result
stamp = time.time()
def pint(num):
print(f'*Test [num] - seconds: time.time() - stamp')
for i in range(10):
foo(90, 90)
pint(0)
stamp = time.time()
manager = multiprocessing.Manager()
return_dict = manager.dict()
processes = []
for i in range(10):
p = multiprocessing.Process(target=foo, args=(90, 90, i, return_dict))
processes.append(p)
p.start()
for p in processes:
p.join()
x0, x1, x2, x3, x4, x5, x6, x7, x8, x9 = return_dict.values()
pint(1)
My output is:
*Test [0] - seconds: 26.120166301727295
*Test [1] - seconds: 8.343111753463745
Process finished with exit code 0
appreciate any input to this. I am wondering can the same logic apply to my version of foo function? I am asking this because the part I want to run multiprocessing on also contains some numpy functions call.
– mathguy
Mar 7 at 21:20
I edited my post. Did I understand right?? But it only makes sence with big operations so i have done it with really high values. Hope I could help (-;
– apatrck00
Mar 8 at 18:31
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55052363%2fmultiprocessing-with-nested-loops-and-some-numpy-function-calls%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Maby this can help.
import multiprocessing
import numpy as np
import time
import multiprocessing
def data_processing(x, y, z): return np.array([x, y]) * (z ** 0.5)
def foo(n1, n2, id=0, return_dict=[None]):
final_result =
for i in range(n1):
result = np.zeros([n2, n2])
for j1 in range(n2):
for j2 in range(j1):
temp = data_processing(j1, j2, i)
result[j1, j2] = np.prod(temp)
final_result[str(i)] = result
return_dict[id] = final_result
stamp = time.time()
def pint(num):
print(f'*Test [num] - seconds: time.time() - stamp')
for i in range(10):
foo(90, 90)
pint(0)
stamp = time.time()
manager = multiprocessing.Manager()
return_dict = manager.dict()
processes = []
for i in range(10):
p = multiprocessing.Process(target=foo, args=(90, 90, i, return_dict))
processes.append(p)
p.start()
for p in processes:
p.join()
x0, x1, x2, x3, x4, x5, x6, x7, x8, x9 = return_dict.values()
pint(1)
My output is:
*Test [0] - seconds: 26.120166301727295
*Test [1] - seconds: 8.343111753463745
Process finished with exit code 0
appreciate any input to this. I am wondering can the same logic apply to my version of foo function? I am asking this because the part I want to run multiprocessing on also contains some numpy functions call.
– mathguy
Mar 7 at 21:20
I edited my post. Did I understand right?? But it only makes sence with big operations so i have done it with really high values. Hope I could help (-;
– apatrck00
Mar 8 at 18:31
add a comment |
Maby this can help.
import multiprocessing
import numpy as np
import time
import multiprocessing
def data_processing(x, y, z): return np.array([x, y]) * (z ** 0.5)
def foo(n1, n2, id=0, return_dict=[None]):
final_result =
for i in range(n1):
result = np.zeros([n2, n2])
for j1 in range(n2):
for j2 in range(j1):
temp = data_processing(j1, j2, i)
result[j1, j2] = np.prod(temp)
final_result[str(i)] = result
return_dict[id] = final_result
stamp = time.time()
def pint(num):
print(f'*Test [num] - seconds: time.time() - stamp')
for i in range(10):
foo(90, 90)
pint(0)
stamp = time.time()
manager = multiprocessing.Manager()
return_dict = manager.dict()
processes = []
for i in range(10):
p = multiprocessing.Process(target=foo, args=(90, 90, i, return_dict))
processes.append(p)
p.start()
for p in processes:
p.join()
x0, x1, x2, x3, x4, x5, x6, x7, x8, x9 = return_dict.values()
pint(1)
My output is:
*Test [0] - seconds: 26.120166301727295
*Test [1] - seconds: 8.343111753463745
Process finished with exit code 0
appreciate any input to this. I am wondering can the same logic apply to my version of foo function? I am asking this because the part I want to run multiprocessing on also contains some numpy functions call.
– mathguy
Mar 7 at 21:20
I edited my post. Did I understand right?? But it only makes sence with big operations so i have done it with really high values. Hope I could help (-;
– apatrck00
Mar 8 at 18:31
add a comment |
Maby this can help.
import multiprocessing
import numpy as np
import time
import multiprocessing
def data_processing(x, y, z): return np.array([x, y]) * (z ** 0.5)
def foo(n1, n2, id=0, return_dict=[None]):
final_result =
for i in range(n1):
result = np.zeros([n2, n2])
for j1 in range(n2):
for j2 in range(j1):
temp = data_processing(j1, j2, i)
result[j1, j2] = np.prod(temp)
final_result[str(i)] = result
return_dict[id] = final_result
stamp = time.time()
def pint(num):
print(f'*Test [num] - seconds: time.time() - stamp')
for i in range(10):
foo(90, 90)
pint(0)
stamp = time.time()
manager = multiprocessing.Manager()
return_dict = manager.dict()
processes = []
for i in range(10):
p = multiprocessing.Process(target=foo, args=(90, 90, i, return_dict))
processes.append(p)
p.start()
for p in processes:
p.join()
x0, x1, x2, x3, x4, x5, x6, x7, x8, x9 = return_dict.values()
pint(1)
My output is:
*Test [0] - seconds: 26.120166301727295
*Test [1] - seconds: 8.343111753463745
Process finished with exit code 0
Maby this can help.
import multiprocessing
import numpy as np
import time
import multiprocessing
def data_processing(x, y, z): return np.array([x, y]) * (z ** 0.5)
def foo(n1, n2, id=0, return_dict=[None]):
final_result =
for i in range(n1):
result = np.zeros([n2, n2])
for j1 in range(n2):
for j2 in range(j1):
temp = data_processing(j1, j2, i)
result[j1, j2] = np.prod(temp)
final_result[str(i)] = result
return_dict[id] = final_result
stamp = time.time()
def pint(num):
print(f'*Test [num] - seconds: time.time() - stamp')
for i in range(10):
foo(90, 90)
pint(0)
stamp = time.time()
manager = multiprocessing.Manager()
return_dict = manager.dict()
processes = []
for i in range(10):
p = multiprocessing.Process(target=foo, args=(90, 90, i, return_dict))
processes.append(p)
p.start()
for p in processes:
p.join()
x0, x1, x2, x3, x4, x5, x6, x7, x8, x9 = return_dict.values()
pint(1)
My output is:
*Test [0] - seconds: 26.120166301727295
*Test [1] - seconds: 8.343111753463745
Process finished with exit code 0
edited Mar 8 at 18:30
answered Mar 7 at 20:55
apatrck00apatrck00
113
113
appreciate any input to this. I am wondering can the same logic apply to my version of foo function? I am asking this because the part I want to run multiprocessing on also contains some numpy functions call.
– mathguy
Mar 7 at 21:20
I edited my post. Did I understand right?? But it only makes sence with big operations so i have done it with really high values. Hope I could help (-;
– apatrck00
Mar 8 at 18:31
add a comment |
appreciate any input to this. I am wondering can the same logic apply to my version of foo function? I am asking this because the part I want to run multiprocessing on also contains some numpy functions call.
– mathguy
Mar 7 at 21:20
I edited my post. Did I understand right?? But it only makes sence with big operations so i have done it with really high values. Hope I could help (-;
– apatrck00
Mar 8 at 18:31
appreciate any input to this. I am wondering can the same logic apply to my version of foo function? I am asking this because the part I want to run multiprocessing on also contains some numpy functions call.
– mathguy
Mar 7 at 21:20
appreciate any input to this. I am wondering can the same logic apply to my version of foo function? I am asking this because the part I want to run multiprocessing on also contains some numpy functions call.
– mathguy
Mar 7 at 21:20
I edited my post. Did I understand right?? But it only makes sence with big operations so i have done it with really high values. Hope I could help (-;
– apatrck00
Mar 8 at 18:31
I edited my post. Did I understand right?? But it only makes sence with big operations so i have done it with really high values. Hope I could help (-;
– apatrck00
Mar 8 at 18:31
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55052363%2fmultiprocessing-with-nested-loops-and-some-numpy-function-calls%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown