Ggtcf

Output visual diagram of picture

Why didn’t Eve recognize the little cockroach as a living organism?

Derivative of an interpolated function

Capacitor electron flow

Do native speakers use "ultima" and "proxima" frequently in spoken English?

Why doesn't Gödel's incompleteness theorem apply to false statements?

Why do Radio Buttons not fill the entire outer circle?

Did I make a mistake by ccing email to boss to others?

Why is indicated airspeed rather than ground speed used during the takeoff roll?

Showing mass murder in a kid's book

Can a Knock spell open the door to Mordenkainen's Magnificent Mansion?

Center page as a whole without centering each element individually

Turning a hard to access nut?

Non-Borel set in arbitrary metric space

Would a primitive species be able to learn English from reading books alone?

Asserting that Atheism and Theism are both faith based positions

How to split IPA spelling into syllables

How do you justify more code being written by following clean code practices?

Mortal danger in mid-grade literature

What is the tangent at a sharp point on a curve?

Should I be concerned about student access to a test bank?

Trouble reading roman numeral notation with flats

What is the purpose of using a decision tree?

What should be the ideal length of sentences in a blog post for ease of reading?

Multiprocessing with nested loops and some numpy function calls

Calling a function of a module by using its name (a string)What's the best way to break from nested (for) loops?Is there a NumPy function to return the first index of something in an array?Breaking out of nested loopsDynamic processes in PythonHow do I break out of nested loops in Java?Calling remove in foreach loop in Javapython multiprocessing vs threading for cpu bound work on windows and linuxUse numpy array in shared memory for multiprocessingWhat is the difference between flatten and ravel functions in numpy?

1

I have read some coding examples about multiprocessing and am stil quite confused about it. Here is my contrived example:

import numpy as np

def data_processing(x,y,z): return np.array([x,y])*(z**0.5)

def foo(n1,n2):

 final_result = 

 for i in range(n1):
 result = np.zeros([n2,n2])

 for j1 in range(n2):
 for j2 in range(j1):
 temp= data_processing(j1,j2,i)
 result[j1,j2] = np.prod(temp)

 final_result[str(i)] = result

 return final_result

if __name__ == '__main__':
 X = foo(9,9)

If I want to run this piece of code while utilizing all of the cpu cores, what should I change? Thank you in advance

python loops numpy multiprocessing nested-loops

share|improve this question

asked Mar 7 at 20:33

mathguymathguy

1197

add a comment | 

1

I have read some coding examples about multiprocessing and am stil quite confused about it. Here is my contrived example:

import numpy as np

def data_processing(x,y,z): return np.array([x,y])*(z**0.5)

def foo(n1,n2):

 final_result = 

 for i in range(n1):
 result = np.zeros([n2,n2])

 for j1 in range(n2):
 for j2 in range(j1):
 temp= data_processing(j1,j2,i)
 result[j1,j2] = np.prod(temp)

 final_result[str(i)] = result

 return final_result

if __name__ == '__main__':
 X = foo(9,9)

If I want to run this piece of code while utilizing all of the cpu cores, what should I change? Thank you in advance

python loops numpy multiprocessing nested-loops

share|improve this question

asked Mar 7 at 20:33

mathguymathguy

1197

add a comment | 

I have read some coding examples about multiprocessing and am stil quite confused about it. Here is my contrived example:

import numpy as np

def data_processing(x,y,z): return np.array([x,y])*(z**0.5)

def foo(n1,n2):

 final_result = 

 for i in range(n1):
 result = np.zeros([n2,n2])

 for j1 in range(n2):
 for j2 in range(j1):
 temp= data_processing(j1,j2,i)
 result[j1,j2] = np.prod(temp)

 final_result[str(i)] = result

 return final_result

if __name__ == '__main__':
 X = foo(9,9)

If I want to run this piece of code while utilizing all of the cpu cores, what should I change? Thank you in advance

python loops numpy multiprocessing nested-loops

share|improve this question

asked Mar 7 at 20:33

mathguymathguy

1197

import numpy as np

def data_processing(x,y,z): return np.array([x,y])*(z**0.5)

def foo(n1,n2):

 final_result = 

 for i in range(n1):
 result = np.zeros([n2,n2])

 for j1 in range(n2):
 for j2 in range(j1):
 temp= data_processing(j1,j2,i)
 result[j1,j2] = np.prod(temp)

 final_result[str(i)] = result

 return final_result

if __name__ == '__main__':
 X = foo(9,9)

share|improve this question

asked Mar 7 at 20:33

mathguymathguy

1197

share|improve this question

asked Mar 7 at 20:33

mathguymathguy

1197

asked Mar 7 at 20:33

mathguymathguy

1197

asked Mar 7 at 20:33

mathguymathguy

1197

1 Answer
1

active

oldest

votes

1

Maby this can help.

import multiprocessing
import numpy as np
import time
import multiprocessing


def data_processing(x, y, z): return np.array([x, y]) * (z ** 0.5)


def foo(n1, n2, id=0, return_dict=[None]):
 final_result = 

 for i in range(n1):
 result = np.zeros([n2, n2])

 for j1 in range(n2):
 for j2 in range(j1):
 temp = data_processing(j1, j2, i)
 result[j1, j2] = np.prod(temp)

 final_result[str(i)] = result

 return_dict[id] = final_result



stamp = time.time()
def pint(num):
 print(f'*Test [num] - seconds: time.time() - stamp')

for i in range(10):
 foo(90, 90)
pint(0)

stamp = time.time()

manager = multiprocessing.Manager()
return_dict = manager.dict()
processes = []

for i in range(10):
 p = multiprocessing.Process(target=foo, args=(90, 90, i, return_dict))
 processes.append(p)
 p.start()

for p in processes:
 p.join()
x0, x1, x2, x3, x4, x5, x6, x7, x8, x9 = return_dict.values()
pint(1)

My output is:

*Test [0] - seconds: 26.120166301727295
*Test [1] - seconds: 8.343111753463745

Process finished with exit code 0

share|improve this answer

edited Mar 8 at 18:30

answered Mar 7 at 20:55

apatrck00apatrck00

113

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  appreciate any input to this. I am wondering can the same logic apply to my version of foo function? I am asking this because the part I want to run multiprocessing on also contains some numpy functions call.
  
  – mathguy
  Mar 7 at 21:20
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  I edited my post. Did I understand right?? But it only makes sence with big operations so i have done it with really high values. Hope I could help (-;
  
  – apatrck00
  Mar 8 at 18:31
  

  
  
  
  

add a comment | 

Your Answer

StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55052363%2fmultiprocessing-with-nested-loops-and-some-numpy-function-calls%23new-answer', 'question_page');

);

Post as a guest

Name

Email

Required, but never shown

1

Maby this can help.

import multiprocessing
import numpy as np
import time
import multiprocessing


def data_processing(x, y, z): return np.array([x, y]) * (z ** 0.5)


def foo(n1, n2, id=0, return_dict=[None]):
 final_result = 

 for i in range(n1):
 result = np.zeros([n2, n2])

 for j1 in range(n2):
 for j2 in range(j1):
 temp = data_processing(j1, j2, i)
 result[j1, j2] = np.prod(temp)

 final_result[str(i)] = result

 return_dict[id] = final_result



stamp = time.time()
def pint(num):
 print(f'*Test [num] - seconds: time.time() - stamp')

for i in range(10):
 foo(90, 90)
pint(0)

stamp = time.time()

manager = multiprocessing.Manager()
return_dict = manager.dict()
processes = []

for i in range(10):
 p = multiprocessing.Process(target=foo, args=(90, 90, i, return_dict))
 processes.append(p)
 p.start()

for p in processes:
 p.join()
x0, x1, x2, x3, x4, x5, x6, x7, x8, x9 = return_dict.values()
pint(1)

My output is:

*Test [0] - seconds: 26.120166301727295
*Test [1] - seconds: 8.343111753463745

Process finished with exit code 0

share|improve this answer

edited Mar 8 at 18:30

answered Mar 7 at 20:55

apatrck00apatrck00

113

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  appreciate any input to this. I am wondering can the same logic apply to my version of foo function? I am asking this because the part I want to run multiprocessing on also contains some numpy functions call.
  
  – mathguy
  Mar 7 at 21:20
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  I edited my post. Did I understand right?? But it only makes sence with big operations so i have done it with really high values. Hope I could help (-;
  
  – apatrck00
  Mar 8 at 18:31
  

  
  
  
  

add a comment | 

1

Maby this can help.

import multiprocessing
import numpy as np
import time
import multiprocessing


def data_processing(x, y, z): return np.array([x, y]) * (z ** 0.5)


def foo(n1, n2, id=0, return_dict=[None]):
 final_result = 

 for i in range(n1):
 result = np.zeros([n2, n2])

 for j1 in range(n2):
 for j2 in range(j1):
 temp = data_processing(j1, j2, i)
 result[j1, j2] = np.prod(temp)

 final_result[str(i)] = result

 return_dict[id] = final_result



stamp = time.time()
def pint(num):
 print(f'*Test [num] - seconds: time.time() - stamp')

for i in range(10):
 foo(90, 90)
pint(0)

stamp = time.time()

manager = multiprocessing.Manager()
return_dict = manager.dict()
processes = []

for i in range(10):
 p = multiprocessing.Process(target=foo, args=(90, 90, i, return_dict))
 processes.append(p)
 p.start()

for p in processes:
 p.join()
x0, x1, x2, x3, x4, x5, x6, x7, x8, x9 = return_dict.values()
pint(1)

My output is:

*Test [0] - seconds: 26.120166301727295
*Test [1] - seconds: 8.343111753463745

Process finished with exit code 0

share|improve this answer

edited Mar 8 at 18:30

answered Mar 7 at 20:55

apatrck00apatrck00

113

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  appreciate any input to this. I am wondering can the same logic apply to my version of foo function? I am asking this because the part I want to run multiprocessing on also contains some numpy functions call.
  
  – mathguy
  Mar 7 at 21:20
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  I edited my post. Did I understand right?? But it only makes sence with big operations so i have done it with really high values. Hope I could help (-;
  
  – apatrck00
  Mar 8 at 18:31
  

  
  
  
  

add a comment | 

Maby this can help.

import multiprocessing
import numpy as np
import time
import multiprocessing


def data_processing(x, y, z): return np.array([x, y]) * (z ** 0.5)


def foo(n1, n2, id=0, return_dict=[None]):
 final_result = 

 for i in range(n1):
 result = np.zeros([n2, n2])

 for j1 in range(n2):
 for j2 in range(j1):
 temp = data_processing(j1, j2, i)
 result[j1, j2] = np.prod(temp)

 final_result[str(i)] = result

 return_dict[id] = final_result



stamp = time.time()
def pint(num):
 print(f'*Test [num] - seconds: time.time() - stamp')

for i in range(10):
 foo(90, 90)
pint(0)

stamp = time.time()

manager = multiprocessing.Manager()
return_dict = manager.dict()
processes = []

for i in range(10):
 p = multiprocessing.Process(target=foo, args=(90, 90, i, return_dict))
 processes.append(p)
 p.start()

for p in processes:
 p.join()
x0, x1, x2, x3, x4, x5, x6, x7, x8, x9 = return_dict.values()
pint(1)

My output is:

*Test [0] - seconds: 26.120166301727295
*Test [1] - seconds: 8.343111753463745

Process finished with exit code 0

share|improve this answer

edited Mar 8 at 18:30

answered Mar 7 at 20:55

apatrck00apatrck00

113

import multiprocessing
import numpy as np
import time
import multiprocessing


def data_processing(x, y, z): return np.array([x, y]) * (z ** 0.5)


def foo(n1, n2, id=0, return_dict=[None]):
 final_result = 

 for i in range(n1):
 result = np.zeros([n2, n2])

 for j1 in range(n2):
 for j2 in range(j1):
 temp = data_processing(j1, j2, i)
 result[j1, j2] = np.prod(temp)

 final_result[str(i)] = result

 return_dict[id] = final_result



stamp = time.time()
def pint(num):
 print(f'*Test [num] - seconds: time.time() - stamp')

for i in range(10):
 foo(90, 90)
pint(0)

stamp = time.time()

manager = multiprocessing.Manager()
return_dict = manager.dict()
processes = []

for i in range(10):
 p = multiprocessing.Process(target=foo, args=(90, 90, i, return_dict))
 processes.append(p)
 p.start()

for p in processes:
 p.join()
x0, x1, x2, x3, x4, x5, x6, x7, x8, x9 = return_dict.values()
pint(1)

*Test [0] - seconds: 26.120166301727295
*Test [1] - seconds: 8.343111753463745

Process finished with exit code 0

share|improve this answer

edited Mar 8 at 18:30

answered Mar 7 at 20:55

apatrck00apatrck00

113

answered Mar 7 at 20:55

apatrck00apatrck00

113

answered Mar 7 at 20:55

apatrck00apatrck00

113