An attempt has been made to start a new process before the current process has finished its bootstrapping phasewhere to put freeze_support() in a Python script?Python: Start new command prompt on Windows and wait for it finish/exitStart a new thread in a specific processParamiko finish process before reading all outputPython: Wait for process to finish before proceeding in loop?Redirect log to file before process finishedPoor performance with DaskProcess for one user starting a form and another finishing itPython multiprocessing - Allocate a new function to a finished process?Start a new process and Killing the current process in pythonException kills all workers and scheduler on dask SLURM cluster
Bob has never been a M before
Why did the EU agree to delay the Brexit deadline?
How will losing mobility of one hand affect my career as a programmer?
Could the E-bike drivetrain wear down till needing replacement after 400 km?
How can Trident be so inexpensive? Will it orbit Triton or just do a (slow) flyby?
Why has "pence" been used in this sentence, not "pences"?
Can I use my Chinese passport to enter China after I acquired another citizenship?
How do I extrude a face to a single vertex
Greco-Roman egalitarianism
Why is Arduino resetting while driving motors?
Diode in opposite direction?
Is it possible to use .desktop files to open local pdf files on specific pages with a browser?
Is it improper etiquette to ask your opponent what his/her rating is before the game?
Why did the HMS Bounty go back to a time when whales are already rare?
What's the difference between 違法 and 不法?
Find last 3 digits of this monster number
Varistor? Purpose and principle
When quoting, must I also copy hyphens used to divide words that continue on the next line?
How to align and center standalone amsmath equations?
How much character growth crosses the line into breaking the character
Can the Supreme Court overturn an impeachment?
What linear sensor for a keyboard?
Longest common substring in linear time
How do you respond to a colleague from another team when they're wrongly expecting that you'll help them?
An attempt has been made to start a new process before the current process has finished its bootstrapping phase
where to put freeze_support() in a Python script?Python: Start new command prompt on Windows and wait for it finish/exitStart a new thread in a specific processParamiko finish process before reading all outputPython: Wait for process to finish before proceeding in loop?Redirect log to file before process finishedPoor performance with DaskProcess for one user starting a form and another finishing itPython multiprocessing - Allocate a new function to a finished process?Start a new process and Killing the current process in pythonException kills all workers and scheduler on dask SLURM cluster
I am new to dask and I found so nice to have a module that makes it easy to get parallelization. I am working on a project where I was able to parallelize in a single machine a loop as you can see here . However, I would like to move over to dask.distributed
. I applied the following changes to the class above:
diff --git a/mlchem/fingerprints/gaussian.py b/mlchem/fingerprints/gaussian.py
index ce6a72b..89f8638 100644
--- a/mlchem/fingerprints/gaussian.py
+++ b/mlchem/fingerprints/gaussian.py
@@ -6,7 +6,7 @@ from sklearn.externals import joblib
from .cutoff import Cosine
from collections import OrderedDict
import dask
-import dask.multiprocessing
+from dask.distributed import Client
import time
@@ -141,13 +141,14 @@ class Gaussian(object):
for image in images.items():
computations.append(self.fingerprints_per_image(image))
+ client = Client()
if self.scaler is None:
- feature_space = dask.compute(*computations, scheduler='processes',
+ feature_space = dask.compute(*computations, scheduler='distributed',
num_workers=self.cores)
feature_space = OrderedDict(feature_space)
else:
stacked_features = dask.compute(*computations,
- scheduler='processes',
+ scheduler='distributed',
num_workers=self.cores)
stacked_features = numpy.array(stacked_features)
Doing so generates this error:
File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/spawn.py", line 136, in _check_not_importing_main
is not going to be frozen to produce an executable.''')
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
I have tried different ways of adding if __name__ == '__main__':
without any success. This can be reproduced by running this example. I would appreciate if anyone could help me to figure this out. I have no clue on how I should change my code to make it work.
Thanks.
Edit: The example is cu_training.py
.
python dask dask-distributed
add a comment |
I am new to dask and I found so nice to have a module that makes it easy to get parallelization. I am working on a project where I was able to parallelize in a single machine a loop as you can see here . However, I would like to move over to dask.distributed
. I applied the following changes to the class above:
diff --git a/mlchem/fingerprints/gaussian.py b/mlchem/fingerprints/gaussian.py
index ce6a72b..89f8638 100644
--- a/mlchem/fingerprints/gaussian.py
+++ b/mlchem/fingerprints/gaussian.py
@@ -6,7 +6,7 @@ from sklearn.externals import joblib
from .cutoff import Cosine
from collections import OrderedDict
import dask
-import dask.multiprocessing
+from dask.distributed import Client
import time
@@ -141,13 +141,14 @@ class Gaussian(object):
for image in images.items():
computations.append(self.fingerprints_per_image(image))
+ client = Client()
if self.scaler is None:
- feature_space = dask.compute(*computations, scheduler='processes',
+ feature_space = dask.compute(*computations, scheduler='distributed',
num_workers=self.cores)
feature_space = OrderedDict(feature_space)
else:
stacked_features = dask.compute(*computations,
- scheduler='processes',
+ scheduler='distributed',
num_workers=self.cores)
stacked_features = numpy.array(stacked_features)
Doing so generates this error:
File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/spawn.py", line 136, in _check_not_importing_main
is not going to be frozen to produce an executable.''')
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
I have tried different ways of adding if __name__ == '__main__':
without any success. This can be reproduced by running this example. I would appreciate if anyone could help me to figure this out. I have no clue on how I should change my code to make it work.
Thanks.
Edit: The example is cu_training.py
.
python dask dask-distributed
add a comment |
I am new to dask and I found so nice to have a module that makes it easy to get parallelization. I am working on a project where I was able to parallelize in a single machine a loop as you can see here . However, I would like to move over to dask.distributed
. I applied the following changes to the class above:
diff --git a/mlchem/fingerprints/gaussian.py b/mlchem/fingerprints/gaussian.py
index ce6a72b..89f8638 100644
--- a/mlchem/fingerprints/gaussian.py
+++ b/mlchem/fingerprints/gaussian.py
@@ -6,7 +6,7 @@ from sklearn.externals import joblib
from .cutoff import Cosine
from collections import OrderedDict
import dask
-import dask.multiprocessing
+from dask.distributed import Client
import time
@@ -141,13 +141,14 @@ class Gaussian(object):
for image in images.items():
computations.append(self.fingerprints_per_image(image))
+ client = Client()
if self.scaler is None:
- feature_space = dask.compute(*computations, scheduler='processes',
+ feature_space = dask.compute(*computations, scheduler='distributed',
num_workers=self.cores)
feature_space = OrderedDict(feature_space)
else:
stacked_features = dask.compute(*computations,
- scheduler='processes',
+ scheduler='distributed',
num_workers=self.cores)
stacked_features = numpy.array(stacked_features)
Doing so generates this error:
File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/spawn.py", line 136, in _check_not_importing_main
is not going to be frozen to produce an executable.''')
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
I have tried different ways of adding if __name__ == '__main__':
without any success. This can be reproduced by running this example. I would appreciate if anyone could help me to figure this out. I have no clue on how I should change my code to make it work.
Thanks.
Edit: The example is cu_training.py
.
python dask dask-distributed
I am new to dask and I found so nice to have a module that makes it easy to get parallelization. I am working on a project where I was able to parallelize in a single machine a loop as you can see here . However, I would like to move over to dask.distributed
. I applied the following changes to the class above:
diff --git a/mlchem/fingerprints/gaussian.py b/mlchem/fingerprints/gaussian.py
index ce6a72b..89f8638 100644
--- a/mlchem/fingerprints/gaussian.py
+++ b/mlchem/fingerprints/gaussian.py
@@ -6,7 +6,7 @@ from sklearn.externals import joblib
from .cutoff import Cosine
from collections import OrderedDict
import dask
-import dask.multiprocessing
+from dask.distributed import Client
import time
@@ -141,13 +141,14 @@ class Gaussian(object):
for image in images.items():
computations.append(self.fingerprints_per_image(image))
+ client = Client()
if self.scaler is None:
- feature_space = dask.compute(*computations, scheduler='processes',
+ feature_space = dask.compute(*computations, scheduler='distributed',
num_workers=self.cores)
feature_space = OrderedDict(feature_space)
else:
stacked_features = dask.compute(*computations,
- scheduler='processes',
+ scheduler='distributed',
num_workers=self.cores)
stacked_features = numpy.array(stacked_features)
Doing so generates this error:
File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/spawn.py", line 136, in _check_not_importing_main
is not going to be frozen to produce an executable.''')
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
I have tried different ways of adding if __name__ == '__main__':
without any success. This can be reproduced by running this example. I would appreciate if anyone could help me to figure this out. I have no clue on how I should change my code to make it work.
Thanks.
Edit: The example is cu_training.py
.
python dask dask-distributed
python dask dask-distributed
edited Mar 8 at 8:07
muammar
asked Mar 8 at 6:38
muammarmuammar
1681317
1681317
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
The Client
command starts up new processes, so it will have to be within the if __name__ == '__main__':
block as described in this SO question or this GitHub issue
This is the same as with the multiprocessing module
Thanks, @MRocklin. I had read the links you have sent in your answer. However, I have not found a way yet to change my code to make this work.
– muammar
Mar 10 at 3:04
I finally understood you, @MRocklin. I fixed it here github.com/muammar/mlchem/commit/… I will try to refactor my code because I don't like very much that I have to do a function for running the calculations, but maybe this is just the intended way on using distributed. Not sure yet. Great tool by the way.
– muammar
Mar 10 at 5:52
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55057957%2fan-attempt-has-been-made-to-start-a-new-process-before-the-current-process-has-f%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
The Client
command starts up new processes, so it will have to be within the if __name__ == '__main__':
block as described in this SO question or this GitHub issue
This is the same as with the multiprocessing module
Thanks, @MRocklin. I had read the links you have sent in your answer. However, I have not found a way yet to change my code to make this work.
– muammar
Mar 10 at 3:04
I finally understood you, @MRocklin. I fixed it here github.com/muammar/mlchem/commit/… I will try to refactor my code because I don't like very much that I have to do a function for running the calculations, but maybe this is just the intended way on using distributed. Not sure yet. Great tool by the way.
– muammar
Mar 10 at 5:52
add a comment |
The Client
command starts up new processes, so it will have to be within the if __name__ == '__main__':
block as described in this SO question or this GitHub issue
This is the same as with the multiprocessing module
Thanks, @MRocklin. I had read the links you have sent in your answer. However, I have not found a way yet to change my code to make this work.
– muammar
Mar 10 at 3:04
I finally understood you, @MRocklin. I fixed it here github.com/muammar/mlchem/commit/… I will try to refactor my code because I don't like very much that I have to do a function for running the calculations, but maybe this is just the intended way on using distributed. Not sure yet. Great tool by the way.
– muammar
Mar 10 at 5:52
add a comment |
The Client
command starts up new processes, so it will have to be within the if __name__ == '__main__':
block as described in this SO question or this GitHub issue
This is the same as with the multiprocessing module
The Client
command starts up new processes, so it will have to be within the if __name__ == '__main__':
block as described in this SO question or this GitHub issue
This is the same as with the multiprocessing module
answered Mar 9 at 23:54
MRocklinMRocklin
27k1471130
27k1471130
Thanks, @MRocklin. I had read the links you have sent in your answer. However, I have not found a way yet to change my code to make this work.
– muammar
Mar 10 at 3:04
I finally understood you, @MRocklin. I fixed it here github.com/muammar/mlchem/commit/… I will try to refactor my code because I don't like very much that I have to do a function for running the calculations, but maybe this is just the intended way on using distributed. Not sure yet. Great tool by the way.
– muammar
Mar 10 at 5:52
add a comment |
Thanks, @MRocklin. I had read the links you have sent in your answer. However, I have not found a way yet to change my code to make this work.
– muammar
Mar 10 at 3:04
I finally understood you, @MRocklin. I fixed it here github.com/muammar/mlchem/commit/… I will try to refactor my code because I don't like very much that I have to do a function for running the calculations, but maybe this is just the intended way on using distributed. Not sure yet. Great tool by the way.
– muammar
Mar 10 at 5:52
Thanks, @MRocklin. I had read the links you have sent in your answer. However, I have not found a way yet to change my code to make this work.
– muammar
Mar 10 at 3:04
Thanks, @MRocklin. I had read the links you have sent in your answer. However, I have not found a way yet to change my code to make this work.
– muammar
Mar 10 at 3:04
I finally understood you, @MRocklin. I fixed it here github.com/muammar/mlchem/commit/… I will try to refactor my code because I don't like very much that I have to do a function for running the calculations, but maybe this is just the intended way on using distributed. Not sure yet. Great tool by the way.
– muammar
Mar 10 at 5:52
I finally understood you, @MRocklin. I fixed it here github.com/muammar/mlchem/commit/… I will try to refactor my code because I don't like very much that I have to do a function for running the calculations, but maybe this is just the intended way on using distributed. Not sure yet. Great tool by the way.
– muammar
Mar 10 at 5:52
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55057957%2fan-attempt-has-been-made-to-start-a-new-process-before-the-current-process-has-f%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown