SparkLauncher launching only one application even when submitting multiple applications2019 Community Moderator ElectionSample Hadoop Config for 4GB server?How to run this code on Spark Cluster modehow to execute multiple applications(jars) in spark simultaneously?spark out of memory multiple iterationsSpark 2 on YARN is utilizing more cluster resource automaticallyspark.master configuration via REST job submission in standalone cluster is ignoredWhere to specify Spark configs when running Spark app in EMR clusterDeploying application with spark-submit: Application is added to the scheduler and is not yet activatedSubmitting application on Spark Cluster using spark submitPyspark - DataFrame persist() errors out java.lang.OutOfMemoryError: GC overhead limit exceeded
Do Bugbears' arms literally get longer when it's their turn?
validation vs test vs training accuracy, which one to compare for claiming overfit?
Should we release the security issues we found in our product as CVE or we can just update those on weekly release notes?
Touchscreen-controlled dentist office snowman collector game
What is the blue range indicating on this manifold pressure gauge?
If Invisibility ends because the original caster casts a non-concentration spell, does Invisibility also end on other targets of the original casting?
How do anti-virus programs start at Windows boot?
Potentiometer like component
Plywood subfloor won't screw down in a trailer home
Draw arrow on sides of triangle
It's a yearly task, alright
How is the Swiss post e-voting system supposed to work, and how was it wrong?
How can I discourage/prevent PCs from using door choke-points?
Humans have energy, but not water. What happens?
If the Captain's screens are out, does he switch seats with the co-pilot?
Why doesn't the EU now just force the UK to choose between referendum and no-deal?
How does Dispel Magic work against Stoneskin?
Excess Zinc in garden soil
What injury would be of little consequence to a biped but terrible for a quadruped?
Running a subshell from the middle of the current command
What Happens when Passenger Refuses to Fly Boeing 737 Max?
When two POV characters meet
Need some help with my first LaTeX drawing…
How could a female member of a species produce eggs unto death?
SparkLauncher launching only one application even when submitting multiple applications
2019 Community Moderator ElectionSample Hadoop Config for 4GB server?How to run this code on Spark Cluster modehow to execute multiple applications(jars) in spark simultaneously?spark out of memory multiple iterationsSpark 2 on YARN is utilizing more cluster resource automaticallyspark.master configuration via REST job submission in standalone cluster is ignoredWhere to specify Spark configs when running Spark app in EMR clusterDeploying application with spark-submit: Application is added to the scheduler and is not yet activatedSubmitting application on Spark Cluster using spark submitPyspark - DataFrame persist() errors out java.lang.OutOfMemoryError: GC overhead limit exceeded
I am submitting/running multiple application through spark launcher in my java web app. it seems to submit only one app.Here is my code
Runnable run = new Runnable()
public void run()
try
SparkAppHandle sparkApp = new SparkLauncher()
.setAppResource("C:\BigData\spark\examples\jars\spark-examples_2.11-2.4.0.jar")
.setMainClass("org.apache.spark.examples.SparkPi")
.setMaster("spark://192.168.2.233:7077")
.setConf("spark.scheduler.mode", "FAIR")
.setVerbose(true)
.setConf("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.setConf("spark.sql.inMemoryColumnarStorage.batchSize", "10000")
.setConf("spark.sql.codegen","false")
.setConf("spark.submit.deployMode", "client")
.setConf("spark.executor.memory", "1g")
.setConf("spark.driver.memory", "1g")
.setConf("spark.cores.max", "1")
.setConf("spark.executor.cores", "1")
.setConf("spark.executor.instances", "1")
.setConf("spark.driver.host","192.168.2.233")
// .setConf("spark.dynamicAllocation.enabled", "true")
// .setConf("spark.shuffle.service.enabled", "true")
.startApplication();
System.out.println(sparkApp.getState());
catch (IOException e)
// TODO Auto-generated catch block
e.printStackTrace();
;
//running two times so as to submit two parallel application
//in the application logic , we would pass different args to different app
new Thread(run).start();
new Thread(run).start();`
I have a standalone cluster with one worker node1(8gb,4cores) and another worker node2(8gb, 2 cores). Master is running on node1 and driver is also node1 only.
It seems that even though second thread launch application, nothing happens to it and the second application doesn't even appear in the WAITING state which would have been understandable.
java apache-spark apache-spark-2.0 spark-launcher
add a comment |
I am submitting/running multiple application through spark launcher in my java web app. it seems to submit only one app.Here is my code
Runnable run = new Runnable()
public void run()
try
SparkAppHandle sparkApp = new SparkLauncher()
.setAppResource("C:\BigData\spark\examples\jars\spark-examples_2.11-2.4.0.jar")
.setMainClass("org.apache.spark.examples.SparkPi")
.setMaster("spark://192.168.2.233:7077")
.setConf("spark.scheduler.mode", "FAIR")
.setVerbose(true)
.setConf("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.setConf("spark.sql.inMemoryColumnarStorage.batchSize", "10000")
.setConf("spark.sql.codegen","false")
.setConf("spark.submit.deployMode", "client")
.setConf("spark.executor.memory", "1g")
.setConf("spark.driver.memory", "1g")
.setConf("spark.cores.max", "1")
.setConf("spark.executor.cores", "1")
.setConf("spark.executor.instances", "1")
.setConf("spark.driver.host","192.168.2.233")
// .setConf("spark.dynamicAllocation.enabled", "true")
// .setConf("spark.shuffle.service.enabled", "true")
.startApplication();
System.out.println(sparkApp.getState());
catch (IOException e)
// TODO Auto-generated catch block
e.printStackTrace();
;
//running two times so as to submit two parallel application
//in the application logic , we would pass different args to different app
new Thread(run).start();
new Thread(run).start();`
I have a standalone cluster with one worker node1(8gb,4cores) and another worker node2(8gb, 2 cores). Master is running on node1 and driver is also node1 only.
It seems that even though second thread launch application, nothing happens to it and the second application doesn't even appear in the WAITING state which would have been understandable.
java apache-spark apache-spark-2.0 spark-launcher
add a comment |
I am submitting/running multiple application through spark launcher in my java web app. it seems to submit only one app.Here is my code
Runnable run = new Runnable()
public void run()
try
SparkAppHandle sparkApp = new SparkLauncher()
.setAppResource("C:\BigData\spark\examples\jars\spark-examples_2.11-2.4.0.jar")
.setMainClass("org.apache.spark.examples.SparkPi")
.setMaster("spark://192.168.2.233:7077")
.setConf("spark.scheduler.mode", "FAIR")
.setVerbose(true)
.setConf("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.setConf("spark.sql.inMemoryColumnarStorage.batchSize", "10000")
.setConf("spark.sql.codegen","false")
.setConf("spark.submit.deployMode", "client")
.setConf("spark.executor.memory", "1g")
.setConf("spark.driver.memory", "1g")
.setConf("spark.cores.max", "1")
.setConf("spark.executor.cores", "1")
.setConf("spark.executor.instances", "1")
.setConf("spark.driver.host","192.168.2.233")
// .setConf("spark.dynamicAllocation.enabled", "true")
// .setConf("spark.shuffle.service.enabled", "true")
.startApplication();
System.out.println(sparkApp.getState());
catch (IOException e)
// TODO Auto-generated catch block
e.printStackTrace();
;
//running two times so as to submit two parallel application
//in the application logic , we would pass different args to different app
new Thread(run).start();
new Thread(run).start();`
I have a standalone cluster with one worker node1(8gb,4cores) and another worker node2(8gb, 2 cores). Master is running on node1 and driver is also node1 only.
It seems that even though second thread launch application, nothing happens to it and the second application doesn't even appear in the WAITING state which would have been understandable.
java apache-spark apache-spark-2.0 spark-launcher
I am submitting/running multiple application through spark launcher in my java web app. it seems to submit only one app.Here is my code
Runnable run = new Runnable()
public void run()
try
SparkAppHandle sparkApp = new SparkLauncher()
.setAppResource("C:\BigData\spark\examples\jars\spark-examples_2.11-2.4.0.jar")
.setMainClass("org.apache.spark.examples.SparkPi")
.setMaster("spark://192.168.2.233:7077")
.setConf("spark.scheduler.mode", "FAIR")
.setVerbose(true)
.setConf("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.setConf("spark.sql.inMemoryColumnarStorage.batchSize", "10000")
.setConf("spark.sql.codegen","false")
.setConf("spark.submit.deployMode", "client")
.setConf("spark.executor.memory", "1g")
.setConf("spark.driver.memory", "1g")
.setConf("spark.cores.max", "1")
.setConf("spark.executor.cores", "1")
.setConf("spark.executor.instances", "1")
.setConf("spark.driver.host","192.168.2.233")
// .setConf("spark.dynamicAllocation.enabled", "true")
// .setConf("spark.shuffle.service.enabled", "true")
.startApplication();
System.out.println(sparkApp.getState());
catch (IOException e)
// TODO Auto-generated catch block
e.printStackTrace();
;
//running two times so as to submit two parallel application
//in the application logic , we would pass different args to different app
new Thread(run).start();
new Thread(run).start();`
I have a standalone cluster with one worker node1(8gb,4cores) and another worker node2(8gb, 2 cores). Master is running on node1 and driver is also node1 only.
It seems that even though second thread launch application, nothing happens to it and the second application doesn't even appear in the WAITING state which would have been understandable.
java apache-spark apache-spark-2.0 spark-launcher
java apache-spark apache-spark-2.0 spark-launcher
edited Mar 7 at 11:43
AngryLeo
asked Mar 7 at 10:46
AngryLeoAngryLeo
14411
14411
add a comment |
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55041939%2fsparklauncher-launching-only-one-application-even-when-submitting-multiple-appli%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55041939%2fsparklauncher-launching-only-one-application-even-when-submitting-multiple-appli%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown