SparkLauncher launching only one application even when submitting multiple applications2019 Community Moderator ElectionSample Hadoop Config for 4GB server?How to run this code on Spark Cluster modehow to execute multiple applications(jars) in spark simultaneously?spark out of memory multiple iterationsSpark 2 on YARN is utilizing more cluster resource automaticallyspark.master configuration via REST job submission in standalone cluster is ignoredWhere to specify Spark configs when running Spark app in EMR clusterDeploying application with spark-submit: Application is added to the scheduler and is not yet activatedSubmitting application on Spark Cluster using spark submitPyspark - DataFrame persist() errors out java.lang.OutOfMemoryError: GC overhead limit exceeded

Do Bugbears' arms literally get longer when it's their turn?

validation vs test vs training accuracy, which one to compare for claiming overfit?

Should we release the security issues we found in our product as CVE or we can just update those on weekly release notes?

Touchscreen-controlled dentist office snowman collector game

What is the blue range indicating on this manifold pressure gauge?

If Invisibility ends because the original caster casts a non-concentration spell, does Invisibility also end on other targets of the original casting?

How do anti-virus programs start at Windows boot?

Potentiometer like component

Plywood subfloor won't screw down in a trailer home

Draw arrow on sides of triangle

It's a yearly task, alright

How is the Swiss post e-voting system supposed to work, and how was it wrong?

How can I discourage/prevent PCs from using door choke-points?

Humans have energy, but not water. What happens?

If the Captain's screens are out, does he switch seats with the co-pilot?

Why doesn't the EU now just force the UK to choose between referendum and no-deal?

How does Dispel Magic work against Stoneskin?

Excess Zinc in garden soil

What injury would be of little consequence to a biped but terrible for a quadruped?

Running a subshell from the middle of the current command

What Happens when Passenger Refuses to Fly Boeing 737 Max?

When two POV characters meet

Need some help with my first LaTeX drawing…

How could a female member of a species produce eggs unto death?



SparkLauncher launching only one application even when submitting multiple applications



2019 Community Moderator ElectionSample Hadoop Config for 4GB server?How to run this code on Spark Cluster modehow to execute multiple applications(jars) in spark simultaneously?spark out of memory multiple iterationsSpark 2 on YARN is utilizing more cluster resource automaticallyspark.master configuration via REST job submission in standalone cluster is ignoredWhere to specify Spark configs when running Spark app in EMR clusterDeploying application with spark-submit: Application is added to the scheduler and is not yet activatedSubmitting application on Spark Cluster using spark submitPyspark - DataFrame persist() errors out java.lang.OutOfMemoryError: GC overhead limit exceeded










1















I am submitting/running multiple application through spark launcher in my java web app. it seems to submit only one app.Here is my code



Runnable run = new Runnable() 
public void run()
try
SparkAppHandle sparkApp = new SparkLauncher()
.setAppResource("C:\BigData\spark\examples\jars\spark-examples_2.11-2.4.0.jar")
.setMainClass("org.apache.spark.examples.SparkPi")
.setMaster("spark://192.168.2.233:7077")
.setConf("spark.scheduler.mode", "FAIR")
.setVerbose(true)
.setConf("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.setConf("spark.sql.inMemoryColumnarStorage.batchSize", "10000")
.setConf("spark.sql.codegen","false")
.setConf("spark.submit.deployMode", "client")
.setConf("spark.executor.memory", "1g")
.setConf("spark.driver.memory", "1g")
.setConf("spark.cores.max", "1")
.setConf("spark.executor.cores", "1")
.setConf("spark.executor.instances", "1")
.setConf("spark.driver.host","192.168.2.233")
// .setConf("spark.dynamicAllocation.enabled", "true")
// .setConf("spark.shuffle.service.enabled", "true")
.startApplication();


System.out.println(sparkApp.getState());

catch (IOException e)
// TODO Auto-generated catch block
e.printStackTrace();


;

//running two times so as to submit two parallel application
//in the application logic , we would pass different args to different app
new Thread(run).start();
new Thread(run).start();`


I have a standalone cluster with one worker node1(8gb,4cores) and another worker node2(8gb, 2 cores). Master is running on node1 and driver is also node1 only.



It seems that even though second thread launch application, nothing happens to it and the second application doesn't even appear in the WAITING state which would have been understandable.










share|improve this question




























    1















    I am submitting/running multiple application through spark launcher in my java web app. it seems to submit only one app.Here is my code



    Runnable run = new Runnable() 
    public void run()
    try
    SparkAppHandle sparkApp = new SparkLauncher()
    .setAppResource("C:\BigData\spark\examples\jars\spark-examples_2.11-2.4.0.jar")
    .setMainClass("org.apache.spark.examples.SparkPi")
    .setMaster("spark://192.168.2.233:7077")
    .setConf("spark.scheduler.mode", "FAIR")
    .setVerbose(true)
    .setConf("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
    .setConf("spark.sql.inMemoryColumnarStorage.batchSize", "10000")
    .setConf("spark.sql.codegen","false")
    .setConf("spark.submit.deployMode", "client")
    .setConf("spark.executor.memory", "1g")
    .setConf("spark.driver.memory", "1g")
    .setConf("spark.cores.max", "1")
    .setConf("spark.executor.cores", "1")
    .setConf("spark.executor.instances", "1")
    .setConf("spark.driver.host","192.168.2.233")
    // .setConf("spark.dynamicAllocation.enabled", "true")
    // .setConf("spark.shuffle.service.enabled", "true")
    .startApplication();


    System.out.println(sparkApp.getState());

    catch (IOException e)
    // TODO Auto-generated catch block
    e.printStackTrace();


    ;

    //running two times so as to submit two parallel application
    //in the application logic , we would pass different args to different app
    new Thread(run).start();
    new Thread(run).start();`


    I have a standalone cluster with one worker node1(8gb,4cores) and another worker node2(8gb, 2 cores). Master is running on node1 and driver is also node1 only.



    It seems that even though second thread launch application, nothing happens to it and the second application doesn't even appear in the WAITING state which would have been understandable.










    share|improve this question


























      1












      1








      1


      1






      I am submitting/running multiple application through spark launcher in my java web app. it seems to submit only one app.Here is my code



      Runnable run = new Runnable() 
      public void run()
      try
      SparkAppHandle sparkApp = new SparkLauncher()
      .setAppResource("C:\BigData\spark\examples\jars\spark-examples_2.11-2.4.0.jar")
      .setMainClass("org.apache.spark.examples.SparkPi")
      .setMaster("spark://192.168.2.233:7077")
      .setConf("spark.scheduler.mode", "FAIR")
      .setVerbose(true)
      .setConf("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
      .setConf("spark.sql.inMemoryColumnarStorage.batchSize", "10000")
      .setConf("spark.sql.codegen","false")
      .setConf("spark.submit.deployMode", "client")
      .setConf("spark.executor.memory", "1g")
      .setConf("spark.driver.memory", "1g")
      .setConf("spark.cores.max", "1")
      .setConf("spark.executor.cores", "1")
      .setConf("spark.executor.instances", "1")
      .setConf("spark.driver.host","192.168.2.233")
      // .setConf("spark.dynamicAllocation.enabled", "true")
      // .setConf("spark.shuffle.service.enabled", "true")
      .startApplication();


      System.out.println(sparkApp.getState());

      catch (IOException e)
      // TODO Auto-generated catch block
      e.printStackTrace();


      ;

      //running two times so as to submit two parallel application
      //in the application logic , we would pass different args to different app
      new Thread(run).start();
      new Thread(run).start();`


      I have a standalone cluster with one worker node1(8gb,4cores) and another worker node2(8gb, 2 cores). Master is running on node1 and driver is also node1 only.



      It seems that even though second thread launch application, nothing happens to it and the second application doesn't even appear in the WAITING state which would have been understandable.










      share|improve this question
















      I am submitting/running multiple application through spark launcher in my java web app. it seems to submit only one app.Here is my code



      Runnable run = new Runnable() 
      public void run()
      try
      SparkAppHandle sparkApp = new SparkLauncher()
      .setAppResource("C:\BigData\spark\examples\jars\spark-examples_2.11-2.4.0.jar")
      .setMainClass("org.apache.spark.examples.SparkPi")
      .setMaster("spark://192.168.2.233:7077")
      .setConf("spark.scheduler.mode", "FAIR")
      .setVerbose(true)
      .setConf("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
      .setConf("spark.sql.inMemoryColumnarStorage.batchSize", "10000")
      .setConf("spark.sql.codegen","false")
      .setConf("spark.submit.deployMode", "client")
      .setConf("spark.executor.memory", "1g")
      .setConf("spark.driver.memory", "1g")
      .setConf("spark.cores.max", "1")
      .setConf("spark.executor.cores", "1")
      .setConf("spark.executor.instances", "1")
      .setConf("spark.driver.host","192.168.2.233")
      // .setConf("spark.dynamicAllocation.enabled", "true")
      // .setConf("spark.shuffle.service.enabled", "true")
      .startApplication();


      System.out.println(sparkApp.getState());

      catch (IOException e)
      // TODO Auto-generated catch block
      e.printStackTrace();


      ;

      //running two times so as to submit two parallel application
      //in the application logic , we would pass different args to different app
      new Thread(run).start();
      new Thread(run).start();`


      I have a standalone cluster with one worker node1(8gb,4cores) and another worker node2(8gb, 2 cores). Master is running on node1 and driver is also node1 only.



      It seems that even though second thread launch application, nothing happens to it and the second application doesn't even appear in the WAITING state which would have been understandable.







      java apache-spark apache-spark-2.0 spark-launcher






      share|improve this question















      share|improve this question













      share|improve this question




      share|improve this question








      edited Mar 7 at 11:43







      AngryLeo

















      asked Mar 7 at 10:46









      AngryLeoAngryLeo

      14411




      14411






















          0






          active

          oldest

          votes











          Your Answer






          StackExchange.ifUsing("editor", function ()
          StackExchange.using("externalEditor", function ()
          StackExchange.using("snippets", function ()
          StackExchange.snippets.init();
          );
          );
          , "code-snippets");

          StackExchange.ready(function()
          var channelOptions =
          tags: "".split(" "),
          id: "1"
          ;
          initTagRenderer("".split(" "), "".split(" "), channelOptions);

          StackExchange.using("externalEditor", function()
          // Have to fire editor after snippets, if snippets enabled
          if (StackExchange.settings.snippets.snippetsEnabled)
          StackExchange.using("snippets", function()
          createEditor();
          );

          else
          createEditor();

          );

          function createEditor()
          StackExchange.prepareEditor(
          heartbeatType: 'answer',
          autoActivateHeartbeat: false,
          convertImagesToLinks: true,
          noModals: true,
          showLowRepImageUploadWarning: true,
          reputationToPostImages: 10,
          bindNavPrevention: true,
          postfix: "",
          imageUploader:
          brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
          contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
          allowUrls: true
          ,
          onDemand: true,
          discardSelector: ".discard-answer"
          ,immediatelyShowMarkdownHelp:true
          );



          );













          draft saved

          draft discarded


















          StackExchange.ready(
          function ()
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55041939%2fsparklauncher-launching-only-one-application-even-when-submitting-multiple-appli%23new-answer', 'question_page');

          );

          Post as a guest















          Required, but never shown

























          0






          active

          oldest

          votes








          0






          active

          oldest

          votes









          active

          oldest

          votes






          active

          oldest

          votes















          draft saved

          draft discarded
















































          Thanks for contributing an answer to Stack Overflow!


          • Please be sure to answer the question. Provide details and share your research!

          But avoid …


          • Asking for help, clarification, or responding to other answers.

          • Making statements based on opinion; back them up with references or personal experience.

          To learn more, see our tips on writing great answers.




          draft saved


          draft discarded














          StackExchange.ready(
          function ()
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55041939%2fsparklauncher-launching-only-one-application-even-when-submitting-multiple-appli%23new-answer', 'question_page');

          );

          Post as a guest















          Required, but never shown





















































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown

































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown







          Popular posts from this blog

          Thal And Out Agency railway station See also References External links Navigation menuOfficial Web Site of Pakistan RailwaysArchivedOfficial Web Site of Pakistan Railwayseeexpanding ite

          Posting a File and Associated Data to a RESTful WebService preferably as JSON2019 Community Moderator ElectionREST URI convention - Singular or plural name of resource while creating itUploading both data and files in one form using Ajax?How do I upload a file with metadata using a REST web service?REST API - file (ie images) processing - best practicesSpring: JSON data and file in the same requestREST API Put large dataPosting a file and JSON data to Spring rest serviceRESTful Web Service Upload/Download Large Data With JSONREST API Design sending JSON data and a file to the api in same requestPost JSONArray to REST servicePUT vs. POST in RESTParsing values from a JSON file?JavaScript/jQuery to download file via POST with JSON dataHow do I upload a file with metadata using a REST web service?How to POST JSON data with Curl from Terminal/Commandline to Test Spring REST?Separate REST JSON API server and client?what's the correct way to send a file from REST web service to client?How do I write JSON data to a file?What is “406-Not Acceptable Response” in HTTP?REST API - file (ie images) processing - best practices

          How can I change the color of pagination dots of UIPageControl?How to change UIPageControl dotsIs there a way to change page indicator dots colorCustomize dot with image of UIPageControl at index 0 of UIPageControlNo visible @interface for 'NSObject<PageControlDelegate>' declares the selector 'pageControlPageDidChange:'How to change the color of pagination dots in UIPageControl with a different color per pagepagecontrol indicator custom image instead of DefaultChanging the colour of UIPageControl dots in MonoTouchpagecontrol selectable page visibility color?Alternative way to load ViewControllers on a UIPageControlHow to set only layer.border-color for UIpage control dots in swiftHow can I develop for iPhone using a Windows development machine?How to change the name of an iOS app?UITableView - change section header coloruipagecontrol indicator(dot)issueCustom UIPageControl dots color not changingchange the interspace between UIPageControl dotsHow to change Status Bar text color in iOSHow can I change image tintColor in iOS and WatchKitUIPageControl dots with larger space in between each dotsHow to change the color of pagination dots in UIPageControl with a different color per page