saveasTextFile(“path”) in scalaSaving a string to HDFS creates line feeds for each characterSpark - load CSV file as DataFrame?saveAsTextFile for non rdd spark scalaHow to save a Map result to a text file in Spark scala?How to save Iterable[String] to hdfs in spark-scalasaveAsTextFile not working with variable pathhow to use mapPartitions followed by saveAsTextFilesUsing iterated writing in HDFS file by using Spark/Scalaerror: value saveAsTextFile is not a member of scala.collection.Map[String,Long]Save output from foreach[unit] to HDFS as text file : Scala Spark
Why does AES have exactly 10 rounds for a 128-bit key, 12 for 192 bits and 14 for a 256-bit key size?
Stack Interview Code methods made from class Node and Smart Pointers
Why is the Sun approximated as a black body at ~ 5800 K?
Why should universal income be universal?
Why do ¬, ∀ and ∃ have the same precedence?
Has the laser at Magurele, Romania reached a tenth of the Sun's power?
How to explain what's wrong with this application of the chain rule?
Why can't the Brexit deadlock in the UK parliament be solved with a plurality vote?
The IT department bottlenecks progress, how should I handle this?
PTIJ: Why is Haman obsessed with Bose?
Is my low blitz game drawing rate at www.chess.com an indicator that I am weak in chess?
How does electrical safety system work on ISS?
Non-trope happy ending?
Does an advisor owe his/her student anything? Will an advisor keep a PhD student only out of pity?
Can I cause damage to electrical appliances by unplugging them when they are turned on?
What is going on with gets(stdin) on the site coderbyte?
Taxes on Dividends in a Roth IRA
Is it necessary to use pronouns with the verb "essere"?
Does Doodling or Improvising on the Piano Have Any Benefits?
What is the difference between lands and mana?
Which Article Helped Get Rid of Technobabble in RPGs?
Do we have to expect a queue for the shuttle from Watford Junction to Harry Potter Studio?
Mimic lecturing on blackboard, facing audience
Why do Radio Buttons not fill the entire outer circle?
saveasTextFile(“path”) in scala
Saving a string to HDFS creates line feeds for each characterSpark - load CSV file as DataFrame?saveAsTextFile for non rdd spark scalaHow to save a Map result to a text file in Spark scala?How to save Iterable[String] to hdfs in spark-scalasaveAsTextFile not working with variable pathhow to use mapPartitions followed by saveAsTextFilesUsing iterated writing in HDFS file by using Spark/Scalaerror: value saveAsTextFile is not a member of scala.collection.Map[String,Long]Save output from foreach[unit] to HDFS as text file : Scala Spark
I used Scala in Spark, and I tried to save my file in HDFS file, but I got an error.
I have tried rdd.saveAsTextFile("path"); sc.saveAsTextFile("pathe");
saveAsTextFile("path")
scala> inputJPG.map(x=>x.split("")).map(array=>array(0)).sc.saveAsTextFile("/loudacre/iplist")
<console>:28: error: value sc is not a member of Array[String]
inputJPG.map(x=>x.split("")).map(array=>array(0)).sc.saveAsTextFile("/loudacre/iplist")
scala apache-spark hdfs
asked Mar 7 at 23:51
Eric.PEric.P
122
add a comment |
I used Scala in Spark, and I tried to save my file in HDFS file, but I got an error.
I have tried rdd.saveAsTextFile("path"); sc.saveAsTextFile("pathe");
saveAsTextFile("path")
scala> inputJPG.map(x=>x.split("")).map(array=>array(0)).sc.saveAsTextFile("/loudacre/iplist")
<console>:28: error: value sc is not a member of Array[String]
inputJPG.map(x=>x.split("")).map(array=>array(0)).sc.saveAsTextFile("/loudacre/iplist")
scala apache-spark hdfs
asked Mar 7 at 23:51
Eric.PEric.P
122
1
Can you be a little more specific? What is the structure of your file and what kind of data is in there? Are you simply trying to write a dataframe to hadoop? Have you read this page? spark.apache.org/docs/latest/sql-getting-started.html
– Tony Fraser
Mar 8 at 0:57
2
You should not include.schere, just use.saveAsTextFile(...)on the RDD. However, it looks like you work with anArray[String]in which case you can't use RDD functions.
– Shaido
Mar 8 at 6:02
add a comment |
I used Scala in Spark, and I tried to save my file in HDFS file, but I got an error.
I have tried rdd.saveAsTextFile("path"); sc.saveAsTextFile("pathe");
saveAsTextFile("path")
scala> inputJPG.map(x=>x.split("")).map(array=>array(0)).sc.saveAsTextFile("/loudacre/iplist")
<console>:28: error: value sc is not a member of Array[String]
inputJPG.map(x=>x.split("")).map(array=>array(0)).sc.saveAsTextFile("/loudacre/iplist")
scala apache-spark hdfs
asked Mar 7 at 23:51
Eric.PEric.P
122
I used Scala in Spark, and I tried to save my file in HDFS file, but I got an error.
I have tried rdd.saveAsTextFile("path"); sc.saveAsTextFile("pathe");
saveAsTextFile("path")
scala> inputJPG.map(x=>x.split("")).map(array=>array(0)).sc.saveAsTextFile("/loudacre/iplist")
<console>:28: error: value sc is not a member of Array[String]
inputJPG.map(x=>x.split("")).map(array=>array(0)).sc.saveAsTextFile("/loudacre/iplist")
scala apache-spark hdfs
scala apache-spark hdfs
asked Mar 7 at 23:51
Eric.PEric.P
122
asked Mar 7 at 23:51
Eric.PEric.P
122
asked Mar 7 at 23:51
Eric.PEric.P
122
asked Mar 7 at 23:51
Eric.PEric.P
122
asked Mar 7 at 23:51
Eric.PEric.P
122
122
1
Can you be a little more specific? What is the structure of your file and what kind of data is in there? Are you simply trying to write a dataframe to hadoop? Have you read this page? spark.apache.org/docs/latest/sql-getting-started.html
– Tony Fraser
Mar 8 at 0:57
2
You should not include.schere, just use.saveAsTextFile(...)on the RDD. However, it looks like you work with anArray[String]in which case you can't use RDD functions.
– Shaido
Mar 8 at 6:02
add a comment |
1
Can you be a little more specific? What is the structure of your file and what kind of data is in there? Are you simply trying to write a dataframe to hadoop? Have you read this page? spark.apache.org/docs/latest/sql-getting-started.html
– Tony Fraser
Mar 8 at 0:57
2
You should not include.schere, just use.saveAsTextFile(...)on the RDD. However, it looks like you work with anArray[String]in which case you can't use RDD functions.
– Shaido
Mar 8 at 6:02
1
1
Can you be a little more specific? What is the structure of your file and what kind of data is in there? Are you simply trying to write a dataframe to hadoop? Have you read this page? spark.apache.org/docs/latest/sql-getting-started.html
– Tony Fraser
Mar 8 at 0:57
Can you be a little more specific? What is the structure of your file and what kind of data is in there? Are you simply trying to write a dataframe to hadoop? Have you read this page? spark.apache.org/docs/latest/sql-getting-started.html
– Tony Fraser
Mar 8 at 0:57
2
2
You should not include
.sc here, just use .saveAsTextFile(...) on the RDD. However, it looks like you work with an Array[String] in which case you can't use RDD functions.– Shaido
Mar 8 at 6:02
You should not include
.sc here, just use .saveAsTextFile(...) on the RDD. However, it looks like you work with an Array[String] in which case you can't use RDD functions.– Shaido
Mar 8 at 6:02
add a comment |
1 Answer
1
active
oldest
votes
I'm a new student and still learning Scala, so not familiar with RDD and Scala function.
Turn to my problem.I found that is because I have val xx=data.collect()before which is not RDD, so I can't do data.saveAsTextFile in Spark.
So I moved the .collect() function => val xx=data
Then data.savaAsTextFile("path") works.
answered Mar 9 at 4:28
Eric.PEric.P
122
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55054662%2fsaveastextfilepath-in-scala%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
I'm a new student and still learning Scala, so not familiar with RDD and Scala function.
Turn to my problem.I found that is because I have val xx=data.collect()before which is not RDD, so I can't do data.saveAsTextFile in Spark.
So I moved the .collect() function => val xx=data
Then data.savaAsTextFile("path") works.
answered Mar 9 at 4:28
Eric.PEric.P
122
add a comment |
I'm a new student and still learning Scala, so not familiar with RDD and Scala function.
Turn to my problem.I found that is because I have val xx=data.collect()before which is not RDD, so I can't do data.saveAsTextFile in Spark.
So I moved the .collect() function => val xx=data
Then data.savaAsTextFile("path") works.
answered Mar 9 at 4:28
Eric.PEric.P
122
add a comment |
I'm a new student and still learning Scala, so not familiar with RDD and Scala function.
Turn to my problem.I found that is because I have val xx=data.collect()before which is not RDD, so I can't do data.saveAsTextFile in Spark.
So I moved the .collect() function => val xx=data
Then data.savaAsTextFile("path") works.
answered Mar 9 at 4:28
Eric.PEric.P
122
I'm a new student and still learning Scala, so not familiar with RDD and Scala function.
Turn to my problem.I found that is because I have val xx=data.collect()before which is not RDD, so I can't do data.saveAsTextFile in Spark.
So I moved the .collect() function => val xx=data
Then data.savaAsTextFile("path") works.
answered Mar 9 at 4:28
Eric.PEric.P
122
answered Mar 9 at 4:28
Eric.PEric.P
122
answered Mar 9 at 4:28
Eric.PEric.P
122
answered Mar 9 at 4:28
Eric.PEric.P
122
122
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55054662%2fsaveastextfilepath-in-scala%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
Can you be a little more specific? What is the structure of your file and what kind of data is in there? Are you simply trying to write a dataframe to hadoop? Have you read this page? spark.apache.org/docs/latest/sql-getting-started.html
– Tony Fraser
Mar 8 at 0:57
2
You should not include
.schere, just use.saveAsTextFile(...)on the RDD. However, it looks like you work with anArray[String]in which case you can't use RDD functions.– Shaido
Mar 8 at 6:02