Ggtcf

I'm Iterating a Dataset<Row> using ForeachFunction while in the iteration I don't know how to append some custom columns to the Row and and append it to another Dataset<Row> in spark Java

Code:

groupedDataset.foreach((ForeachFunction<Row>) row -> 

 double average = //some value

 // the Row has four columns
 // All I want is to have a new Dataset<Row> with specific columns
 // from the Row i.e row(0),row(1),row(3) and average value

 Dataset<Row> newDs = row.getString("ID"),row.getString("time"),row.getInt("value"),average;

);

I have tried a lot but I couldn't able to solve it.

Thank you!

Rows are not supposed to be modified directly (it is possible but not convenient). When manipulating dataframes (Dataset of rows), you are supposed to use the SparkSQL API for two main reasons: 1. it's easy to use 2. it allows spark to perform a lot of optimizations on your requests.

Now, here is an example that seem to look like what you are trying to achieve. Basically I create a dataset with three columns and I use a select to average the result of two of them and discard the last one. Let me know if you need more details.

SparkSession spark = SparkSession.builder().getOrCreate();
Dataset<Row> data = spark
 .range(10)
 .select(col("id").as("id"),
 col("id").cast("string").as("str"),
 col("id").plus(5).as("id5") );
 data.show();

 Dataset<Row> result = data
 .select(col("id"), col("id5"),
 col("id").plus(col("id5")).divide(2).as("avg"));

 result.show();

which yields:

+---+---+---+
| id|str|id5|
+---+---+---+
| 0| 0| 5|
| 1| 1| 6|
| 2| 2| 7|
+---+---+---+ 

+---+---+---+
| id|id5|avg|
+---+---+---+
| 0| 5|2.5|
| 1| 6|3.5|
| 2| 7|4.5|
+---+---+---+