Spark nested json parsing getting null records2019 Community Moderator ElectionParsing JSON with Unix toolsHow to parse JSON in JavaParsing values from a JSON file?How can I parse a JSON file with PHP?Parse JSON in JavaScript?Parsing a JSON string in RubyHow to parse JSON using Node.js?Parse JSON in PythonHow do I get ASP.NET Web API to return JSON instead of XML using Chrome?Parsing JSON giving “unexpected token o” error
Why are special aircraft used for the carriers in the United States Navy?
Quitting employee has privileged access to critical information
How do you make a gun that shoots melee weapons and/or swords?
What is the meaning of option 'by' in TikZ Intersections
What is "desert glass" and what does it do to the PCs?
If nine coins are tossed, what is the probability that the number of heads is even?
Does the in-code argument passing conventions used on PDP-11's have a name?
Why would the IRS ask for birth certificates or even audit a small tax return?
Why do phishing e-mails use faked e-mail addresses instead of the real one?
Computing the volume of a simplex-like object with constraints
The past tense for the quoting particle って
What does "rhumatis" mean?
Create chunks from an array
PTIJ: Aliyot for the deceased
The (Easy) Road to Code
Practical reasons to have both a large police force and bounty hunting network?
Convert an array of objects to array of the objects' values
The need of reserving one's ability in job interviews
What's the difference between Compensation, Indemnity, and Reparations?
Questions of the type "What do you think other people would think?"
3.5% Interest Student Loan or use all of my savings on Tuition?
Why can't we use freedom of speech and expression to incite people to rebel against government in India?
Infinitive vs Gerund
How do we objectively assess if a dialogue sounds unnatural or cringy?
Spark nested json parsing getting null records
2019 Community Moderator ElectionParsing JSON with Unix toolsHow to parse JSON in JavaParsing values from a JSON file?How can I parse a JSON file with PHP?Parse JSON in JavaScript?Parsing a JSON string in RubyHow to parse JSON using Node.js?Parse JSON in PythonHow do I get ASP.NET Web API to return JSON instead of XML using Chrome?Parsing JSON giving “unexpected token o” error
When I am parsing the Json data using the below schema. I am getting the null records for the Products Filed.Can anyone give me some suggestion if I need to do any schema change for parsing JSON
val fields: StructType = StructType(
Array(StructField("Errors", StringType, true ),
StructField("Products",
StructType(Array(StructField("shop",StringType, true),
StructField("country",StringType, true),
StructField("location",StringType, true),
StructField("id",StringType, true),
StructField("downloaded_on",StringType, true),
StructField("run_id",StringType, true),
StructField("display_url",StringType, true),
StructField("request",
StructType(Array(StructField("url",StringType, true)))
),
StructField("resource",
StructType(Array(StructField("uri",StringType, true),
StructField("resource_type",StringType, true),
StructField("version",StringType, true))
)),
StructField("pricing",
StructType(Array(StructField("prices",
StructType(Array(StructField("unit",StringType, true),
StructField("is_main_price",StringType, true),
StructField("regular",StringType, true)))
),
StructField("vendor",StringType, true),
StructField("availability_strings",StringType, true),
StructField("prices",
StructType(Array(StructField("unit",StringType, true),
StructField("is_main_price",StringType, true),
StructField("regular",StringType, true)))),
StructField("vendor",StringType, true),
StructField("availability_strings",StringType, true))))
))),
StructField("uri",StringType, true)))
val data: _root_.org.apache.spark.sql.DataFrame = spark.read.schema(fields).json("file:///C:\Users\WELCOME\Desktop\sample_data.json")
data.printSchema()
data.show()
Sample_data :
"errors": [], "products": ["shop": "bol", "country": "be", "location": "be", "id": "9200000061636906", "downloaded_on": 1541387646, "run_id": 1541379910, "display_url": "https://www.bol.com/nl/p/levis-muurverf-ambiance-marmerwit-satijn-2-5-l/9200000061636906/prijsoverzicht/?filter=all&sort=price&country=BE&sortOrder=asc&approved=true", "request": "url": "https://www.bol.com/nl/p/levis-muurverf-ambiance-marmerwit-satijn-2-5-l/9200000061636906/prijsoverzicht/?filter=all&sort=price&country=BE&sortOrder=asc&approved=true", "resource": "uri": "s3://daltix-hive-production/resources/bol/be/be/20181105/1541379910/offer_page/1541387646.080048.html", "resource_type": "offer", "version": "default", "pricing": ["prices": ["unit": null, "is_main_price": true, "regular": 52.25], "vendor": "HandigeHarry.com", "availability_strings": ["Op voorraad. Voor 16:00 uur besteld, morgen in huisnWe doen er alles aan om dit artikel op tijd te bezorgen. Het is echter in een enkel geval mogelijk dat door omstandigheden de bezorging vertraagd is.nDe eventueel getoonde doorgestreepte prijs is de consumentenadviesprijs van onze leverancier."], "prices": ["unit": null, "is_main_price": true, "regular": 60.65], "vendor": "shop.vermeersch-deconinck.be", "availability_strings": ["Op voorraad. Voor 12:00 uur besteld, morgen in huisnWe doen er alles aan om dit artikel op tijd te bezorgen. Het is echter in een enkel geval mogelijk dat door omstandigheden de bezorging vertraagd is.nDe eventueel getoonde doorgestreepte prijs is de consumentenadviesprijs van onze leverancier."]]], "uri": "s3://daltix-hive-production/resources/bol/be/be/20181105/1541379910/offer_page/1541387646.080048.html"
"errors": [], "products": ["shop": "amazon", "country": "fr", "location": "fr", "id": "B079FL58GY", "downloaded_on": 1541387644, "run_id": 1541385102, "display_url": "https://www.amazon.fr/SUGU-AMZ62298-Avocat-Sheet-Masque/dp/B079FL58GY?ie=UTF8&sr=8-1&keywords=5037200062298", "request": "url": "https://www.amazon.fr/SUGU-AMZ62298-Avocat-Sheet-Masque/dp/B079FL58GY?ie=UTF8&sr=8-1&keywords=5037200062298", "resource": "uri": "s3://daltix-hive-production/resources/amazon/fr/fr/20181105/1541385102/batch_page/1541387644.0574694.html", "resource_type": "batch", "version": "default", "name": "Sugu Avocat Sheet Masque", "brand": "SUGU", "pricing": ["prices": ["unit": null, "is_main_price": true, "regular": 10.21]]], "uri": "s3://daltix-hive-production/resources/amazon/fr/fr/20181105/1541385102/batch_page/1541387644.0574694.html"
json apache-spark parsing apache-spark-sql
edited yesterday
howie
594316
asked yesterday
vishnu pavanvishnu pavan
11
New contributor
vishnu pavan is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
|
show 1 more comment
When I am parsing the Json data using the below schema. I am getting the null records for the Products Filed.Can anyone give me some suggestion if I need to do any schema change for parsing JSON
val fields: StructType = StructType(
Array(StructField("Errors", StringType, true ),
StructField("Products",
StructType(Array(StructField("shop",StringType, true),
StructField("country",StringType, true),
StructField("location",StringType, true),
StructField("id",StringType, true),
StructField("downloaded_on",StringType, true),
StructField("run_id",StringType, true),
StructField("display_url",StringType, true),
StructField("request",
StructType(Array(StructField("url",StringType, true)))
),
StructField("resource",
StructType(Array(StructField("uri",StringType, true),
StructField("resource_type",StringType, true),
StructField("version",StringType, true))
)),
StructField("pricing",
StructType(Array(StructField("prices",
StructType(Array(StructField("unit",StringType, true),
StructField("is_main_price",StringType, true),
StructField("regular",StringType, true)))
),
StructField("vendor",StringType, true),
StructField("availability_strings",StringType, true),
StructField("prices",
StructType(Array(StructField("unit",StringType, true),
StructField("is_main_price",StringType, true),
StructField("regular",StringType, true)))),
StructField("vendor",StringType, true),
StructField("availability_strings",StringType, true))))
))),
StructField("uri",StringType, true)))
val data: _root_.org.apache.spark.sql.DataFrame = spark.read.schema(fields).json("file:///C:\Users\WELCOME\Desktop\sample_data.json")
data.printSchema()
data.show()
Sample_data :
"errors": [], "products": ["shop": "bol", "country": "be", "location": "be", "id": "9200000061636906", "downloaded_on": 1541387646, "run_id": 1541379910, "display_url": "https://www.bol.com/nl/p/levis-muurverf-ambiance-marmerwit-satijn-2-5-l/9200000061636906/prijsoverzicht/?filter=all&sort=price&country=BE&sortOrder=asc&approved=true", "request": "url": "https://www.bol.com/nl/p/levis-muurverf-ambiance-marmerwit-satijn-2-5-l/9200000061636906/prijsoverzicht/?filter=all&sort=price&country=BE&sortOrder=asc&approved=true", "resource": "uri": "s3://daltix-hive-production/resources/bol/be/be/20181105/1541379910/offer_page/1541387646.080048.html", "resource_type": "offer", "version": "default", "pricing": ["prices": ["unit": null, "is_main_price": true, "regular": 52.25], "vendor": "HandigeHarry.com", "availability_strings": ["Op voorraad. Voor 16:00 uur besteld, morgen in huisnWe doen er alles aan om dit artikel op tijd te bezorgen. Het is echter in een enkel geval mogelijk dat door omstandigheden de bezorging vertraagd is.nDe eventueel getoonde doorgestreepte prijs is de consumentenadviesprijs van onze leverancier."], "prices": ["unit": null, "is_main_price": true, "regular": 60.65], "vendor": "shop.vermeersch-deconinck.be", "availability_strings": ["Op voorraad. Voor 12:00 uur besteld, morgen in huisnWe doen er alles aan om dit artikel op tijd te bezorgen. Het is echter in een enkel geval mogelijk dat door omstandigheden de bezorging vertraagd is.nDe eventueel getoonde doorgestreepte prijs is de consumentenadviesprijs van onze leverancier."]]], "uri": "s3://daltix-hive-production/resources/bol/be/be/20181105/1541379910/offer_page/1541387646.080048.html"
"errors": [], "products": ["shop": "amazon", "country": "fr", "location": "fr", "id": "B079FL58GY", "downloaded_on": 1541387644, "run_id": 1541385102, "display_url": "https://www.amazon.fr/SUGU-AMZ62298-Avocat-Sheet-Masque/dp/B079FL58GY?ie=UTF8&sr=8-1&keywords=5037200062298", "request": "url": "https://www.amazon.fr/SUGU-AMZ62298-Avocat-Sheet-Masque/dp/B079FL58GY?ie=UTF8&sr=8-1&keywords=5037200062298", "resource": "uri": "s3://daltix-hive-production/resources/amazon/fr/fr/20181105/1541385102/batch_page/1541387644.0574694.html", "resource_type": "batch", "version": "default", "name": "Sugu Avocat Sheet Masque", "brand": "SUGU", "pricing": ["prices": ["unit": null, "is_main_price": true, "regular": 10.21]]], "uri": "s3://daltix-hive-production/resources/amazon/fr/fr/20181105/1541385102/batch_page/1541387644.0574694.html"
json apache-spark parsing apache-spark-sql
edited yesterday
howie
594316
asked yesterday
vishnu pavanvishnu pavan
11
New contributor
vishnu pavan is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
Can you try without schema i.e spark.read.json("filePath")?
– hayat
yesterday
I have Tried Without using the schema iam getting null records
– vishnu pavan
yesterday
Its working for me I'm using linux system, seems issue is OS path related val df = spark.read.json("path") df.printSchema() df.show() +------+--------------------+--------------------+ |errors| products| uri| +------+--------------------+--------------------+ | []|[[be, ..|s3://daltix-hive-...| +------+--------------------+--------------------+
– hayat
yesterday
Only Root Fields are coming but child fileds are unable to parse using the schema
– vishnu pavan
yesterday
1
Your data contains a bunch ofarraysofstructswhich are incorrectly marked asstructsin your manually defined schema. You can start with fixing these.
– user10958683
yesterday
|
show 1 more comment
When I am parsing the Json data using the below schema. I am getting the null records for the Products Filed.Can anyone give me some suggestion if I need to do any schema change for parsing JSON
val fields: StructType = StructType(
Array(StructField("Errors", StringType, true ),
StructField("Products",
StructType(Array(StructField("shop",StringType, true),
StructField("country",StringType, true),
StructField("location",StringType, true),
StructField("id",StringType, true),
StructField("downloaded_on",StringType, true),
StructField("run_id",StringType, true),
StructField("display_url",StringType, true),
StructField("request",
StructType(Array(StructField("url",StringType, true)))
),
StructField("resource",
StructType(Array(StructField("uri",StringType, true),
StructField("resource_type",StringType, true),
StructField("version",StringType, true))
)),
StructField("pricing",
StructType(Array(StructField("prices",
StructType(Array(StructField("unit",StringType, true),
StructField("is_main_price",StringType, true),
StructField("regular",StringType, true)))
),
StructField("vendor",StringType, true),
StructField("availability_strings",StringType, true),
StructField("prices",
StructType(Array(StructField("unit",StringType, true),
StructField("is_main_price",StringType, true),
StructField("regular",StringType, true)))),
StructField("vendor",StringType, true),
StructField("availability_strings",StringType, true))))
))),
StructField("uri",StringType, true)))
val data: _root_.org.apache.spark.sql.DataFrame = spark.read.schema(fields).json("file:///C:\Users\WELCOME\Desktop\sample_data.json")
data.printSchema()
data.show()
Sample_data :
"errors": [], "products": ["shop": "bol", "country": "be", "location": "be", "id": "9200000061636906", "downloaded_on": 1541387646, "run_id": 1541379910, "display_url": "https://www.bol.com/nl/p/levis-muurverf-ambiance-marmerwit-satijn-2-5-l/9200000061636906/prijsoverzicht/?filter=all&sort=price&country=BE&sortOrder=asc&approved=true", "request": "url": "https://www.bol.com/nl/p/levis-muurverf-ambiance-marmerwit-satijn-2-5-l/9200000061636906/prijsoverzicht/?filter=all&sort=price&country=BE&sortOrder=asc&approved=true", "resource": "uri": "s3://daltix-hive-production/resources/bol/be/be/20181105/1541379910/offer_page/1541387646.080048.html", "resource_type": "offer", "version": "default", "pricing": ["prices": ["unit": null, "is_main_price": true, "regular": 52.25], "vendor": "HandigeHarry.com", "availability_strings": ["Op voorraad. Voor 16:00 uur besteld, morgen in huisnWe doen er alles aan om dit artikel op tijd te bezorgen. Het is echter in een enkel geval mogelijk dat door omstandigheden de bezorging vertraagd is.nDe eventueel getoonde doorgestreepte prijs is de consumentenadviesprijs van onze leverancier."], "prices": ["unit": null, "is_main_price": true, "regular": 60.65], "vendor": "shop.vermeersch-deconinck.be", "availability_strings": ["Op voorraad. Voor 12:00 uur besteld, morgen in huisnWe doen er alles aan om dit artikel op tijd te bezorgen. Het is echter in een enkel geval mogelijk dat door omstandigheden de bezorging vertraagd is.nDe eventueel getoonde doorgestreepte prijs is de consumentenadviesprijs van onze leverancier."]]], "uri": "s3://daltix-hive-production/resources/bol/be/be/20181105/1541379910/offer_page/1541387646.080048.html"
"errors": [], "products": ["shop": "amazon", "country": "fr", "location": "fr", "id": "B079FL58GY", "downloaded_on": 1541387644, "run_id": 1541385102, "display_url": "https://www.amazon.fr/SUGU-AMZ62298-Avocat-Sheet-Masque/dp/B079FL58GY?ie=UTF8&sr=8-1&keywords=5037200062298", "request": "url": "https://www.amazon.fr/SUGU-AMZ62298-Avocat-Sheet-Masque/dp/B079FL58GY?ie=UTF8&sr=8-1&keywords=5037200062298", "resource": "uri": "s3://daltix-hive-production/resources/amazon/fr/fr/20181105/1541385102/batch_page/1541387644.0574694.html", "resource_type": "batch", "version": "default", "name": "Sugu Avocat Sheet Masque", "brand": "SUGU", "pricing": ["prices": ["unit": null, "is_main_price": true, "regular": 10.21]]], "uri": "s3://daltix-hive-production/resources/amazon/fr/fr/20181105/1541385102/batch_page/1541387644.0574694.html"
json apache-spark parsing apache-spark-sql
edited yesterday
howie
594316
asked yesterday
vishnu pavanvishnu pavan
11
New contributor
vishnu pavan is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
When I am parsing the Json data using the below schema. I am getting the null records for the Products Filed.Can anyone give me some suggestion if I need to do any schema change for parsing JSON
val fields: StructType = StructType(
Array(StructField("Errors", StringType, true ),
StructField("Products",
StructType(Array(StructField("shop",StringType, true),
StructField("country",StringType, true),
StructField("location",StringType, true),
StructField("id",StringType, true),
StructField("downloaded_on",StringType, true),
StructField("run_id",StringType, true),
StructField("display_url",StringType, true),
StructField("request",
StructType(Array(StructField("url",StringType, true)))
),
StructField("resource",
StructType(Array(StructField("uri",StringType, true),
StructField("resource_type",StringType, true),
StructField("version",StringType, true))
)),
StructField("pricing",
StructType(Array(StructField("prices",
StructType(Array(StructField("unit",StringType, true),
StructField("is_main_price",StringType, true),
StructField("regular",StringType, true)))
),
StructField("vendor",StringType, true),
StructField("availability_strings",StringType, true),
StructField("prices",
StructType(Array(StructField("unit",StringType, true),
StructField("is_main_price",StringType, true),
StructField("regular",StringType, true)))),
StructField("vendor",StringType, true),
StructField("availability_strings",StringType, true))))
))),
StructField("uri",StringType, true)))
val data: _root_.org.apache.spark.sql.DataFrame = spark.read.schema(fields).json("file:///C:\Users\WELCOME\Desktop\sample_data.json")
data.printSchema()
data.show()
Sample_data :
"errors": [], "products": ["shop": "bol", "country": "be", "location": "be", "id": "9200000061636906", "downloaded_on": 1541387646, "run_id": 1541379910, "display_url": "https://www.bol.com/nl/p/levis-muurverf-ambiance-marmerwit-satijn-2-5-l/9200000061636906/prijsoverzicht/?filter=all&sort=price&country=BE&sortOrder=asc&approved=true", "request": "url": "https://www.bol.com/nl/p/levis-muurverf-ambiance-marmerwit-satijn-2-5-l/9200000061636906/prijsoverzicht/?filter=all&sort=price&country=BE&sortOrder=asc&approved=true", "resource": "uri": "s3://daltix-hive-production/resources/bol/be/be/20181105/1541379910/offer_page/1541387646.080048.html", "resource_type": "offer", "version": "default", "pricing": ["prices": ["unit": null, "is_main_price": true, "regular": 52.25], "vendor": "HandigeHarry.com", "availability_strings": ["Op voorraad. Voor 16:00 uur besteld, morgen in huisnWe doen er alles aan om dit artikel op tijd te bezorgen. Het is echter in een enkel geval mogelijk dat door omstandigheden de bezorging vertraagd is.nDe eventueel getoonde doorgestreepte prijs is de consumentenadviesprijs van onze leverancier."], "prices": ["unit": null, "is_main_price": true, "regular": 60.65], "vendor": "shop.vermeersch-deconinck.be", "availability_strings": ["Op voorraad. Voor 12:00 uur besteld, morgen in huisnWe doen er alles aan om dit artikel op tijd te bezorgen. Het is echter in een enkel geval mogelijk dat door omstandigheden de bezorging vertraagd is.nDe eventueel getoonde doorgestreepte prijs is de consumentenadviesprijs van onze leverancier."]]], "uri": "s3://daltix-hive-production/resources/bol/be/be/20181105/1541379910/offer_page/1541387646.080048.html"
"errors": [], "products": ["shop": "amazon", "country": "fr", "location": "fr", "id": "B079FL58GY", "downloaded_on": 1541387644, "run_id": 1541385102, "display_url": "https://www.amazon.fr/SUGU-AMZ62298-Avocat-Sheet-Masque/dp/B079FL58GY?ie=UTF8&sr=8-1&keywords=5037200062298", "request": "url": "https://www.amazon.fr/SUGU-AMZ62298-Avocat-Sheet-Masque/dp/B079FL58GY?ie=UTF8&sr=8-1&keywords=5037200062298", "resource": "uri": "s3://daltix-hive-production/resources/amazon/fr/fr/20181105/1541385102/batch_page/1541387644.0574694.html", "resource_type": "batch", "version": "default", "name": "Sugu Avocat Sheet Masque", "brand": "SUGU", "pricing": ["prices": ["unit": null, "is_main_price": true, "regular": 10.21]]], "uri": "s3://daltix-hive-production/resources/amazon/fr/fr/20181105/1541385102/batch_page/1541387644.0574694.html"
json apache-spark parsing apache-spark-sql
json apache-spark parsing apache-spark-sql
edited yesterday
howie
594316
asked yesterday
vishnu pavanvishnu pavan
11
New contributor
vishnu pavan is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
edited yesterday
howie
594316
asked yesterday
vishnu pavanvishnu pavan
11
New contributor
vishnu pavan is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
edited yesterday
howie
594316
edited yesterday
howie
594316
edited yesterday
howie
594316
594316
asked yesterday
vishnu pavanvishnu pavan
11
New contributor
vishnu pavan is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked yesterday
vishnu pavanvishnu pavan
11
asked yesterday
vishnu pavanvishnu pavan
11
11
New contributor
vishnu pavan is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
New contributor
vishnu pavan is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
vishnu pavan is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
Can you try without schema i.e spark.read.json("filePath")?
– hayat
yesterday
I have Tried Without using the schema iam getting null records
– vishnu pavan
yesterday
Its working for me I'm using linux system, seems issue is OS path related val df = spark.read.json("path") df.printSchema() df.show() +------+--------------------+--------------------+ |errors| products| uri| +------+--------------------+--------------------+ | []|[[be, ..|s3://daltix-hive-...| +------+--------------------+--------------------+
– hayat
yesterday
Only Root Fields are coming but child fileds are unable to parse using the schema
– vishnu pavan
yesterday
1
Your data contains a bunch ofarraysofstructswhich are incorrectly marked asstructsin your manually defined schema. You can start with fixing these.
– user10958683
yesterday
|
show 1 more comment
Can you try without schema i.e spark.read.json("filePath")?
– hayat
yesterday
I have Tried Without using the schema iam getting null records
– vishnu pavan
yesterday
Its working for me I'm using linux system, seems issue is OS path related val df = spark.read.json("path") df.printSchema() df.show() +------+--------------------+--------------------+ |errors| products| uri| +------+--------------------+--------------------+ | []|[[be, ..|s3://daltix-hive-...| +------+--------------------+--------------------+
– hayat
yesterday
Only Root Fields are coming but child fileds are unable to parse using the schema
– vishnu pavan
yesterday
1
Your data contains a bunch ofarraysofstructswhich are incorrectly marked asstructsin your manually defined schema. You can start with fixing these.
– user10958683
yesterday
Can you try without schema i.e spark.read.json("filePath")?
– hayat
yesterday
Can you try without schema i.e spark.read.json("filePath")?
– hayat
yesterday
I have Tried Without using the schema iam getting null records
– vishnu pavan
yesterday
I have Tried Without using the schema iam getting null records
– vishnu pavan
yesterday
Its working for me I'm using linux system, seems issue is OS path related val df = spark.read.json("path") df.printSchema() df.show() +------+--------------------+--------------------+ |errors| products| uri| +------+--------------------+--------------------+ | []|[[be, ..|s3://daltix-hive-...| +------+--------------------+--------------------+
– hayat
yesterday
Its working for me I'm using linux system, seems issue is OS path related val df = spark.read.json("path") df.printSchema() df.show() +------+--------------------+--------------------+ |errors| products| uri| +------+--------------------+--------------------+ | []|[[be, ..|s3://daltix-hive-...| +------+--------------------+--------------------+
– hayat
yesterday
Only Root Fields are coming but child fileds are unable to parse using the schema
– vishnu pavan
yesterday
Only Root Fields are coming but child fileds are unable to parse using the schema
– vishnu pavan
yesterday
1
1
Your data contains a bunch of
arrays of structs which are incorrectly marked as structs in your manually defined schema. You can start with fixing these.– user10958683
yesterday
Your data contains a bunch of
arrays of structs which are incorrectly marked as structs in your manually defined schema. You can start with fixing these.– user10958683
yesterday
|
show 1 more comment
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
vishnu pavan is a new contributor. Be nice, and check out our Code of Conduct.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55023308%2fspark-nested-json-parsing-getting-null-records%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
vishnu pavan is a new contributor. Be nice, and check out our Code of Conduct.
vishnu pavan is a new contributor. Be nice, and check out our Code of Conduct.
vishnu pavan is a new contributor. Be nice, and check out our Code of Conduct.
vishnu pavan is a new contributor. Be nice, and check out our Code of Conduct.
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55023308%2fspark-nested-json-parsing-getting-null-records%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Can you try without schema i.e spark.read.json("filePath")?
– hayat
yesterday
I have Tried Without using the schema iam getting null records
– vishnu pavan
yesterday
Its working for me I'm using linux system, seems issue is OS path related val df = spark.read.json("path") df.printSchema() df.show() +------+--------------------+--------------------+ |errors| products| uri| +------+--------------------+--------------------+ | []|[[be, ..|s3://daltix-hive-...| +------+--------------------+--------------------+
– hayat
yesterday
Only Root Fields are coming but child fileds are unable to parse using the schema
– vishnu pavan
yesterday
1
Your data contains a bunch of
arraysofstructswhich are incorrectly marked asstructsin your manually defined schema. You can start with fixing these.– user10958683
yesterday