Ggtcf

I have a file in a subset of YAML with data such as the below:

# This is a comment
# This is another comment


spark:spark.ui.enabled: 'false'
spark:spark.sql.adaptive.enabled: 'true'
yarn:yarn.nodemanager.log.retain-seconds: '259200'

I need to convert that into a JSON document looking like this (note that strings containing booleans and integers still remain strings):

 "spark:spark.ui.enabled": "false",
 "spark:spark.sql.adaptive.enabled": "true",
 "yarn:yarn.nodemanager.log.retain-seconds", "259200"

The closest I got was this:

cat << EOF > ./file.yaml
> # This is a comment
> # This is another comment
> 
> 
> spark:spark.ui.enabled: 'false'
> spark:spark.sql.adaptive.enabled: 'true'
> yarn:yarn.nodemanager.log.retain-seconds: '259200'
> EOF
echo grep -o '^[^#]*' 

which apart from looking rather gnarly doesn't give the correct answer, it returns:

"spark:spark.ui.enabled": 'false',"spark:spark.sql.adaptive.enabled": 'true',"dataproc:dataproc.monitoring.stackdriver.enable": 'true',"spark:spark.submit.deployMode": 'cluster'

which, if I pipe to jq causes a parse error.

I'm hoping I'm missing a much much easier way of doing this but I can't figure it out. Can anyone help?

Implemented in pure jq (tested with version 1.6):

#!/usr/bin/env bash

jq_script=$(cat <<'EOF'
def content_for_line:
 "^[[:space:]]*([#]|$)" as $ignore_re | # regex for comments, blank lines
 "^(?<key>.*): (?<value>.*)$" as $content_re | # regex for actual k/v pairs
 "^'(?<value>.*)'$" as $quoted_re | # regex for values in single quotes
 if test($ignore_re) then else # empty lines add nothing to the data
 if test($content_re) then ( # non-empty: match against $content_re
 capture($content_re) as $content | # ...and put the groups into $content
 $content.key as $key | # string before ": " becomes $key
 (if ($content.value | test($quoted_re)) then # if value contains literal quotes...
 ($content.value | capture($quoted_re)).value # ...take string from inside quotes
 else
 $content.value # no quotes to strip
 end) as $value | # result of the above block becomes $value
 "($key)": "($value)" # and return a map from one key to one value
 ) else
 # we get here if a line didn't match $ignore_re *or* $content_re
 error("Line (.) is not recognized as a comment, empty, or valid content")
 end
 end;

# iterate over our input lines, passing each one to content_for_line and merging the result
# into the object we're building, which we eventually return as our result.
reduce inputs as $item (; . + ($item | content_for_line))
EOF
)

# jq -R: read input as raw strings
# jq -n: don't read from stdin until requested with "input" or "inputs"
jq -Rn "$jq_script" <file.yaml >file.json

Unlike syntax-unaware tools, this can never generate output that isn't valid JSON; and it can easily be extended with application-specific logic (f/e, to emit some values but not others as numeric literals rather than string literals) by adding an additional filter stage to inspect and modify the output of content_for_line.

Here's a no-frills but simple solution:

def tidy: sub("^ *'?";"") | sub(" *'?$";"");
def kv: split(":") | [ (.[:-1] | join(":")), (.[-1]|tidy)];

reduce (inputs| select( test("^ *#|^ *$")|not) | kv) as $row (;
 .[$row[0]] = $row[1] )

Invocation

jq -n -R -f tojson.jq input.txt

You can do it all in awk using gsub and sprintf, for example:

(edit to add "," separating json records)

awk 'BEGIN ol=0; print "" 
/^[^#]/ 
 if (ol) print ","
 gsub ("47", "42")
 $1 = sprintf (" "%s":", substr ($1, 1, length ($1) - 1))
 printf "%s %s", $1, $2
 ol++ 
 
END print "n" ' file.yaml

(note: though jq is the proper tool for json formatting)

Explanation

• 
  awk 'BEGIN { ol=0; print "" call awk setting the output line variable ol=0 for "," output control and printing the header "",


• 
  /^[^#]/ only match non-comment lines,


• 
  if (ol) print "," if the output line ol is greater than zero, output a trailing ","
  


• 
  gsub ("47", "42") replace all single-quotes with double-quotes,


• 
  $1 = sprintf (" "%s":", substr ($1, 1, length ($1) - 1)) add 2 leading spaces and double-quotes around the first field (except for the last character) and then append a ':' at the end.


• 
  print $1, $2 output the reformatted fields,


• 
  ol++ increment the output line count, and


• 
  END print "" ' close by printing the "" footer

Example Use/Output

Just select/paste the awk command above (changing the filename as needed)

$ awk 'BEGIN ol=0; print "" 
> /^[^#]/ 
> if (ol) print ","
> gsub ("47", "42")
> $1 = sprintf (" "%s":", substr ($1, 1, length ($1) - 1))
> printf "%s %s", $1, $2
> ol++
> 
> END print "n" ' file.yaml

 "spark:spark.ui.enabled": "false",
 "spark:spark.sql.adaptive.enabled": "true"