Versioning on Parquet files from AWS Glue for external table querying with Redshift Spectrum2019 Community Moderator ElectionSkipping header rows in AWS Redshift External TablesAWS Spectrum giving blank result for parquet files generated by AWS GlueAWS Glue ETL Job fails with AnalysisException: u'Unable to infer schema for Parquet. It must be specified manually.;'How to create an external table for nested Parquet type in redshift spectrumRedshift Spectrum : Getting no values/ empty while select using ParquetAWS Glue: How to handle nested JSON with varying schemashow to view data catalog table in S3 using redshift spectrumAWS Glue: How to ETL non-scalar JSON with varying schemasAthena returns empty results from Firehose > Glue > S3 parquet setupMove data from PostgreSQL to AWS S3 and analyze with RedShift Spectrum

How to recover against Snake as a heavyweight character?

Unfamiliar notation in Diabelli's "Duet in D" for piano

How to make sure I'm assertive enough in contact with subordinates?

Is there a math expression equivalent to the conditional ternary operator?

Short story about an infectious indestructible metal bar?

Is the differential, dp, exact or not?

Create chunks from an array

Why restrict private health insurance?

Is "cogitate" used appropriately in "I cogitate that success relies on hard work"?

Giving a talk in my old university, how prominently should I tell students my salary?

Was this cameo in Captain Marvel computer generated?

What is the oldest European royal house?

What does *dead* mean in *What do you mean, dead?*?

How to write a chaotic neutral protagonist and prevent my readers from thinking they are evil?

Does an unused member variable take up memory?

Has a sovereign Communist government ever run, and conceded loss, on a fair election?

An Undercover Army

Is this Paypal Github SDK reference really a dangerous site?

How to install "rounded" brake pads

Is there a logarithm base for which the logarithm becomes an identity function?

Should I apply for my boss's promotion?

Professor forcing me to attend a conference, I can't afford even with 50% funding

How would an energy-based "projectile" blow up a spaceship?

Why do phishing e-mails use faked e-mail addresses instead of the real one?



Versioning on Parquet files from AWS Glue for external table querying with Redshift Spectrum



2019 Community Moderator ElectionSkipping header rows in AWS Redshift External TablesAWS Spectrum giving blank result for parquet files generated by AWS GlueAWS Glue ETL Job fails with AnalysisException: u'Unable to infer schema for Parquet. It must be specified manually.;'How to create an external table for nested Parquet type in redshift spectrumRedshift Spectrum : Getting no values/ empty while select using ParquetAWS Glue: How to handle nested JSON with varying schemashow to view data catalog table in S3 using redshift spectrumAWS Glue: How to ETL non-scalar JSON with varying schemasAthena returns empty results from Firehose > Glue > S3 parquet setupMove data from PostgreSQL to AWS S3 and analyze with RedShift Spectrum










1















I am working on an ETL pipeline to transform JSON files into Parquet format (storage cost) using AWS Glue batch job, and then query the S3 bucket with Parquet files using RedShift External tables.



My roadblock is how to handle multiple versioning of the Parquet files, so that the external table queries the latest version of the Parquet file or force AWS Glue to overwrite Parquet files and always maintain latest version.




--------------------------------------------------------------------------------------------------------------------
| Day | JSON (S3) | JSON Ver |Parquet (S3) | External Table Query Data |
--------------------------------------------------------------------------------------------------------------------
| 1 | temp.json | 1.0 |part-00000-9n83b353-c000.snappy.parquet| Count 1, includes Parquet i.e. 1.0 |
--------------------------------------------------------------------------------------------------------------------
| 2 | temp.json | 2.0 |part-00000-7s34s653-c000.snappy.parquet| Count 2, includes both version, Parquet 1.0 & 2.0|
--------------------------------------------------------------------------------------------------------------------


As a service, S3-AWSGlue-Redshift Spectrum meets my implementation scenario but would like to have a recommendation on how to handle versioning issue.










share|improve this question


























    1















    I am working on an ETL pipeline to transform JSON files into Parquet format (storage cost) using AWS Glue batch job, and then query the S3 bucket with Parquet files using RedShift External tables.



    My roadblock is how to handle multiple versioning of the Parquet files, so that the external table queries the latest version of the Parquet file or force AWS Glue to overwrite Parquet files and always maintain latest version.




    --------------------------------------------------------------------------------------------------------------------
    | Day | JSON (S3) | JSON Ver |Parquet (S3) | External Table Query Data |
    --------------------------------------------------------------------------------------------------------------------
    | 1 | temp.json | 1.0 |part-00000-9n83b353-c000.snappy.parquet| Count 1, includes Parquet i.e. 1.0 |
    --------------------------------------------------------------------------------------------------------------------
    | 2 | temp.json | 2.0 |part-00000-7s34s653-c000.snappy.parquet| Count 2, includes both version, Parquet 1.0 & 2.0|
    --------------------------------------------------------------------------------------------------------------------


    As a service, S3-AWSGlue-Redshift Spectrum meets my implementation scenario but would like to have a recommendation on how to handle versioning issue.










    share|improve this question
























      1












      1








      1








      I am working on an ETL pipeline to transform JSON files into Parquet format (storage cost) using AWS Glue batch job, and then query the S3 bucket with Parquet files using RedShift External tables.



      My roadblock is how to handle multiple versioning of the Parquet files, so that the external table queries the latest version of the Parquet file or force AWS Glue to overwrite Parquet files and always maintain latest version.




      --------------------------------------------------------------------------------------------------------------------
      | Day | JSON (S3) | JSON Ver |Parquet (S3) | External Table Query Data |
      --------------------------------------------------------------------------------------------------------------------
      | 1 | temp.json | 1.0 |part-00000-9n83b353-c000.snappy.parquet| Count 1, includes Parquet i.e. 1.0 |
      --------------------------------------------------------------------------------------------------------------------
      | 2 | temp.json | 2.0 |part-00000-7s34s653-c000.snappy.parquet| Count 2, includes both version, Parquet 1.0 & 2.0|
      --------------------------------------------------------------------------------------------------------------------


      As a service, S3-AWSGlue-Redshift Spectrum meets my implementation scenario but would like to have a recommendation on how to handle versioning issue.










      share|improve this question














      I am working on an ETL pipeline to transform JSON files into Parquet format (storage cost) using AWS Glue batch job, and then query the S3 bucket with Parquet files using RedShift External tables.



      My roadblock is how to handle multiple versioning of the Parquet files, so that the external table queries the latest version of the Parquet file or force AWS Glue to overwrite Parquet files and always maintain latest version.




      --------------------------------------------------------------------------------------------------------------------
      | Day | JSON (S3) | JSON Ver |Parquet (S3) | External Table Query Data |
      --------------------------------------------------------------------------------------------------------------------
      | 1 | temp.json | 1.0 |part-00000-9n83b353-c000.snappy.parquet| Count 1, includes Parquet i.e. 1.0 |
      --------------------------------------------------------------------------------------------------------------------
      | 2 | temp.json | 2.0 |part-00000-7s34s653-c000.snappy.parquet| Count 2, includes both version, Parquet 1.0 & 2.0|
      --------------------------------------------------------------------------------------------------------------------


      As a service, S3-AWSGlue-Redshift Spectrum meets my implementation scenario but would like to have a recommendation on how to handle versioning issue.







      amazon-web-services amazon-s3 parquet aws-glue amazon-redshift-spectrum






      share|improve this question













      share|improve this question











      share|improve this question




      share|improve this question










      asked 2 days ago









      SunSmilesSunSmiles

      867




      867






















          0






          active

          oldest

          votes











          Your Answer






          StackExchange.ifUsing("editor", function ()
          StackExchange.using("externalEditor", function ()
          StackExchange.using("snippets", function ()
          StackExchange.snippets.init();
          );
          );
          , "code-snippets");

          StackExchange.ready(function()
          var channelOptions =
          tags: "".split(" "),
          id: "1"
          ;
          initTagRenderer("".split(" "), "".split(" "), channelOptions);

          StackExchange.using("externalEditor", function()
          // Have to fire editor after snippets, if snippets enabled
          if (StackExchange.settings.snippets.snippetsEnabled)
          StackExchange.using("snippets", function()
          createEditor();
          );

          else
          createEditor();

          );

          function createEditor()
          StackExchange.prepareEditor(
          heartbeatType: 'answer',
          autoActivateHeartbeat: false,
          convertImagesToLinks: true,
          noModals: true,
          showLowRepImageUploadWarning: true,
          reputationToPostImages: 10,
          bindNavPrevention: true,
          postfix: "",
          imageUploader:
          brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
          contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
          allowUrls: true
          ,
          onDemand: true,
          discardSelector: ".discard-answer"
          ,immediatelyShowMarkdownHelp:true
          );



          );













          draft saved

          draft discarded


















          StackExchange.ready(
          function ()
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55027518%2fversioning-on-parquet-files-from-aws-glue-for-external-table-querying-with-redsh%23new-answer', 'question_page');

          );

          Post as a guest















          Required, but never shown

























          0






          active

          oldest

          votes








          0






          active

          oldest

          votes









          active

          oldest

          votes






          active

          oldest

          votes















          draft saved

          draft discarded
















































          Thanks for contributing an answer to Stack Overflow!


          • Please be sure to answer the question. Provide details and share your research!

          But avoid


          • Asking for help, clarification, or responding to other answers.

          • Making statements based on opinion; back them up with references or personal experience.

          To learn more, see our tips on writing great answers.




          draft saved


          draft discarded














          StackExchange.ready(
          function ()
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55027518%2fversioning-on-parquet-files-from-aws-glue-for-external-table-querying-with-redsh%23new-answer', 'question_page');

          );

          Post as a guest















          Required, but never shown





















































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown

































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown







          Popular posts from this blog

          Identity Server 4 is not redirecting to Angular app after login2019 Community Moderator ElectionIdentity Server 4 and dockerIdentityserver implicit flow unauthorized_clientIdentityServer Hybrid Flow - Access Token is null after user successful loginIdentity Server to MVC client : Page Redirect After loginLogin with Steam OpenId(oidc-client-js)Identity Server 4+.NET Core 2.0 + IdentityIdentityServer4 post-login redirect not working in Edge browserCall to IdentityServer4 generates System.NullReferenceException: Object reference not set to an instance of an objectIdentityServer4 without HTTPS not workingHow to get Authorization code from identity server without login form

          2005 Ahvaz unrest Contents Background Causes Casualties Aftermath See also References Navigation menue"At Least 10 Are Killed by Bombs in Iran""Iran"Archived"Arab-Iranians in Iran to make April 15 'Day of Fury'"State of Mind, State of Order: Reactions to Ethnic Unrest in the Islamic Republic of Iran.10.1111/j.1754-9469.2008.00028.x"Iran hangs Arab separatists"Iran Overview from ArchivedConstitution of the Islamic Republic of Iran"Tehran puzzled by forged 'riots' letter""Iran and its minorities: Down in the second class""Iran: Handling Of Ahvaz Unrest Could End With Televised Confessions""Bombings Rock Iran Ahead of Election""Five die in Iran ethnic clashes""Iran: Need for restraint as anniversary of unrest in Khuzestan approaches"Archived"Iranian Sunni protesters killed in clashes with security forces"Archived

          Can't initialize raids on a new ASUS Prime B360M-A motherboard2019 Community Moderator ElectionSimilar to RAID config yet more like mirroring solution?Can't get motherboard serial numberWhy does the BIOS entry point start with a WBINVD instruction?UEFI performance Asus Maximus V Extreme