hive analyze query taking lot of time2019 Community Moderator ElectionMyISAM versus InnoDBImprove INSERT-per-second performance of SQLite?Speed comparison with Project Euler: C vs Python vs Erlang vs HaskellWhy are elementwise additions much faster in separate loops than in a combined loop?Swift Beta performance: sorting arraysReplacing a 32-bit loop counter with 64-bit introduces crazy performance deviationsWhy is “1000000000000000 in range(1000000000000001)” so fast in Python 3?Optimize a long running hive SQL query - has a join with same tableC++ code for testing the Collatz conjecture faster than hand-written assembly - why?How to run a query on the view in Hive?
How to generate globally unique ids for different tables of the same database?
Define, (actually define) the "stability" and "energy" of a compound
Identifying the interval from A♭ to D♯
Is it possible that AIC = BIC?
Do I need life insurance if I can cover my own funeral costs?
What is IP squat space
Co-worker team leader wants to inject his friend's awful software into our development. What should I say to our common boss?
What is this large pipe coming out of my roof?
Make a transparent 448*448 image
Humanity loses the vast majority of its technology, information, and population in the year 2122. How long does it take to rebuild itself?
Why does Deadpool say "You're welcome, Canada," after shooting Ryan Reynolds in the end credits?
Bash: What does "masking return values" mean?
Can elves maintain concentration in a trance?
It's a yearly task, alright
How do anti-virus programs start at Windows boot?
Use of プラトニック in this sentence?
Sword in the Stone story where the sword was held in place by electromagnets
Is it possible to upcast ritual spells?
Good allowance savings plan?
Can anyone tell me why this program fails?
Welcoming 2019 Pi day: How to draw the letter π?
Why do Australian milk farmers need to protest supermarkets' milk price?
Latest web browser compatible with Windows 98
Is a lawful good "antagonist" effective?
hive analyze query taking lot of time
2019 Community Moderator ElectionMyISAM versus InnoDBImprove INSERT-per-second performance of SQLite?Speed comparison with Project Euler: C vs Python vs Erlang vs HaskellWhy are elementwise additions much faster in separate loops than in a combined loop?Swift Beta performance: sorting arraysReplacing a 32-bit loop counter with 64-bit introduces crazy performance deviationsWhy is “1000000000000000 in range(1000000000000001)” so fast in Python 3?Optimize a long running hive SQL query - has a join with same tableC++ code for testing the Collatz conjecture faster than hand-written assembly - why?How to run a query on the view in Hive?
In order to speed up ETL queries on large tables, we run many analyze queries on these tables and date columns in the evening.
but these analyze queries on columns take lot of memory and time.
we are using tez.
is there any way to optimize analyze query also like some set commands.
performance hadoop hive query-tuning apache-tez
edited Mar 9 at 7:32
leftjoin
9,45522155
asked Mar 7 at 12:21
KumarKumar
273
add a comment |
In order to speed up ETL queries on large tables, we run many analyze queries on these tables and date columns in the evening.
but these analyze queries on columns take lot of memory and time.
we are using tez.
is there any way to optimize analyze query also like some set commands.
performance hadoop hive query-tuning apache-tez
edited Mar 9 at 7:32
leftjoin
9,45522155
asked Mar 7 at 12:21
KumarKumar
273
add a comment |
In order to speed up ETL queries on large tables, we run many analyze queries on these tables and date columns in the evening.
but these analyze queries on columns take lot of memory and time.
we are using tez.
is there any way to optimize analyze query also like some set commands.
performance hadoop hive query-tuning apache-tez
edited Mar 9 at 7:32
leftjoin
9,45522155
asked Mar 7 at 12:21
KumarKumar
273
In order to speed up ETL queries on large tables, we run many analyze queries on these tables and date columns in the evening.
but these analyze queries on columns take lot of memory and time.
we are using tez.
is there any way to optimize analyze query also like some set commands.
performance hadoop hive query-tuning apache-tez
performance hadoop hive query-tuning apache-tez
edited Mar 9 at 7:32
leftjoin
9,45522155
asked Mar 7 at 12:21
KumarKumar
273
edited Mar 9 at 7:32
leftjoin
9,45522155
asked Mar 7 at 12:21
KumarKumar
273
edited Mar 9 at 7:32
leftjoin
9,45522155
edited Mar 9 at 7:32
leftjoin
9,45522155
edited Mar 9 at 7:32
leftjoin
9,45522155
9,45522155
asked Mar 7 at 12:21
KumarKumar
273
asked Mar 7 at 12:21
KumarKumar
273
asked Mar 7 at 12:21
KumarKumar
273
273
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
If you are loading tables using insert overwrite then statistics can be gathered automatically by setting hive.stats.autogather=true during insert overwrite queries.
If the table is partitioned and partitions are being loaded incrementally, then you can analyze only last partitions.
ANALYZE TABLE [db_name.]tablename [PARTITION(partcol1[=val1], partcol2[=val2], ...)]
See examples here: https://cwiki.apache.org/confluence/display/Hive/StatsDev
For ORC files it's possible to specify hive.stats.gather.num.threads to incraase parallelism.
See full list of statistic settings here: https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-Statistics
answered Mar 9 at 8:01
leftjoinleftjoin
9,45522155
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55043680%2fhive-analyze-query-taking-lot-of-time%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
If you are loading tables using insert overwrite then statistics can be gathered automatically by setting hive.stats.autogather=true during insert overwrite queries.
If the table is partitioned and partitions are being loaded incrementally, then you can analyze only last partitions.
ANALYZE TABLE [db_name.]tablename [PARTITION(partcol1[=val1], partcol2[=val2], ...)]
See examples here: https://cwiki.apache.org/confluence/display/Hive/StatsDev
For ORC files it's possible to specify hive.stats.gather.num.threads to incraase parallelism.
See full list of statistic settings here: https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-Statistics
answered Mar 9 at 8:01
leftjoinleftjoin
9,45522155
add a comment |
If you are loading tables using insert overwrite then statistics can be gathered automatically by setting hive.stats.autogather=true during insert overwrite queries.
If the table is partitioned and partitions are being loaded incrementally, then you can analyze only last partitions.
ANALYZE TABLE [db_name.]tablename [PARTITION(partcol1[=val1], partcol2[=val2], ...)]
See examples here: https://cwiki.apache.org/confluence/display/Hive/StatsDev
For ORC files it's possible to specify hive.stats.gather.num.threads to incraase parallelism.
See full list of statistic settings here: https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-Statistics
answered Mar 9 at 8:01
leftjoinleftjoin
9,45522155
add a comment |
If you are loading tables using insert overwrite then statistics can be gathered automatically by setting hive.stats.autogather=true during insert overwrite queries.
If the table is partitioned and partitions are being loaded incrementally, then you can analyze only last partitions.
ANALYZE TABLE [db_name.]tablename [PARTITION(partcol1[=val1], partcol2[=val2], ...)]
See examples here: https://cwiki.apache.org/confluence/display/Hive/StatsDev
For ORC files it's possible to specify hive.stats.gather.num.threads to incraase parallelism.
See full list of statistic settings here: https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-Statistics
answered Mar 9 at 8:01
leftjoinleftjoin
9,45522155
If you are loading tables using insert overwrite then statistics can be gathered automatically by setting hive.stats.autogather=true during insert overwrite queries.
If the table is partitioned and partitions are being loaded incrementally, then you can analyze only last partitions.
ANALYZE TABLE [db_name.]tablename [PARTITION(partcol1[=val1], partcol2[=val2], ...)]
See examples here: https://cwiki.apache.org/confluence/display/Hive/StatsDev
For ORC files it's possible to specify hive.stats.gather.num.threads to incraase parallelism.
See full list of statistic settings here: https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-Statistics
answered Mar 9 at 8:01
leftjoinleftjoin
9,45522155
answered Mar 9 at 8:01
leftjoinleftjoin
9,45522155
answered Mar 9 at 8:01
leftjoinleftjoin
9,45522155
answered Mar 9 at 8:01
leftjoinleftjoin
9,45522155
9,45522155
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55043680%2fhive-analyze-query-taking-lot-of-time%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
