Using grouper to group a timestamp in a specific range2019 Community Moderator ElectionHow do you get a timestamp in JavaScript?Should I use the datetime or timestamp data type in MySQL?How can one change the timestamp of an old commit in Git?“Large data” work flows using pandasHow to count number of rows per group (and other statistics) in pandas group by?Convert range to timestamp in PandasHow to group dataframe by hour using timestamp with PandasPandas - Group by weeks from first/last entrypandas - how to organised dataframe based on date and assign new values to columndf grouping rows with same dates
Theorems like the Lovász Local Lemma?
When do we add an hyphen (-) to a complex adjective word?
Why do Australian milk farmers need to protest supermarkets' milk price?
Ban on all campaign finance?
Did CPM support custom hardware using device drivers?
How could a scammer know the apps on my phone / iTunes account?
Professor being mistaken for a grad student
What are the possible solutions of the given equation?
Official degrees of earth’s rotation per day
PTIJ: Who should pay for Uber rides: the child or the parent?
Sword in the Stone story where the sword was held in place by electromagnets
Dot in front of file
Replacing Windows 7 security updates with anti-virus?
Is having access to past exams cheating and, if yes, could it be proven just by a good grade?
Why does Deadpool say "You're welcome, Canada," after shooting Ryan Reynolds in the end credits?
What is IP squat space
Brexit - No Deal Rejection
Does the statement `int val = (++i > ++j) ? ++i : ++j;` invoke undefined behavior?
RegionDifference for Cylinder and Cuboid
How could a female member of a species produce eggs unto death?
Can anyone tell me why this program fails?
Is a lawful good "antagonist" effective?
How to explain that I do not want to visit a country due to personal safety concern?
My adviser wants to be the first author
Using grouper to group a timestamp in a specific range
2019 Community Moderator ElectionHow do you get a timestamp in JavaScript?Should I use the datetime or timestamp data type in MySQL?How can one change the timestamp of an old commit in Git?“Large data” work flows using pandasHow to count number of rows per group (and other statistics) in pandas group by?Convert range to timestamp in PandasHow to group dataframe by hour using timestamp with PandasPandas - Group by weeks from first/last entrypandas - how to organised dataframe based on date and assign new values to columndf grouping rows with same dates
Suppose that I have a data-frame (DF). Index of this data-frame is timestamp from 11 AM to 6 PM every day and this data-frame contains 30 days. I want to group it every 30 minutes. This is the function I'm using:
out = DF.groupby(pd.Grouper(freq='30min'))
The start date of output is correct, but it considers the whole day (24h) for grouping. For example, In the new timestamp, I have something like this:
11:00:00
11:30:00
12:00:00
12:30:00
...
18:00:00
18:30:00
...
23:00:00
23:30:00
...
2:00:00
2:30:00
...
...
10:30:00
11:00:00
11:30:00
As a result, many outputs are empty because from 6:00 PM to 11 AM, I don't have any data.
pandas timestamp pandas-groupby
asked Mar 7 at 12:23
user2991243user2991243
1,40963067
add a comment |
Suppose that I have a data-frame (DF). Index of this data-frame is timestamp from 11 AM to 6 PM every day and this data-frame contains 30 days. I want to group it every 30 minutes. This is the function I'm using:
out = DF.groupby(pd.Grouper(freq='30min'))
The start date of output is correct, but it considers the whole day (24h) for grouping. For example, In the new timestamp, I have something like this:
11:00:00
11:30:00
12:00:00
12:30:00
...
18:00:00
18:30:00
...
23:00:00
23:30:00
...
2:00:00
2:30:00
...
...
10:30:00
11:00:00
11:30:00
As a result, many outputs are empty because from 6:00 PM to 11 AM, I don't have any data.
pandas timestamp pandas-groupby
asked Mar 7 at 12:23
user2991243user2991243
1,40963067
Can you add some data sample? Can you explain24hfor grouping iffreq='30min'?
– jezrael
Mar 7 at 12:26
1
The output is correct and as expected. If you do not want to keep the empty intervals simply filter them away afterwards.
– John Sloper
Mar 7 at 12:35
@JohnSloper Is there any way to handle that ingroupby?
– user2991243
Mar 7 at 12:36
add a comment |
Suppose that I have a data-frame (DF). Index of this data-frame is timestamp from 11 AM to 6 PM every day and this data-frame contains 30 days. I want to group it every 30 minutes. This is the function I'm using:
out = DF.groupby(pd.Grouper(freq='30min'))
The start date of output is correct, but it considers the whole day (24h) for grouping. For example, In the new timestamp, I have something like this:
11:00:00
11:30:00
12:00:00
12:30:00
...
18:00:00
18:30:00
...
23:00:00
23:30:00
...
2:00:00
2:30:00
...
...
10:30:00
11:00:00
11:30:00
As a result, many outputs are empty because from 6:00 PM to 11 AM, I don't have any data.
pandas timestamp pandas-groupby
asked Mar 7 at 12:23
user2991243user2991243
1,40963067
Suppose that I have a data-frame (DF). Index of this data-frame is timestamp from 11 AM to 6 PM every day and this data-frame contains 30 days. I want to group it every 30 minutes. This is the function I'm using:
out = DF.groupby(pd.Grouper(freq='30min'))
The start date of output is correct, but it considers the whole day (24h) for grouping. For example, In the new timestamp, I have something like this:
11:00:00
11:30:00
12:00:00
12:30:00
...
18:00:00
18:30:00
...
23:00:00
23:30:00
...
2:00:00
2:30:00
...
...
10:30:00
11:00:00
11:30:00
As a result, many outputs are empty because from 6:00 PM to 11 AM, I don't have any data.
pandas timestamp pandas-groupby
pandas timestamp pandas-groupby
asked Mar 7 at 12:23
user2991243user2991243
1,40963067
asked Mar 7 at 12:23
user2991243user2991243
1,40963067
edited Mar 7 at 12:29
user2991243
asked Mar 7 at 12:23
user2991243user2991243
1,40963067
asked Mar 7 at 12:23
user2991243user2991243
1,40963067
asked Mar 7 at 12:23
user2991243user2991243
1,40963067
1,40963067
Can you add some data sample? Can you explain24hfor grouping iffreq='30min'?
– jezrael
Mar 7 at 12:26
1
The output is correct and as expected. If you do not want to keep the empty intervals simply filter them away afterwards.
– John Sloper
Mar 7 at 12:35
@JohnSloper Is there any way to handle that ingroupby?
– user2991243
Mar 7 at 12:36
add a comment |
Can you add some data sample? Can you explain24hfor grouping iffreq='30min'?
– jezrael
Mar 7 at 12:26
1
The output is correct and as expected. If you do not want to keep the empty intervals simply filter them away afterwards.
– John Sloper
Mar 7 at 12:35
@JohnSloper Is there any way to handle that ingroupby?
– user2991243
Mar 7 at 12:36
Can you add some data sample? Can you explain
24h for grouping if freq='30min' ?– jezrael
Mar 7 at 12:26
Can you add some data sample? Can you explain
24h for grouping if freq='30min' ?– jezrael
Mar 7 at 12:26
1
1
The output is correct and as expected. If you do not want to keep the empty intervals simply filter them away afterwards.
– John Sloper
Mar 7 at 12:35
The output is correct and as expected. If you do not want to keep the empty intervals simply filter them away afterwards.
– John Sloper
Mar 7 at 12:35
@JohnSloper Is there any way to handle that in
groupby?– user2991243
Mar 7 at 12:36
@JohnSloper Is there any way to handle that in
groupby?– user2991243
Mar 7 at 12:36
add a comment |
2 Answers
2
active
oldest
votes
One possible solution should be DatetimeIndex.floor:
out = DF.groupby(DF.index.floor('30min'))
Or use dropna after aggregate function:
out = DF.groupby(pd.Grouper(freq='30min')).mean().dropna()
answered Mar 7 at 12:40
jezraeljezrael
347k25302378
add a comment |
As mentioned in comment to original post this is as expected. If you want to remove empty groups simply slice them afterwards. Assuming in this case you are using count to aggregate:
df = df.groupby(pd.Grouper(freq='30min')).count()
df = df[df > 0]
answered Mar 7 at 12:42
John SloperJohn Sloper
803510
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55043724%2fusing-grouper-to-group-a-timestamp-in-a-specific-range%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
One possible solution should be DatetimeIndex.floor:
out = DF.groupby(DF.index.floor('30min'))
Or use dropna after aggregate function:
out = DF.groupby(pd.Grouper(freq='30min')).mean().dropna()
answered Mar 7 at 12:40
jezraeljezrael
347k25302378
add a comment |
One possible solution should be DatetimeIndex.floor:
out = DF.groupby(DF.index.floor('30min'))
Or use dropna after aggregate function:
out = DF.groupby(pd.Grouper(freq='30min')).mean().dropna()
answered Mar 7 at 12:40
jezraeljezrael
347k25302378
add a comment |
One possible solution should be DatetimeIndex.floor:
out = DF.groupby(DF.index.floor('30min'))
Or use dropna after aggregate function:
out = DF.groupby(pd.Grouper(freq='30min')).mean().dropna()
answered Mar 7 at 12:40
jezraeljezrael
347k25302378
One possible solution should be DatetimeIndex.floor:
out = DF.groupby(DF.index.floor('30min'))
Or use dropna after aggregate function:
out = DF.groupby(pd.Grouper(freq='30min')).mean().dropna()
answered Mar 7 at 12:40
jezraeljezrael
347k25302378
answered Mar 7 at 12:40
jezraeljezrael
347k25302378
answered Mar 7 at 12:40
jezraeljezrael
347k25302378
answered Mar 7 at 12:40
jezraeljezrael
347k25302378
347k25302378
add a comment |
add a comment |
As mentioned in comment to original post this is as expected. If you want to remove empty groups simply slice them afterwards. Assuming in this case you are using count to aggregate:
df = df.groupby(pd.Grouper(freq='30min')).count()
df = df[df > 0]
answered Mar 7 at 12:42
John SloperJohn Sloper
803510
add a comment |
As mentioned in comment to original post this is as expected. If you want to remove empty groups simply slice them afterwards. Assuming in this case you are using count to aggregate:
df = df.groupby(pd.Grouper(freq='30min')).count()
df = df[df > 0]
answered Mar 7 at 12:42
John SloperJohn Sloper
803510
add a comment |
As mentioned in comment to original post this is as expected. If you want to remove empty groups simply slice them afterwards. Assuming in this case you are using count to aggregate:
df = df.groupby(pd.Grouper(freq='30min')).count()
df = df[df > 0]
answered Mar 7 at 12:42
John SloperJohn Sloper
803510
As mentioned in comment to original post this is as expected. If you want to remove empty groups simply slice them afterwards. Assuming in this case you are using count to aggregate:
df = df.groupby(pd.Grouper(freq='30min')).count()
df = df[df > 0]
answered Mar 7 at 12:42
John SloperJohn Sloper
803510
answered Mar 7 at 12:42
John SloperJohn Sloper
803510
answered Mar 7 at 12:42
John SloperJohn Sloper
803510
answered Mar 7 at 12:42
John SloperJohn Sloper
803510
803510
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55043724%2fusing-grouper-to-group-a-timestamp-in-a-specific-range%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Can you add some data sample? Can you explain
24hfor grouping iffreq='30min'?– jezrael
Mar 7 at 12:26
1
The output is correct and as expected. If you do not want to keep the empty intervals simply filter them away afterwards.
– John Sloper
Mar 7 at 12:35
@JohnSloper Is there any way to handle that in
groupby?– user2991243
Mar 7 at 12:36