Fast way of Selecting Indices from Numpy ArrayHow to randomly select an item from a list?Is there a NumPy function to return the first index of something in an array?How to print the full NumPy array, without truncation?Numpy array dimensionsDump a NumPy array into a csv fileHow do I get indices of N maximum values in a NumPy array?Select rows from a DataFrame based on values in a column in pandasExecution time difference in matrix multiplication caused by parenthesesComparing zip operation with pandas slice operationWhy is Cython so much slower than Numba when iterating over NumPy arrays?
Which one is correct as adjective “protruding” or “protruded”?
How can Trident be so inexpensive? Will it orbit Triton or just do a (slow) flyby?
Non-trope happy ending?
When were female captains banned from Starfleet?
Fear of getting stuck on one programming language / technology that is not used in my country
250 Floor Tower
Not using 's' for he/she/it
Electoral considerations aside, what are potential benefits, for the US, of policy changes proposed by the tweet recognizing Golan annexation?
Calculating Wattage for Resistor in High Frequency Application?
Problem with TransformedDistribution
Is it improper etiquette to ask your opponent what his/her rating is before the game?
Is there a single word describing earning money through any means?
Does a 'pending' US visa application constitute a denial?
What should you do if you miss a job interview (deliberately)?
A social experiment. What is the worst that can happen?
Is it possible to have a strip of cold climate in the middle of a planet?
WiFi Thermostat, No C Terminal on Furnace
Yosemite Fire Rings - What to Expect?
How to implement a feedback to keep the DC gain at zero for this conceptual passive filter?
Longest common substring in linear time
Why Shazam when there is already Superman?
The IT department bottlenecks progress. How should I handle this?
Why is it that I can sometimes guess the next note?
Why did the Mercure fail?
Fast way of Selecting Indices from Numpy Array
How to randomly select an item from a list?Is there a NumPy function to return the first index of something in an array?How to print the full NumPy array, without truncation?Numpy array dimensionsDump a NumPy array into a csv fileHow do I get indices of N maximum values in a NumPy array?Select rows from a DataFrame based on values in a column in pandasExecution time difference in matrix multiplication caused by parenthesesComparing zip operation with pandas slice operationWhy is Cython so much slower than Numba when iterating over NumPy arrays?
I have a large array, X, out of which I must select interesting indices, as below. X is 60000 x 784. I'm wasting about a second running this at every call during a loop.
%timeit subX = X[:,indices]
890 ms ± 23.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
How can I access my array faster? indices is a list and can range from 500 numbers to 784.
python numpy indexing
asked Mar 8 at 4:42
saadsaad
622927
add a comment |
I have a large array, X, out of which I must select interesting indices, as below. X is 60000 x 784. I'm wasting about a second running this at every call during a loop.
%timeit subX = X[:,indices]
890 ms ± 23.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
How can I access my array faster? indices is a list and can range from 500 numbers to 784.
python numpy indexing
asked Mar 8 at 4:42
saadsaad
622927
Changing the array order to F order might help
– busybear
Mar 8 at 4:45
@busybear can you please elaborate?
– saad
Mar 8 at 4:46
1
In F order, the array is stored in memory with the first axis as contiguous data. Since you are accessing the entire first axis (column), it may be more efficient to have the data stored in this fashion--so it's not jumping around the memory space to retrieve data. The documentation will give a more thorough explanation. Not sure how much (or if) you'll get a performance boost with this though. Worth trying.
– busybear
Mar 8 at 4:50
1
@busybear just tried it. Down to 230ms!
– saad
Mar 8 at 4:57
Nice! That's a decent boost. You do have a lot of data so it's going to be hard to avoid having some slow down.
– busybear
Mar 8 at 5:03
add a comment |
I have a large array, X, out of which I must select interesting indices, as below. X is 60000 x 784. I'm wasting about a second running this at every call during a loop.
%timeit subX = X[:,indices]
890 ms ± 23.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
How can I access my array faster? indices is a list and can range from 500 numbers to 784.
python numpy indexing
asked Mar 8 at 4:42
saadsaad
622927
I have a large array, X, out of which I must select interesting indices, as below. X is 60000 x 784. I'm wasting about a second running this at every call during a loop.
%timeit subX = X[:,indices]
890 ms ± 23.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
How can I access my array faster? indices is a list and can range from 500 numbers to 784.
python numpy indexing
python numpy indexing
asked Mar 8 at 4:42
saadsaad
622927
asked Mar 8 at 4:42
saadsaad
622927
asked Mar 8 at 4:42
saadsaad
622927
asked Mar 8 at 4:42
saadsaad
622927
asked Mar 8 at 4:42
saadsaad
622927
622927
Changing the array order to F order might help
– busybear
Mar 8 at 4:45
@busybear can you please elaborate?
– saad
Mar 8 at 4:46
1
In F order, the array is stored in memory with the first axis as contiguous data. Since you are accessing the entire first axis (column), it may be more efficient to have the data stored in this fashion--so it's not jumping around the memory space to retrieve data. The documentation will give a more thorough explanation. Not sure how much (or if) you'll get a performance boost with this though. Worth trying.
– busybear
Mar 8 at 4:50
1
@busybear just tried it. Down to 230ms!
– saad
Mar 8 at 4:57
Nice! That's a decent boost. You do have a lot of data so it's going to be hard to avoid having some slow down.
– busybear
Mar 8 at 5:03
add a comment |
Changing the array order to F order might help
– busybear
Mar 8 at 4:45
@busybear can you please elaborate?
– saad
Mar 8 at 4:46
1
In F order, the array is stored in memory with the first axis as contiguous data. Since you are accessing the entire first axis (column), it may be more efficient to have the data stored in this fashion--so it's not jumping around the memory space to retrieve data. The documentation will give a more thorough explanation. Not sure how much (or if) you'll get a performance boost with this though. Worth trying.
– busybear
Mar 8 at 4:50
1
@busybear just tried it. Down to 230ms!
– saad
Mar 8 at 4:57
Nice! That's a decent boost. You do have a lot of data so it's going to be hard to avoid having some slow down.
– busybear
Mar 8 at 5:03
Changing the array order to F order might help
– busybear
Mar 8 at 4:45
Changing the array order to F order might help
– busybear
Mar 8 at 4:45
@busybear can you please elaborate?
– saad
Mar 8 at 4:46
@busybear can you please elaborate?
– saad
Mar 8 at 4:46
1
1
In F order, the array is stored in memory with the first axis as contiguous data. Since you are accessing the entire first axis (column), it may be more efficient to have the data stored in this fashion--so it's not jumping around the memory space to retrieve data. The documentation will give a more thorough explanation. Not sure how much (or if) you'll get a performance boost with this though. Worth trying.
– busybear
Mar 8 at 4:50
In F order, the array is stored in memory with the first axis as contiguous data. Since you are accessing the entire first axis (column), it may be more efficient to have the data stored in this fashion--so it's not jumping around the memory space to retrieve data. The documentation will give a more thorough explanation. Not sure how much (or if) you'll get a performance boost with this though. Worth trying.
– busybear
Mar 8 at 4:50
1
1
@busybear just tried it. Down to 230ms!
– saad
Mar 8 at 4:57
@busybear just tried it. Down to 230ms!
– saad
Mar 8 at 4:57
Nice! That's a decent boost. You do have a lot of data so it's going to be hard to avoid having some slow down.
– busybear
Mar 8 at 5:03
Nice! That's a decent boost. You do have a lot of data so it's going to be hard to avoid having some slow down.
– busybear
Mar 8 at 5:03
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55056848%2ffast-way-of-selecting-indices-from-numpy-array%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55056848%2ffast-way-of-selecting-indices-from-numpy-array%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Changing the array order to F order might help
– busybear
Mar 8 at 4:45
@busybear can you please elaborate?
– saad
Mar 8 at 4:46
1
In F order, the array is stored in memory with the first axis as contiguous data. Since you are accessing the entire first axis (column), it may be more efficient to have the data stored in this fashion--so it's not jumping around the memory space to retrieve data. The documentation will give a more thorough explanation. Not sure how much (or if) you'll get a performance boost with this though. Worth trying.
– busybear
Mar 8 at 4:50
1
@busybear just tried it. Down to 230ms!
– saad
Mar 8 at 4:57
Nice! That's a decent boost. You do have a lot of data so it's going to be hard to avoid having some slow down.
– busybear
Mar 8 at 5:03