Python: Efficient search in 3d numpy array2019 Community Moderator ElectionCalling an external command in PythonWhat are metaclasses in Python?Finding the index of an item given a list containing it in PythonDifference between append vs. extend list methods in PythonHow can I safely create a nested directory in Python?Does Python have a ternary conditional operator?How to get the current time in PythonHow can I make a time delay in Python?How to do case insensitive search in VimDoes Python have a string 'contains' substring method?
How to balance a monster modification (zombie)?
Why are there no stars visible in cislunar space?
Is there any common country to visit for uk and schengen visa?
Can "few" be used as a subject? If so, what is the rule?
Did Nintendo change its mind about 68000 SNES?
Do I need to convey a moral for each of my blog post?
Does convergence of polynomials imply that of its coefficients?
Why didn't Héctor fade away after this character died in the movie Coco?
Someone scrambled my calling sign- who am I?
Triple Trouble Tribond
pipe commands inside find -exec?
Does fire aspect on a sword, destroy mob drops?
How to test the sharpness of a knife?
Friend wants my recommendation but I don't want to
Isn't the word "experience" wrongly used in this context?
Why do I have a large white artefact on the rendered image?
Would this string work as string?
CLI: Get information Ubuntu releases
Print last inputted byte
Difficulty understanding group delay concept
Homology of the fiber
Why is this tree refusing to shed its dead leaves?
Norwegian Refugee travel document
Animating wave motion in water
Python: Efficient search in 3d numpy array
2019 Community Moderator ElectionCalling an external command in PythonWhat are metaclasses in Python?Finding the index of an item given a list containing it in PythonDifference between append vs. extend list methods in PythonHow can I safely create a nested directory in Python?Does Python have a ternary conditional operator?How to get the current time in PythonHow can I make a time delay in Python?How to do case insensitive search in VimDoes Python have a string 'contains' substring method?
I have a 3d numpy array (shape (z,y,x): 137,601,1200)) and face the challenge of finding the index of the value that is closest to 500 in every vertical column.
For instance I'd like to find the index that is closest to 500 for the array (:,30,112). The numbers are ordered from bottom to top in descending order. For instance some of the 137 values of (:,30,112) could look as follows: [1033.91 1031.35 ... 0.01].
My first approach is to apply a search function through every vertical column by means of a for loop. But I'm sure, there must be a faster way! Here is my first try:
from bisect import bisect_left
def takeClosest(myList, myNumber):
pos = bisect_left(myList, myNumber)
if pos == 0:
return 0
if pos == len(myList):
return -1
before = myList[pos - 1]
after = myList[pos]
if after - myNumber < myNumber - before:
return pos
else:
return pos-1
def calcOptK(tnsr):
# 2d array which saves the optimal levels
optKs = np.zeros([601,1200])
#iterate through the tensor and call on each point the search function
for y in range (0, 601):
for x in range (0, 1200):
optKs[y,x]=136-takeClosest(p3d[:,y,x][::-1],500)
return optKs;
optKs=calcOptK(p3d)
UPDATE: Thanks to @hpaulj and @Mstaino.
This works perfectly:
optKs = np.argmin(np.abs(p3d-500), axis=0)
python numpy search multidimensional-array
asked Mar 7 at 18:37
SevSev
76
|
show 2 more comments
I have a 3d numpy array (shape (z,y,x): 137,601,1200)) and face the challenge of finding the index of the value that is closest to 500 in every vertical column.
For instance I'd like to find the index that is closest to 500 for the array (:,30,112). The numbers are ordered from bottom to top in descending order. For instance some of the 137 values of (:,30,112) could look as follows: [1033.91 1031.35 ... 0.01].
My first approach is to apply a search function through every vertical column by means of a for loop. But I'm sure, there must be a faster way! Here is my first try:
from bisect import bisect_left
def takeClosest(myList, myNumber):
pos = bisect_left(myList, myNumber)
if pos == 0:
return 0
if pos == len(myList):
return -1
before = myList[pos - 1]
after = myList[pos]
if after - myNumber < myNumber - before:
return pos
else:
return pos-1
def calcOptK(tnsr):
# 2d array which saves the optimal levels
optKs = np.zeros([601,1200])
#iterate through the tensor and call on each point the search function
for y in range (0, 601):
for x in range (0, 1200):
optKs[y,x]=136-takeClosest(p3d[:,y,x][::-1],500)
return optKs;
optKs=calcOptK(p3d)
UPDATE: Thanks to @hpaulj and @Mstaino.
This works perfectly:
optKs = np.argmin(np.abs(p3d-500), axis=0)
python numpy search multidimensional-array
asked Mar 7 at 18:37
SevSev
76
3
How aboutnp.argmin(np.abs(arr-500), axis=?)? That is get the distance from 500 for all points, and find the smallest along the relevant axis.
– hpaulj
Mar 7 at 18:50
I sense from your code you are trying to find the closest but larger than value, not the absolute closest. If that is the case, please update the question.
– Mstaino
Mar 7 at 19:33
@hpaulj's method should be significantly faster than your loop since it's vectorized. And since it's only 137 numbers I doubt using a binary search within Python will be faster than numpy's C loop.
– Rocky Li
Mar 7 at 20:08
Thanks @hpaulj for the idea. But how and where exactly would I fit it into my code?
– Sev
Mar 8 at 10:49
@Mstaino I try to find the absolute closest.
– Sev
Mar 8 at 10:49
|
show 2 more comments
I have a 3d numpy array (shape (z,y,x): 137,601,1200)) and face the challenge of finding the index of the value that is closest to 500 in every vertical column.
For instance I'd like to find the index that is closest to 500 for the array (:,30,112). The numbers are ordered from bottom to top in descending order. For instance some of the 137 values of (:,30,112) could look as follows: [1033.91 1031.35 ... 0.01].
My first approach is to apply a search function through every vertical column by means of a for loop. But I'm sure, there must be a faster way! Here is my first try:
from bisect import bisect_left
def takeClosest(myList, myNumber):
pos = bisect_left(myList, myNumber)
if pos == 0:
return 0
if pos == len(myList):
return -1
before = myList[pos - 1]
after = myList[pos]
if after - myNumber < myNumber - before:
return pos
else:
return pos-1
def calcOptK(tnsr):
# 2d array which saves the optimal levels
optKs = np.zeros([601,1200])
#iterate through the tensor and call on each point the search function
for y in range (0, 601):
for x in range (0, 1200):
optKs[y,x]=136-takeClosest(p3d[:,y,x][::-1],500)
return optKs;
optKs=calcOptK(p3d)
UPDATE: Thanks to @hpaulj and @Mstaino.
This works perfectly:
optKs = np.argmin(np.abs(p3d-500), axis=0)
python numpy search multidimensional-array
asked Mar 7 at 18:37
SevSev
76
I have a 3d numpy array (shape (z,y,x): 137,601,1200)) and face the challenge of finding the index of the value that is closest to 500 in every vertical column.
For instance I'd like to find the index that is closest to 500 for the array (:,30,112). The numbers are ordered from bottom to top in descending order. For instance some of the 137 values of (:,30,112) could look as follows: [1033.91 1031.35 ... 0.01].
My first approach is to apply a search function through every vertical column by means of a for loop. But I'm sure, there must be a faster way! Here is my first try:
from bisect import bisect_left
def takeClosest(myList, myNumber):
pos = bisect_left(myList, myNumber)
if pos == 0:
return 0
if pos == len(myList):
return -1
before = myList[pos - 1]
after = myList[pos]
if after - myNumber < myNumber - before:
return pos
else:
return pos-1
def calcOptK(tnsr):
# 2d array which saves the optimal levels
optKs = np.zeros([601,1200])
#iterate through the tensor and call on each point the search function
for y in range (0, 601):
for x in range (0, 1200):
optKs[y,x]=136-takeClosest(p3d[:,y,x][::-1],500)
return optKs;
optKs=calcOptK(p3d)
UPDATE: Thanks to @hpaulj and @Mstaino.
This works perfectly:
optKs = np.argmin(np.abs(p3d-500), axis=0)
python numpy search multidimensional-array
python numpy search multidimensional-array
asked Mar 7 at 18:37
SevSev
76
asked Mar 7 at 18:37
SevSev
76
edited Mar 9 at 14:57
Sev
asked Mar 7 at 18:37
SevSev
76
asked Mar 7 at 18:37
SevSev
76
asked Mar 7 at 18:37
SevSev
76
76
3
How aboutnp.argmin(np.abs(arr-500), axis=?)? That is get the distance from 500 for all points, and find the smallest along the relevant axis.
– hpaulj
Mar 7 at 18:50
I sense from your code you are trying to find the closest but larger than value, not the absolute closest. If that is the case, please update the question.
– Mstaino
Mar 7 at 19:33
@hpaulj's method should be significantly faster than your loop since it's vectorized. And since it's only 137 numbers I doubt using a binary search within Python will be faster than numpy's C loop.
– Rocky Li
Mar 7 at 20:08
Thanks @hpaulj for the idea. But how and where exactly would I fit it into my code?
– Sev
Mar 8 at 10:49
@Mstaino I try to find the absolute closest.
– Sev
Mar 8 at 10:49
|
show 2 more comments
3
How aboutnp.argmin(np.abs(arr-500), axis=?)? That is get the distance from 500 for all points, and find the smallest along the relevant axis.
– hpaulj
Mar 7 at 18:50
I sense from your code you are trying to find the closest but larger than value, not the absolute closest. If that is the case, please update the question.
– Mstaino
Mar 7 at 19:33
@hpaulj's method should be significantly faster than your loop since it's vectorized. And since it's only 137 numbers I doubt using a binary search within Python will be faster than numpy's C loop.
– Rocky Li
Mar 7 at 20:08
Thanks @hpaulj for the idea. But how and where exactly would I fit it into my code?
– Sev
Mar 8 at 10:49
@Mstaino I try to find the absolute closest.
– Sev
Mar 8 at 10:49
3
3
How about
np.argmin(np.abs(arr-500), axis=?)? That is get the distance from 500 for all points, and find the smallest along the relevant axis.– hpaulj
Mar 7 at 18:50
How about
np.argmin(np.abs(arr-500), axis=?)? That is get the distance from 500 for all points, and find the smallest along the relevant axis.– hpaulj
Mar 7 at 18:50
I sense from your code you are trying to find the closest but larger than value, not the absolute closest. If that is the case, please update the question.
– Mstaino
Mar 7 at 19:33
I sense from your code you are trying to find the closest but larger than value, not the absolute closest. If that is the case, please update the question.
– Mstaino
Mar 7 at 19:33
@hpaulj's method should be significantly faster than your loop since it's vectorized. And since it's only 137 numbers I doubt using a binary search within Python will be faster than numpy's C loop.
– Rocky Li
Mar 7 at 20:08
@hpaulj's method should be significantly faster than your loop since it's vectorized. And since it's only 137 numbers I doubt using a binary search within Python will be faster than numpy's C loop.
– Rocky Li
Mar 7 at 20:08
Thanks @hpaulj for the idea. But how and where exactly would I fit it into my code?
– Sev
Mar 8 at 10:49
Thanks @hpaulj for the idea. But how and where exactly would I fit it into my code?
– Sev
Mar 8 at 10:49
@Mstaino I try to find the absolute closest.
– Sev
Mar 8 at 10:49
@Mstaino I try to find the absolute closest.
– Sev
Mar 8 at 10:49
|
show 2 more comments
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55050657%2fpython-efficient-search-in-3d-numpy-array%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55050657%2fpython-efficient-search-in-3d-numpy-array%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
3
How about
np.argmin(np.abs(arr-500), axis=?)? That is get the distance from 500 for all points, and find the smallest along the relevant axis.– hpaulj
Mar 7 at 18:50
I sense from your code you are trying to find the closest but larger than value, not the absolute closest. If that is the case, please update the question.
– Mstaino
Mar 7 at 19:33
@hpaulj's method should be significantly faster than your loop since it's vectorized. And since it's only 137 numbers I doubt using a binary search within Python will be faster than numpy's C loop.
– Rocky Li
Mar 7 at 20:08
Thanks @hpaulj for the idea. But how and where exactly would I fit it into my code?
– Sev
Mar 8 at 10:49
@Mstaino I try to find the absolute closest.
– Sev
Mar 8 at 10:49