read .pkl files from IBM object storage2019 Community Moderator ElectionPHP Softlayer Object Storage Upload SSL errorHow do I do bulk file storage with IBM Object Storage?IBM Object Storage with PythonAuthorization Header - IBM Cloud Object StorageIBM Cloud - Object Storage , nosuchkeyhow can I save a binary file from cloud object storage to the notebook filesystem?How to set (basic http) authentication for IBM Cloud Object Storage (COS) files?How to download multiple objects from IBM Cloud Object Storage?create object with python ibm-cos-sdk get doesn't workIBM Object Storage Archive
When two POV characters meet
What does it mean when multiple 々 marks follow a 、?
How do anti-virus programs start at Windows boot?
Is having access to past exams cheating and, if yes, could it be proven just by a good grade?
Want to switch to tankless, but can I use my existing wiring?
Should we release the security issues we found in our product as CVE or we can just update those on weekly release notes?
Making a sword in the stone, in a medieval world without magic
Best mythical creature to use as livestock?
How could a female member of a species produce eggs unto death?
As a monk, can you make a melee attack roll using your Strength modifier, but roll damage with your Dexterity modifier?
Why does Deadpool say "You're welcome, Canada," after shooting Ryan Reynolds in the end credits?
Is a lawful good "antagonist" effective?
If the Captain's screens are out, does he switch seats with the co-pilot?
Identifying the interval from A♭ to D♯
Single word request: Harming the benefactor
validation vs test vs training accuracy, which one to compare for claiming overfit?
Potentiometer like component
Can infringement of a trademark be pursued for using a company's name in a sentence?
What is the dot in “1.2.4."
Time dilation for a moving electronic clock
Is going from continuous data to categorical always wrong?
Excess Zinc in garden soil
Is "history" a male-biased word ("his+story")?
Ban on all campaign finance?
read .pkl files from IBM object storage
2019 Community Moderator ElectionPHP Softlayer Object Storage Upload SSL errorHow do I do bulk file storage with IBM Object Storage?IBM Object Storage with PythonAuthorization Header - IBM Cloud Object StorageIBM Cloud - Object Storage , nosuchkeyhow can I save a binary file from cloud object storage to the notebook filesystem?How to set (basic http) authentication for IBM Cloud Object Storage (COS) files?How to download multiple objects from IBM Cloud Object Storage?create object with python ibm-cos-sdk get doesn't workIBM Object Storage Archive
I am trying to read '.pkl' files stored in IBM object storage into IBM cloud/jupyter notebook using function given below. I am using python 2.7.
from ibm_botocore.client import Config
import ibm_boto3
COS_ENDPOINT = "https://s3.private.us.cloud-object-
torage.appdomain.cloud"
COS_API_KEY_ID = "xxxxxxxxxxxx"
COS_AUTH_ENDPOINT = "https://iam.ng.bluemix.net/oidc/token"
COS_RESOURCE_CRN = "xxxxxxxxxxxxxxxxx"
COS_BUCKET_LOCATION = "us-geo"
# Create resource
cos = ibm_boto3.resource("s3",
ibm_api_key_id=COS_API_KEY_ID,
ibm_service_instance_id=COS_RESOURCE_CRN,
ibm_auth_endpoint=COS_AUTH_ENDPOINT,
config=Config(signature_version="oauth"),
endpoint_url=COS_ENDPOINT
)
def get_item(bucket_name, item_name):
print("Retrieving item from bucket: 0, key:
1".format(bucket_name, item_name))
try:
file = cos.Object(bucket_name, item_name).get()
print("File Contents: 0".format(file["Body"].read()))
except ClientError as be:
print("CLIENT ERROR: 0n".format(be))
except Exception as e:
print("Unable to retrieve file contents: 0".format(e))
get_item('bucket_name', 'item_name')
The function get_item is returning empty string when trying to read '.pkl' file. But using same function I am able to read a '.txt' file. The '.pkl' file is 30 MB in size so I am not able to download them into IBM cloud because of size constraint. Could anyone please let me know how to stream cloud object storage data without downloading them.
python-2.7 ibm-cloud object-storage
add a comment |
I am trying to read '.pkl' files stored in IBM object storage into IBM cloud/jupyter notebook using function given below. I am using python 2.7.
from ibm_botocore.client import Config
import ibm_boto3
COS_ENDPOINT = "https://s3.private.us.cloud-object-
torage.appdomain.cloud"
COS_API_KEY_ID = "xxxxxxxxxxxx"
COS_AUTH_ENDPOINT = "https://iam.ng.bluemix.net/oidc/token"
COS_RESOURCE_CRN = "xxxxxxxxxxxxxxxxx"
COS_BUCKET_LOCATION = "us-geo"
# Create resource
cos = ibm_boto3.resource("s3",
ibm_api_key_id=COS_API_KEY_ID,
ibm_service_instance_id=COS_RESOURCE_CRN,
ibm_auth_endpoint=COS_AUTH_ENDPOINT,
config=Config(signature_version="oauth"),
endpoint_url=COS_ENDPOINT
)
def get_item(bucket_name, item_name):
print("Retrieving item from bucket: 0, key:
1".format(bucket_name, item_name))
try:
file = cos.Object(bucket_name, item_name).get()
print("File Contents: 0".format(file["Body"].read()))
except ClientError as be:
print("CLIENT ERROR: 0n".format(be))
except Exception as e:
print("Unable to retrieve file contents: 0".format(e))
get_item('bucket_name', 'item_name')
The function get_item is returning empty string when trying to read '.pkl' file. But using same function I am able to read a '.txt' file. The '.pkl' file is 30 MB in size so I am not able to download them into IBM cloud because of size constraint. Could anyone please let me know how to stream cloud object storage data without downloading them.
python-2.7 ibm-cloud object-storage
add a comment |
I am trying to read '.pkl' files stored in IBM object storage into IBM cloud/jupyter notebook using function given below. I am using python 2.7.
from ibm_botocore.client import Config
import ibm_boto3
COS_ENDPOINT = "https://s3.private.us.cloud-object-
torage.appdomain.cloud"
COS_API_KEY_ID = "xxxxxxxxxxxx"
COS_AUTH_ENDPOINT = "https://iam.ng.bluemix.net/oidc/token"
COS_RESOURCE_CRN = "xxxxxxxxxxxxxxxxx"
COS_BUCKET_LOCATION = "us-geo"
# Create resource
cos = ibm_boto3.resource("s3",
ibm_api_key_id=COS_API_KEY_ID,
ibm_service_instance_id=COS_RESOURCE_CRN,
ibm_auth_endpoint=COS_AUTH_ENDPOINT,
config=Config(signature_version="oauth"),
endpoint_url=COS_ENDPOINT
)
def get_item(bucket_name, item_name):
print("Retrieving item from bucket: 0, key:
1".format(bucket_name, item_name))
try:
file = cos.Object(bucket_name, item_name).get()
print("File Contents: 0".format(file["Body"].read()))
except ClientError as be:
print("CLIENT ERROR: 0n".format(be))
except Exception as e:
print("Unable to retrieve file contents: 0".format(e))
get_item('bucket_name', 'item_name')
The function get_item is returning empty string when trying to read '.pkl' file. But using same function I am able to read a '.txt' file. The '.pkl' file is 30 MB in size so I am not able to download them into IBM cloud because of size constraint. Could anyone please let me know how to stream cloud object storage data without downloading them.
python-2.7 ibm-cloud object-storage
I am trying to read '.pkl' files stored in IBM object storage into IBM cloud/jupyter notebook using function given below. I am using python 2.7.
from ibm_botocore.client import Config
import ibm_boto3
COS_ENDPOINT = "https://s3.private.us.cloud-object-
torage.appdomain.cloud"
COS_API_KEY_ID = "xxxxxxxxxxxx"
COS_AUTH_ENDPOINT = "https://iam.ng.bluemix.net/oidc/token"
COS_RESOURCE_CRN = "xxxxxxxxxxxxxxxxx"
COS_BUCKET_LOCATION = "us-geo"
# Create resource
cos = ibm_boto3.resource("s3",
ibm_api_key_id=COS_API_KEY_ID,
ibm_service_instance_id=COS_RESOURCE_CRN,
ibm_auth_endpoint=COS_AUTH_ENDPOINT,
config=Config(signature_version="oauth"),
endpoint_url=COS_ENDPOINT
)
def get_item(bucket_name, item_name):
print("Retrieving item from bucket: 0, key:
1".format(bucket_name, item_name))
try:
file = cos.Object(bucket_name, item_name).get()
print("File Contents: 0".format(file["Body"].read()))
except ClientError as be:
print("CLIENT ERROR: 0n".format(be))
except Exception as e:
print("Unable to retrieve file contents: 0".format(e))
get_item('bucket_name', 'item_name')
The function get_item is returning empty string when trying to read '.pkl' file. But using same function I am able to read a '.txt' file. The '.pkl' file is 30 MB in size so I am not able to download them into IBM cloud because of size constraint. Could anyone please let me know how to stream cloud object storage data without downloading them.
python-2.7 ibm-cloud object-storage
python-2.7 ibm-cloud object-storage
asked Mar 7 at 11:07
VinayVinay
33
33
add a comment |
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55042362%2fread-pkl-files-from-ibm-object-storage%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55042362%2fread-pkl-files-from-ibm-object-storage%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown