AWS copy command does not work in EC2 instance with python psycopg2How to install psycopg2 with “pip” on Python?scp (secure copy) to ec2 instance without passwordChange key pair for ec2 instanceTrying to SSH into an Amazon Ec2 instance - permission errorCopying data from S3 to AWS redshift using python and psycopg2Redshift copy command from S3 works, but no data uploadedCopy Redshift table from S3 csv file using Python?How to execute AWS S3 to Redshift Copy command from SQL script?vb.net stuck on AWS Redshift Copy commandConnecting to a remote Redshift cluster from an EC2 with psycopg2 via UserData in CloudFormation template
Biological Blimps: Propulsion
Quoting Keynes in a lecture
Can I say "fingers" when referring to toes?
15% tax on $7.5k earnings. Is that right?
Can disgust be a key component of horror?
Is aluminum electrical wire used on aircraft?
Why is it that I can sometimes guess the next note?
How does a computer interpret real numbers?
What is the highest possible scrabble score for placing a single tile
Yosemite Fire Rings - What to Expect?
Is this toilet slogan correct usage of the English language?
Temporarily disable WLAN internet access for children, but allow it for adults
Strong empirical falsification of quantum mechanics based on vacuum energy density
Does the Linux kernel need a file system to run?
Why Shazam when there is already Superman?
Invalid date error by date command
Unexpected behavior of the procedure `Area` on the object 'Polygon'
Why did the EU agree to delay the Brexit deadline?
Plot of a tornado-shaped surface
Open a doc from terminal, but not by its name
What if a revenant (monster) gains fire resistance?
Does Doodling or Improvising on the Piano Have Any Benefits?
Is there a RAID 0 Equivalent for RAM?
Hero deduces identity of a killer
AWS copy command does not work in EC2 instance with python psycopg2
How to install psycopg2 with “pip” on Python?scp (secure copy) to ec2 instance without passwordChange key pair for ec2 instanceTrying to SSH into an Amazon Ec2 instance - permission errorCopying data from S3 to AWS redshift using python and psycopg2Redshift copy command from S3 works, but no data uploadedCopy Redshift table from S3 csv file using Python?How to execute AWS S3 to Redshift Copy command from SQL script?vb.net stuck on AWS Redshift Copy commandConnecting to a remote Redshift cluster from an EC2 with psycopg2 via UserData in CloudFormation template
When I'm using SQL workbench, the AWS copy command works well.
The purpose is to copy S3 data to Redshift table.
Here's the copy command I'm using:
copy oao_features_usbank_v2 from 's3://my_bucket/test_data.csv' iam_role 'arn:aws:iam::accountid:role/my_role' CSV IGNOREHEADER 1 MAXERROR 10 DELIMITER as '|';
Then when using python psycopg2. I have tried multiple connection methods, none of them work. Such as:
conn_string = 'postgresql://username:pwd@host:port/db'
copy_query = 'copy oao_features_usbank_v2 from 's3://my_bucket/test_data.csv' iam_role 'arn:aws:iam::accountid:role/my_role' CSV IGNOREHEADER 1 MAXERROR 10 DELIMITER as '|';'
with psycopg2.connect(conn_string) as conn:
with conn.cursor() as curs:
curs.execute(copy_query)
conn.commit()
python-3.x amazon-s3 amazon-ec2 amazon-redshift psycopg2
asked Mar 8 at 1:49
Cherry WuCherry Wu
92531937
add a comment |
When I'm using SQL workbench, the AWS copy command works well.
The purpose is to copy S3 data to Redshift table.
Here's the copy command I'm using:
copy oao_features_usbank_v2 from 's3://my_bucket/test_data.csv' iam_role 'arn:aws:iam::accountid:role/my_role' CSV IGNOREHEADER 1 MAXERROR 10 DELIMITER as '|';
Then when using python psycopg2. I have tried multiple connection methods, none of them work. Such as:
conn_string = 'postgresql://username:pwd@host:port/db'
copy_query = 'copy oao_features_usbank_v2 from 's3://my_bucket/test_data.csv' iam_role 'arn:aws:iam::accountid:role/my_role' CSV IGNOREHEADER 1 MAXERROR 10 DELIMITER as '|';'
with psycopg2.connect(conn_string) as conn:
with conn.cursor() as curs:
curs.execute(copy_query)
conn.commit()
python-3.x amazon-s3 amazon-ec2 amazon-redshift psycopg2
asked Mar 8 at 1:49
Cherry WuCherry Wu
92531937
add a comment |
When I'm using SQL workbench, the AWS copy command works well.
The purpose is to copy S3 data to Redshift table.
Here's the copy command I'm using:
copy oao_features_usbank_v2 from 's3://my_bucket/test_data.csv' iam_role 'arn:aws:iam::accountid:role/my_role' CSV IGNOREHEADER 1 MAXERROR 10 DELIMITER as '|';
Then when using python psycopg2. I have tried multiple connection methods, none of them work. Such as:
conn_string = 'postgresql://username:pwd@host:port/db'
copy_query = 'copy oao_features_usbank_v2 from 's3://my_bucket/test_data.csv' iam_role 'arn:aws:iam::accountid:role/my_role' CSV IGNOREHEADER 1 MAXERROR 10 DELIMITER as '|';'
with psycopg2.connect(conn_string) as conn:
with conn.cursor() as curs:
curs.execute(copy_query)
conn.commit()
python-3.x amazon-s3 amazon-ec2 amazon-redshift psycopg2
asked Mar 8 at 1:49
Cherry WuCherry Wu
92531937
When I'm using SQL workbench, the AWS copy command works well.
The purpose is to copy S3 data to Redshift table.
Here's the copy command I'm using:
copy oao_features_usbank_v2 from 's3://my_bucket/test_data.csv' iam_role 'arn:aws:iam::accountid:role/my_role' CSV IGNOREHEADER 1 MAXERROR 10 DELIMITER as '|';
Then when using python psycopg2. I have tried multiple connection methods, none of them work. Such as:
conn_string = 'postgresql://username:pwd@host:port/db'
copy_query = 'copy oao_features_usbank_v2 from 's3://my_bucket/test_data.csv' iam_role 'arn:aws:iam::accountid:role/my_role' CSV IGNOREHEADER 1 MAXERROR 10 DELIMITER as '|';'
with psycopg2.connect(conn_string) as conn:
with conn.cursor() as curs:
curs.execute(copy_query)
conn.commit()
python-3.x amazon-s3 amazon-ec2 amazon-redshift psycopg2
python-3.x amazon-s3 amazon-ec2 amazon-redshift psycopg2
asked Mar 8 at 1:49
Cherry WuCherry Wu
92531937
asked Mar 8 at 1:49
Cherry WuCherry Wu
92531937
asked Mar 8 at 1:49
Cherry WuCherry Wu
92531937
asked Mar 8 at 1:49
Cherry WuCherry Wu
92531937
asked Mar 8 at 1:49
Cherry WuCherry Wu
92531937
92531937
add a comment |
add a comment |
2 Answers
2
active
oldest
votes
Try something like this:
import psycopg2
db = "your_database"
host = "redshift_end_point"
port = "5439"
user = "your_user_here"
pwd = "your_password_here"
connstr = "dbname=%s host=%s port=%s user=%s password=%s" % (db, host, port, user, pwd)
con = psycopg2.connect(connstr)
The Redshift end point can be found in the AWS console. It will look something like:
clustername.sofhsdl34534.us-east-1.redshift.amazonaws.com
Your EC2 instance should also have a Security Group attached that allows connecting to Redshift.
answered Mar 8 at 2:52
Nathan GriffithsNathan Griffiths
8,76022239
add a comment |
Try inserting your copy_query into multiple line string like below:
copy_query = """copy oao_features_usbank_v2 from 's3://my_bucket/test_data.csv' iam_role 'arn:aws:iam::accountid:role/my_role' CSV IGNOREHEADER 1 MAXERROR 10 DELIMITER as '|';' """
So that python code will take all the string inside of three quotes as your copy command
answered yesterday
Vin OdhVin Odh
26915
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55055598%2faws-copy-command-does-not-work-in-ec2-instance-with-python-psycopg2%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
Try something like this:
import psycopg2
db = "your_database"
host = "redshift_end_point"
port = "5439"
user = "your_user_here"
pwd = "your_password_here"
connstr = "dbname=%s host=%s port=%s user=%s password=%s" % (db, host, port, user, pwd)
con = psycopg2.connect(connstr)
The Redshift end point can be found in the AWS console. It will look something like:
clustername.sofhsdl34534.us-east-1.redshift.amazonaws.com
Your EC2 instance should also have a Security Group attached that allows connecting to Redshift.
answered Mar 8 at 2:52
Nathan GriffithsNathan Griffiths
8,76022239
add a comment |
Try something like this:
import psycopg2
db = "your_database"
host = "redshift_end_point"
port = "5439"
user = "your_user_here"
pwd = "your_password_here"
connstr = "dbname=%s host=%s port=%s user=%s password=%s" % (db, host, port, user, pwd)
con = psycopg2.connect(connstr)
The Redshift end point can be found in the AWS console. It will look something like:
clustername.sofhsdl34534.us-east-1.redshift.amazonaws.com
Your EC2 instance should also have a Security Group attached that allows connecting to Redshift.
answered Mar 8 at 2:52
Nathan GriffithsNathan Griffiths
8,76022239
add a comment |
Try something like this:
import psycopg2
db = "your_database"
host = "redshift_end_point"
port = "5439"
user = "your_user_here"
pwd = "your_password_here"
connstr = "dbname=%s host=%s port=%s user=%s password=%s" % (db, host, port, user, pwd)
con = psycopg2.connect(connstr)
The Redshift end point can be found in the AWS console. It will look something like:
clustername.sofhsdl34534.us-east-1.redshift.amazonaws.com
Your EC2 instance should also have a Security Group attached that allows connecting to Redshift.
answered Mar 8 at 2:52
Nathan GriffithsNathan Griffiths
8,76022239
Try something like this:
import psycopg2
db = "your_database"
host = "redshift_end_point"
port = "5439"
user = "your_user_here"
pwd = "your_password_here"
connstr = "dbname=%s host=%s port=%s user=%s password=%s" % (db, host, port, user, pwd)
con = psycopg2.connect(connstr)
The Redshift end point can be found in the AWS console. It will look something like:
clustername.sofhsdl34534.us-east-1.redshift.amazonaws.com
Your EC2 instance should also have a Security Group attached that allows connecting to Redshift.
answered Mar 8 at 2:52
Nathan GriffithsNathan Griffiths
8,76022239
answered Mar 8 at 2:52
Nathan GriffithsNathan Griffiths
8,76022239
answered Mar 8 at 2:52
Nathan GriffithsNathan Griffiths
8,76022239
answered Mar 8 at 2:52
Nathan GriffithsNathan Griffiths
8,76022239
8,76022239
add a comment |
add a comment |
Try inserting your copy_query into multiple line string like below:
copy_query = """copy oao_features_usbank_v2 from 's3://my_bucket/test_data.csv' iam_role 'arn:aws:iam::accountid:role/my_role' CSV IGNOREHEADER 1 MAXERROR 10 DELIMITER as '|';' """
So that python code will take all the string inside of three quotes as your copy command
answered yesterday
Vin OdhVin Odh
26915
add a comment |
Try inserting your copy_query into multiple line string like below:
copy_query = """copy oao_features_usbank_v2 from 's3://my_bucket/test_data.csv' iam_role 'arn:aws:iam::accountid:role/my_role' CSV IGNOREHEADER 1 MAXERROR 10 DELIMITER as '|';' """
So that python code will take all the string inside of three quotes as your copy command
answered yesterday
Vin OdhVin Odh
26915
add a comment |
Try inserting your copy_query into multiple line string like below:
copy_query = """copy oao_features_usbank_v2 from 's3://my_bucket/test_data.csv' iam_role 'arn:aws:iam::accountid:role/my_role' CSV IGNOREHEADER 1 MAXERROR 10 DELIMITER as '|';' """
So that python code will take all the string inside of three quotes as your copy command
answered yesterday
Vin OdhVin Odh
26915
Try inserting your copy_query into multiple line string like below:
copy_query = """copy oao_features_usbank_v2 from 's3://my_bucket/test_data.csv' iam_role 'arn:aws:iam::accountid:role/my_role' CSV IGNOREHEADER 1 MAXERROR 10 DELIMITER as '|';' """
So that python code will take all the string inside of three quotes as your copy command
answered yesterday
Vin OdhVin Odh
26915
answered yesterday
Vin OdhVin Odh
26915
answered yesterday
Vin OdhVin Odh
26915
answered yesterday
Vin OdhVin Odh
26915
26915
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55055598%2faws-copy-command-does-not-work-in-ec2-instance-with-python-psycopg2%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
