How do I code an standalone Athena API start_query_execution asynchronous call in Python2019 Community Moderator ElectionHow can I represent an 'Enum' in Python?How do I use raw_input in Python 3How to install pip with Python 3?Call a REST API in PHPWhat does “SyntaxError: Missing parentheses in call to 'print'” mean in Python?Return JSON from Athena Query via APIMultiple Await in Python Async FunctionAWS Athena asynchronous pagination in Pythonpython asyncronous images download (multiple urls)TypeError: _request() got an unexpected keyword argument 'cookies' (aiohttp)

Is going from continuous data to categorical always wrong?

What is the dot in “1.2.4."

Why doesn't the EU now just force the UK to choose between referendum and no-deal?

Is all copper pipe pretty much the same?

What has been your most complicated TikZ drawing?

Deleting missing values from a dataset

What is the difference between "shut" and "close"?

Time dilation for a moving electronic clock

If the Captain's screens are out, does he switch seats with the co-pilot?

The three point beverage

What does it mean when multiple 々 marks follow a 、?

Should we release the security issues we found in our product as CVE or we can just update those on weekly release notes?

Best approach to update all entries in a list that is paginated?

Good allowance savings plan?

Coworker uses her breast-pump everywhere in the office

Humans have energy, but not water. What happens?

Welcoming 2019 Pi day: How to draw the letter π?

Is it illegal in Germany to take sick leave if you caused your own illness with food?

Counter-example to the existence of left Bousfield localization of combinatorial model category

Want to switch to tankless, but can I use my existing wiring?

Does Linux have system calls to access all the features of the file systems it supports?

What exactly is the purpose of connection links straped between the rocket and the launch pad

How does Dispel Magic work against Stoneskin?

Is it true that real estate prices mainly go up?



How do I code an standalone Athena API start_query_execution asynchronous call in Python



2019 Community Moderator ElectionHow can I represent an 'Enum' in Python?How do I use raw_input in Python 3How to install pip with Python 3?Call a REST API in PHPWhat does “SyntaxError: Missing parentheses in call to 'print'” mean in Python?Return JSON from Athena Query via APIMultiple Await in Python Async FunctionAWS Athena asynchronous pagination in Pythonpython asyncronous images download (multiple urls)TypeError: _request() got an unexpected keyword argument 'cookies' (aiohttp)










0















I'm attempting to code an asyncronous API call to use athena to run the action - start_query_execution. I am not using boto3 or any other library.



Any support on this query would be appreciated.



I've tried to change the following code by Mathew Marcus, which uses the equivalent to the lambda.invoke using what I am wanting to do. However, I have been unsuccessful in converting the code to an athena.start_query_execution instance.



SAMPLE OF MY CURRENT CODE:



request_parameters = ''
request_parameters += 'QueryString: "SELECT * FROM <table_name> LIMIT 1;",'
request_parameters += 'ResultConfiguration: "OutputLocation": "outputlocation"'
request_parameters += ''

creds = Session().get_credentials()
ATHENA_ENDPOINT_BASE = 'https://athena.region.amazonaws.com/'

def create_signed_headers(url, request_parameters):
host_segments = urlparse(url).netloc.split('.')
print('6. host_segments: ' , host_segments)
service = host_segments[0]
print('7. service: ' , service)
region = host_segments[1]
print('8. region: ' , region)
request = AWSRequest(method ='POST',
url = url,
data=request_parameters)
print('9. request: ', request)
SigV4Auth(creds, service, region).add_auth(request)
return dict(request.headers.items())

#query-string and result-configuration[output location] are *REQUIRED for start query execution

request_parameters

async def start_query_execution(url, request_parameters ,session):
signed_headers = create_signed_headers(url, request_parameters)
print('10. signed_headers: ' , signed_headers)
async with session.post(url,
json = request_parameters,
headers = signed_headers) as response:
return await response.json()

def generate_queries(name_and_request, base_url, session):
for name, request_parameters in name_and_request:
url = join(base_url, name, 'queries')
print('5. url: ' , url)
yield start_query_execution(url, request_parameters, session)

def start_all(name_and_request, region = 'eu-west-1'):
base_url = ATHENA_ENDPOINT_BASE.format(region = region)
print('3. base_url: ' , base_url)
async def wrapped():
async with ClientSession(raise_for_status=True) as session:
queries = generate_queries(name_and_request,
base_url,
session)
print('4. queries: ', queries)
return await asyncio.gather(*queries)

return asyncio.get_event_loop().run_until_complete(wrapped())

def main():
name = 'hello-world-'
name_and_request = ((name.format(i), dict(hello=i)) for i in range(10))
print('1. name_and_request: ' , name_and_request)

athena_responses = start_all(name_and_request)
print('2. athena_responses: ', athena_responses)

if __name__ == '__main__':
main()









share|improve this question
























  • I think you need to provide at least some minimal context about what it is that is not working. Writing your own AWS client is a pretty big task, and anything could be wrong from request signing to spelling mistakes in the API calls. I would strongly urge you not to do it and just use boto3.

    – Theo
    Mar 8 at 19:31











  • Thank you for your response @Theo, my initial attempts at finding solution did involve and still does involve using boto3, however there has been indications that boto3 does not support asynchronous calls within the context of the version of Python. Hence my attempt at using a direct call. As to providing context of what is not working, I agree there is not enough here. The reason for this is due to the fact that I could not isolate the exact origin of the error.

    – Brent
    Mar 9 at 20:08
















0















I'm attempting to code an asyncronous API call to use athena to run the action - start_query_execution. I am not using boto3 or any other library.



Any support on this query would be appreciated.



I've tried to change the following code by Mathew Marcus, which uses the equivalent to the lambda.invoke using what I am wanting to do. However, I have been unsuccessful in converting the code to an athena.start_query_execution instance.



SAMPLE OF MY CURRENT CODE:



request_parameters = ''
request_parameters += 'QueryString: "SELECT * FROM <table_name> LIMIT 1;",'
request_parameters += 'ResultConfiguration: "OutputLocation": "outputlocation"'
request_parameters += ''

creds = Session().get_credentials()
ATHENA_ENDPOINT_BASE = 'https://athena.region.amazonaws.com/'

def create_signed_headers(url, request_parameters):
host_segments = urlparse(url).netloc.split('.')
print('6. host_segments: ' , host_segments)
service = host_segments[0]
print('7. service: ' , service)
region = host_segments[1]
print('8. region: ' , region)
request = AWSRequest(method ='POST',
url = url,
data=request_parameters)
print('9. request: ', request)
SigV4Auth(creds, service, region).add_auth(request)
return dict(request.headers.items())

#query-string and result-configuration[output location] are *REQUIRED for start query execution

request_parameters

async def start_query_execution(url, request_parameters ,session):
signed_headers = create_signed_headers(url, request_parameters)
print('10. signed_headers: ' , signed_headers)
async with session.post(url,
json = request_parameters,
headers = signed_headers) as response:
return await response.json()

def generate_queries(name_and_request, base_url, session):
for name, request_parameters in name_and_request:
url = join(base_url, name, 'queries')
print('5. url: ' , url)
yield start_query_execution(url, request_parameters, session)

def start_all(name_and_request, region = 'eu-west-1'):
base_url = ATHENA_ENDPOINT_BASE.format(region = region)
print('3. base_url: ' , base_url)
async def wrapped():
async with ClientSession(raise_for_status=True) as session:
queries = generate_queries(name_and_request,
base_url,
session)
print('4. queries: ', queries)
return await asyncio.gather(*queries)

return asyncio.get_event_loop().run_until_complete(wrapped())

def main():
name = 'hello-world-'
name_and_request = ((name.format(i), dict(hello=i)) for i in range(10))
print('1. name_and_request: ' , name_and_request)

athena_responses = start_all(name_and_request)
print('2. athena_responses: ', athena_responses)

if __name__ == '__main__':
main()









share|improve this question
























  • I think you need to provide at least some minimal context about what it is that is not working. Writing your own AWS client is a pretty big task, and anything could be wrong from request signing to spelling mistakes in the API calls. I would strongly urge you not to do it and just use boto3.

    – Theo
    Mar 8 at 19:31











  • Thank you for your response @Theo, my initial attempts at finding solution did involve and still does involve using boto3, however there has been indications that boto3 does not support asynchronous calls within the context of the version of Python. Hence my attempt at using a direct call. As to providing context of what is not working, I agree there is not enough here. The reason for this is due to the fact that I could not isolate the exact origin of the error.

    – Brent
    Mar 9 at 20:08














0












0








0


1






I'm attempting to code an asyncronous API call to use athena to run the action - start_query_execution. I am not using boto3 or any other library.



Any support on this query would be appreciated.



I've tried to change the following code by Mathew Marcus, which uses the equivalent to the lambda.invoke using what I am wanting to do. However, I have been unsuccessful in converting the code to an athena.start_query_execution instance.



SAMPLE OF MY CURRENT CODE:



request_parameters = ''
request_parameters += 'QueryString: "SELECT * FROM <table_name> LIMIT 1;",'
request_parameters += 'ResultConfiguration: "OutputLocation": "outputlocation"'
request_parameters += ''

creds = Session().get_credentials()
ATHENA_ENDPOINT_BASE = 'https://athena.region.amazonaws.com/'

def create_signed_headers(url, request_parameters):
host_segments = urlparse(url).netloc.split('.')
print('6. host_segments: ' , host_segments)
service = host_segments[0]
print('7. service: ' , service)
region = host_segments[1]
print('8. region: ' , region)
request = AWSRequest(method ='POST',
url = url,
data=request_parameters)
print('9. request: ', request)
SigV4Auth(creds, service, region).add_auth(request)
return dict(request.headers.items())

#query-string and result-configuration[output location] are *REQUIRED for start query execution

request_parameters

async def start_query_execution(url, request_parameters ,session):
signed_headers = create_signed_headers(url, request_parameters)
print('10. signed_headers: ' , signed_headers)
async with session.post(url,
json = request_parameters,
headers = signed_headers) as response:
return await response.json()

def generate_queries(name_and_request, base_url, session):
for name, request_parameters in name_and_request:
url = join(base_url, name, 'queries')
print('5. url: ' , url)
yield start_query_execution(url, request_parameters, session)

def start_all(name_and_request, region = 'eu-west-1'):
base_url = ATHENA_ENDPOINT_BASE.format(region = region)
print('3. base_url: ' , base_url)
async def wrapped():
async with ClientSession(raise_for_status=True) as session:
queries = generate_queries(name_and_request,
base_url,
session)
print('4. queries: ', queries)
return await asyncio.gather(*queries)

return asyncio.get_event_loop().run_until_complete(wrapped())

def main():
name = 'hello-world-'
name_and_request = ((name.format(i), dict(hello=i)) for i in range(10))
print('1. name_and_request: ' , name_and_request)

athena_responses = start_all(name_and_request)
print('2. athena_responses: ', athena_responses)

if __name__ == '__main__':
main()









share|improve this question
















I'm attempting to code an asyncronous API call to use athena to run the action - start_query_execution. I am not using boto3 or any other library.



Any support on this query would be appreciated.



I've tried to change the following code by Mathew Marcus, which uses the equivalent to the lambda.invoke using what I am wanting to do. However, I have been unsuccessful in converting the code to an athena.start_query_execution instance.



SAMPLE OF MY CURRENT CODE:



request_parameters = ''
request_parameters += 'QueryString: "SELECT * FROM <table_name> LIMIT 1;",'
request_parameters += 'ResultConfiguration: "OutputLocation": "outputlocation"'
request_parameters += ''

creds = Session().get_credentials()
ATHENA_ENDPOINT_BASE = 'https://athena.region.amazonaws.com/'

def create_signed_headers(url, request_parameters):
host_segments = urlparse(url).netloc.split('.')
print('6. host_segments: ' , host_segments)
service = host_segments[0]
print('7. service: ' , service)
region = host_segments[1]
print('8. region: ' , region)
request = AWSRequest(method ='POST',
url = url,
data=request_parameters)
print('9. request: ', request)
SigV4Auth(creds, service, region).add_auth(request)
return dict(request.headers.items())

#query-string and result-configuration[output location] are *REQUIRED for start query execution

request_parameters

async def start_query_execution(url, request_parameters ,session):
signed_headers = create_signed_headers(url, request_parameters)
print('10. signed_headers: ' , signed_headers)
async with session.post(url,
json = request_parameters,
headers = signed_headers) as response:
return await response.json()

def generate_queries(name_and_request, base_url, session):
for name, request_parameters in name_and_request:
url = join(base_url, name, 'queries')
print('5. url: ' , url)
yield start_query_execution(url, request_parameters, session)

def start_all(name_and_request, region = 'eu-west-1'):
base_url = ATHENA_ENDPOINT_BASE.format(region = region)
print('3. base_url: ' , base_url)
async def wrapped():
async with ClientSession(raise_for_status=True) as session:
queries = generate_queries(name_and_request,
base_url,
session)
print('4. queries: ', queries)
return await asyncio.gather(*queries)

return asyncio.get_event_loop().run_until_complete(wrapped())

def main():
name = 'hello-world-'
name_and_request = ((name.format(i), dict(hello=i)) for i in range(10))
print('1. name_and_request: ' , name_and_request)

athena_responses = start_all(name_and_request)
print('2. athena_responses: ', athena_responses)

if __name__ == '__main__':
main()






python-3.x amazon-web-services api call amazon-athena






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Mar 7 at 14:26







Brent

















asked Mar 7 at 10:57









Brent Brent

133




133












  • I think you need to provide at least some minimal context about what it is that is not working. Writing your own AWS client is a pretty big task, and anything could be wrong from request signing to spelling mistakes in the API calls. I would strongly urge you not to do it and just use boto3.

    – Theo
    Mar 8 at 19:31











  • Thank you for your response @Theo, my initial attempts at finding solution did involve and still does involve using boto3, however there has been indications that boto3 does not support asynchronous calls within the context of the version of Python. Hence my attempt at using a direct call. As to providing context of what is not working, I agree there is not enough here. The reason for this is due to the fact that I could not isolate the exact origin of the error.

    – Brent
    Mar 9 at 20:08


















  • I think you need to provide at least some minimal context about what it is that is not working. Writing your own AWS client is a pretty big task, and anything could be wrong from request signing to spelling mistakes in the API calls. I would strongly urge you not to do it and just use boto3.

    – Theo
    Mar 8 at 19:31











  • Thank you for your response @Theo, my initial attempts at finding solution did involve and still does involve using boto3, however there has been indications that boto3 does not support asynchronous calls within the context of the version of Python. Hence my attempt at using a direct call. As to providing context of what is not working, I agree there is not enough here. The reason for this is due to the fact that I could not isolate the exact origin of the error.

    – Brent
    Mar 9 at 20:08

















I think you need to provide at least some minimal context about what it is that is not working. Writing your own AWS client is a pretty big task, and anything could be wrong from request signing to spelling mistakes in the API calls. I would strongly urge you not to do it and just use boto3.

– Theo
Mar 8 at 19:31





I think you need to provide at least some minimal context about what it is that is not working. Writing your own AWS client is a pretty big task, and anything could be wrong from request signing to spelling mistakes in the API calls. I would strongly urge you not to do it and just use boto3.

– Theo
Mar 8 at 19:31













Thank you for your response @Theo, my initial attempts at finding solution did involve and still does involve using boto3, however there has been indications that boto3 does not support asynchronous calls within the context of the version of Python. Hence my attempt at using a direct call. As to providing context of what is not working, I agree there is not enough here. The reason for this is due to the fact that I could not isolate the exact origin of the error.

– Brent
Mar 9 at 20:08






Thank you for your response @Theo, my initial attempts at finding solution did involve and still does involve using boto3, however there has been indications that boto3 does not support asynchronous calls within the context of the version of Python. Hence my attempt at using a direct call. As to providing context of what is not working, I agree there is not enough here. The reason for this is due to the fact that I could not isolate the exact origin of the error.

– Brent
Mar 9 at 20:08













0






active

oldest

votes











Your Answer






StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);













draft saved

draft discarded


















StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55042157%2fhow-do-i-code-an-standalone-athena-api-start-query-execution-asynchronous-call-i%23new-answer', 'question_page');

);

Post as a guest















Required, but never shown

























0






active

oldest

votes








0






active

oldest

votes









active

oldest

votes






active

oldest

votes















draft saved

draft discarded
















































Thanks for contributing an answer to Stack Overflow!


  • Please be sure to answer the question. Provide details and share your research!

But avoid


  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55042157%2fhow-do-i-code-an-standalone-athena-api-start-query-execution-asynchronous-call-i%23new-answer', 'question_page');

);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







Popular posts from this blog

Identity Server 4 is not redirecting to Angular app after login2019 Community Moderator ElectionIdentity Server 4 and dockerIdentityserver implicit flow unauthorized_clientIdentityServer Hybrid Flow - Access Token is null after user successful loginIdentity Server to MVC client : Page Redirect After loginLogin with Steam OpenId(oidc-client-js)Identity Server 4+.NET Core 2.0 + IdentityIdentityServer4 post-login redirect not working in Edge browserCall to IdentityServer4 generates System.NullReferenceException: Object reference not set to an instance of an objectIdentityServer4 without HTTPS not workingHow to get Authorization code from identity server without login form

2005 Ahvaz unrest Contents Background Causes Casualties Aftermath See also References Navigation menue"At Least 10 Are Killed by Bombs in Iran""Iran"Archived"Arab-Iranians in Iran to make April 15 'Day of Fury'"State of Mind, State of Order: Reactions to Ethnic Unrest in the Islamic Republic of Iran.10.1111/j.1754-9469.2008.00028.x"Iran hangs Arab separatists"Iran Overview from ArchivedConstitution of the Islamic Republic of Iran"Tehran puzzled by forged 'riots' letter""Iran and its minorities: Down in the second class""Iran: Handling Of Ahvaz Unrest Could End With Televised Confessions""Bombings Rock Iran Ahead of Election""Five die in Iran ethnic clashes""Iran: Need for restraint as anniversary of unrest in Khuzestan approaches"Archived"Iranian Sunni protesters killed in clashes with security forces"Archived

Can't initialize raids on a new ASUS Prime B360M-A motherboard2019 Community Moderator ElectionSimilar to RAID config yet more like mirroring solution?Can't get motherboard serial numberWhy does the BIOS entry point start with a WBINVD instruction?UEFI performance Asus Maximus V Extreme