Amazon and websocket recording best practices The Next CEO of Stack OverflowCommonly accepted best practices around code organization in JavaScriptTrying to SSH into an Amazon Ec2 instance - permission errorWhat is the simplest way to process WebSockets messages on AWS?Amazon S3 + Lambda + DynamoDB Website HostingAWS “Serverless” architecture for real time client-server messengingHow to execute Amazon Lambda functions on dedicated EC2 server?Is it possible to subscribe to a WebSocket in a serverless fashion using AWS?How would I configure an EC2 instance To Download From MYSQL and Process The Data Based On A Script?Schedule to start an EC2 instance and run a python script within itUDP/TCP server in AWS
Would a completely good Muggle be able to use a wand?
What day is it again?
Film where the government was corrupt with aliens, people sent to kill aliens are given rigged visors not showing the right aliens
What CSS properties can the br tag have?
Why the last AS PATH item always is `I` or `?`?
My ex-girlfriend uses my Apple ID to login to her iPad, do I have to give her my Apple ID password to reset it?
What difference does it make using sed with/without whitespaces?
Spaces in which all closed sets are regular closed
How do I fit a non linear curve?
Why is the US ranked as #45 in Press Freedom ratings, despite its extremely permissive free speech laws?
How to find image of a complex function with given constraints?
Are the names of these months realistic?
From jafe to El-Guest
Audio Conversion With ADS1243
Is there a way to save my career from absolute disaster?
"Eavesdropping" vs "Listen in on"
Help/tips for a first time writer?
Is there a reasonable and studied concept of reduction between regular languages?
Aggressive Under-Indexing and no data for missing index
Computationally populating tables with probability data
Is dried pee considered dirt?
What steps are necessary to read a Modern SSD in Medieval Europe?
Easy to read palindrome checker
Does destroying a Lich's phylactery destroy the soul within it?
Amazon and websocket recording best practices
The Next CEO of Stack OverflowCommonly accepted best practices around code organization in JavaScriptTrying to SSH into an Amazon Ec2 instance - permission errorWhat is the simplest way to process WebSockets messages on AWS?Amazon S3 + Lambda + DynamoDB Website HostingAWS “Serverless” architecture for real time client-server messengingHow to execute Amazon Lambda functions on dedicated EC2 server?Is it possible to subscribe to a WebSocket in a serverless fashion using AWS?How would I configure an EC2 instance To Download From MYSQL and Process The Data Based On A Script?Schedule to start an EC2 instance and run a python script within itUDP/TCP server in AWS
I'm consuming and processing websocket data from markets 24/7, and I'm currently finding my way around AWS and trying to determine best practices. This is one of those "spent so much time thinking how to do it, but never considered if I should do it". I'd like to use what AWS has to offer, even though it's probably overkill for what I'm doing currently.
At the moment, I'm simply running a single, full-time Python script which does everything from a single EC2 instance, and it interacts with a separate RDS database to store raw data. I'd like to break this apart, as I'm now going to actually work with incoming data to maintain a market orderbook "state", probably in DynamoDB (since it's free and seems to tick the necessary boxes). I don't need bleeding edge speed and response times, but would appreciate answers that additionally cover that future, potential need.
Is there a better/leaner way than having an EC2 instance running a 24/7 Python script to receive and push any received websocket data to RDS? Do Docker images/instances come into play at all in my needs?
If I push everything I receive into an SQS queue, would I use one or more lambda functions to clean up the received data, and then have it store the result on RDS?
The market websockets I record from often drop out, so I'd like to make a solution whereby websocket receiver instances/images can be duplicated, and all feed their respective data into this system, and de-duped.
If I want to work with the received data, and store the market state on DynamoDB (as well as store it in its raw format in RDS), could this all be achieved with Lambda functions?
Would I just be better off doing all the work on the inital EC2 instance/Python script, or try as best as possible to utilise SQS/Lambda/others? Are there additional components I should know about?
Thanks!
amazon-web-services design-patterns amazon-ec2 aws-lambda amazon-dynamodb
asked Mar 8 at 17:16
RobRob
13
add a comment |
I'm consuming and processing websocket data from markets 24/7, and I'm currently finding my way around AWS and trying to determine best practices. This is one of those "spent so much time thinking how to do it, but never considered if I should do it". I'd like to use what AWS has to offer, even though it's probably overkill for what I'm doing currently.
At the moment, I'm simply running a single, full-time Python script which does everything from a single EC2 instance, and it interacts with a separate RDS database to store raw data. I'd like to break this apart, as I'm now going to actually work with incoming data to maintain a market orderbook "state", probably in DynamoDB (since it's free and seems to tick the necessary boxes). I don't need bleeding edge speed and response times, but would appreciate answers that additionally cover that future, potential need.
Is there a better/leaner way than having an EC2 instance running a 24/7 Python script to receive and push any received websocket data to RDS? Do Docker images/instances come into play at all in my needs?
If I push everything I receive into an SQS queue, would I use one or more lambda functions to clean up the received data, and then have it store the result on RDS?
The market websockets I record from often drop out, so I'd like to make a solution whereby websocket receiver instances/images can be duplicated, and all feed their respective data into this system, and de-duped.
If I want to work with the received data, and store the market state on DynamoDB (as well as store it in its raw format in RDS), could this all be achieved with Lambda functions?
Would I just be better off doing all the work on the inital EC2 instance/Python script, or try as best as possible to utilise SQS/Lambda/others? Are there additional components I should know about?
Thanks!
amazon-web-services design-patterns amazon-ec2 aws-lambda amazon-dynamodb
asked Mar 8 at 17:16
RobRob
13
add a comment |
I'm consuming and processing websocket data from markets 24/7, and I'm currently finding my way around AWS and trying to determine best practices. This is one of those "spent so much time thinking how to do it, but never considered if I should do it". I'd like to use what AWS has to offer, even though it's probably overkill for what I'm doing currently.
At the moment, I'm simply running a single, full-time Python script which does everything from a single EC2 instance, and it interacts with a separate RDS database to store raw data. I'd like to break this apart, as I'm now going to actually work with incoming data to maintain a market orderbook "state", probably in DynamoDB (since it's free and seems to tick the necessary boxes). I don't need bleeding edge speed and response times, but would appreciate answers that additionally cover that future, potential need.
Is there a better/leaner way than having an EC2 instance running a 24/7 Python script to receive and push any received websocket data to RDS? Do Docker images/instances come into play at all in my needs?
If I push everything I receive into an SQS queue, would I use one or more lambda functions to clean up the received data, and then have it store the result on RDS?
The market websockets I record from often drop out, so I'd like to make a solution whereby websocket receiver instances/images can be duplicated, and all feed their respective data into this system, and de-duped.
If I want to work with the received data, and store the market state on DynamoDB (as well as store it in its raw format in RDS), could this all be achieved with Lambda functions?
Would I just be better off doing all the work on the inital EC2 instance/Python script, or try as best as possible to utilise SQS/Lambda/others? Are there additional components I should know about?
Thanks!
amazon-web-services design-patterns amazon-ec2 aws-lambda amazon-dynamodb
asked Mar 8 at 17:16
RobRob
13
I'm consuming and processing websocket data from markets 24/7, and I'm currently finding my way around AWS and trying to determine best practices. This is one of those "spent so much time thinking how to do it, but never considered if I should do it". I'd like to use what AWS has to offer, even though it's probably overkill for what I'm doing currently.
At the moment, I'm simply running a single, full-time Python script which does everything from a single EC2 instance, and it interacts with a separate RDS database to store raw data. I'd like to break this apart, as I'm now going to actually work with incoming data to maintain a market orderbook "state", probably in DynamoDB (since it's free and seems to tick the necessary boxes). I don't need bleeding edge speed and response times, but would appreciate answers that additionally cover that future, potential need.
Is there a better/leaner way than having an EC2 instance running a 24/7 Python script to receive and push any received websocket data to RDS? Do Docker images/instances come into play at all in my needs?
If I push everything I receive into an SQS queue, would I use one or more lambda functions to clean up the received data, and then have it store the result on RDS?
The market websockets I record from often drop out, so I'd like to make a solution whereby websocket receiver instances/images can be duplicated, and all feed their respective data into this system, and de-duped.
If I want to work with the received data, and store the market state on DynamoDB (as well as store it in its raw format in RDS), could this all be achieved with Lambda functions?
Would I just be better off doing all the work on the inital EC2 instance/Python script, or try as best as possible to utilise SQS/Lambda/others? Are there additional components I should know about?
Thanks!
amazon-web-services design-patterns amazon-ec2 aws-lambda amazon-dynamodb
amazon-web-services design-patterns amazon-ec2 aws-lambda amazon-dynamodb
asked Mar 8 at 17:16
RobRob
13
asked Mar 8 at 17:16
RobRob
13
asked Mar 8 at 17:16
RobRob
13
asked Mar 8 at 17:16
RobRob
13
asked Mar 8 at 17:16
RobRob
13
13
add a comment |
add a comment |
0
active
oldest
votes
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55068004%2famazon-and-websocket-recording-best-practices%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55068004%2famazon-and-websocket-recording-best-practices%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
