wolf4earth

wolf4earth

AWS: Best practices implementing fan-out with SNS and SQS

At work we plan to replace a totally overkill Kafka instance with a combination of SNS and SQS. I don’t want to get into a discussion on the feature difference between these two, for our particular use-case SNS and SQS are a good fit.

Let me lay it out real quick:

  • a single application publishes events to a single SNS topic
  • multiple other apps are interested in these events
  • for each interested app a SQS queue is created which subscribes to the SNS topic

This way we effectively implemented a fan-out pattern using SNS and SQS. With a queue for each subscriber.

The thing I’m wondering now - and wasn’t able to find any meaningful content on - is how to other people are running this particular setup. We will setup most of this up using Terraform, and while this is perfectly fine for the SNS topic I’m not so 100% certain about the SQS queues.

As we have a bunch of apps interested in these events - with a single SQS queue for each one - setting the SQS queues up in Terraform would mean we would have to remember creating a new queue for each new app which wants these events.

From this perspective it seems easier to have some shared logic which provisions a queue for each new app as soon as this app wants to start consuming events. As we’re running in Kubernetes this would most likely be an initContainer which ensures the SQS queue exists for this particular service.

Now here is the thing: while this sounds good on paper, I have no idea if this actually is a good idea (or maybe even an anti-pattern?). Do you have any experience with an approach like this, or input on how you’re running a similar setup at work?

Where Next?

Popular Backend topics Top

chasekaylee
Hi there everyone! Recently, I have fallen in love with programming with Elixir and have been having so much fun with it. I have been do...
New
IhorYachmenov
Hello. I have an iOS app where needs a proxying website through private server(HTTP / HTTPS proxy), but its idea each time has some trou...
New
Jsdr3398
I’m trying to create a router where everything is in a collection of routes (similar to how I do my routes in expressjs). But it doesn’t ...
New
New
sampu
I have a use case where a client is invoking a Rest endpoint via a load balancer, which in turn invokes a third party endpoint which is r...
New
Fl4m3Ph03n1x
Background While playing around with dialyzer, typespecs and currying, I was able to create an example of a false positive in dialyzer. ...
New
Fl4m3Ph03n1x
Background So, I am playing around with a concept named “NewType” and I am taking inspiration from languages like F# and Scala. My objec...
New
osbre
Hello everyone I’m trying to implement a “magic link” or “one-time login link” functionality I wonder what a secure way to implement it...
New
AstonJ
If when trying to create (or recreate) your dev db with rails db:create you are getting: PG::ConnectionBad: connection to server on soc...
New
Fl4m3Ph03n1x
Background I have a module that uses TypedStruct to create structs. This is the code: defmodule Shared.Data.Authorization do @moduledo...
New

Other popular topics Top

AstonJ
Or looking forward to? :nerd_face:
483 11975 256
New
Exadra37
Please tell us what is your preferred monitor setup for programming(not gaming) and why you have chosen it. Does your monitor have eye p...
New
DevotionGeo
I know that -t flag is used along with -i flag for getting an interactive shell. But I cannot digest what the man page for docker run com...
New
AstonJ
There’s a whole world of custom keycaps out there that I didn’t know existed! Check out all of our Keycaps threads here: https://forum....
New
AstonJ
Just done a fresh install of macOS Big Sur and on installing Erlang I am getting: asdf install erlang 23.1.2 Configure failed. checking ...
New
First poster: bot
zig/http.zig at 7cf2cbb33ef34c1d211135f56d30fe23b6cacd42 · ziglang/zig. General-purpose programming language and toolchain for maintaini...
New
CommunityNews
A Brief Review of the Minisforum V3 AMD Tablet. Update: I have created an awesome-minisforum-v3 GitHub repository to list information fo...
New
AstonJ
If you’re getting errors like this: psql: error: connection to server on socket “/tmp/.s.PGSQL.5432” failed: No such file or directory ...
New
Margaret
Ask Me Anything with Mark Volkmann @mvolkmann On February 24 and 25, we are giving you a chance to ask questions of PragProg author M...
New
PragmaticBookshelf
A concise guide to MySQL 9 database administration, covering fundamental concepts, techniques, and best practices. Neil Smyth MySQL...
New