wolf4earth

wolf4earth

AWS: Best practices implementing fan-out with SNS and SQS

At work we plan to replace a totally overkill Kafka instance with a combination of SNS and SQS. I don’t want to get into a discussion on the feature difference between these two, for our particular use-case SNS and SQS are a good fit.

Let me lay it out real quick:

  • a single application publishes events to a single SNS topic
  • multiple other apps are interested in these events
  • for each interested app a SQS queue is created which subscribes to the SNS topic

This way we effectively implemented a fan-out pattern using SNS and SQS. With a queue for each subscriber.

The thing I’m wondering now - and wasn’t able to find any meaningful content on - is how to other people are running this particular setup. We will setup most of this up using Terraform, and while this is perfectly fine for the SNS topic I’m not so 100% certain about the SQS queues.

As we have a bunch of apps interested in these events - with a single SQS queue for each one - setting the SQS queues up in Terraform would mean we would have to remember creating a new queue for each new app which wants these events.

From this perspective it seems easier to have some shared logic which provisions a queue for each new app as soon as this app wants to start consuming events. As we’re running in Kubernetes this would most likely be an initContainer which ensures the SQS queue exists for this particular service.

Now here is the thing: while this sounds good on paper, I have no idea if this actually is a good idea (or maybe even an anti-pattern?). Do you have any experience with an approach like this, or input on how you’re running a similar setup at work?

Where Next?

Popular Backend topics Top

AstonJ
Just done a fresh install of macOS Big Sur and on installing Erlang I am getting: asdf install erlang 23.1.2 Configure failed. checking ...
New
Jsdr3398
I’m trying to create a router where everything is in a collection of routes (similar to how I do my routes in expressjs). But it doesn’t ...
New
Jsdr3398
Are there any databases that require no setup (can be shipped in a small zip together with the project)?
New
New
Jsdr3398
I’ve been working on and rewriting my messaging platform several times for the past two years. With Discords new rebranding, it has reall...
New
MarkIden
Hi, Recommend pls your favorite learning resources in Go, with best books, podcasts etc.
/go
New
GermaVinsmoke
Does anyone know beginner friendly Elixir/Phoenix Open source projects? For learning purpose :slightly_smiling_face:
New
Fl4m3Ph03n1x
Background I am a fan of dialyzer and friends (looking at Gradient) and I try to have sepcs in my code as much as I can. To this end, I a...
New
Fl4m3Ph03n1x
Background I have a release file inside a tarball. However I want the final release to have some additional files and to move things aro...
New
Fl4m3Ph03n1x
Background As I often do, I read books to learn and improve myself. I also enjoy teaching and helping others when I can, so this is somet...
New

Other popular topics Top

AstonJ
A thread that every forum needs! Simply post a link to a track on YouTube (or SoundCloud or Vimeo amongst others!) on a separate line an...
New
PragmaticBookshelf
A PragProg Hero’s Journey with Brian P. Hogan @bphogan Have you ever worried that your only legacy will be in the form of legacy...
New
Exadra37
I am thinking in building or buy a desktop computer for programing, both professionally and on my free time, and my choice of OS is Linux...
New
Exadra37
Please tell us what is your preferred monitor setup for programming(not gaming) and why you have chosen it. Does your monitor have eye p...
New
Rainer
My first contact with Erlang was about 2 years ago when I used RabbitMQ, which is written in Erlang, for my job. This made me curious and...
New
PragmaticBookshelf
Learn different ways of writing concurrent code in Elixir and increase your application's performance, without sacrificing scalability or...
New
Maartz
Hi folks, I don’t know if I saw this here but, here’s a new programming language, called Roc Reminds me a bit of Elm and thus Haskell. ...
New
husaindevelop
Inside our android webview app, we are trying to paste the copied content from another app eg (notes) using navigator.clipboard.readtext ...
New
PragmaticBookshelf
Author Spotlight: VM Brasseur @vmbrasseur We have a treat for you today! We turn the spotlight onto Open Source as we sit down with V...
New
AstonJ
This is a very quick guide, you just need to: Download LM Studio: https://lmstudio.ai/ Click on search Type DeepSeek, then select the o...
New