wolf4earth

wolf4earth

AWS: Best practices implementing fan-out with SNS and SQS

At work we plan to replace a totally overkill Kafka instance with a combination of SNS and SQS. I don’t want to get into a discussion on the feature difference between these two, for our particular use-case SNS and SQS are a good fit.

Let me lay it out real quick:

  • a single application publishes events to a single SNS topic
  • multiple other apps are interested in these events
  • for each interested app a SQS queue is created which subscribes to the SNS topic

This way we effectively implemented a fan-out pattern using SNS and SQS. With a queue for each subscriber.

The thing I’m wondering now - and wasn’t able to find any meaningful content on - is how to other people are running this particular setup. We will setup most of this up using Terraform, and while this is perfectly fine for the SNS topic I’m not so 100% certain about the SQS queues.

As we have a bunch of apps interested in these events - with a single SQS queue for each one - setting the SQS queues up in Terraform would mean we would have to remember creating a new queue for each new app which wants these events.

From this perspective it seems easier to have some shared logic which provisions a queue for each new app as soon as this app wants to start consuming events. As we’re running in Kubernetes this would most likely be an initContainer which ensures the SQS queue exists for this particular service.

Now here is the thing: while this sounds good on paper, I have no idea if this actually is a good idea (or maybe even an anti-pattern?). Do you have any experience with an approach like this, or input on how you’re running a similar setup at work?

Where Next?

Popular Backend topics Top

Rainer
Is there somewhere a good introduction to rust for experienced programmers (with years of C++/C#/Java experience)? Wanted to give it a t...
New
Kurisu
Following on an old discussion I started on Elixir Forum here, I finally made my mind to learn Ruby on Rails in addition to Elixir/Phoen...
New
Jsdr3398
I’ve been working on and rewriting my messaging platform several times for the past two years. With Discords new rebranding, it has reall...
New
Fl4m3Ph03n1x
Background While playing around with dialyzer, typespecs and currying, I was able to create an example of a false positive in dialyzer. ...
New
JimmyCarterSon
I am following this tutorial . I have followed carefully correcting errors as I go. The app allows you to create a blog post and add comm...
New
Fl4m3Ph03n1x
Background So, I am playing around with a concept named “NewType” and I am taking inspiration from languages like F# and Scala. My objec...
New
AstonJ
If when trying to create (or recreate) your dev db with rails db:create you are getting: PG::ConnectionBad: connection to server on soc...
New
Fl4m3Ph03n1x
Background I have a personal project that is an elixir desktop application for PC Windows. It works pretty well, but now I want to give i...
New
Fl4m3Ph03n1x
Background When trying to execute mix release on a Windows 11 machine for a Phoenix project I get the following error: * assembling mark...
New
AstonJ
If you’re getting errors like this: psql: error: connection to server on socket “/tmp/.s.PGSQL.5432” failed: No such file or directory ...
New

Other popular topics Top

Exadra37
Please tell us what is your preferred monitor setup for programming(not gaming) and why you have chosen it. Does your monitor have eye p...
New
AstonJ
poll poll Be sure to check out @Dusty’s article posted here: An Introduction to Alternative Keyboard Layouts It’s one of the best write-...
New
AstonJ
There’s a whole world of custom keycaps out there that I didn’t know existed! Check out all of our Keycaps threads here: https://forum....
New
AstonJ
Thanks to @foxtrottwist’s and @Tomas’s posts in this thread: Poll: Which code editor do you use? I bought Onivim! :nerd_face: https://on...
New
AstonJ
Continuing the discussion from Thinking about learning Crystal, let’s discuss - I was wondering which languages don’t GC - maybe we can c...
New
PragmaticBookshelf
Create efficient, elegant software tests in pytest, Python's most powerful testing framework. Brian Okken @brianokken Edited by Kat...
New
PragmaticBookshelf
Author Spotlight Rebecca Skinner @RebeccaSkinner Welcome to our latest author spotlight, where we sit down with Rebecca Skinner, auth...
New
New
AstonJ
This is cool! DEEPSEEK-V3 ON M4 MAC: BLAZING FAST INFERENCE ON APPLE SILICON We just witnessed something incredible: the largest open-s...
New
mindriot
Ok, well here are some thoughts and opinions on some of the ergonomic keyboards I have, I guess like mini review of each that I use enoug...
New