wolf4earth

wolf4earth

AWS: Best practices implementing fan-out with SNS and SQS

At work we plan to replace a totally overkill Kafka instance with a combination of SNS and SQS. I don’t want to get into a discussion on the feature difference between these two, for our particular use-case SNS and SQS are a good fit.

Let me lay it out real quick:

  • a single application publishes events to a single SNS topic
  • multiple other apps are interested in these events
  • for each interested app a SQS queue is created which subscribes to the SNS topic

This way we effectively implemented a fan-out pattern using SNS and SQS. With a queue for each subscriber.

The thing I’m wondering now - and wasn’t able to find any meaningful content on - is how to other people are running this particular setup. We will setup most of this up using Terraform, and while this is perfectly fine for the SNS topic I’m not so 100% certain about the SQS queues.

As we have a bunch of apps interested in these events - with a single SQS queue for each one - setting the SQS queues up in Terraform would mean we would have to remember creating a new queue for each new app which wants these events.

From this perspective it seems easier to have some shared logic which provisions a queue for each new app as soon as this app wants to start consuming events. As we’re running in Kubernetes this would most likely be an initContainer which ensures the SQS queue exists for this particular service.

Now here is the thing: while this sounds good on paper, I have no idea if this actually is a good idea (or maybe even an anti-pattern?). Do you have any experience with an approach like this, or input on how you’re running a similar setup at work?

Where Next?

Popular Backend topics Top

DevotionGeo
I know that -t flag is used along with -i flag for getting an interactive shell. But I cannot digest what the man page for docker run com...
New
Jsdr3398
I’m trying to create a router where everything is in a collection of routes (similar to how I do my routes in expressjs). But it doesn’t ...
New
s2k
I have this code in a file that’s used to … render templates. require 'erb' require 'ostruct' MISSING_CONFIG_MARKER = :config_key_and_v...
New
Ookma-Kyi
I am learning Elixir at the moment but am finding it difficult to learn. The issue is there aren’t any good tutorial sites out there when...
New
Fl4m3Ph03n1x
Background I am trying to get a Github Action working with Windows and Bakeware because I am trying to create a release using it. Howeve...
New
Fl4m3Ph03n1x
Background I am trying to up my Functional Programming (FP) skills and one of the things that newcomers first learn in FP is the Option T...
New
Fl4m3Ph03n1x
Background I am trying to encode a structure into json format using the Jason library. However, this is not working as expected. Code L...
New
osbre
Hello everyone I’m trying to implement a “magic link” or “one-time login link” functionality I wonder what a secure way to implement it...
New
harwind
I received this error for a binary search programme in C, despite the fact that it requested for inputs and produced the right output. Th...
/c
New
jaeyson
Hi! I have clarifications (please correct me, as I mostly mix/confuse this details) with the following: The term RAG here where it read...
New

Other popular topics Top

New
axelson
I’ve been really enjoying obsidian.md: It is very snappy (even though it is based on Electron). I love that it is all local by defaul...
New
AstonJ
poll poll Be sure to check out @Dusty’s article posted here: An Introduction to Alternative Keyboard Layouts It’s one of the best write-...
New
AstonJ
I’ve been hearing quite a lot of comments relating to the sound of a keyboard, with one of the most desirable of these called ‘thock’, he...
New
AstonJ
Do the test and post your score :nerd_face: :keyboard: If possible, please add info such as the keyboard you’re using, the layout (Qw...
New
Rainer
Not sure if following fits exactly this thread, or if we should have a hobby thread… For many years I’m designing and building model air...
New
mafinar
Crystal recently reached version 1. I had been following it for awhile but never got to really learn it. Most languages I picked up out o...
New
AstonJ
We’ve talked about his book briefly here but it is quickly becoming obsolete - so he’s decided to create a series of 7 podcasts, the firs...
New
PragmaticBookshelf
Author Spotlight: VM Brasseur @vmbrasseur We have a treat for you today! We turn the spotlight onto Open Source as we sit down with V...
New
Fl4m3Ph03n1x
Background Lately I am in a quest to find a good quality TTS ai generation tool to run locally in order to create audio for some videos I...
New