wolf4earth

wolf4earth

AWS: Best practices implementing fan-out with SNS and SQS

At work we plan to replace a totally overkill Kafka instance with a combination of SNS and SQS. I don’t want to get into a discussion on the feature difference between these two, for our particular use-case SNS and SQS are a good fit.

Let me lay it out real quick:

  • a single application publishes events to a single SNS topic
  • multiple other apps are interested in these events
  • for each interested app a SQS queue is created which subscribes to the SNS topic

This way we effectively implemented a fan-out pattern using SNS and SQS. With a queue for each subscriber.

The thing I’m wondering now - and wasn’t able to find any meaningful content on - is how to other people are running this particular setup. We will setup most of this up using Terraform, and while this is perfectly fine for the SNS topic I’m not so 100% certain about the SQS queues.

As we have a bunch of apps interested in these events - with a single SQS queue for each one - setting the SQS queues up in Terraform would mean we would have to remember creating a new queue for each new app which wants these events.

From this perspective it seems easier to have some shared logic which provisions a queue for each new app as soon as this app wants to start consuming events. As we’re running in Kubernetes this would most likely be an initContainer which ensures the SQS queue exists for this particular service.

Now here is the thing: while this sounds good on paper, I have no idea if this actually is a good idea (or maybe even an anti-pattern?). Do you have any experience with an approach like this, or input on how you’re running a similar setup at work?

Where Next?

Popular Backend topics Top

DevotionGeo
I know that -t flag is used along with -i flag for getting an interactive shell. But I cannot digest what the man page for docker run com...
New
Fl4m3Ph03n1x
Background I am trying to encode a structure into json format using the Jason library. However, this is not working as expected. Code L...
New
JimmyCarterSon
I am following this tutorial . I have followed carefully correcting errors as I go. The app allows you to create a blog post and add comm...
New
Fl4m3Ph03n1x
Background I have recently been delving into more functional code. My objective right now is to get something similar to the IO Monad (in...
New
sona11
In Java, if I try to do.equals() on a null string, a null pointer error is issued. I’m wondering whether I can perform the following if I...
New
sona11
I’m having a difficulty. I want to modify an attribute’s data type from String to Array. { “id”: “trn:tarb:tradingpartner:uuid:00000...
New
Fl4m3Ph03n1x
Background I am a fan of dialyzer and friends (looking at Gradient) and I try to have sepcs in my code as much as I can. To this end, I a...
New
harwind
I have a large SQL database with millions of records, and I’ve identified duplicate entries. What’s the most efficient way to find and re...
New
Fl4m3Ph03n1x
Background I have an umbrella project, where I run mix test from the root. In one of the apps, I am mocking the File module using the Mo...
New
ogoldberg
Any recommendations on good resources for learning Elixir, Phoenix, and Ash?
New

Other popular topics Top

dasdom
No chair. I have a standing desk. This post was split into a dedicated thread from our thread about chairs :slight_smile:
New
AstonJ
SpaceVim seems to be gaining in features and popularity and I just wondered how it compares with SpaceMacs in 2020 - anyone have any thou...
New
AstonJ
You might be thinking we should just ask who’s not using VSCode :joy: however there are some new additions in the space that might give V...
New
AstonJ
poll poll Be sure to check out @Dusty’s article posted here: An Introduction to Alternative Keyboard Layouts It’s one of the best write-...
New
AstonJ
Continuing the discussion from Thinking about learning Crystal, let’s discuss - I was wondering which languages don’t GC - maybe we can c...
New
Margaret
Hello everyone! This thread is to tell you about what authors from The Pragmatic Bookshelf are writing on Medium.
1145 27688 760
New
AstonJ
Biggest jackpot ever apparently! :upside_down_face: I don’t (usually) gamble/play the lottery, but working on a program to predict the...
New
New
PragmaticBookshelf
Author Spotlight Jamis Buck @jamis This month, we have the pleasure of spotlighting author Jamis Buck, who has written Mazes for Prog...
New
sir.laksmana_wenk
I’m able to do the “artistic” part of game-development; character designing/modeling, music, environment modeling, etc. However, I don’t...
New