Concurrent Data Processing in Elixir (Pragmatic Bookshelf)

PragmaticBookshelf
Learn different ways of writing concurrent code in Elixir and increase your application's performance, without sacrificing scalability or fault-tolerance. Leverage concurrency by using OTP, GenStage, Flow, and Broadway to build high-performance applications and data processing pipelines.

Svilen Gospodinov @svilen

edited by Jacquelyn Carter @jkcarter

Learn different ways of writing concurrent code in Elixir and increase your application’s performance, without sacrificing scalability or fault-tolerance. Most projects benefit from running background tasks and processing data concurrently, but the world of OTP and various libraries can be challenging. Which Supervisor and what strategy to use? What about GenServer? Maybe you need back-pressure, but is GenStage, Flow, or Broadway a better choice? You will learn everything you need to know to answer these questions, start building highly concurrent applications in no time, and write code that’s not only fast, but also resilient to errors and easy to scale.

Whether you are building a high-frequency stock trading application or a consumer web app, you need to know how to leverage concurrency to build applications that are fast and efficient. Elixir and the OTP offer a range of powerful tools, and this guide will show you how to choose the best tool for each job, and use it effectively to quickly start building highly concurrent applications.

Learn about Tasks, supervision trees, and the different types of Supervisors available to you. Understand why processes and process linking are the building blocks of concurrency in Elixir. Get comfortable with the OTP and use the GenServer behaviour to maintain process state for long-running jobs. Easily scale the number of running processes using the Registry. Handle large volumes of data and traffic spikes with GenStage, using back-pressure to your advantage. Create your first multi-stage data processing pipeline using producer, consumer, and producer-consumer stages. Process large collections with Flow, using MapReduce and more in parallel. Thanks to Broadway, you will see how easy it is to integrate with popular message broker systems, or even existing GenStage producers.

Start building the high-performance and fault-tolerant applications Elixir is famous for today.


Don’t forget you can get 35% off with your Devtalk discount! Just use the coupon code “devtalk.com" at checkout :+1:

Latest Threads About This Book Top

Cichacz
@svilen The book says: Currently, async_stream/3 assumes we want the results in the same order as they were originally. This order pre...
New
sprite2005
@svilen warning: Using a mapper operation, such as map/filter/reject, after reduce/3 is deprecated. Use Flow.on_trigger/2 instead (flow...
New
markthequark
@svilen def via(id) do {:via, Registry, {ProducerConsumerRegistry, id} end is missing a }. This is also the case in the downloadable...
New
smahi
Spining multiple PageConsumer using the book provided code, raise an error children = [ PageProducer, Supervisor.child_sp...
New
pgeraghty
@svilen Book version: P1.0 (August 2021) At the end of the definition of init, I believe GenStage.ParitionDispatcher should be GenStage...
New
axelson
Hello Svilen Gospodinov @svilen, The book currently states: When using GenServer.cast/2, we always get :ok as a reply. In fact, the r...
New
mafinar
Concurrent Data Processing in Elixir is now content complete and I finally found the time I’ve been looking for to dedicate behind readin...
New
New
kunna
On page 147. the book says “Using Task.async_stream/3 is a great way to process large collections of data, and provide back-pressure at t...
New
New
chriseyre2000
Towards the end of Chapter 3 (B2) the via function is implemented as: def via(id) do {{:via, Registry, {ProducerConsumerRegistry, id}},...
New
lancejjohnson
Page 3 We are going to use the Task module later to develop some if its functionality. s/if/of/
New
lancejjohnson
Page viii It introduces the Task module, processes, timeouts, and other topics that lay the foundation for the following chapters to bu...
New
lancejjohnson
You will also discover a range new of new techniques, that will help you simplify your product, Delete “new” after “range”.
New
mattjg908
Title: Concurrent Data Processing in Elixir: typo (31-32) I believe (page 31-32): def handle_cast({:send, email}, state) do . . {:no...
New
justincjohnson
In Chapter 1 at the end of “Managing Series of Tasks” “We talked about timeout exceptions, which happen when the processes takes too lon...
New
kunna
After finishing Chapter 2, Long-Running Processes Using GenServer, I was able to successfully run the code. but if I see the process obse...
New
chriseyre2000
The first handle_cast({:send, email}, state) does not update the state so the example will not work as described. It’s missing: state =...
New
mratliff
“ Not only we have processed the pages faster, ” should be: “ Not only have we processed the pages faster, ”
New
mratliff
The text currently reads: “ This means that the handle_events/3 callback will be called with a list of 500 events initially, followed by...
New
herminiotorres
Hi @Margaret , On page VII the book tells us the example and snippets will be all using Elixir version 1.11 But on page 3 almost the en...
New
mratliff
Not a big deal but might want to mention that you should add :crypto to the extra_applications portion of mix like so: def application d...
New
New
New
New
New
PragmaticBookshelf
Learn different ways of writing concurrent code in Elixir and increase your application's performance, without sacrificing scalability or...
New

Most Active This Week Top

Most Active This Month Top

Most Active This Year Top

Most Active Last Three Years Top

sprite2005
@svilen warning: Using a mapper operation, such as map/filter/reject, after reduce/3 is deprecated. Use Flow.on_trigger/2 instead (flow...
New
Cichacz
@svilen The book says: Currently, async_stream/3 assumes we want the results in the same order as they were originally. This order pre...
New

Most Active Over Three Years Top

New
PragmaticBookshelf
Learn different ways of writing concurrent code in Elixir and increase your application's performance, without sacrificing scalability or...
New
mafinar
Concurrent Data Processing in Elixir is now content complete and I finally found the time I’ve been looking for to dedicate behind readin...
New
herminiotorres
Hi @Margaret , On page VII the book tells us the example and snippets will be all using Elixir version 1.11 But on page 3 almost the en...
New
kunna
After finishing Chapter 2, Long-Running Processes Using GenServer, I was able to successfully run the code. but if I see the process obse...
New
mattjg908
Title: Concurrent Data Processing in Elixir: typo (31-32) I believe (page 31-32): def handle_cast({:send, email}, state) do . . {:no...
New
New
chriseyre2000
The first handle_cast({:send, email}, state) does not update the state so the example will not work as described. It’s missing: state =...
New
axelson
Hello Svilen Gospodinov @svilen, The book currently states: When using GenServer.cast/2, we always get :ok as a reply. In fact, the r...
New
New
mratliff
Not a big deal but might want to mention that you should add :crypto to the extra_applications portion of mix like so: def application d...
New
pgeraghty
@svilen Book version: P1.0 (August 2021) At the end of the definition of init, I believe GenStage.ParitionDispatcher should be GenStage...
New
kunna
On page 147. the book says “Using Task.async_stream/3 is a great way to process large collections of data, and provide back-pressure at t...
New
New
smahi
Spining multiple PageConsumer using the book provided code, raise an error children = [ PageProducer, Supervisor.child_sp...
New

Get money off!

The Pragmatic Bookshelf

35% off any eBook

Simply use coupon code "devtalk.com" at checkout. See full details here.

Latest in Concurrent Data Processing in Elixir

View all Concurrent Data Processing in Elixir threads ❯

Related Portals