sprite2005

sprite2005

Concurrent Data Processing in Elixir: Using a mapper operation after reduce/3 is deprecated (page 114)

@svilen

warning: Using a mapper operation, such as map/filter/reject, after reduce/3 is deprecated. Use Flow.on_trigger/2 instead
(flow 1.2.0) lib/flow.ex:1916: Flow.add_mapper/3
(airports 0.1.0) lib/airports.ex:25: Airports.open_airports/0
(stdlib 4.0.1) erl_eval.erl:744: :erl_eval.do_apply/7
(elixir 1.13.4) src/elixir.erl:296: :elixir.recur_eval/3
(elixir 1.13.4) src/elixir.erl:274: :elixir.eval_forms/3
(iex 1.13.4) lib/iex/evaluator.ex:310: IEx.Evaluator.handle_eval/3

How would we rewrite this as an on_trigger function?

Most Liked

andybab

andybab

This should work, but is longer than Flow.map version from book:

...
     |> Flow.reject(&(&1.type == "closed"))
     |> Flow.partition(key: {:key, :country})
     |> Flow.group_by(& &1.country)
     |> Flow.on_trigger(fn map ->
        country_data = Enum.map(map, fn {country, data} -> {country, Enum.count(data)} end)
        {country_data, map}
      end)
     #|> Flow.map(fn {country, data} -> {country, Enum.count(data)} end)
     |> Enum.to_list()
rschenk

rschenk

@svilen

I ran into this problem as well. While @andybab’s solution will produce the correct answer, I believe the use of Enum.map means that this part of the computation will not be calculated concurrently, whereas (I think?) the version in the book which uses Flow.map will be run in parallel. I could be wrong, I’m totally new at this.

It sure would be nice if the author of the book could provide a version that is both concurrent and not deprecated.

Edit: looks like my thinking was wrong. If you throw an IO.inspect(self()) into the andybab’s on_trigger callback, you can see that this does get split in parallel across multiple stages. For a noob, it’s confusing why/how this works and why you can’t use Flow.map instead, hopefully the author can update this section with an explanation.

Where Next?

Popular Pragmatic Bookshelf topics Top

jon
Some minor things in the paper edition that says “3 2020” on the title page verso, not mentioned in the book’s errata online: p. 186 But...
New
telemachus
Python Testing With Pytest - Chapter 2, warnings for “unregistered custom marks” While running the smoke tests in Chapter 2, I get these...
New
lirux
Hi Jamis, I think there’s an issue with a test on chapter 6. I own the ebook, version P1.0 Feb. 2019. This test doesn’t pass for me: ...
New
raul
Hi Travis! Thank you for the cool book! :slight_smile: I made a list of issues and thought I could post them chapter by chapter. I’m rev...
New
joepstender
The generated iex result below should list products instead of product for the metadata. (page 67) iex> product = %Product{} %Pento....
New
Chrichton
Dear Sophie. I tried to do the “Authorization” exercise and have two questions: When trying to plug in an email-service, I found the ...
New
swlaschin
The book has the same “Problem space/Solution space” diagram on page 18 as is on page 17. The correct Problem/Solution space diagrams ar...
New
jskubick
I’m under the impression that when the reader gets to page 136 (“View Data with the Database Inspector”), the code SHOULD be able to buil...
New
AufHe
I’m a newbie to Rails 7 and have hit an issue with the bin/Dev script mentioned on pages 112-113. Iteration A1 - Seeing the list of prod...
New
ggerico
I got this error when executing the plot files on macOS Ventura 13.0.1 with Python 3.10.8 and matplotlib 3.6.1: programming_ML/code/03_...
New

Other popular topics Top

PragmaticBookshelf
Write Elixir tests that you can be proud of. Dive into Elixir’s test philosophy and gain mastery over the terminology and concepts that u...
New
Exadra37
I am thinking in building or buy a desktop computer for programing, both professionally and on my free time, and my choice of OS is Linux...
New
PragmaticBookshelf
Design and develop sophisticated 2D games that are as much fun to make as they are to play. From particle effects and pathfinding to soci...
New
AstonJ
I’ve been hearing quite a lot of comments relating to the sound of a keyboard, with one of the most desirable of these called ‘thock’, he...
New
PragmaticBookshelf
Develop, deploy, and debug BEAM applications using BEAMOps: a new paradigm that focuses on scalability, fault tolerance, and owning each ...
New
AstonJ
Curious what kind of results others are getting, I think actually prefer the 7B model to the 32B model, not only is it faster but the qua...
New
NewsBot
Node.js v22.14.0 has been released. Link: Release 2025-02-11, Version 22.14.0 'Jod' (LTS), @aduh95 · nodejs/node · GitHub
New
PragmaticBookshelf
As digital systems increasingly run the world, mastery of the recurring patterns of software development risk is the key to fast and effe...
New
PragmaticBookshelf
Quickly turn complex problems into simple, working solutions using functional programming, safe concurrency, and the expressive tools of ...
New
xiji2646-netizen
Woke up to this today: Claude Code’s complete source code exposed via npm source map. Not a snippet. All 512,000 lines. 1,900 TypeScript ...
New

Sub Categories: