kunna

kunna

Concurrent Data Processing in Elixir:147

On page 147. the book says “Using Task.async_stream/3 is a great way to process large collections of data, and provide back-pressure at the same time.”

but while using Task.async_stream/3, I found some unexpected memory usages and ran some tests. and my test results show Task.async_stream is not handling back-pressure well. It works fine with fewer concurrency, but it fails to manage the back-pressure when I increased the max_concurrency option to 100.

This is my test code.

defmodule Playground do
  def stream do
    Stream.resource(
      fn -> Enum.to_list(0..100) end,
      fn list ->
        case Enum.split(list, 10) do
          {[], _} ->
            {:halt, []}
          {head, tail} ->
            IO.inspect("emit")
            {head, tail}
        end
      end,
      fn _ -> :foo end
    )
  end

  def run_async do
    stream()
    |> Task.async_stream(__MODULE__, :do_it, [], max_concurrency: 100, ordered: false)
    |> Stream.run()
  end

  def do_it(i) do
    Process.sleep(:rand.uniform(1000))
    IO.inspect(i)
  end
end

and the printed out result

iex(5)> Playground.run_async
"emit"
"emit"
"emit"
"emit"
"emit"
"emit"
"emit"
"emit"
"emit"
"emit"
"emit"
0
51
5
35
52
68
65
21
98
85
87
10
27
29
47
.
.
.

Where Next?

Popular Pragmatic Bookshelf topics Top

mikecargal
Title: Hands-On Rust (Chap 8 (Adding a Heads Up Display) It looks like ​.with_simple_console_no_bg​(SCREEN_WIDTH*2, SCREEN_HEIGHT*2...
New
jdufour
Hello! On page xix of the preface, it says there is a community forum "… for help if your’re stuck on one of the exercises in this book… ...
New
raul
Hi Travis! Thank you for the cool book! :slight_smile: I made a list of issues and thought I could post them chapter by chapter. I’m rev...
New
herminiotorres
Hi! I know not the intentions behind this narrative when called, on page XI: mount() |> handle_event() |> render() but the correc...
New
joepstender
The generated iex result below should list products instead of product for the metadata. (page 67) iex> product = %Product{} %Pento....
New
jeremyhuiskamp
Title: Web Development with Clojure, Third Edition, vB17.0 (p9) The create table guestbook syntax suggested doesn’t seem to be accepted ...
New
adamwoolhether
When trying to generate the protobuf .go file, I receive this error: Unknown flag: --go_opt libprotoc 3.12.3 MacOS 11.3.1 Googling ...
New
brunogirin
When installing Cards as an editable package, I get the following error: ERROR: File “setup.py” not found. Directory cannot be installe...
New
taguniversalmachine
It seems the second code snippet is missing the code to set the current_user: current_user: Accounts.get_user_by_session_token(session["...
New
davetron5000
Hello faithful readers! If you have tried to follow along in the book, you are asked to start up the dev environment via dx/build and ar...
New

Other popular topics Top

AstonJ
A thread that every forum needs! Simply post a link to a track on YouTube (or SoundCloud or Vimeo amongst others!) on a separate line an...
New
New
AstonJ
What chair do you have while working… and why? Is there a ‘best’ type of chair or working position for developers?
New
siddhant3030
I’m thinking of buying a monitor that I can rotate to use as a vertical monitor? Also, I want to know if someone is using it for program...
New
PragmaticBookshelf
Design and develop sophisticated 2D games that are as much fun to make as they are to play. From particle effects and pathfinding to soci...
New
AstonJ
Just done a fresh install of macOS Big Sur and on installing Erlang I am getting: asdf install erlang 23.1.2 Configure failed. checking ...
New
AstonJ
In case anyone else is wondering why Ruby 3 doesn’t show when you do asdf list-all ruby :man_facepalming: do this first: asdf plugin-upd...
New
New
PragmaticBookshelf
Author Spotlight: VM Brasseur @vmbrasseur We have a treat for you today! We turn the spotlight onto Open Source as we sit down with V...
New
AstonJ
This is a very quick guide, you just need to: Download LM Studio: https://lmstudio.ai/ Click on search Type DeepSeek, then select the o...
New

Sub Categories: