PragProg Customers

Next-Level A/B is Testing: 'inconclusive results' is ambiguous (p. 12)

dlindelof

dlindelof

Next-Level A/B is Testing: 'inconclusive results' is ambiguous (p. 12)

@leemay

The book refers to ‘inconclusive results’ starting on page 12. I would caution against introducing this term without carefully defining it. At my dayjob I frequently deal with product managers who believe that an A/B experiment with a neutral outcome is ‘inconclusive’, when in fact it just says that there is no statistically significant difference between the two arms.

2 comments
/book-next-level-a-b-testing
1 15 2

Most Liked

leemay

leemay

Author of Practical A/B Testing

That’s a really good point. I’ll do exactly as you suggest, the details matter. I’ll define and give caveats of what it looks like in practice/what that could mean. Luckily we’re still in beta so I can make these changes this week. Thank you @dlindelof!

1
Post #3

Popular Prag Prog topics Top

belgoros
PragProg Customers

Effective Testing with RSpec 3: Running Sequel migrations fails (page 84)
Following the steps described in Chapter 6 of the book, I’m stuck with running the migration as described on page 84: bundle exec sequel...
#errata/book-effective-testing-with-rspec-3
4 1149 1
New
HarryDeveloper
PragProg Customers

Programming Kotlin: Difference between SupervisorJob() and supervisorScope (324, 325)
Hi @venkats, It has been mentioned in the description of ‘Supervisory Job’ title that 2 things as mentioned below result in the same eff...
#question/book-programming-kotlin
0 2210 2
New
cro
PragProg Customers

Programming Phoenix LiveView: P27, live_path confusion
I am working on the “Your Turn” for chapter one and building out the restart button talked about on page 27. It recommends looking into ...
#question/book-programming-phoenix-liveview
7 3474 14
New
jskubick
PragProg Customers

Kotlin and Android Development featuring Jetpack: $kotlin_version ? (page 8)
I’m running Android Studio “Arctic Fox” 2020.3.1 Patch 2, and I’m embarrassed to admit that I only made it to page 8 before running into ...
#errata/book-kotlin-and-android-development-featuring-jetpack
0 1695 3
New
akraut
PragProg Customers

Programming Phoenix LiveView: Display Image Uploads (pdf p142)
The markup used to display the uploaded image results in a Phoenix.LiveView.HTMLTokenizer.ParseError error. lib/pento_web/live/product_l...
#errata/book-programming-phoenix-liveview
2 2080 9
New
taguniversalmachine
PragProg Customers

Real-Time Phoenix: ProductChannelTest - undefined function describe/2 (pg 203)
Hi, I am getting an error I cannot figure out on my test. I have what I think is the exact code from the book, other than I changed “us...
#question/book-real-time-phoenix
0 1757 6
New
EdBorn
PragProg Customers

Agile Web Development with Rails 7: (page 70)
Title: Agile Web Development with Rails 7: (page 70) I am running windows 11 pro with rails 7.0.3 and ruby 3.1.2p20 (2022-04-12 revision...
#question/book-agile-web-development-with-rails-7
0 1125 1
New
andreheijstek
PragProg Customers

Modern front-end development for Rails, second edition - Struggling to get the first chapter to work
After running /bin/setup, the first error was: The foreman' command exists in these Ruby versions: That was easy to fix: gem install fore...
#errata/book-modern-front-end-development-for-rails
0 1105 3
New
Keton
PragProg Customers

Hands-on Rust: How do I debug a call chain in Rust?
When running the program in chapter 8, “Implementing Combat”, the printout Health before attack was never printed so I assumed something ...
#question/book-hands-on-rust
2 1004 2
New
ggerico
PragProg Customers

Programming Machine Learning: some plot files are throwing error
I got this error when executing the plot files on macOS Ventura 13.0.1 with Python 3.10.8 and matplotlib 3.6.1: programming_ML/code/03_...
#question/book-programming-machine-learning
1 3024 5
New

Other popular topics Top

ohm
Tech Chat

Games! Which do you play?
Which, if any, games do you play? On what platform? I just bought (and completed) Minecraft Dungeons for my Nintendo Switch. Other than ...
#games
245 5072 101
New
siddhant3030
General Dev Chat

Which vertical monitor do you use?
I’m thinking of buying a monitor that I can rotate to use as a vertical monitor? Also, I want to know if someone is using it for program...
#monitors#programming
51 4075 20
New
AstonJ
Backend Developer Forum>Questions/Help

Erlang's not installing on macOS Big Sur "You are natively building Erlang/OTP for a later version of MacOSX than current version"
Just done a fresh install of macOS Big Sur and on installing Erlang I am getting: asdf install erlang 23.1.2 Configure failed. checking ...
/erlang#asdf#macos#big-sur
10 5357 8
New
Exadra37
General Dev Chat

What is the Best Laptop to develop with a Linux OS?
I am a Linux user since 2012, more or less, and I always use Ubuntu on my computers, and my last 2 laptops have been used Thinkpads, wher...
#developer-experience#linux#laptop
64 2918 29
New
AstonJ
Backend Developer Forum>Chat/Discussions

Rails console using 100% CPU in dev (fix)
If you are experiencing Rails console using 100% CPU on your dev machine, then updating your development and test gems might fix the issu...
/ruby/rails
3 3436 3
New
PragmaticBookshelf
In The Spotlight

Spotlight: Chris Pine (Author)
A Hero’s Journey with Chris Pine @chrispine Chris Pine, author of Learn to Program, Third Edition, discusses his journey to beco...
#author-spotlight/book-learn-to-program-third-edition/ruby
81 3229 26
New
AstonJ
Hardware

CharaChorder - type at the speed of thought?
Saw this on TikTok of all places! :lol: Anyone heard of them before? Lite:
/keyboards#charachorder
13 3705 4
New
Maartz
General Dev Chat

Roc Language - a new purely functional programming language built for speed and ergonomics
Hi folks, I don’t know if I saw this here but, here’s a new programming language, called Roc Reminds me a bit of Elm and thus Haskell. ...
#programminguages#functional-programming
49 4145 14
New
DevotionGeo
General Questions/Help

Do you prefer regular mechanical keyboards or low profile mechanical keyboards and why?
I have always used antique keyboards like Cherry MX 1800 or Cherry MX 8100 and almost always have modified the switches in some way, like...
/keyboards#mechanical-keyboards
27 2436 9
New
PragmaticBookshelf
In The Spotlight

Spotlight: Sophie DeBenedetto (Author) Interview and AMA!
Author Spotlight: Sophie DeBenedetto @SophieDeBenedetto The days of the traditional request-response web application are long gone, b...
/elixir/phoenix#liveview/book-programming-phoenix-liveview
37 2988 14
New
Back to top

Latest in PragProg

Hotwire Native for Rails Developers: how can i hide a tab bar? (page 76)
PragProg Customers
Ash Framework: Incorrect return of {:ok, ...} for bang (!) function (page 20)
PragProg Customers
Powerful Command-Line Applications in Go: ListAction example missing args (page 241)
PragProg Customers
Agile Web Development with Rails 8_B6.0(page 148)
PragProg Customers
Advanced Hands-on Rust: Workspace Dependencies (B4.0 page 41)
PragProg Customers
Agile Web Development with Rails 8_B6.0(page 139)
PragProg Customers
Effective Haskell: duplicate extension (page xx)
PragProg Customers
Agile Web Development with Rails 8_B6.0(page 126)
PragProg Customers
Hotwire Native for Rails Developers: env.local? true -> false (page 237)
PragProg Customers
Kotlin Coroutine Confidence: error handling suggestion (page 43) (Solved)
PragProg Customers
Kotlin Coroutine Confidence: misleading image or text (p 20ish)
PragProg Customers
Kotlin Coroutine Confidence: not using kotlin style builder pattern (page 19ish)
PragProg Customers
Kotlin Coroutine Confidence: TimerApplication: won't app finish before timer?
PragProg Customers
Real-World Event Sourcing: duplicated paragraph (p. 30)
PragProg Customers
Agile Web Development with Rails 8_B6.0 "headless_chrome" related (page 189)
PragProg Customers
Real World Event Sourcing: typo, data elided for readability (p. 32)
PragProg Customers
I cannot download PDF versions of two books I purchased
PragProg Customers
Real-World Event Sourcing: Command to run EventStore on Docker doesn't allow access to Stream Browser (B4.0, page 81)
PragProg Customers
Hotwire Native for Rails Developers: import for R shortcut (pages 94-95)
PragProg Customers
Hotwire Native for Rails Developers: Tailwind 4 (page 44)
PragProg Customers

View all threads ❯

Latest (all)

Kotlin v2.1.21-RC released!
Backend News
WebSharper 9.0.0.547 released!
Backend News
Spring v7.0.0-M4 and v6.1.19 released!
Backend News
Microsoft Copilot can now ‘see’ what’s on your screen in Edge
In The News
Run your tests against Django's main!
Backend News
Smarter Kotlin Development With JetBrains AI: Junie and AI Assistant in IntelliJ IDEA
Backend News
Arch Linux: Valkey to replace Redis in the [extra] Repository
Linux News
(All) Databases Are Just Files. Postgres Too
In The News
Erlang OTP-26.2.5.11 and OTP-25.3.2.20 released!
Backend News
Quarkus 3.22.0.CR1 and 3.21.3 released!
Backend News
Crystal 1.16.1 released!
Backend News
Crystal 1.16.1 is released!
Backend News
KotlinConf 2025: Last Tickets, Epic Talks, an Evening to Remember, and More
Backend News
Clojure Deref (Apr 11, 2025)
Backend News
PostgreSQL: The Schedule is out for POSETTE: An Event for Postgres 2025!
Backend News

View all threads ❯

We ❤️ helpful members!

We reward our most helpful members via our MOTM scheme - by giving away a whopping 25 books per year!

Member of the Month

Devtalk Sponsors

Pragmatic Bookshelf

Practical resources that improve the lives of professional developers.

Erlang Solutions

We build trusted fault-tolerant systems that scale to billions of users.

AppSignal

Catch errors, track performance, monitor hosts and more.

WyeWorks

Enabling companies to succeed by building software people love.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

Categories:

Popular Portals

Devtalk Sponsors

Pragmatic Bookshelf

Practical resources that improve the lives of professional developers.

Erlang Solutions

We build trusted fault-tolerant systems that scale to billions of users.

AppSignal

Catch errors, track performance, monitor hosts and more.

WyeWorks

Enabling companies to succeed by building software people love.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

We're in Beta

About us Mission Statement See our Roadmap