PragProg Customers

Next-Level A/B is Testing: 'inconclusive results' is ambiguous (p. 12)

dlindelof

dlindelof

Next-Level A/B is Testing: 'inconclusive results' is ambiguous (p. 12)

@leemay

The book refers to ‘inconclusive results’ starting on page 12. I would caution against introducing this term without carefully defining it. At my dayjob I frequently deal with product managers who believe that an A/B experiment with a neutral outcome is ‘inconclusive’, when in fact it just says that there is no statistically significant difference between the two arms.

2 comments
/book-next-level-a-b-testing
1 15 2

Most Liked

leemay

leemay

Author of Practical A/B Testing

That’s a really good point. I’ll do exactly as you suggest, the details matter. I’ll define and give caveats of what it looks like in practice/what that could mean. Luckily we’re still in beta so I can make these changes this week. Thank you @dlindelof!

1
Post #3

Popular Pragmatic topics Top

belgoros
PragProg Customers

Effective Testing with RSpec 3: Running Sequel migrations fails (page 84)
Following the steps described in Chapter 6 of the book, I’m stuck with running the migration as described on page 84: bundle exec sequel...
#errata/book-effective-testing-with-rspec-3
4 1149 1
New
joepstender
PragProg Customers

Programming Phoenix LiveView: errata and remarks Version B2.0
The generated iex result below should list products instead of product for the metadata. (page 67) iex> product = %Product{} %Pento....
#errata/book-programming-phoenix-liveview
0 1798 20
New
alanq
PragProg Customers

Modern Front-End Development for Rails: Can't get TURBO_STREAM format in some cases
This isn’t directly about the book contents so maybe not the right forum…but in some of the code apps (e.g. turbo/06) it sends a TURBO_ST...
#question/book-modern-front-end-development-for-rails
1 1364 7
New
leba0495
PragProg Customers

A Common-Sense Guide to Data Structures and Algorithms, Second Edition: (pg460)
Hello! Thanks for the great book. I was attempting the Trie (chap 17) exercises and for number 4 the solution provided for the autocorre...
#errata/book-a-common-sense-guide-to-data-structures-and-algorithms-second-edition
0 1148 3
New
AndyDavis3416
PragProg Customers

Modern Front-End Development for Rails: ERROR in tsconfig.json Chapter_01/01
@noelrappin Running the webpack dev server, I receive the following warning: ERROR in tsconfig.json TS18003: No inputs were found in c...
/book-modern-front-end-development-for-rails
3 1433 3
New
Henrai
PragProg Customers

The Ray Tracer Challenge: Cannot get rid of acne When implement Shadows in Chapter 8.
Hi, I’m working on the Chapter 8 of the book. After I add add the point_offset, I’m still able to see acne: In the image above, I re...
#question/book-the-ray-tracer-challenge
1 1019 4
New
andreheijstek
PragProg Customers

Modern front-end development for Rails, second edition - Struggling to get the first chapter to work
After running /bin/setup, the first error was: The foreman' command exists in these Ruby versions: That was easy to fix: gem install fore...
#errata/book-modern-front-end-development-for-rails
1 1108 3
New
redconfetti
PragProg Customers

Docker for Rails Developers - 'docker-machine' command deprecated (page 156)
Docker-Machine became part of the Docker Toolbox, which was deprecated in 2020, long after Docker Desktop supported Docker Engine nativel...
/book-docker-for-rails-developers#suggestion
0 894 0
New
mcpierce
PragProg Customers

Kotlin and Android Development featuring Jetpack: Unique constraint error when running chapter 5 code changes
@mfazio23 I’ve applied the changes from Chapter 5 of the book and everything builds correctly and runs. But, when I try to start a game,...
/book-kotlin-and-android-development-featuring-jetpack
0 1152 10
New
davetron5000
PragProg Customers

Bug with dx/build - how to workaround until fix is up
Hello faithful readers! If you have tried to follow along in the book, you are asked to start up the dev environment via dx/build and ar...
/book-ruby-on-rails-background-jobs-with-sidekiq#errata
3 1149 19
New

Other popular topics Top

AstonJ
Hardware

Which keyboard do you have?
If it’s a mechanical keyboard, which switches do you have? Would you recommend it? Why? What will your next keyboard be? Pics always w...
#hardware/keyboards#mechanical-keyboards#sticky
144 8284 50
New
ohm
Tech Chat

Games! Which do you play?
Which, if any, games do you play? On what platform? I just bought (and completed) Minecraft Dungeons for my Nintendo Switch. Other than ...
#games
245 5129 101
New
AstonJ
Tech Chat

What are you watching?
Or looking forward to? :nerd_face:
#community
480 9175 251
New
AstonJ
General Dev Chat

Which language or framework do you want to learn next?
Curious to know which languages and frameworks you’re all thinking about learning next :upside_down_face: Perhaps if there’s enough peop...
#community#learning
239 5590 90
New
AstonJ
Hardware

Seen any cool new keyboards?
We have a thread about the keyboards we have, but what about nice keyboards we come across that we want? If you have seen any that look n...
/keyboards#mechanical-keyboards
44 5055 37
New
AstonJ
Hardware

Planck vs Preonic vs Subatomic (Keyboards)
I ended up cancelling my Moonlander order as I think it’s just going to be a bit too bulky for me. I think the Planck and the Preonic (o...
/keyboards#mechanical-keyboards#ortholinear#planck#preonic
105 14115 47
New
DevotionGeo
General Dev Chat

The V Programming Language
The V Programming Language Simple language for building maintainable programs V is already mentioned couple of times in the forum, but I...
#programminguages/v
21 10890 7
New
PragmaticBookshelf
In The Spotlight

Spotlight: Mike Riley (Author) Interview and AMA!
Author Spotlight Mike Riley @mriley This month, we turn the spotlight on Mike Riley, author of Portable Python Projects. Mike’s book ...
#author-spotlight/python/book-portable-python-projects#iot#internet-of-things
62 6172 19
New
PragmaticBookshelf
In The Spotlight

Spotlight: David Bryant Copeland (Author) Interview and AMA!
Author Spotlight: David Bryant Copeland @davetron5000 We’re so happy to bring you another Author Spotlight, a series where we sit dow...
#author-spotlight/book-agile-web-development-with-rails-6/book-rails-angular-postgres-and-bootstrap-second-edition/book-sustainable-web-development-with-ruby-on-rails
19 3230 15
New
PragmaticBookshelf
In The Spotlight

Spotlight: Tammy Coron (Author) Interview and AMA!
Author Spotlight: Tammy Coron @Paradox927 Gaming, and writing games in particular, is about passion, vision, experience, and immersio...
/swift/book-apple-game-frameworks-and-technologies
36 3275 18
New
Back to top

Latest in PragProg

Hotwire Native for Rails Developers: Collected Errata from our class at the University of Chicago
PragProg Customers
Agile Web Development with Rails 7 p1.0—May2023: foreman Gem (page 73)
PragProg Customers
The Definitive ANTLR 4 Reference: No download link provided for TestSimple.java (Page 240)
PragProg Customers
Programming Phoenix LiveView: Wrong closing element tag (page 106)
PragProg Customers
Programming Phoenix LiveView: Typo (page 103)
PragProg Customers
Programming Phoenix LiveView: Wrong HTML comment tag in example (page 102)
PragProg Customers
Programming Phoenix LiveView: typo missing word (page 101)
PragProg Customers
Agile Web Development with Rails 8 B7.0: order_mailer_test failing (page 196)
PragProg Customers
Simplicity: Let’s Change Our World — Again
PragProg Customers
Simplicity: "detail"→"derail" (page 74, B2.0)
PragProg Customers
Agile Web Development with Rails 8: Page 9
PragProg Customers
Kotlin Coroutine Confidence: Code does not run (pg 225)
PragProg Customers
Agile Web Development with Rails 8 B7.0: partials folder (page 151)
PragProg Customers
Kotlin Coroutine Confidence: Change function name, maybe (pg 182)
PragProg Customers
Modern Front-End Development for Rails, Second Edition: can't get code to run
PragProg Customers
Kotlin Coroutine Confidence: Book & source code do not match (pg 180)
PragProg Customers
Kotlin Coroutine Confidence: Add formatter off/on comments (pg 162)
PragProg Customers
Kotlin Coroutine Confidence: What's the IntelliJ IDEA inspection? (pg 133 & 141)
PragProg Customers
Kotlin Coroutine Confidence: Add `parallel/v9` (pg 125)
PragProg Customers
Agile Web Development with Rails 8_B7.0 (pages 93/94)
PragProg Customers

View all threads ❯

Latest (all)

Wikidive - AI-Guided Wikipedia Exploration
AI In The News
Node.js v24.0.1 released!
Backend News (official)
Mind-reading AI recreates what you're looking at with amazing accuracy
AI In The News
Why do LLMs have emergent properties?
AI In The News
Announcing Ivar: Ruby’s Missing Instance Variable Typo Warnings
Backend In The News
DOGE software engineer’s computer infected by info-stealing malware
In The News
Google hits back after Apple exec says AI is hurting search
macOS In The News
Present and Future of Kotlin for Web
Backend News (official)
Plasma 6.3.5 update for Kubuntu 25.04 available via PPA
Linux News (official)
Elon Musk is responsible for “killing the world’s poorest children,” says Bill Gates
In The News
Memory Safety Features in Zig
Backend In The News
Hardening GitHub Actions: Lessons from Recent Attacks | Wiz Blog
In The News
Erlang OTP-27.3.4 released!
Backend News (official)
Scala 3.3.6 LTS is now available!
Backend News (official)
AI is Like Cars
Blogs/Articles/Talks/Podcasts

View all threads ❯

We ❤️ helpful members!

We reward our most helpful members via our MOTM scheme - by giving away a whopping 25 books per year!

Member of the Month

Devtalk Sponsors

Pragmatic Bookshelf

Practical resources that improve the lives of professional developers.

Erlang Solutions

We build trusted fault-tolerant systems that scale to billions of users.

Manning Publications

Develop your skills with books, videos, and courses.

AppSignal

Catch errors, track performance, monitor hosts and more.

WyeWorks

Enabling companies to succeed by building software people love.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

Categories:

Popular Portals

Devtalk Sponsors

Pragmatic Bookshelf

Practical resources that improve the lives of professional developers.

Erlang Solutions

We build trusted fault-tolerant systems that scale to billions of users.

Manning Publications

Develop your skills with books, videos, and courses.

AppSignal

Catch errors, track performance, monitor hosts and more.

WyeWorks

Enabling companies to succeed by building software people love.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

We're in Beta

About us Mission Statement See our Roadmap