ManningBooks

ManningBooks

Devtalk Sponsor

AI Model Evaluation (Manning)

Before deploying an AI model into production, you need to know more than just its accuracy. Will it be fast enough for your users? Will it scale under real-world traffic? Can you trust its decisions in critical scenarios? AI Model Evaluation (Manning Publications) gives you the practical tools and strategies to answer these questions—and more—so you can ship AI systems that actually work in the real world.

Leemay Nassery

Before deploying an AI model into production, you need to know more than just its accuracy. Will it be fast enough for your users? Will it scale under real-world traffic? Can you trust its decisions in critical scenarios? AI Model Evaluation (Manning Publications) gives you the practical tools and strategies to answer these questions—and more—so you can ship AI systems that actually work in the real world.

What you’ll learn in AI Model Evaluation:

  • Build diagnostic offline evaluations to uncover hidden model behaviors
  • Use shadow traffic to simulate production conditions safely
  • Design A/B tests to measure real business and product impact
  • Spot nuanced failures with human-in-the-loop feedback
  • Scale evaluations with LLMs as automated judges

Author Leemay Nassery (Spotify, Comcast, Dropbox, Etsy) shares real-world insights on what it really takes to prepare models for production. You’ll go beyond standard accuracy metrics to evaluate latency, user experience, and long-term impact on product goals.

Inside the book:
Each chapter explores a different evaluation method, from offline testing and A/B experiments to shadow deployments and qualitative analysis. Hands-on examples, including a movie recommendation engine, make it easy to apply these techniques to your own AI projects.


Don’t forget you can get 45% off with your Devtalk discount! Just use the coupon code “devtalk.com” at checkout :+1:

Most Liked

peterchancc

peterchancc

We started exploring AI apps with LLMs, so this book should be a good reference for evaluating the open-source LLMs that we plan to use.

ManningBooks

ManningBooks

Devtalk Sponsor

Definitely. Here are some questions to help your team that the book addresses clearly:

  1. What happens if your model is “accurate” offline but tanks your engagement metrics in production — how would you know why?
    (Follow-up: Do you have evaluation strategies beyond just accuracy or F1?)

  2. When was the last time your team measured the system latency impact of a new AI model before launching it?
    (And what if the model slowed down page load time by 200ms — would you catch it before it hits users?)

  3. If a model makes worse predictions for a specific user segment, do you catch that in your current evaluation process? Or are those failures only visible after a launch?

  4. Before you ship a model, do you know how it affects:

  • Feature latency?
  • Cold start performance?
  • Infrastructure cost at scale?
    (Or are you finding out during the fire drill after launch?)

Are you still using the same evaluation metrics your team used 3 years ago?
(What if the nature of your product or user behavior has changed — and your evaluations are now stale?)

Hope this helps.

Cheers

peterchancc

peterchancc

Thanks!

Where Next?

Popular Ai topics Top

ManningBooks
Before deploying an AI model into production, you need to know more than just its accuracy. Will it be fast enough for your users? Will i...
New
ManningBooks
In Build a Reasoning Model (From Scratch), acclaimed ML research engineer Sebastian Raschka takes you inside the black box of reasoning-e...
New
ManningBooks
Based on Ilya Sutskever’s famous “must-read” list of ~30 AI papers, this book walks you through the research that shaped today’s deep lea...
New
ManningBooks
Learn AI Data Engineering in a Month of Lunches is a fast, friendly guide to integrating large language models into your data workflows. ...
New
New

Other popular topics Top

dasdom
No chair. I have a standing desk. This post was split into a dedicated thread from our thread about chairs :slight_smile:
New
AstonJ
There’s a whole world of custom keycaps out there that I didn’t know existed! Check out all of our Keycaps threads here: https://forum....
New
AstonJ
I’ve been hearing quite a lot of comments relating to the sound of a keyboard, with one of the most desirable of these called ‘thock’, he...
New
AstonJ
Just done a fresh install of macOS Big Sur and on installing Erlang I am getting: asdf install erlang 23.1.2 Configure failed. checking ...
New
New
husaindevelop
Inside our android webview app, we are trying to paste the copied content from another app eg (notes) using navigator.clipboard.readtext ...
New
hilfordjames
There appears to have been an update that has changed the terminology for what has previously been known as the Taskbar Overflow - this h...
New
PragmaticBookshelf
Develop, deploy, and debug BEAM applications using BEAMOps: a new paradigm that focuses on scalability, fault tolerance, and owning each ...
New
CommunityNews
A Brief Review of the Minisforum V3 AMD Tablet. Update: I have created an awesome-minisforum-v3 GitHub repository to list information fo...
New
Margaret
Ask Me Anything with Mark Volkmann @mvolkmann On February 24 and 25, we are giving you a chance to ask questions of PragProg author M...
New