wasshuber

wasshuber

Programming Machine Learning: Help: weird results I don't understand

I encountered something that I can’t explain. Any help, tips, or explanations would be great.

I followed the one hidden layer example with 100 nodes and sigmoid activation function. Works great and I can get to 98.6% accuracy with a learning rate of 1.0, a batch size of 1000, and 100 epochs.

I then decided to exchange the sigmoid activation function with the ReLU. This is not done in the book at this point but it is easy enough to program the ReLU and its derivative. Here is the Python code I used:

def relu(z):
    return np.maximum(0.0,z)
def relu_gradient(z):
    return (z > 0)*1

Works fine as long as one reduces the learning rate which I did reduce to 0.1. It reaches about the same level of accuracy as with the sigmoid. I then did one insignificant change in the gradient of the ReLU. Instead of z > 0 I wrote z >= 0. So the code for the gradient was now:

def relu_gradient(z):
    return (z >= 0)*1

This I thought should not make any difference because how often would z be exactly zero? How often would the weighted sum of all inputs in the floating point format be exactly zero? Perhaps never. Even if it is zero occasionally it should hardly make any big difference. But to my surprise, it makes a profound difference. I can only get to about 95%. Why? Why is there almost 4% difference in accuracy for this insignificant change? There must be something weird happening.

I tried this several times to rule out that somehow the random initialization was unusual. I tried it with different learning rates and different batch sizes. None made any difference in the result. I checked for dead neurons. Found none. If somebody can tell me what is going on here I would really appreciate it.

Most Liked

wasshuber

wasshuber

Turns out it was a bug. Using the nomenclature of the book I was feeding h into the gradient function when I should have fed a into it. With the >= comparison this made all the gradients 1 and thus it acted like the linear activation function. (The linear activation function does produce about 94% accuracy.) Properly using the gradient function produces the expected results. It doesn’t matter if one uses > or >=.

I am happy I found this bug. But this is also part of why your book is so great. Programming it yourself forces one to understand the little details and allows one to change and modify the algorithms at the very core, which leads to much deeper understanding of how this all works.

Here is an insight that my experimentation produced. I tested a bunch of different activation functions including weird piecewise linear ones, periodic ones with sin and cos, combinations thereof etc. It surprised me that many work just as good as ReLU or sigmoid with a single hidden layer. (I intend to extend this experimentation to multiple hidden layers.) For example, it is kind of shocking at first that the absolute-value-function works just as good as ReLU. This kind of makes sense in the biological case. A neuron being a cell would not be completely identical to its neighbor neuron. Neurons in nature would certainly have different activation functions. Perhaps not as different as I experimented with but they would perhaps be noisy and distorted versions of sigmoid or ReLU. It doesn’t matter, it still works fine.

Further, this makes me wonder if perhaps that variation in activation functions in nature is a benefit. I am wondering if folks have tried to make nets where each activation function of each neuron is different. Perhaps that confers a training advantage to the network because not everything behaves in exactly the same way? I will try to explore this question. But first I need to extend the code to allow for multiple hidden layers.

This is one critique I have to make. In my opinion, it would have been better to go further with the code and extend it to multiple hidden layers than to switch to libraries. The point of the book is programming it yourself to allow full unmitigated experimentation. I would have added one or two chapters to extend the code further even if that would have meant leaving out libraries altogether. Numpy should be fast enough to explore multilayer networks on a single average computer.

Where Next?

Popular Pragmatic Bookshelf topics Top

brianokken
Many tasks_proj/tests directories exist in chapters 2, 3, 5 that have tests that use the custom markers smoke and get, which are not decl...
New
johnp
Hi Brian, Looks like the api for tinydb has changed a little. Noticed while working on chapter 7 that the .purge() call to the db throws...
New
yulkin
your book suggests to use Image.toByteData() to convert image to bytes, however I get the following error: "the getter ‘toByteData’ isn’t...
New
conradwt
First, the code resources: Page 237: rumbl_umbrella/apps/rumbl/mix.exs Note: That this file is missing. Page 238: rumbl_umbrella/app...
New
leba0495
Hello! Thanks for the great book. I was attempting the Trie (chap 17) exercises and for number 4 the solution provided for the autocorre...
New
digitalbias
Title: Build a Weather Station with Elixir and Nerves: Problem connecting to Postgres with Grafana on (page 64) If you follow the defau...
New
New
s2k
Hi all, currently I wonder how the Tailwind colours work (or don’t work). For example, in app/views/layouts/application.html.erb I have...
New
gorkaio
root_layout: {PentoWeb.LayoutView, :root}, This results in the following following error: no “root” html template defined for PentoWeb...
New
mcpierce
@mfazio23 I’ve applied the changes from Chapter 5 of the book and everything builds correctly and runs. But, when I try to start a game,...
New

Other popular topics Top

AstonJ
A thread that every forum needs! Simply post a link to a track on YouTube (or SoundCloud or Vimeo amongst others!) on a separate line an...
New
wolf4earth
@AstonJ prompted me to open this topic after I mentioned in the lockdown thread how I started to do a lot more for my fitness. https://f...
New
Exadra37
I am thinking in building or buy a desktop computer for programing, both professionally and on my free time, and my choice of OS is Linux...
New
AstonJ
This looks like a stunning keycap set :orange_heart: A LEGENDARY KEYBOARD LIVES ON When you bought an Apple Macintosh computer in the e...
New
PragmaticBookshelf
“Finding the Boundaries” Hero’s Journey with Noel Rappin @noelrappin Even when you’re ultimately right about what the future ho...
New
DevotionGeo
I have always used antique keyboards like Cherry MX 1800 or Cherry MX 8100 and almost always have modified the switches in some way, like...
New
New
AstonJ
If you’re getting errors like this: psql: error: connection to server on socket “/tmp/.s.PGSQL.5432” failed: No such file or directory ...
New
AstonJ
This is cool! DEEPSEEK-V3 ON M4 MAC: BLAZING FAST INFERENCE ON APPLE SILICON We just witnessed something incredible: the largest open-s...
New
AstonJ
This is a very quick guide, you just need to: Download LM Studio: https://lmstudio.ai/ Click on search Type DeepSeek, then select the o...
New

Sub Categories: