Programming Machine Learning: Help: weird results I don't understand

I encountered something that I can’t explain. Any help, tips, or explanations would be great.

I followed the one hidden layer example with 100 nodes and sigmoid activation function. Works great and I can get to 98.6% accuracy with a learning rate of 1.0, a batch size of 1000, and 100 epochs.

I then decided to exchange the sigmoid activation function with the ReLU. This is not done in the book at this point but it is easy enough to program the ReLU and its derivative. Here is the Python code I used:

def relu(z):
    return np.maximum(0.0,z)
def relu_gradient(z):
    return (z > 0)*1

Works fine as long as one reduces the learning rate which I did reduce to 0.1. It reaches about the same level of accuracy as with the sigmoid. I then did one insignificant change in the gradient of the ReLU. Instead of z > 0 I wrote z >= 0. So the code for the gradient was now:

def relu_gradient(z):
    return (z >= 0)*1

This I thought should not make any difference because how often would z be exactly zero? How often would the weighted sum of all inputs in the floating point format be exactly zero? Perhaps never. Even if it is zero occasionally it should hardly make any big difference. But to my surprise, it makes a profound difference. I can only get to about 95%. Why? Why is there almost 4% difference in accuracy for this insignificant change? There must be something weird happening.

I tried this several times to rule out that somehow the random initialization was unusual. I tried it with different learning rates and different batch sizes. None made any difference in the result. I checked for dead neurons. Found none. If somebody can tell me what is going on here I would really appreciate it.