A nice and practical introduction to Artificial Neural Networks (ANN) can be found in Andrej Karpathy’s blog (1)

Single Neuron

The equation representing a single neuron is

$$o = act(w.i+b)$$

where:

• $i$ is the input
• $w$ and $b$ are respectively the weight and the bias of the neuron
• $act$ is the activation function
• $o$ is the ouptut

Neural Net peoples enjoy representing their toys with block diagrams. A neuron can be described by the following diagram:

FigXXX - Single Neuron.

Activation Function

A whole range of activation functions are commonly used. The following plot ( created with this code ) illustrate those available in keras.

FigXXX - Activation functions.

Effect of Weights and Biases

By varying the weight and bias of a neuron, one is able to change the shape of the input/output function of the neuron, as shown in figure (XXX) ( created with this code )

FigXXX - Weights.

Neural Network

Neurons are commonly organized in layers such as illustrated in figure (XXX). The equation can be matricially summarized as:

Input and ouput can be multi dimensional (tensors)

Layers can be stacked into a network

FigXXX - Response of a dual layers network.

Network Fitting

back propagation

Examples

Learning a sine

code

FigXXX - identity activation.

FigXXX - logistic activation.

FigXXX - tanh activation.

FigXXX - relu activation.

Bibliography

• (1) Andrej Karpathy, Hacker’s guide to Neural Networks http://karpathy.github.io/neuralnets/