Neural network basics. The perceptron:

Neural networks: history and basic idea. Relationship between biology and mathematics. The perceptron: basic construction, training, activation functions. Practice: intro to TensorFlow and Keras.

Feedforward neural networks:

Feedforward neural networks. Gradient descent basics. Computation graph and computing gradients on the computation graph (backpropagation). Why deep learning is hard. Practice: a feedforward neural network on the MNIST dataset.

Optimization in neural networks:

Gradient descent and its problems. Nesterov’s momentum. Second order methods. Adaptive methods of gradient descent: Adagrad, Adadelta, Adam. Practice: comparing gradient descent variations.

Regularisation in neural networks:

Regularization: L1, L2, early stopping. Dropout. Practice: comparing regularisers.

Weight initialisation and batchnorm:

Weight initialization: supervised pre training idea, why straightforward random init fails, Xavier initialisation. Covariate shift and batch normalisation. Practice: putting everything together.

Convolutional neural networks II:

Modern convolutional architectures. AlexNet, VGG, Network in network, Inception. Residual connections and ResNet. Practice: image recognition.

Recurrent neural networks I:

Sequence-based problems. Recurrent neural networks: idea, backprop in RNNs. Simple RNNs and their problems. Vanishing and exploding gradients. Practice: seq2seq.

Recurrent neural networks II:

How to fix vanishing gradients. Constant error carousel: LSTM, GRU, and other architectures. Practice: sentiment analysis with RNNs.

Autoencoders:

Autoencoders. Sparse autoencoders, regularisation, denoising autoencoders. Deconvolution and convolutional autoencoders. Practice: autoencoders.

Generative adversarial networks:

Generative models and neural networks. Types of generative models. Generative adversarial networks: idea, DCGAN, AAE, modern applications. Practice: AAE on MNIST.

Deep reinforcement learning:

Reinforcement learning. Multiarmed bandits. Markov decision processes, the Bellman equations, policy iteration methods. Practice: multiarmed bandits.

Deep reinforcement learning II:

TD-learning, Q-learning. Reinforcement learning with neural networks: DQN and tricks (Double DQN, experience replay etc.). Policy gradient and actor-critic algorithms. Practice: OpenAI Gym.

Bayesian methods and neural networks:

Neuro bayesian methods. Variational autoencoders. A Bayesian look at dropout and dropout in RNNs. Practice: generating numbers with a variational autoencoder.

Final test:

Putting everything together on a real-world problem.

Convolutional neural networks I:

Convolutional architectures: idea and structure. Examples. Deconvolution and visualization in CNNs. Practice: CNNs for MNIST.