Differentiable Programming and Deep Learning

Learning Outcomes

Knowledge and Understanding

Having successfully completed this module, you will be able to demonstrate knowledge and understanding of:

• The key factors that have made deep learning successful for various applications
• Underlying mathematical and algorithmic principles of deep learning

Subject Specific Practical Skills

Having successfully completed this module you will be able to:

• Apply existing deep learning models to real datasets
• Gain facility in working with differentiable programming and deep learning libraries in order to create and evaluate differentiable programmes and deep networks

Subject Specific Intellectual and Research Skills

Having successfully completed this module you will be able to:

• Critically appraise the merits and shortcomings of model architectures on specific problems
• Critically appraise recent scientific literature in deep learning

Historical Developments

• MLPs
• CNNs, LeNet and the ImageNet competition
• RNNs
• Attention mechanisms and transformers

Mechanics

• Backpropagation
• Automatic differentiation

Learning Algorithms

• Initialisation
• SGD, Momentum, etc.

Auto-encoders

• variational
• denoising

CNNs

• Architectures
• Region Propositions
• Semantic Segmentation

Sequence Modelling

• Linear Embeddings
• RNNs

∗ LSTMs

∗ GRUs

∗ back-prop through time

• Transformers

Deep Learning Applications

• Computer Vision
• Natural Language Processing & Generation
• Speech

Differentiable Relaxations

• Straight-through operator
• Sampling

Current research challenges

Lectures, labs and guided self-study

Summative

This is how we’ll formally assess what you have learned in this module.

Referral

This is how we’ll assess you if you don’t meet the criteria to pass this module.

Repeat

An internal repeat is where you take all of your modules again, including any you passed. An external repeat is where you only re-take the modules you failed.