Transformers for machine learning: a deep dive

Transformers are becoming a core part of many neural network architectures, employed in a wide range of applications such as NLP, Speech Recognition, Time Series, and Computer Vision. Transformers have gone through many adaptations and alterations, resulting in newer techniques and methods. Transformers for Machine Learning: A Deep Dive is the first comprehensive book on transformers. Key Features: A comprehensive reference book for detailed explanations for every algorithm and techniques related to the transformers. 60+ transformer architectures covered in a comprehensive manner. A book for understanding how to apply the transformer techniques in speech, text, time series, and computer vision. Practical tips and tricks for each architecture and how to use it in the real world. Hands-on case studies and code snippets for theory and practical real-world analysis using the tools and libraries, all ready to run in Google Colab. The theoretical explanations of the state-of-the-art transformer architectures will appeal to postgraduate students and researchers (academic and industry) as it will provide a single entry point with deep discussions of a quickly moving field. The practical hands-on case studies and code will appeal to undergraduate students, practitioners, and professionals as it allows for quick experimentation and lowers the barrier to entry into the field.

List of Figures List of Tables Author Bios Foreword Preface Contributors Deep Learning and Transformers: An Introduction 1.1 DEEP LEARNING: A HISTORIC PERSPECTIVE1.2 TRANSFORMERS AND TAXONOMY 1.2.1 Modified Transformer Architecture 1.2.1.1 Transformer block changes 1.2.1.2 Transformer sublayer changes 1.2.2 Pretraining Methods and Applications 1.3 RESOURCES 1.3.1 Libraries and Implementations 1.3.2 Books 1.3.3 Courses, Tutorials, and Lectures 1.3.4 Case Studies and Details Transformers: Basics and Introduction 2.1 ENCODER-DECODER ARCHITECTURE 2.2 SEQUENCE TO SEQUENCE 2.2.1 Encoder 2.2.2 Decoder 2.2.3 Training 2.2.4 Issues with RNN-based Encoder Decoder 2.3 ATTENTION MECHANISM 2.3.1 Background 2.3.2 Types of Score-Based Attention 2.3.2.1 Dot Product (multiplicative) 2.3.2.2 Scaled Dot Product or multiplicative 2.3.2.3 Linear, MLP, or additive 2.3.3 Attention-based Sequence to Sequence 2.4 TRANSFORMER 2.4.1 Source and Target Representation 2.4.1.1 Word Embedding 2.4.1.2 Positional Encoding 2.4.2 Attention Layers 2.4.2.1 Self-Attention 2.4.2.2 Multi-Head Attention 2.4.2.3 Masked Multi-Head Attention 2.4.2.4 Encoder-Decoder Multi-Head Attention 2.4.3 Residuals and Layer Normalization …