The Transformer - Part 3
-
- 0 Rating
- 0 Reviews
- 0 Students Enrolled
The Transformer - Part 3
Transformers build on stacked self-attention and feed-forward layers with residuals, layer norm, and positional encodings, enabling parallel sequence modeling. Attention (self, multi-head, masked) captures dependencies and order, while major variants include encoder-only BERT, decoder-only GPT, and encoder–decoder T5 and successors.
-
- 0 Rating
- 0 Reviews
- 0 Students Enrolled
Requirements
- see DELTA course requirements
General Overview
Description
Overview & building blocks of the Transformer: A stack of self-attention and position-wise feed-forward layers with residual connections and layer normalization, operating on token embeddings plus positional encodings. Architectures appear as encoder–decoder (sequence-to-sequence), encoder-only (understanding), or decoder-only (generation), enabling parallel computation over tokens without recurrence.
Transformer’s use of attention: Self-attention models token–token dependencies; multi-head attention captures diverse relations; masking enforces causality or ignores padding; positional encoding injects order information absent in attention.
Some popular Transformers: BERT (encoder-only, bidirectional, masked-LM pretraining), GPT (decoder-only, autoregressive generation with causal masks), and beyond—T5 (encoder–decoder, text-to-text), RoBERTa/DeBERTa (improved pretraining), Longformer/XLNet (long context and alternative objectives).
Courses that include this CL
Recommended for you
Meet the instructors !
Stefan received a PhD from the University of Hamburg in 2007, where he also completed his habilitation on decision analysis and support using ensemble forecasting models in 2012. He then joined the Humboldt-University of Berlin in 2014, where he heads the Chair of Information Systems at the School of Business and Economics. He serves as an associate editor for the International Journal of Business Analytics, Digital Finance, and the International Journal of Forecasting, and as department editor of Business and Information System Engineering (BISE). Stefan has secured substantial amounts of research funding and published several papers in leading international journals and conferences. His research concerns the support of managerial decision-making using quantitative empirical methods. He specializes in applications of (deep) machine learning techniques in the broad scope of marketing and risk analytics. Stefan actively participates in knowledge transfer and consulting projects with industry partners; from start-up companies to global players and not-for-profit organizations.