Transformer Architecture Analysis

The attention mechanism computes:

Attention(Q, K, V) = softmax(QK^T / sqrt(d_k)) * V

Performance Results

Dataset	BLEU Score	Parameters
WMT 2014 EN-DE	28.4	65M
WMT 2014 EN-FR	41.0	213M

The scaled dot-product attention uses scaling factor 1/sqrt(d_k) where d_k is the dimension of keys.

Multi-head attention: MultiHead(Q,K,V) = Concat(head_1,...,head_h)W^O