This frame is this line
256 key words then g·256 query words. Key word j draws column j; each row is the average of g consecutive query words. Pause to edit: click any word and replace it.
How a line becomes a frame
x = RMSNorm(embed[ids]) q = RoPE(q_norm(x · Wq[head 0])) k = RoPE(k_norm(x · Wk[kv 0])) logits = q[256:] · k[:256]ᵀ / √128 # queries' rows over the keys' columns frame = mean over each g consecutive rows # 256 × 256
No softmax, no mask: the frame is the logit matrix itself, and every key comes before every query, so a causal mask
would hide nothing in it. Before RoPE, q and k depend only on the token, so this page
ships them for the 50,903 single-token words of Qwen3's vocabulary (26 MB, fp16) and does RoPE and Q·Kᵀ itself.
Checked against the dataset's play.py on all 6,570 frames: at most 1 of 255 display levels apart.
Inspired by brayevalerien's GPT-2 XL attention maps, this version draws the video from plain text. Every line was found by
exact coordinate descent over real tokens. After k_norm Qwen3's vocabulary keys are
nearly one-dimensional, so each row is a RoPE Fourier series whose coefficients the query words pick — hence the streaks.
Shuffle the keys of any frame and watch the picture fall apart.
16 heads, 1024² — too big for a browser tab
The mosaic16 text gives each of the 16 query heads of layer 0 one 256² tile of a 1024² frame (frames
1800–2699, 20,480 tokens per line). Running it here would need q/k tables for 16 heads (≈ 300 MB), so this is the
pre-rendered video of it, made by play.py --mosaic. The dataset's mosaic64 does the same with
the 64 heads of Qwen3-32B: a 2048² frame, the whole video
(watch it on YouTube).