01 / AN OPEN QUESTION
What’s inside the sequence?
A transformer compresses an image into learned queries.
Reconstruction and pretrained visual features guide the learning.
Pick a query. Follow its effect.
See its attention and what changes when it is masked.
Couldn’t load this query.
The same query, across images.
Across ImageNet validation
Each tile is one query. The highlighted query follows the selection above.
One sequence. Complementary roles.
Content
Attention follows the subject.
Detail
Stable attention, local reconstruction effects.
02 / FROM OBSERVATION TO DESIGN
Different roles.
Different processing paths.
Token features choose the route.
Reconstruction and visual regularization shape how the experts learn.
Swipe to follow the encoder layers →
03 / THE DESIGN CHOICES
What makes it work?
04 / BEYOND THE TOKENIZER
Discrete tokens.
Serious generation.
MoETok also powers a unified autoregressive model,
predicting text and image tokens with the same next-token objective.
GenEval2
40.04 GM — 4.56 points from the
closed Gemini reference.
Leading generation among the unified discrete models evaluated in our paper.