Papperoni

2019

exBERT: A Visual Analysis Tool to Explore Learned Representations in Transformers Models

Ben Hoover, Hendrik Strobelt, Sebastian Gehrmann

Open PDF Google Scholar

citations

Cite Score

11

AI summary

This paper introduces EXBERT, an interactive visual analysis tool for Transformer models like BERT, that explores learned representations and attention mechanisms by matching user-specified inputs to similar contexts in a large annotated dataset, providing intuitive explanations of attention-head functions.

Main Contributions

Introduces EXBERT, an interactive visualization tool for exploring learned representations in Transformer models.
Combines static analysis with a dynamic and intuitive view of attention and internal representations.
Uses a nearest neighbor search of embeddings on an annotated corpus to provide insights.
Demonstrates applicability of EXBERT to the BERT model using the Wizard of Oz corpus.
Helps understand what information BERT encodes and how it uses attention by probing linguistic features and positional information.

Abstract

Large language models can produce powerful contextual representations that lead to improvements across many NLP tasks. Since these models are typically guided by a sequence of learned self attention mechanisms and may comprise undesired inductive biases, it is paramount to be able to explore what the attention has learned. While static analyses of these models lead to targeted insights, interactive tools are more dynamic and can help humans better gain an intuition for the model-internal reasoning process. We present EXBERT, an interactive tool named after the popular BERT language model, that provides insights into the meaning of the contextual representations by matching a human-specified input to similar contexts in a large annotated dataset. By aggregating the annotations of the matching similar contexts, EXBERT helps intuitively explain what each attention-head has learned.

Citation Graph

Loading graph...

References [23]

Sort:

Filter:

[1]Attention Is All You Need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin - 2017

47 papers in library cite

I mean... it introduced Transformers!

[2]BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding

Jacob Devlin, M. W. Chang, K. Lee, Kristina Toutanova - 2018

39 papers in library cite

Simply amazing. It's very impressive how they make a leap vs. existing stuff (you can see from the references, pretty much no one is doing what they are doing, other than GPT)

[3]RoBERTa: A Robustly Optimized BERT Pretraining Approach

Yibo Liu, M. Ott, N. Goyal, J. Du, M. Joshi, Deli Chen, Omer Levy, Martha Lewis, Luke Zettlemoyer, Veselin Stoyanov - 2019

17 papers in library cite

I liked it a lot! It shows that you don't need to do something completely new to have good results and contribute to science. It could be a 5, but it's a 4 due to not bringing anything new

[4]Language Models Are Unsupervised Multitask Learners

Alec Radford, Jeffrey Wu, Rewon Child, D. Luan, Dario Amodei, Ilya Sutskever - 2019

27 papers in library cite

Amazing! Tons of important contributions. I think they could have explained the models a bit better, and I think this is where OpenAI starts to become evil (and not open)

[5]DistilBERT, a Distilled Version of BERT: Smaller, Faster, Cheaper and Lighter

Thomas Wolf - 2019

6 papers in library cite

Surprisingly simple and amazing results!

[6]Neural Machine Translation of Rare Words with Subword Units

R. Sennrich, B. Haddow, Alexandra Birch - 2016

22 papers in library cite

Very good! Simple, explains quite a lot and good results. Forms the basis for a lot of stuff now!

[7]GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

A. Wang, A. Singh, J. Michael, F. Hill, Omer Levy, Samuel R. Bowman - 2018

26 papers in library cite

I like it, but it's just a mesh of different existing datasets and F1 score. Nothing new really but I get why it's important

[8]Huggingface's Transformers: State-of-the-Art Natural Language Processing

Thomas Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, Sam Shleifer, P. V. Platen, C. Ma, Yacine Jernite, J. Plu, Chenfeng Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, Alexander M. Rush - 2019

7 papers in library cite

It's a solid paper, but these framework papers are usually boring... Yet, it's nice to see that they started with reduced scope (and how large it got now).

[9]SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems

A. Wang, Y. Pruksachatkun, Nikita Nangia, A. Singh, J. Michael, F. Hill, Omer Levy, Samuel R. Bowman - 2019

15 papers in library cite

Nothing too surprising, just getting a bunch of stuff from different places and putting it all together. At least they do a good analysis of the benchmark vs. existing methodologies.

[10]Billion-Scale Similarity Search With Gpus

J. Johnson, M. Douze, Hervé Jégou - 2017

4 papers in library cite

Efficient KNN with GPUs

[11]Ctrl: A Conditional Transformer Language Model for Controllable Generation

Nitish Shirish Keskar, B. Mccann, L. R. Varshney, Caiming Xiong, Richard Socher - 2019

4 papers in library cite

[12]What Do You Learn From Context? Probing for Sentence Structure in Contextualized Word Representations

I. Tenney, P. Xia, Berlin Chen, A. Wang, A. Poliak, R. T. Mccoy, N. Kim, B. V. Durme, S. Bowman, Dipanjan Das, Ellie Pavlick - 2019

4 papers in library cite

Analysis of how transformers learn phrase structure

[13]An Analysis of Encoder Representations in Transformer-Based Machine Translation

A. Raganato, J. Tiedemann - 2018

2 papers in library cite

[14]Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the REST Can Be Pruned

E. Voita, D. Talbot, F. Moiseev, R. Sennrich, T. Ivan - 2019

2 papers in library cite

[15]BERT Rediscovers the Classical NLP Pipeline

I. Tenney, Dipanjan Das, Ellie Pavlick - 2019

2 papers in library cite

[16]Lstmvis: A Tool for Visual Analysis of Hidden State Dynamics in Recurrent Neural Networks

Hendrik Strobelt, Sebastian Gehrmann, H. Pfister, Alexander M. Rush - 2017

2 papers in library cite

[17]A Multiscale Visualization of Attention in the Transformer Model

J. Vig - 2019

1 paper in library cites

[18]Analyzing the Structure of Attention in a Transformer Language Model

J. Vig, Yonatan Belinkov - 2019

1 paper in library cites

[19]Attention Is Not Explanation

Shantanu Jain, B. C. Wallace - 2019

1 paper in library cites

Seems like they disagree with the idea that you can explain models looking at attention

[20]Attention Is Not Not Explanation

S. Wiegreffe, Y. Pinter - 2019

1 paper in library cites

Seems like they disagree with the paper that says that attention is not explanation

[21]On the Validity of Self-Attention as Explanation in Transformer Models

G. Brunner, Yibo Liu, D. Pascual, O. Richter, R. Wattenhofer - 2019

1 paper in library cites

They discuss that looking at attention may not be a good approach

[22]What Does BERT Look At? An Analysis of bert's Attention

K. Clark, U. Khandelwal, Omer Levy, Christopher D. Manning - 2019

1 paper in library cites

[23]Seq2seq-Vis: A Visual Debugging Tool for Sequence-to-Sequence Models

Hendrik Strobelt, Sebastian Gehrmann, M. Behrisch, A. Perer, H. Pfister, Alexander M. Rush - 2018

1 paper in library cites

Cited by

1

papers in your library

Cites

22

papers in your library

Read

on January 11, 2026

Ok, it seems like a nice tool, but TBH doesn't seem too useful since it's very exploratory and the phenomena can be a complex mix of heads.

Tags

Paper Aliases

No aliases