Papperoni

2019

Huggingface's Transformers: State-of-the-Art Natural Language Processing

Thomas Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, Sam Shleifer, P. V. Platen, C. Ma, Yacine Jernite, J. Plu, Chenfeng Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, Alexander M. Rush

Open PDF Google Scholar

citations

Cite Score

80

AI summary

This paper introduces the Transformers library, an open-source library that provides carefully engineered Transformer architectures under a unified API and a curated collection of pre-trained models to facilitate NLP research and deployment, achieving great speedups using ONNX.

Main Contributions

Introduces Transformers, an open-source library for Transformer-based architectures.
Provides a unified API for state-of-the-art Transformer architectures.
Offers a curated collection of pre-trained models made by and available for the community.
Demonstrates the library's extensibility, simplicity, and robustness for industrial deployments.
Achieves significant speedups using ONNX for model deployment.

Abstract

Recent progress in natural language processing has been driven by advances in both model architecture and model pretraining. Transformer architectures have facilitated building higher-capacity models and pretraining has made it possible to effectively utilize this capacity for a wide variety of tasks. Transformers is an open-source library with the goal of opening up these advances to the wider machine learning community. The library consists of carefully engineered state-of-the art Transformer architectures under a unified API. Backing this library is a curated collection of pretrained models made by and available for the community. Transformers is designed to be extensible by researchers, simple for practitioners, and fast and robust in industrial deployments.

Citation Graph

Loading graph...

References [45]

Sort:

Filter:

[1]Attention Is All You Need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin - 2017

47 papers in library cite

I mean... it introduced Transformers!

[2]BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding

Jacob Devlin, M. W. Chang, K. Lee, Kristina Toutanova - 2018

39 papers in library cite

Simply amazing. It's very impressive how they make a leap vs. existing stuff (you can see from the references, pretty much no one is doing what they are doing, other than GPT)

[3]RoBERTa: A Robustly Optimized BERT Pretraining Approach

Yibo Liu, M. Ott, N. Goyal, J. Du, M. Joshi, Deli Chen, Omer Levy, Martha Lewis, Luke Zettlemoyer, Veselin Stoyanov - 2019

17 papers in library cite

I liked it a lot! It shows that you don't need to do something completely new to have good results and contribute to science. It could be a 5, but it's a 4 due to not bringing anything new

[4]Language Models Are Unsupervised Multitask Learners

Alec Radford, Jeffrey Wu, Rewon Child, D. Luan, Dario Amodei, Ilya Sutskever - 2019

27 papers in library cite

Amazing! Tons of important contributions. I think they could have explained the models a bit better, and I think this is where OpenAI starts to become evil (and not open)

[5]Deep Contextualized Word Representations

M. E. Peters, M. Neumann, M. Iyyer, Matt Gardner, C. Clark, K. Lee, L. S. Zettlemoyer - 2018

27 papers in library cite

I didn't really like the approach. Seems a bit derivative TBH. BERT seems more elegant.

[6]BART: Denoising Sequence-to-Sequence Pre-Training for Natural Language Generation, Translation, and Comprehension

Martha Lewis, Yibo Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, Omer Levy, Veselin Stoyanov, Luke Zettlemoyer - 2019

6 papers in library cite

I expected SO much less from this paper, but it is simple, intuitive, AND achieves SotA!

[7]XLNet: Generalized Autoregressive Pretraining for Language Understanding

Zhilin Yang, Z. Dai, Yining Yang, J. Carbonell, Ruslan Salakhutdinov, Quoc V. Le - 2019

11 papers in library cite

The method is nice and the results are very good but this paper is just soooo hard to follow...

[8]DistilBERT, a Distilled Version of BERT: Smaller, Faster, Cheaper and Lighter

Thomas Wolf - 2019

6 papers in library cite

Surprisingly simple and amazing results!

[9]ALBERT: A Lite BERT for Self-Supervised Learning of Language Representations

Z. Lan, Mark Chen, S. Goodman, Kevin Gimpel, P. Sharma, Radu Soricut - 2019

8 papers in library cite

I like how they can achieve very close results with very few params! Very nice tricks to do that as well.

[10]GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

A. Wang, A. Singh, J. Michael, F. Hill, Omer Levy, Samuel R. Bowman - 2018

26 papers in library cite

I like it, but it's just a mesh of different existing datasets and F1 score. Nothing new really but I get why it's important

[11]Universal Language Model Fine-Tuning for Text Classification

J. Howard, Sebastian Ruder - 2018

14 papers in library cite

Amazing! Bridging the gap between pre-training/finetuning in CV vs. NLP, plus giving amazing resuts!

[12]Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context

Z. Dai, Zhilin Yang, Yining Yang, W. Cohen, J. Carbonell, Quoc Le, Ruslan Salakhutdinov - 2019

9 papers in library cite

It's so cool to see context expansion without the need to actually expand context! Such a simple context and so effective!

[13]SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems

A. Wang, Y. Pruksachatkun, Nikita Nangia, A. Singh, J. Michael, F. Hill, Omer Levy, Samuel R. Bowman - 2019

15 papers in library cite

Nothing too surprising, just getting a bunch of stuff from different places and putting it all together. At least they do a good analysis of the benchmark vs. existing methodologies.

[14]Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

M. Shoeybi, M. Patwary, Raul Puri, P. Legresley, J. Casper, Bryan Catanzaro - 2019

3 papers in library cite

Simply amazing! very well explained and easy to follow + nice results, and shows that scaling is good for transformers

[15]SpanBERT: Improving Pre-Training by Representing and Predicting Spans

M. Joshi, Deli Chen, Yibo Liu, D. Weld, Luke Zettlemoyer, Omer Levy - 2019

5 papers in library cite

Yet another BERT, and it's not surprising since I've seen this in other papers. But is good nonetheless.

[16]Cross-Lingual Language Model Pretraining

G. Lample, Alexis Conneau - 2019

5 papers in library cite

I like how they learn cross-lingual stuff via vocabulary sharing. it's probably the start of the multi-language LMs.

[17]Learned in Translation: Contextualized Word Vectors

B. Mccann, J. Bradbury, Caiming Xiong, Richard Socher - 2017

14 papers in library cite

Doesn't seem too revolutionary tbh. It's a nice methodology though, and it's nice that they do it in a supervised way

[18]exBERT: A Visual Analysis Tool to Explore Learned Representations in Transformers Models

Ben Hoover, Hendrik Strobelt, Sebastian Gehrmann - 2019

1 paper in library cites

Ok, it seems like a nice tool, but TBH doesn't seem too useful since it's very exploratory and the phenomena can be a complex mix of heads.

[19]Exploring the Limits of Transfer Learning With a Unified Text-to-Text Transformer

Colin Raffel, Noam Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, Wentao Li, P. J. Liu - 2019

17 papers in library cite

44 pages; T5 paper - "unifying framework where all text-based NLP problems are cast as text-to-text tasks"

[20]Longformer: The Long-Document Transformer

I. Beltagy, M. E. Peters, A. Cohan - 2020

1 paper in library cites

A ton of citations! Seems relevant

[21]Reformer: The Efficient Transformer

N. Kitaev, Lukasz Kaiser, A. Levskaya - 2020

1 paper in library cites

Efficient transformer? Sounds interesting

[22]fairseq: A Fast, Extensible Toolkit for Sequence Modeling

M. Ott, S. Edunov, A. Baevski, A. Fan, S. Gross, N. Ng, D. Grangier, Michael Auli - 2019

4 papers in library cite

[23]Model Cards for Model Reporting

M. Mitchell, S. Wu, A. Zaldivar, P. Barnes, L. Vasserman, B. Hutchinson, E. Spitzer, I. D. Raji, T. Gebru - 2018

5 papers in library cite

Seems like something foundational - we use at nubank!

[24]OpenNMT: Open-Source Toolkit for Neural Machine Translation

G. Klein, Yoon Kim, Y. Deng, J. Senellart, A. Rush - 2017

4 papers in library cite

Framework that I never heard before...

[25]Allennlp: A Deep Semantic Natural Language Processing Platform

Matt Gardner, J. Grus, M. Neumann, Oyvind Tafjord, P. Dasigi, N. Liu, M. Peters, M. Schmitz, Luke Zettlemoyer - 2018

5 papers in library cite

Framework from alenai

[26]Marian: Fast Neural Machine Translation in C++

M. J. Dowmunt, R. Grundkiewicz, T. Dwojak, H. Hoang, K. Heafield, T. Neckermann, F. Seide, U. Germann, A. F. Aji, N. Bogoychev, A. F. Martins, Alexandra Birch - 2018

1 paper in library cites

Sounds interesting

[27]BERT Rediscovers the Classical NLP Pipeline

I. Tenney, Dipanjan Das, Ellie Pavlick - 2019

2 papers in library cite

[28]Lstmvis: A Tool for Visual Analysis of Hidden State Dynamics in Recurrent Neural Networks

Hendrik Strobelt, Sebastian Gehrmann, H. Pfister, Alexander M. Rush - 2017

2 papers in library cite

[29]The stanford coreNLP Natural Language Processing Toolkit

Christopher D. Manning, M. Surdeanu, J. Bauer, J. Finkel, S. J. Bethard, D. Mcclosky - 2014

6 papers in library cite

[30]Jiant: A Software Toolkit for Research on General-Purpose Text Understanding Models

Y. Pruksachatkun, P. Yeres, Haozhe Liu, Jason Phang, Phu Mon Htut, A. Wang, I. Tenney, Samuel R. Bowman - 2020

3 papers in library cite

[31]spaCy 2: Natural Language Understanding With Bloom Embeddings, Convolutional Neural Networks and Incremental Parsing

M. Honnibal, I. Montani - 2017

3 papers in library cite

[32]Comet: Commonsense Transformers for Automatic Knowledge Graph Construction

Antoine Bosselut, Hannah Rashkin, Maarten Sap, C. Malaviya, A. Elikyilmaz, Yejin Choi - 2019

2 papers in library cite

[33]ELECTRA: Pre-Training Text Encoders as Discriminators Rather Than Generators

K. Clark, M. T. Luong, Quoc V. Le, Christopher D. Manning - 2020

2 papers in library cite

[34]NLTK: The Natural Language Toolkit

E. Loper, S. Bird - 2002

2 papers in library cite

[35]{TVM}: An Automated End-to-End Optimizing Compiler for Deep Learning

T. Chen, T. Moreau, Zhejun Jiang, L. Zheng, E. Yan, H. Shen, M. Cowan, Lisa Wang, Y. Hu, L. Ceze - 2018

1 paper in library cites

[36]Flair: An Easy-to-Use Framework for State-of-the-Art NLP

A. Akbik, T. Bergmann, D. Blythe, K. Rasul, S. Schweter, R. Vollgraf - 2019

1 paper in library cites

[37]Flaubert: Unsupervised Language Model Pre-Training for French

H. Le, L. Vial, J. Frej, V. Segonne, M. Coavoux, B. Lecouteux, A. Allauzen, B. Crabbe, L. Besacier, D. Schwab - 2020

1 paper in library cites

[38]JAX: Composable Transformations of Python+NumPy Programs

J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, S. W. Milne - 2018

1 paper in library cites

Missing author listMissing year

[39]PyTorch Hub

1 paper in library cites

[40]SciBERT: A Pretrained Language Model for Scientific Text

I. Beltagy, K. Lo, A. Cohan - 2019

1 paper in library cites

[41]Stanza: A Python Natural Language Processing Toolkit for Many Human Languages

P. Qi, Y. Z. Zhang, Y. Z. Zhang, J. Bolton, Christopher D. Manning - 2020

1 paper in library cites

[42]Supervised Multimodal Bitransformers for Classifying Images and Text

Douwe Kiela, S. Bhooshan, H. Firooz, D. Testuggine - 2019

1 paper in library cites

[43]Tensor2tensor for Neural Machine Translation

Ashish Vaswani, Samy Bengio, E. Brevdo, Francois Chollet, Aidan N. Gomez, S. Gouws, Llion Jones, Lukasz Kaiser, N. Kalchbrenner, Niki Parmar, R. Sepassi, Noam Shazeer, Jakob Uszkoreit - 2018

1 paper in library cites

Missing author listMissing year

[44]TensorFlow Hub

1 paper in library cites

[45]Texar: A Modularized, Versatile, and Extensible Toolkit for Text Generation

Z. Hu, H. Shi, B. Tan, Wenyi Wang, Zhilin Yang, T. Z. Zhao, J. He, Lianhui Qin, D. Wang, X. Ma, Ze Liu, X. Liang, Wei Zhu, D. S. Sachan, E. P. Xing - 2018

1 paper in library cites

Cited by

7

papers in your library

Cites

28

papers in your library

Read

on December 25, 2025

It's a solid paper, but these framework papers are usually boring... Yet, it's nice to see that they started with reduced scope (and how large it got now).

Tags

Paper Aliases

No aliases