Papperoni

[1]Deep Residual Learning for Image Recognition

K. He, X. Zhang, S. Ren, Jian Sun - 2016

20 papers in library cite

★★★★★

This is simply amazing. Very very simple idea, totally revolutionary. No maths, just "it works!". Amazing.

[2]Adam: A Method for Stochastic Optimization

D. P. Kingma, Jimmy Lei Ba - 2014

49 papers in library cite

★★★★★

Google Scholar

Amazing paper! Very well explained and huge impact. I am amazed that they made something so simple even when it requires a lot of background mathematical knowledge

[3]Attention Is All You Need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin - 2017

47 papers in library cite

★★★★★

AttentionTransformersMachine Translation

Google Scholar

I mean... it introduced Transformers!

[4]Very Deep Convolutional Networks for Large-Scale Image Recognition

K. Simonyan, Andrew Zisserman - 2014

20 papers in library cite

★★★★★

Google Scholar

This is very good! The great thing here is small filters and depth analysis, but truly they do some other stuff as well: SotA, generalization for other tasks, and open source their models. Very nice.

[5]BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding

Jacob Devlin, M. W. Chang, K. Lee, Kristina Toutanova - 2018

39 papers in library cite

★★★★★

Google Scholar

Simply amazing. It's very impressive how they make a leap vs. existing stuff (you can see from the references, pretty much no one is doing what they are doing, other than GPT)

[6]ImageNet Classification With Deep Convolutional Neural Networks

Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton - 2012

71 papers in library cite

★★★★★

Google Scholar

I'm giving this a 5 just because of the impact, but this is VEEERY derivative of earlier work. Kudos for them for putting it all together, but really there's nothing revolutionary here.

[7]Long Short-Term Memory

Sepp Hochreiter, Jürgen Schmidhuber - 1997

94 papers in library cite

★★★★★

Google Scholar

LSTMs FTW!

[8]Deep Learning

I. Goodfellow, Yoshua Bengio, Y. A. Courville, A. Aaron - 2016

5 papers in library cite

★★★★★

Google Scholar

It's an awesome review for people that are not familiar with deep learning, but it's very basic as well. Not too relevant given the context of other papers I read, but huge impact in terms of scientific communication.

[9]ImageNet: A Large-Scale Hierarchical Image Database

J. Deng, W. Dong, Richard Socher, L. J. Li, K. Li, Li Fei Fei - 2009

28 papers in library cite

★★★★★

Google Scholar

Very nice idea and huge impact!

[10]Generative Adversarial Nets

Ian J. Goodfellow, J. P. Abadie, M. Mirza, B. Xu, D. W. Farley, S. Ozair, Aaron Courville, Yoshua Bengio - 2014

2 papers in library cite

★★★★☆

Google Scholar

The idea is surprisingly simple. I was a bit afraid of the math, but it's very simple. There's proof that this converges and finds an optimum. I just wish they explained more, or gave better results. It just seems that it finishes and I'm still hungry.

[11]Gradient-Based Learning Applied to Document Recognition

Yann Lecun, Leon Bottou, Yoshua Bengio, Patrick Haffner - 1998

65 papers in library cite

★☆☆☆☆

Google Scholar

I absolutely hated this paper. Has ~50 pages but seems like 200 pages. Takes too long to explain some things that really is just repeating itself. Also doesn't seem to add too much on top of LeNet-5. Also, focuses a lot on GTNs, which really didn't stick.

[12]Language Models Are Few-Shot Learners

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, Dario Amodei - 2020

21 papers in library cite

★★★☆☆

Vetto Study

Google Scholar

It's just training the GPT arch with more data and more params. Nothing too surprising, but kudos for identifying and formalizing few-shot learning.

[13]Going Deeper With Convolutions

Christian Szegedy, Weizhou Liu, Y. Jia, P. Sermanet, S. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew Rabinovich - 2015

20 papers in library cite

★★★★☆

Google Scholar

Introduced the inception algorithm, which is nice. The paper is quite good, but I had to google some stuff to understand it fully. Nice contribution and SotA, but TBH I felt that it wasn't toooo good of a read.

[14]Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

S. Ioffe, Christian Szegedy - 2015

18 papers in library cite

★★★★★

Google Scholar

Very good paper! Similar feel as ResNets: simple idea, elegant. Not too mathy

[15]Microsoft COCO: Common Objects in Context

T. Y. Lin, M. Maire, S. Belongie, James Hays, Pietro Perona, D. Ramanan, Piotr Dollar, C. L. Zitnick - 2014

14 papers in library cite

★★★★☆

Google Scholar

I liked this paper a lot. It's a bit long and I was already a bit tired, but it was nice overall.

[16]Fully Convolutional Networks for Semantic Segmentation

J. Long, E. Shelhamer, Trevor Darrell - 2015

7 papers in library cite

★★★☆☆

Google Scholar

I didn't really like the way the paper is written and the results seem a bit underwhelming. However, it's nice that they do it in a fully convolutional way and that they increase the speed.

[17]Dropout: A Simple Way to Prevent Neural Networks From Overfitting

N. Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan R. Salakhutdinov - 2014

20 papers in library cite

★★★★☆

Google Scholar

Good paper, but it's mostly a review of the method described in the other paper with more results. It's longer as well, so I would suggest just reading the other one.

[18]Densely Connected Convolutional Networks

G. Huang, Ze Liu, K. Weinberger, Laurens Van Der Maaten - 2017

5 papers in library cite

★★★★★

Google Scholar

I liked this paper so much! The way that it's written makes it very easy to follow. Results are nice, explanations are intuitive. Very nice!

[19]Faster R-CNN: Towards Real-Time Object Detection With Region Proposal Networks

Jian Sun - 2016

2 papers in library cite

★★★★★

Google Scholar

Now, this is the pinnacle of object detection so far! Very nice to see the progress from Deep CNNs -> R-CNN -> Fast R-CNN -> Faster R-CNN. Very nice improvements on detection and speed, and the paper is incredibly written. The abstract is probably the best I've read in a long time.

[20]Efficient Estimation of Word Representations in Vector Space

Tomas Mikolov, K. Chen, G. S. Corrado, Jeffrey Dean - 2013

26 papers in library cite

★★★★★

Google Scholar

Expanded wor2vec. Very nice overall.

[21]Distributed Representations of Words and Phrases and Their Compositionality

Tomas Mikolov, Ilya Sutskever, K. Chen, G. S. Corrado, Jeffrey Dean - 2013

32 papers in library cite

★★★★★

Google Scholar

Introduced word2vec. Game changer.

[22]Visualizing Data Using t-SNE

Geoffrey Hinton - 2008

7 papers in library cite

★★★★★

Google Scholar

Amazing. Simple. Impactful. Easy to understand. Masterpiece.

[23]GloVe: Global Vectors for Word Representation

Jeffrey Pennington, Richard Socher, Christopher D. Manning - 2014

31 papers in library cite

★★☆☆☆

Google Scholar

Not a bad paper, I just don't like the motivation and I think the methodology is poorly explained and hard to follow. I can't deny the good results though...

[24]Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation

Ross Girshick, J. Donahue, Trevor Darrell, Jitendra Malik - 2014

18 papers in library cite

★★★★☆

Google Scholar

Good results, beat overfeat, used pretraining for improving performance. Only issue is that the paper is overly long...

[25]Learning Representations by Back-Propagating Errors

D. E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams - 1986

34 papers in library cite

★★★★★

Google Scholar

Introduced backprop. Short and simple.

[26]Fast R-CNN

Ross Girshick - 2015

2 papers in library cite

★★★★☆

Google Scholar

Very nice improvement over R-CNN! I like that they start pushing towards having an end-to-end framework, but this still falls short for using a separate region proposal module.

[27]Rethinking the Inception Architecture for Computer Vision

Zbigniew Wojna - 2015

5 papers in library cite

★★★☆☆

Google Scholar

It's nice to see all of the performance optimizations they do, but it's very derivative

[28]Proximal Policy Optimization Algorithms

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov - 2017

10 papers in library cite

★★★★★

RLHFVetto Study

Google Scholar

Very simple methodology and very well explained. I also liked that they did a good job on motivating the method.

[29]Neural Machine Translation by Jointly Learning to Align and Translate

D. Bahdanau, Kyunghyun Cho, Yoshua Bengio - 2014

59 papers in library cite

★★★★★

Google Scholar

Introduces the attention mechanism - amazing overall

[30]Human-Level Control Through Deep Reinforcement Learning

V. Mnih - 2015

9 papers in library cite

★★★★★

Google Scholar

Maybe the first instance of RL with NNs! I am impressed that it seems quite straightforward to implement (maybe tricky to get working).

[31]Learning Multiple Layers of Features From Tiny Images

Alex Krizhevsky - 2009

27 papers in library cite

★★★☆☆

Google Scholar

It's alright. It mainly focuses on RBMs and their features and the actual part that describes the dataset is like 1 page. However, it's maybe the best intuitive description of an RBM I have seen. Other than that, it reads very much like an undergraduate thesis.

[32]Learning Phrase Representations Using RNN Encoder-Decoder for Statistical Machine Translation

Kyunghyun Cho, B. V. Merrienboer, C. G. Gulcehre, D. Bahdanau, F. Bougares, Holger Schwenk, Yoshua Bengio - 2014

38 papers in library cite

★★★★★

Google Scholar

Introduces RNN encoder-decoder. I love it :)

[33]Multilayer Feedforward Networks Are Universal Approximators

Kur Hornik, Maxwell Stinchcombe, Halbert White - 1989

3 papers in library cite

★★★☆☆

Google Scholar

It's a bunch of proofs, so not nice to read but it's very impactful.

[34]BLUE: A Method for Automatic Evaluation of Machine Translation

K. Papineni, S. Roukos, T. Ward, Wei Jing Zhu - 2002

19 papers in library cite

★★★★★

Google Scholar

Very cool idea. Simple yet very impactful!

[35]Learning Internal Representations by Error Propagation

D. E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams - 1986

46 papers in library cite

★★★★★

Google Scholar

I expected very little of this, but was so good in explaining concepts! Very good read. It gets a bit boring when it starts explaining things by the end of the chapter, but good nonetheless.

[36]Decoupled Weight Decay Regularization

I. Loshchilov, Frank Hutter - 2017

7 papers in library cite

★★★★★

Google Scholar

Very nice! Very simple to understand despite the complex topic. Very good results also

[37]A Survey on Transfer Learning

Sinno Jialin Pan, Qiang Yang - 2010

1 paper in library cites

★★★★☆

Google Scholar

I love that it formalized what is transfer learning and the main concepts. I think the citations are because of that as well. The part where they survey the methods is a bit boring and TBH most methods are not relevant. Also, transfer learning occurs quite naturally in neural networks, not so much in traditional methods.

[38]Computing Machinery and Intelligence

A. M. Turing - 1950

8 papers in library cite

★★★★☆

Google Scholar

A must-read, but it gets a bit boring halfway through (as he is describing every counter argument).

[39]Sequence to Sequence Learning With Neural Networks

Ilya Sutskever, Oriol Vinyals, Quoc V. Le - 2014

58 papers in library cite

★★★★☆

Google Scholar

Good paper, but I think it only got famous because they set a new good baseline for NNs in MT. Their main contribution was reversing the source sentence TBH.

[40]Neural Networks and Physical Systems With Emergent Collective Computational Abilities

J. J. Hopfield - 1982

8 papers in library cite

★★★☆☆

Google Scholar

Very theory heavy and doesn't seem to add much... Famous because it introduced the "Hopfield network" - "At a time when research in neural networks was facing a lull, Hopfield's paper injected a fresh perspective by demonstrating how a network of simple, interconnected units could exhibit complex computational abilities as a collective, emergent property."

[41]Understanding the Difficulty of Training Deep Feedforward Neural Networks

Yoshua Bengio - 2010

20 papers in library cite

★★★☆☆

Google Scholar

Nice but underwhelming results (they still underperform vs. pretraining). I also didn't really like the way it's written. It's not bad, it's just a bit clunky. Worth the read though.

[42]Rectified Linear Units Improve Restricted Boltzmann Machines

V. Nair, Geoffrey E. Hinton - 2010

18 papers in library cite

★★☆☆☆

Google Scholar

I hate when people introduce a new idea but don't care to explain it! This is terrible compared to bengio's paper.

[43]Explaining and Harnessing Adversarial Examples

Ian J. Goodfellow, J. Shlens, Christian Szegedy - 2015

4 papers in library cite

★★★★★

Google Scholar

It feels that it is an extension of the previous paper, which is nice. It is less original but also easier to read!

[44]Delving Deep Into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

K. He, X. Zhang, S. Ren, Jian Sun - 2015

10 papers in library cite

★★★★☆

Google Scholar

I think the PRELU idea didn't catch on, but the initialization is very nice! Good read.

[45]The PASCAL Visual Object Classes (VOC) Challenge

Mark Everingham, Luc Van Gool, Christopher K. I. Williams, John Winn, Andrew Zisserman - 2010

7 papers in library cite

★★★☆☆

Google Scholar

Good overview of the VOC challenge and good read, but too long and the part that they describe methods and results is very boring. Also, not related to NNs at the time!

[46]Training Language Models to Follow Instructions With Human Feedback

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, C. Wainwright, Pamela Mishkin, Chiyuan Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, Ryan Lowe - 2022

11 papers in library cite

★★★☆☆

RLHFVetto Study

Google Scholar

No new research here. Only true contribution is scaling RLHF to GPT 3.

[47]Distilling the Knowledge in a Neural Network

Jeffrey Dean - 2015

6 papers in library cite

★★★★★

Google Scholar

Very nice, very relevant.

[48]Reducing the Dimensionality of Data With Neural Networks

Geoffrey Hinton, Ruslan Salakhutdinov - 2006

37 papers in library cite

★★★★☆

Google Scholar

I didn't like the way this is written, very hard to understand without a ton of background knowledge. But hey, it's the first deep learning model!

[49]Visualizing and Understanding Convolutional Networks

Matthew D. Zeiler, Rob Fergus - 2014

15 papers in library cite

★★★★☆

Google Scholar

Very good explanation and visualization of CNNs, and also nice that they use their findings to improve the performance. The ablation study is also nice.

[50]MapReduce: Simplified Data Processing on Large Clusters

Jeffrey Dean, Sanjay Ghemawat - 2004

4 papers in library cite

★★★★★

Google Scholar

Amazing paper that discusses how MapReduce works! Very simple, and really nice to read something not related to AI. A shame that it's off-topic.

[51]Mastering the Game of Go With Deep Neural Networks and Tree Search

D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. V. D. Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, Ilya Sutskever, T. Lillicrap, M. Leach, Koray Kavukcuoglu, T. Graepel, Demis Hassabis - 2016

5 papers in library cite

★★★★★

Google Scholar

Surprisingly simple. Years ago when I first heard of AlphaGo it seemed such a different/far away concept - "wow, it's AI!". Now that I understand this, it sounds like something I could have worked on (if I had my current knowledge)

[52]RoBERTa: A Robustly Optimized BERT Pretraining Approach

Yibo Liu, M. Ott, N. Goyal, J. Du, M. Joshi, Deli Chen, Omer Levy, Martha Lewis, Luke Zettlemoyer, Veselin Stoyanov - 2019

17 papers in library cite

★★★★☆

Google Scholar

I liked it a lot! It shows that you don't need to do something completely new to have good results and contribute to science. It could be a 5, but it's a 4 due to not bringing anything new

[53]A Fast Learning Algorithm for Deep Belief Nets

Geoffrey E. Hinton, S. Osindero, Y. Teh - 2006

43 papers in library cite

★★★☆☆

Google Scholar

The paper does not explain anything. It just throws the idea and a bunch of math, but doesn't really care to explain the concepts.

[54]WordNet: A Lexical Database for English

G. Miller - 1995

5 papers in library cite

★★★☆☆

Google Scholar

Meh. It seems like it was publish so that people could cite the dataset somehow. Nothing interesting, but quick read and very used.

[55]The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain

F. Rosenblatt - 1958

3 papers in library cite

★★★★★

Google Scholar

Very nice intro to NNs!

[56]ROUGE: A Package for Automatic Evaluation of Summaries

Chin Yew Lin - 2004

9 papers in library cite

★☆☆☆☆

Google Scholar

Wow, very poorly written and badly explained. This is the exact opposite of BLEU

[57]Intriguing Properties of Neural Networks

Rob Fergus - 2014

7 papers in library cite

★★★★★

Google Scholar

Very nice, and the first to notice how flaky NNs are. I think the end they went overboard with math, but the rest of the paper is very good.

[58]Convolutional Neural Networks for Sentence Classification

Yoon Kim - 2014

8 papers in library cite

★★★★☆

Google Scholar

It's nice, goes straight to the point. I can see why it has tons of citations. However, I am not sure it was as impactful as 20k citations.

[59]Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling

J. Chung, C. G. Gulcehre, Kyunghyun Cho, Yoshua Bengio - 2014

11 papers in library cite

★★★☆☆

Google Scholar

It's a good paper but results are veeeery underwhelming.

[60]Representation Learning: A Review and New Perspectives

Yoshua Bengio, Aaron Courville, A. A. Vincent, P. Pascal - 2013

4 papers in library cite

★★★☆☆

Google Scholar

Good in spirit but is too long and covers a lot of things that are not relevant anymore.

[61]Language Models Are Unsupervised Multitask Learners

Alec Radford, Jeffrey Wu, Rewon Child, D. Luan, Dario Amodei, Ilya Sutskever - 2019

27 papers in library cite

★★★★★

Google Scholar

Amazing! Tons of important contributions. I think they could have explained the models a bit better, and I think this is where OpenAI starts to become evil (and not open)

[62]Backpropagation Applied to Handwritten Zip-Code Recognition

Yann Lecun, B. Boser, John S. Denker, D. Henderson, R. E. Howard, W. Hubbard, L. D. Jackal - 1989

24 papers in library cite

★★★★★

Google Scholar

The first convolution NN! Very simple concept and very simply explained. Very good results and overall a good read.

[63]No Free Lunch Theorems for Optimization

David H. Wolpert, William G. Macready - 1997

1 paper in library cites

★★★★☆

Google Scholar

Nice read, but it gets a bit tough to follow after Section 4. Very nice results thought... I missed some practical implications of this. Like, results are interesting but so what? Seems like this expanded into some pretty interesting interpretations, but they don't discuss much (there is one paragraph that does)

[64]Caffe: Convolutional Architecture for Fast Feature Embedding

Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, Ross Girshick, S. Guadarrama, Trevor Darrell - 2014

12 papers in library cite

★★★☆☆

Google Scholar

Nothing new really, but worth the read. It's nice because it's the precursor to current AI frameworks + has a Python interface. Also good that model representation is separate from implementation

[65]Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition

K. He, X. Zhang, S. Ren, Jian Sun - 2014

6 papers in library cite

★★★★☆

Google Scholar

Very simple, general and effective method. The paper ends at page ~4 TBH, the rest is just results and gets boring. Good contribution though.

[66]Finding Structure in Time

Jeffrey L. Elman - 1990

23 papers in library cite

★★★★☆

Google Scholar

Good paper overall that introduces the concept of an RNN. However, applications and results are still very primitive.

[67]Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks

Chelsea Finn, P. Abbeel, Sergey Levine - 2017

4 papers in library cite

★★★★★

Google Scholar

Very nice and very well explained. At first it seems that it will be tough to follow, but once they explain it it gets very intuitive.

[68]Deep Contextualized Word Representations

M. E. Peters, M. Neumann, M. Iyyer, Matt Gardner, C. Clark, K. Lee, L. S. Zettlemoyer - 2018

27 papers in library cite

★★★☆☆

Google Scholar

I didn't really like the approach. Seems a bit derivative TBH. BERT seems more elegant.

[69]Automatic Differentiation in PyTorch

A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. Devito, Zongyu Lin, A. Desmaison, L. Antiga, Adam Lerer - 2017

3 papers in library cite

★★☆☆☆

Google Scholar

Very boring and poorly explained. It looks like they said "we need to cash in some citations! quick, publish anything you have!!!"

[70]Random Search for Hyper-Parameter Optimization

James Bergstra, Yoshua Bengio - 2012

7 papers in library cite

★★★★★

Google Scholar

It seems crazy that it was only in 2012 that they found out that random search was good! Still, kudos on them for noticing, and the paper is just so easy to follow and enjoyable!

[71]Adaptive Subgradient Methods for Online Learning and Stochastic Optimization

John Duchi, Elad Hazan, Yoram Singer - 2011

19 papers in library cite

★★☆☆☆

Google Scholar

I actually skimmed through most of this. It's not a bad paper, but it's a math paper, not AI.

[72]Layer Normalization

Jimmy Lei Ba, R. Kiros, Geoffrey E. Hinton - 2016

14 papers in library cite

★★★★★

Google Scholar

Very nice! At first I had a little bit of prejudice because it seemed way too mathy, but actually the math is easy to follow and the results are very nice.

[73]Improving Language Understanding by Generative Pre-Training

Alec Radford, K. Narasimhan, T. Salimans, Ilya Sutskever - 2018

23 papers in library cite

★★★★★

Google Scholar

Very simple and very nice! Easy to understand and revolutionary maybe?

[74]A Stochastic Approximation Method

Sutton Monro - 1951

3 papers in library cite

★★★☆☆

Google Scholar

It's math. But it actually does a somewhat good job at explaining (but I don't think they tried too hard). It gets way better near the end.

[75]BART: Denoising Sequence-to-Sequence Pre-Training for Natural Language Generation, Translation, and Comprehension

Martha Lewis, Yibo Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, Omer Levy, Veselin Stoyanov, Luke Zettlemoyer - 2019

6 papers in library cite

★★★★★

Google Scholar

I expected SO much less from this paper, but it is simple, intuitive, AND achieves SotA!

[76]Enriching Word Vectors With Subword Information

Tomas Mikolov - 2017

7 papers in library cite

★★★☆☆

Google Scholar

It's just word2vec with ngrams instead of full words. Unsurprising, but important nonetheless

[77]Identity Mappings in Deep Residual Networks

K. He, X. Zhang, S. Ren, Jian Sun - 2016

4 papers in library cite

★★★★★

Google Scholar

This paper is amazing - simple, easy to follow, intuitive, while also being impactful! ResNets are already a big leap forward, and they can improve on that

[78]Deep Neural Networks for Acoustic Modeling in Speech Recognition

Geoffrey Hinton - 2012

21 papers in library cite

★★★★☆

Google Scholar

The core of the paper itself is a bit boring and doesn't introduce anything new (just RBMs and DBNs again) but I am giving this a 4 because it's probably the best explanation of RBMs and DBNs I've read so far.

[79]Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning

R. Williams - 1992

11 papers in library cite

★★★☆☆

RLHF

Google Scholar

It's alright for formalizing the concept, but it's a bit boring and doesn't add a lot from the middle on. Focuses too much in reviewing existing techniques and in stochastic units.

[80]Show, Attend and Tell: Neural Image Caption Generation With Visual Attention

K. Xu, Jimmy Lei Ba, R. Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, R. Zemel, Yoshua Bengio - 2015

12 papers in library cite

★★★★☆

Google Scholar

It's a nice paper. I liked the soft attention way more than the hard one, and I am a bit mad that it wasn't the best lol And also it's the first paper I read about multimodality, but it seems that this was bustling at the time. Also results are kinda bad.

[81]Learning Long-Term Dependencies With Gradient Descent Is Difficult

Yoshua Bengio, Patrice Simard, Paolo Frasconi - 1994

31 papers in library cite

★★★☆☆

Google Scholar

The first ones to notice that there is a problem with gradient descent, but way too mathy for me.

[82]TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems

M. Abadi, Akshat Agarwal, P. Barham, E. Brevdo, Ziru Chen, C. Citro, G. Corrado, A. Davis, Jeffrey Dean, M. Devin, Sanjay Ghemawat, I. Goodfellow, A. Harp, Geoffrey Irving, M. Isard, Y. Jia, R. Jozefowicz, Lukasz Kaiser, M. Kudlur, J. Levenberg, D. Mane, R. Monga, S. Moore, D. Murray, Christopher Olah, M. Schuster, J. Shlens, B. Steiner, Ilya Sutskever, K. Talwar, P. Tucker, Vincent Vanhoucke, V. Vasudevan, F. Viegas, Oriol Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, Xiaoqiang Zheng - 2015

11 papers in library cite

★★★★★

Google Scholar

This should be the golden standard to what framework papers should be. It's large, but it's not boring at all. Explains the core concepts while not going too deep as to describe unimportant things; explains design decisions and shortcomings... overall amazing

[83]Distributed Representations of Sentences and Documents

Quoc Le, Tomas Mikolov - 2014

13 papers in library cite

★★★★★

Google Scholar

Introduced document embeddings. Very nice overall.

[84]Deep Sparse Rectifier neural Networks

Xavier Glorot, Antoine Bordes, Yoshua Bengio - 2011

17 papers in library cite

★★★★★

Google Scholar

How can you not love Bengio? This paper is everything that Hinton's is not. Really well explained, surprising results, good comparisons... Amazing impact!

[85]Learning Deep Architectures for AI

Yoshua Bengio - 2009

25 papers in library cite

★★★★☆

Google Scholar

It's a nice overview. Some sections get very theoretical, but the first half is very good and I feel that it does a waaaay better job of explaining RBMs and DBNs than other papers. This feels like Bengio is taking your hand and saying "if you don't know what's going on, here you go, everything you need to know to jump into the deep nets train"

[86]Speech Recognition With Deep Recurrent Neural Networks

Geoffrey Hinton - 2013

13 papers in library cite

★★★★★

Google Scholar

Small, simple, nice results. First usage of deep BiLSTM?

[87]A Neural Probabilistic Language Model

Yoshua Bengio, R. Ducharme, Pascal Vincent - 2001

62 papers in library cite

★★★★★

Google Scholar

What started it all. Very simple and elegant.

[88]How Transferable Are Features in Deep Neural Networks?

Hod Lipson - 2014

2 papers in library cite

★★★★★

Google Scholar

Such a good paper to analyze pretraining behavior: simple, intuitive, nice to read

[89]Bidirectional Recurrent Neural Networks

M. Schuster, Kuldip K. Paliwal - 1997

10 papers in library cite

★★★★☆

Google Scholar

Introduced the BRNN concept. Good paper.

[90]Improving Neural Networks by Preventing Co-Adaptation of Feature Detectors

Geoffrey E. Hinton, N. Srivastava, Alex Krizhevsky, Ilya Sutskever, Ruslan R. Salakhutdinov - 2012

25 papers in library cite

★★★★★

Google Scholar

Dropout, super impactful. The idea that you are training many estimators at once is also very nice.

[91]XLNet: Generalized Autoregressive Pretraining for Language Understanding

Zhilin Yang, Z. Dai, Yining Yang, J. Carbonell, Ruslan Salakhutdinov, Quoc V. Le - 2019

11 papers in library cite

★★★☆☆

Google Scholar

The method is nice and the results are very good but this paper is just soooo hard to follow...

[92]Effective Approaches to Attention-Based Neural Machine Translation

T. Luong, H. Pham, Christopher D. Manning - 2015

15 papers in library cite

★★★☆☆

Google Scholar

Good paper, but very derivative. Attention methods start getting very complicated... I understand why Transformers took over TBH

[93]SGDR: Stochastic Gradient Descent With Warm Restarts

Frank Hutter - 2017

4 papers in library cite

★★★★☆

Google Scholar

Very simple, intuitive, and effective. I just hoped they provided a better motivation for the restarts rather than "it exists in other literature"

[94]Multitask Learning

Rich Caruana - 1997

13 papers in library cite

★★★☆☆

Google Scholar

I expected waaaaaay more from this paper. The idea is sooooo simple and the results are underwhelming. Also, 30 pages for something that could be said in 10. The writing style is a bit boring. TBH it seems like it's just a re-writing of Caruana's PhD thesis.

[95]Building a Large Annotated Corpus of English: The Penn Treebank

M. P. Marcus, B. Santorini, Mary Ann Marcinkiewicz - 1993

22 papers in library cite

★★★☆☆

Google Scholar

Well, not really interesting but very cool to see how the peen tree bank was made.

[96]Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories

Svetlana Lazebnik, Cordelia Schmid, Jean Ponce - 2006

14 papers in library cite

★★★★☆

Google Scholar

It's a fun read, but in the end is just an application of the spatial pyramid matching kernel from the other paper.

[97]Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank

Richard Socher, A. Perelygin, Jeffrey Wu, J. Chuang, C. Manning, A. Ng, Christopher Potts - 2013

24 papers in library cite

★★★☆☆

Google Scholar

I didn't really like the first paper and I don't really like this one. I think the dataset is more influential than the methodology. I think Stanford folks are too focused on old school NLP.

[98]Overcoming Catastrophic Forgetting in Neural Networks

J. Kirkpatrick, Razvan Pascanu, N. C. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. G. Barwinska, Demis Hassabis, C. Clopath, D. Kumaran, Raia Hadsell - 2017

5 papers in library cite

★★★★★

Google Scholar

Very nice, intuitive, and impressive results.

[99]Natural Language Processing (Almost) From Scratch

Ronan Collobert, Jason Weston, Leon Bottou, M. Karlen, Koray Kavukcuoglu, P. P. Kuksa - 2011

23 papers in library cite

★★★☆☆

Google Scholar

It's one of those huge papers that gets very tiring by the end. However, it's a very nice contribution. I am biased towards not liking it because it's basically old style NLP using NNs, which to me is a bit meh. However, I think this sets very important foundations for pretraining, embeddings, and proving that NNs rock.

[100]Rectifier Nonlinearities Improve Neural Network Acoustic Models

A. L. Maas, A. Y. Hannun, Andrew Y. Ng - 2013

3 papers in library cite

★★★★☆

Google Scholar

It's a good paper, but I expected SO MUCH MORE! In the end it's like "eh, leaky relus train slightly faster" - uuughhh

[101]DistilBERT, a Distilled Version of BERT: Smaller, Faster, Cheaper and Lighter

Thomas Wolf - 2019

6 papers in library cite

★★★★★

Google Scholar

Surprisingly simple and amazing results!

[102]SQuAD: 100,000+ Questions for Machine Comprehension of Text

P. Rajpurkar, J. Zhang, K. Lopyrev, Percy Liang - 2016

37 papers in library cite

★★★★☆

Google Scholar

Nice paper that introduced an important dataset. Not much else though.

[103]Network in Network

M. Lin, Qinlang Chen, Shuicheng Yan - 2013

11 papers in library cite

★★☆☆☆

Google Scholar

I think this was badly written and explained. The idea is nice but I didn't like the paper at all.

[104]Neocognitron: A Self-Organizing Neural Network Model for a Mechanism of Pattern Recognition Unaffected by Shift in Position

Kunihiko Fukushima - 1980

8 papers in library cite

★★★★☆

Google Scholar

Good read! First conv net? :)

[105]Evaluating Large Language Models Trained on Code

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, Wojciech Zaremba - 2021

9 papers in library cite

★★★★☆

canonBenchmark

Google Scholar

Very nice read! Nothin new in the methodology but very thoughtful and thorough analysis.

[106]Extracting and Composing Robust Features With Denoising Autoencoders

Pascal Vincent, Hugo Larochelle, Yoshua Bengio, Pierre Antoine Manzagol - 2008

25 papers in library cite

★★★★☆

Google Scholar

I am *so* glad we found an alternative to DBNs. Also, introduced the idea of denoising which is nice.

[107]Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network With a Local Denoising Criterion

P. H. Vincent, Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre Antoine Manzagol - 2010

6 papers in library cite

★★★★★

Google Scholar

This is basically a summary of everything that happened from 2006-2010, and also points some interesting things about DBNs! Very well explained as well.

[108]On the Properties of Neural Machine Translation: Encoder-Decoder Approaches

Kyunghyun Cho, B. V. Merrienboer, D. Bahdanau, Yoshua Bengio - 2014

9 papers in library cite

★★★☆☆

Google Scholar

This just builds upon other work, comparing and analyzing previous architectures. This faces a specific problem of the time: long sentences. Not too relevant today.

[109]Neural Machine Translation of Rare Words with Subword Units

R. Sennrich, B. Haddow, Alexandra Birch - 2016

22 papers in library cite

★★★★★

Google Scholar

Very good! Simple, explains quite a lot and good results. Forms the basis for a lot of stuff now!

[110]ALBERT: A Lite BERT for Self-Supervised Learning of Language Representations

Z. Lan, Mark Chen, S. Goodman, Kevin Gimpel, P. Sharma, Radu Soricut - 2019

8 papers in library cite

★★★★☆

Google Scholar

I like how they can achieve very close results with very few params! Very nice tricks to do that as well.

[111]ELIZA - A Computer Program for the Study of Natural Language Communication Between Man and Machine

Joseph Weizenbaum - 1966

4 papers in library cite

★★☆☆☆

Google Scholar

Very boring. Most of the paper is just discussing how they created ELIZA (which is very straightforward and not amazing at all). The discussion is very superficial.

[112]Google's Neural Machine Translation System: Bridging the Gap Between Human and Machine Translation

Yonghui Wu, M. Schuster, Ziru Chen, Quoc V. Le, M. Norouzi, W. Macherey, M. Krikun, Yue Cao, Q. Gao, K. Macherey, J. Klingner, A. Shah, M. J. Johnson, Xiaodong Liu, Lukasz Kaiser, S. Gouws, Y. Kato, T. Kudo, H. Kazawa, K. Stevens, G. Kurian, N. Patil, Wenyi Wang, C. Young, J. Smith, J. Riesa, A. Rudnick, Oriol Vinyals, G. S. Corrado, M. Hughes, Jeffrey Dean - 2016

15 papers in library cite

★★★★☆

Google Scholar

It's a very good paper but TBH doesn't bring anything new other than joining a bunch of existing stuff. I think it ended up being foundational because it's Google and several people used it as a base for future research. Good contribution then :)

[113]Deepseek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

Deepseek Ai - 2025

2 papers in library cite

★★★★★

Vetto Study

Google Scholar

Finally the RLVR paper!

[114]Direct Preference Optimization: Your Language Model Is Secretly a Reward Model

Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn - 2023

3 papers in library cite

★★★★☆

RLHFVetto Study

Google Scholar

Very nice idea and probably very revolutionary but paper is a bit boring.

[115]Learning to Forget: Continual Prediction With LSTM

Felix A. Gers, Jürgen Schmidhuber, Fred Cummins - 2000

13 papers in library cite

★★★★★

Google Scholar

This defined what an LSTM is. Before that the LSTM didn't know how to forget. Very nice.

[116]Measuring Massive Multitask Language Understanding

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, Jacob Steinhardt - 2021

6 papers in library cite

★★★☆☆

Vetto Study

Google Scholar

Ok, good benchmark but nothing surprising. I bet it saturated soon after (it did)

[117]Adadelta: An Adaptive Learning Rate Method

Matthew D. Zeiler - 2012

13 papers in library cite

★★★★☆

Google Scholar

Cool contribution. Simple and straightforward, and not too mathy. Based on ADAGRAD.

[118]Large-Scale Video Classification With Convolutional Neural Networks

A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, Li Fei Fei - 2014

2 papers in library cite

★★★★☆

Google Scholar

I liked it a lot. It's nothing "wow", but a very nice approach, and apparently the first o apply CNN to video, which is nice :)

[119]Deepface: Closing the Gap to Human-Level Performance in Face Verification

Y. Taigman, Michael Yang, Marc'aurelio Ranzato, Lior Wolf - 2014

5 papers in library cite

★★★☆☆

Google Scholar

Very impressive results but boring overall. I think the main thing is that they were the first to use CNN, but it seems like the most important part of their method was the 3D alignment (which is nice, but out of scope for me)

[120]GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

A. Wang, A. Singh, J. Michael, F. Hill, Omer Levy, Samuel R. Bowman - 2018

26 papers in library cite

★★★☆☆

Google Scholar

I like it, but it's just a mesh of different existing datasets and F1 score. Nothing new really but I get why it's important

[121]Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

P. Lewis, Ethan Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Kuttler, Martha Lewis, W. T. Yih, Tim Rocktaschel, Sebastian Riedel, K. Douwe - 2020

5 papers in library cite

★★★☆☆

Google Scholar

It coined the term "RAG" but really it was a very different concept.

[122]Video google: A Text Retrieval Approach to Object Matching in Videos

Josef Sivic, Andrew Zisserman - 2003

5 papers in library cite

★★★★☆

Google Scholar

Fun read! It's not really related to AI, but TBH the way they do search on video is more interesting than the object recog.

[123]Show and Tell: A Neural Image Caption Generator

Dumitru Erhan - 2015

11 papers in library cite

★★★★☆

Google Scholar

It's nice and they beat a ton of SotA. However, I read the one that uses attention first so this is a bit less surprising.

[124]Training Verifiers to Solve Math Word Problems

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, John Schulman - 2021

7 papers in library cite

★★★★★

Google Scholar

Fun read. Simple and nice use of verifiers

[125]LSTM: A Search Space Odyssey

K. Greff, R. K. Srivastava, J. Koutn'ik, B. R. Steunebrink, Jürgen Schmidhuber - 2015

4 papers in library cite

★★★★★

Google Scholar

Very good review on different architectural choices of LSTM -  actually brings some nice insights (as opposed to the other paper that compares GRU and LSTMs)

[126]Scaling Laws for Neural Language Models

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, Dario Amodei - 2020

12 papers in library cite

★★★★☆

Google Scholar

Very nice! An amazing contribution. Problem is, the paper is just like 3 pages of actual interesting content, and 10 pages of detailed results. Boring to read but very good otherwise.

[127]Bridging Nonlinearities and Stochastic Regularizers With Gaussian Error Linear Units

Dan Hendrycks, Kevin Gimpel - 2016

9 papers in library cite

★★★★★

Google Scholar

Very understandable, and very nice! I don't think the justification is good, but hey, it works!

[128]On the Difficulty of Training Recurrent Neural Networks

Razvan Pascanu, Tomas Mikolov, Yoshua Bengio - 2013

21 papers in library cite

★★★★☆

Google Scholar

It starts very mathy but in the end there are some very nice contributions! You don't actually need to understand the math to know what's going on in the end.

[129]The Pascal Visual Object Classes Challenge: A Retrospective

Mark Everingham, S. M. Ali Eslami, Luc Van Gool, C. Williams, John Winn, Andrew Zisserman - 2015

1 paper in library cites

★★★★☆

Google Scholar

Dying with grace. Nice to see what happened at the end of PASCAL

[130]Recurrent Neural Network Based Language Model

Tomas Mikolov, M. Karafiat, Lukas Burget, Jan Cernocky, Sanjeev Khudanpur - 2010

36 papers in library cite

★★★★☆

Google Scholar

The comeback of RNNs for language modeling. Not too exciting but impactful and a short read.

[131]Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)

D. A. Clevert, Thomas Unterthiner, Sepp Hochreiter - 2016

2 papers in library cite

★★★★★

Google Scholar

Very nice paper. I am not sure how influential it is, but I like how they explain the whole idea and motivation behind the ELU. I also like that the math is contained in a single section, and they give you a final summary to help understand what the math means.

[132]Huggingface's Transformers: State-of-the-Art Natural Language Processing

Thomas Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, Sam Shleifer, P. V. Platen, C. Ma, Yacine Jernite, J. Plu, Chenfeng Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, Alexander M. Rush - 2019

7 papers in library cite

★★★☆☆

Google Scholar

It's a solid paper, but these framework papers are usually boring... Yet, it's nice to see that they started with reduced scope (and how large it got now).

[133]A Unified Architecture for Natural Language Processing: Deep Neural Networks With Multitask Learning

Ronan Collobert, Jason Weston - 2008

32 papers in library cite

★★★☆☆

Google Scholar

Really did not add much to the game. I think this was more of a small perf. improvement over other existing things and set a few methodological standards. Maybe main contribution is Multitask Learning + Deep learning

[134]Emergence of Simple-Cell Receptive Field Properties by Learning a Sparse Code for Natural Images

B. Olshausen, D. Field - 1996

5 papers in library cite

★★★☆☆

Google Scholar

I can see the contribution, but overall the paper was uninteresting

[135]The Mythos of Model Interpretability

Zachary C. Lipton - 2016

1 paper in library cites

★★★☆☆

Google Scholar

I expected so much more, but it's just a review on interpretability + some platitudes. It's fine I guess but the title oversells it.

[136]Reading Digits in Natural Images With Unsupervised Feature Learning

Y. Netzer, Tianle Wang, A. Coates, Alessandro Bissacco, Bo Wu, Andrew Y. Ng - 2011

8 papers in library cite

★★★☆☆

Google Scholar

It's a bit meh. It's also 2011 so maybe it was impressive, but I think the main contribution is the dataset.

[137]Connectionist Temporal Classification: Labelling Unsegmented Sequence Data With Recurrent Neural Networks

Alex Graves, Santiago Fernandez, Faustino Gomez, Jürgen Schmidhuber - 2006

7 papers in library cite

★★★☆☆

Google Scholar

It's a bit lukewarm. Nice idea but execution was a bit meh. I also think the prefix search was unnecessarily complex and loses to beam search (as they admit later on)

[138]Efficient Backprop

Yann Lecun, Leon Bottou, G. B. Orr, Klaus Robert Muller - 1998

20 papers in library cite

★★★☆☆

Google Scholar

The first half is very very good. The remainder is very boring.

[139]Backpropagation Through Time: What It Does and How to Do It

P. Werbos - 1990

9 papers in library cite

★★★★★

Google Scholar

Amazing tutorial! Very pragmatic. Explains very basic concepts and focus on implementation

[140]Greedy Layer-Wise Training of Deep Networks

Yoshua Bengio, P. Lamblin, D. Popovici, Hugo Larochelle - 2006

33 papers in library cite

★★★★☆

Google Scholar

Bengio is perfect. This is everything that Hinton's paper hoped to be. Very well explained, and also tying back to real use cases (not just "hey, the math works and it reduced the score")

[141]Curriculum Learning

Yoshua Bengio, J. Louradour, Ronan Collobert, Jason Weston - 2009

6 papers in library cite

★★★★★

Google Scholar

Very nice paper that introduces curriculum learning. Possibly not too relevant, but good nonetheless.

[142]Deep Reinforcement Learning From Human Preferences

Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, Dario Amodei - 2017

11 papers in library cite

★★★★☆

RLHF

Google Scholar

Very nice idea overall!

[143]Handwritten Digit Recognition With a Back-Propagation Network

Yann Lecun, B. Boser, John S. Denker, D. Henderson, R. E. Howard, W. Hubbard, L. D. Jackel - 1990

10 papers in library cite

★★☆☆☆

Google Scholar

Just a minor derivation of the previous paper, adding more layers to the net. Pretty much the same exact text than the other one. Uninteresting if you read the other.

[144]Adaptive Mixtures of Local Experts

Robert A. Jacobs, Michael I. Jordan, S. J. Nowlan, Geoffrey E. Hinton - 1991

5 papers in library cite

★★★★☆

Google Scholar

Very cool idea and it's being used again! It's very early so application is a bit meh, and the part about speech recognition is a bit hard to grasp without context. However, very well explained!

[145]On the Importance of Initialization and Momentum in Deep Learning

Ilya Sutskever, James Martens, G. Dahl, Geoffrey Hinton - 2013

13 papers in library cite

★★★★☆

Google Scholar

They give very good context and it's easy to understand that they are doing this as a counterpoint to HF. Surprising results as well. I just think it was made obsolete by relu

[146]Multi-Column Deep Neural Networks for Image Classification

Dan C. Ciresan, Ueli Meier, Jürgen Schmidhuber - 2012

11 papers in library cite

★★★★★

Google Scholar

Very nice paper! And I am impressed they used CNNs before Hinton's paper. It's a shame there are so few citations. They also propose max-pooling and actually give a good explanation about it.

[147]Learning Word Vectors for Sentiment Analysis

A. L. Maas, R. E. Daly, P. T. Pham, Dong Huang, Andrew Y. Ng, Christopher Potts - 2011

12 papers in library cite

★★☆☆☆

Google Scholar

Very convoluted method. I did not like it and I had to go back twice to remember if I had read it or not. Main contribution is the IMDB movie review dataset. (It is cited by Mikolov as a comparison)

[148]Training Products of Experts by Minimizing Contrastive Divergence

Geoffrey Hinton - 2002

23 papers in library cite

★★★★☆

Google Scholar

Good read, but I think I need to revisit it after I understand RBMs better.

[149]Optimal Brain Damage

Yann Lecun, John Denker, Sara Solla, Richard Howard, Lawrence Jackel - 1990

4 papers in library cite

★★★☆☆

Google Scholar

It's a nice idea but I think it's a bit uninteresting - maybe it influenced other important work later?

[150]Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing

Matei Zaharia, M. Chowdhury, T. Das, A. Dave, J. Ma, M. Mccauley, M. J. Franklin, S. Shenker, Ion Stoica - 2012

2 papers in library cite

★★★★★

Google Scholar

Very nice paper. Interesting throughout, and introduced Spark! Very very nice.

[151]Learning Generative Visual Models From Few Training Examples: An Incremental Bayesian Approach Tested on 101 Object Categories

Li Fei Fei, Rob Fergus, Pietro Perona - 2004

15 papers in library cite

★★☆☆☆

Google Scholar

I think most people cite this thinking this is where the Caltech 101 dataset comes from (it's not). Anyway, it's just an extension of the other dataset and it's very mathy, not NNs, and uninteresting.

[152]Neural Network-Based Face Detection

H. Rowley, S. Baluja, Takeo Kanade - 1998

4 papers in library cite

★★★★☆

Google Scholar

It's nice, but nothing crazy. Just applying CNNs

[153]DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition

J. Donahue, Y. Jia, Oriol Vinyals, J. Hoffman, N. Zhang, E. Tzeng, Trevor Darrell - 2014

15 papers in library cite

★★★★★

Google Scholar

Very nice paper. First I've seen (and based on the text, first ever) about feature extraction for images. It's very nice to see embeddings doing SotA

[154]Understanding Deep Learning Requires Rethinking Generalization

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, Oriol Vinyals - 2016

2 papers in library cite

★★★★☆

Google Scholar

I like the push of rethinking generalization, and very intuitive to read. It is a bit incredible that it took so long for people to realize this - seems so obvious!

[155]Deepseekmath: Pushing the Limits of Mathematical Reasoning in Open Language Models

Zhihong Shao, Peng Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Yiwei Li, Yonghui Wu - 2024

3 papers in library cite

★★★★☆

RLHFVetto Study

Google Scholar

Fun read and GRPO is a very nice idea!

[156]Universal Language Model Fine-Tuning for Text Classification

J. Howard, Sebastian Ruder - 2018

14 papers in library cite

★★★★★

Google Scholar

Amazing! Bridging the gap between pre-training/finetuning in CV vs. NLP, plus giving amazing resuts!

[157]Srilm - An Extensible Language Modeling Toolkit

Andreas Stolcke - 2002

13 papers in library cite

★★☆☆☆

Google Scholar

Toolkit for N-grams. Not too relevant and sounds veeeery simple (sorry for those who implemented it). It's nice to see early implementation of OOP though. The paper is boring and doesn't really say much about the framework, more of a description of how to use the commands and n-gram models.

[158]Energy and Policy Considerations for Deep Learning in NLP

E. Strubell, A. Ganesh, Andrew Mccallum - 2019

3 papers in library cite

★★★★☆

Google Scholar

Very nice push and maybe the first one to actually raise a flag - also, very contemporary and relevant discussion

[159]Generating Sequences With Recurrent Neural Networks

Alex Graves - 2013

27 papers in library cite

★★★★☆

Google Scholar

Very cool and is the first to actually proposed the Attention mechanism! It gets a bit mathy but nothing too crazy. Also has the first examples of good machine generated writing I've seen in these papers, so very nice results.

[160]A Large Annotated Corpus for Learning Natural Language Inference

Samuel R. Bowman, G. Angeli, Christopher Potts, Christopher D. Manning - 2015

25 papers in library cite

★★★☆☆

Google Scholar

Dataset collection is ok. The model that they create seems very low effort.

[161]Linguistic Regularities in Continuous Space Word Representations

Tomas Mikolov, W. T. Yih, Geoffrey Zweig - 2013

8 papers in library cite

★★★★☆

Google Scholar

This builds on Bengio's "Word Representations" paper to show that embeddings can be used to describe syntactic and semantic characteristics of words. Good contribution, but not sure if too impactful.

[162]A Broad-Coverage Challenge Corpus for Sentence Understanding Through Inference

A. Williams, Nikita Nangia, S. Bowman - 2018

19 papers in library cite

★★★★☆

Google Scholar

Very nice paper and cool dataset - good thing they expanded SNLI. Also, they at least tried to have a good baseline, and comparisons of domains are nice.

[163]Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context

Z. Dai, Zhilin Yang, Yining Yang, W. Cohen, J. Carbonell, Quoc Le, Ruslan Salakhutdinov - 2019

9 papers in library cite

★★★★★

Google Scholar

It's so cool to see context expansion without the need to actually expand context! Such a simple context and so effective!

[164]Labelme: A Database and Web-Based Tool for Image Annotation

Bryan C. Russell, Antonio Torralba, Kevin P. Murphy, William T. Freeman - 2008

10 papers in library cite

★★★☆☆

Google Scholar

It's a good paper overall but not worth the read. They just describe the platform (which for the time may have been a paradigm shift from in-house datasets). Maybe the basis for Amazon MT?

[165]Measuring Mathematical Problem Solving With the MATH Dataset

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, Jacob Steinhardt - 2021

8 papers in library cite

★★★☆☆

Google Scholar

Good idea in creating the dataset but it's just a regular and uninteresting dataset description paper

[166]A Convolutional Neural Network for Modelling Sentences

Phil Blunsom, Edward Grefenstette, N. Kalchbrenner - 2014

7 papers in library cite

★★★★☆

Google Scholar

I am usually biased against CNNs, but this was very good! Very well motivated and simple/intuitive explanations!

[167]Recurrent Models of Visual Attention

V. Mnih, N. Heess, Alex Graves - 2014

5 papers in library cite

★★★★★

Google Scholar

It's not as good as the other paper (DRAW), but it's a precursor and it's so nice how the model learns to pay attention. Also very nice to see RL in the mix, and see the possible usages in games and other things.

[168]Large Scale Distributed Deep Networks

Jeffrey Dean, G. S. Corrado, R. Monga, K. Chen, M. Devin, Quoc V. Le, Mark Z. Mao, Marc'aurelio Ranzato, A. Senior, P. Tucker, K. Yang, Andrew Y. Ng - 2012

16 papers in library cite

★★★★☆

Google Scholar

Good paper, nice algorithm. Nothing too crazy, but I understand the impact. I think the work to create the system was larger than the algorithm itself.

[169]Sun Database: Large-Scale Scene Recognition From Abbey to Zoo

Jianxiong Xiao, James Hays, K. Ehinger, Aude Oliva, Antonio Torralba - 2010

2 papers in library cite

★★★☆☆

Google Scholar

I like the dataset and the approach. Solid methodology also. However, I don't really think "scene within a scene" is good for practical use. Also not too sure about impact.

[170]A Maximum Entropy Approach to Natural Language Processing

A. L. Berger, S. A. D. Pietra, Vincent J. Della Pietra - 1996

10 papers in library cite

★★★★★

Google Scholar

This paper is so good! Easy to follow and very nice results. The experiments are a bit meh, but otherwise wonderful.

[171]SentencePiece: A Simple and Language Independent subword Tokenizer and Detokenizer for Neural Text Processing

John Richardson - 2018

3 papers in library cite

★★★★☆

Google Scholar

I like that this is just a simple library that wraps functionality. It's also minimal, so can be included in a bunch of places.

[172]Think You Have Solved Question Answering? Try arc, the Ai2 Reasoning Challenge

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord - 2018

5 papers in library cite

★★★☆☆

Google Scholar

Meh, this is just the dataset definition. I don't see anything special, just a new data source. No new methodology or anything.

[173]Hierarchical Mixtures of Experts and the EM Algorithm

M. Jordan, Rowan Jacobs - 1994

3 papers in library cite

★★☆☆☆

Google Scholar

I think the main contribution here is the introduction + using the MoE. The rest of the paper is just a bunch of math proofs.

[174]Convolutional Sequence to Sequence Learning

J. Gehring, Michael Auli, D. Grangier, D. Yarats, Yann Dauphin - 2017

3 papers in library cite

★★★★☆

Google Scholar

It's a good paper. I liked that it is very similar to the transformer methodology (but they don't use convolutions there). I need to go back and read the transformers paper to make parallels.

[175]Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication

Herbert Jaeger, Harald Haas - 2004

4 papers in library cite

★★☆☆☆

Google Scholar

Bad read. Doesn't explain anything of how things work. It's boring and I didn't like the figures. Maybe the main paper about ESNs is better, but frankly I don't want to read it.

[176]Teaching Machines to Read and Comprehend

K. M. Hermann, T. Kocisky, Edward Grefenstette, L. Espeholt, W. Kay, M. Suleyman, Phil Blunsom - 2015

31 papers in library cite

★★★★☆

Google Scholar

Nice way of converting unsupervised data to train for Q&A - and nice visualizations as well :) But I think their main contribution is the dataset. Maybe with the dataset they "unlocked" summarization?

[177]Pointer Networks

Oriol Vinyals, M. Fortunato, Navdeep Jaitly - 2015

10 papers in library cite

★★★☆☆

Google Scholar

Cool concept. Nice that it works and can find good solutions for TSP.

[178]Hellaswag: Can a Machine Really Finish Your Sentence?

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, Yejin Choi - 2019

6 papers in library cite

★★★★☆

Google Scholar

It's very simple, just a revamp on the SWAG dataset. However, the authors had a lot of foresight into understanding the dataset saturation loop - made me wonder if we stopped using adversarial examples because they just don't make sense anymore.

[179]Phoneme Recognition Using Time-Delay Neural Networks

A. H. Waibel, T. Hanazawa, Geoffrey Hinton, K. Shikano, K. Lang - 1989

13 papers in library cite

★★★☆☆

Google Scholar

I didn't really like this paper, although it's a good early example of using backprop. I think it can be thought as a 1D convolution.

[180]Best Practices for Convolutional Neural Networks Applied to Visual Document Analysis

John C. Platt - 2003

12 papers in library cite

★★★★★

Google Scholar

Very good paper. Simple and pragmatic.

[181]Natural Questions: A Benchmark for Question Answering Research

T. Kwiatkowski, J. Palomaki, O. Rhinehart, Michael Collins, A. P. Parikh, C. Alberti, D. Epstein, Illia Polosukhin, M. Kelcey, Jacob Devlin, K. Lee, K. N. Toutanova, Llion Jones, M. W. Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, Slav Petrov - 2019

9 papers in library cite

★★★★☆

Google Scholar

The dataset and methodology is very nice - it's amazing to see how Google does the summaries in search. However, the paper is too complex with the math stuff - unnecessary.

[182]Recurrent Neural Network Regularization

Wojciech Zaremba, Ilya Sutskever, Oriol Vinyals - 2014

22 papers in library cite

★★★☆☆

Google Scholar

It's a very simple idea and TBH it's nothing different from dropout. It's good that it's a very short paper and very straightforward, but could be a paragraph long.

Missing author list

[183]Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback

2022

4 papers in library cite

★★★☆☆

Vetto StudyRLHF

Google Scholar

It's ok but it's too long. The idea of optimizing HH is nice but nothing new in terms of RLHF.

[184]Context-Dependent Pre-Trained Deep Neural Networks for Large-Vocabulary Speech Recognition

G. Dahl, D. Yu, L. Deng, Alex Acero - 2012

19 papers in library cite

★★★☆☆

Google Scholar

Good paper, very well written and probably the best explanation of RBMs and DBNs I've seen. However, I don't see a lot of impact and seems very derivative from other works.

[185]Get to the Point: Summarization With Pointer-Generator Networks

A. See, P. J. Liu, Christopher D. Manning - 2017

8 papers in library cite

★★★☆☆

Google Scholar

It's a bit of the same thing of the other ones. I am not sure if this was the first or not, but I am getting a bit bored of this

[186]Constitutional AI: Harmlessness From AI Feedback

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron Mckinnon, C. C. Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova Dassarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield Dodds, Benjamin Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom B. Brown, Jared Kaplan - 2022

2 papers in library cite

★★★☆☆

Vetto StudyRLHF

Google Scholar

Nice idea, but it's just a new spin on RLHF. Also, too long.

[187]Know What You Don't Know: Un-Answerable Questions for SQuAD

P. Rajpurkar, R. Jia, Percy Liang - 2018

14 papers in library cite

★★★☆☆

Google Scholar

It's alright... It's an extension to the other paper/dataset. I feel that it didn't need to be a full paper (maybe a 6-pager).

[188]Program Synthesis With Large Language Models

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le - 2021

4 papers in library cite

★★☆☆☆

canonBenchmark

Google Scholar

Boring read. No news. They lost to OpenAI and HumanEval.

[189]Why Does Unsupervised Pre-Training Help Deep Learning?

Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pierre Antoine Manzagol, Pascal Vincent, Samy Bengio - 2010

12 papers in library cite

★★★★☆

Google Scholar

Good paper, easy to follow, and brings some light to the pre-training stuff (layer-by-layer). I just wish it wasn't so long. It's a chore.

[190]Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books

Yuxuan Zhu, R. Kiros, R. Zemel, Ruslan Salakhutdinov, R. Urtasun, Antonio Torralba, Sanja Fidler - 2015

18 papers in library cite

★★★☆☆

Google Scholar

I think their approach was a bit convoluted and didn't really add a lot. Main contribution here is probably BookCorpus

[191]A Neural Attention Model for Abstractive Sentence Summarization

Alexander M. Rush, S. Chopra, Jason Weston - 2015

13 papers in library cite

★★★★☆

Google Scholar

TBH the paper is a bit boring and nothing new after reading a bunch of more modern techniques. I feel that they could have done a better job considering that seq2seq existed at the time. Either way, points for being the first to propose summarization with NNs.

[192]Learning to Summarize from Human Feedback

Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, Paul Christiano - 2020

10 papers in library cite

★★★★★

Vetto StudyRLHF

Google Scholar

Very thoughtful on explaining data collection and worrying about future consequences. Seems much closer to the RLHF we do now vs. what is in the human preferences paper.

[193]Catastrophic Forgetting in Connectionist Networks

Robert M. French - 1999

2 papers in library cite

★★★★★

Google Scholar

Such a good review/survey on catastrophic forgetting! Very nice

[194]Regularization of Neural Networks Using Dropconnect

L. Wan, M. Zeiler, S. Zhang, Rob Fergus - 2013

8 papers in library cite

★★★☆☆

Google Scholar

I feel that the method is very complex and does not improve much on top of regular dropout.

[195]Language Models as Knowledge Bases?

F. Petroni, Tim Rocktaschel, P. Lewis, A. Bakhtin, Yonghui Wu, A. H. Miller, Sebastian Riedel - 2019

4 papers in library cite

★★★★☆

Google Scholar

Very nice, but I expected a bit more. I thought it would be more of a philosophical discussion rather than a benchmark analysis. Still, probably the first ones to notice that LMs contain knowledge.

[196]Training Very Deep Networks

R. K. Srivastava, K. Greff, Jürgen Schmidhuber - 2015

6 papers in library cite

★★★★☆

Google Scholar

The idea of using the same concepts of LSTM to feedforward nets is so simple yet it took so long! Very cool idea but the results are underwhelming (otherwise would be a 5!)

[197]TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension

M. Joshi, E. Choi, D. Weld, Luke Zettlemoyer - 2017

18 papers in library cite

★★★★☆

Google Scholar

I like the way they collect the data, and I think this is a nice dataset. However, it seems like they didn't even try to make a good baseline.

[198]Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

Noam Shazeer, Azalia Mirhoseini, K. Maziarz, A. Davis, Quoc Le, Geoffrey Hinton, Jeffrey Dean - 2017

9 papers in library cite

★★★★☆

Google Scholar

It's nice, but there's an important section in the middle about batch sizes that I don't quite understand. Not sure if I am missing some background knowledge or if they explain it poorly, and seems foundational to their main method... Either way, I did understand the methodology of the paper, and they have nice results :)

[199]Let's Verify Step by Step

Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, Karl Cobbe - 2023

4 papers in library cite

★★★★☆

RLHFVetto Study

Google Scholar

Fun read but nothing really surprising in terms of methodology. Just proving PRMs are better than ORM (again, not surprising)

[200]Pointer Sentinel Mixture Models

S. Merity, Caiming Xiong, J. Bradbury, Richard Socher - 2017

12 papers in library cite

★★★★☆

Google Scholar

I really liked the methodology, but I had to read it a few times to understand it intuitively - I think they should have done a better job at explaining it.

[201]One-Shot Learning of Object Categories

Li Fei Fei, Rob Fergus, Pietro Perona - 2006

5 papers in library cite

★★★☆☆

Google Scholar

It's a nice problem they are trying to solve, though I doubt the effectiveness. Maybe it was really relevant in a time where they didn't have data. Problem is it focuses too much on the methodology, and I am not too interested in it.

[202]Neural Turing Machines

Alex Graves, G. Wayne, Ivo Danihelka - 2014

18 papers in library cite

★★★★★

Google Scholar

This paper is amazing. If someone told me that NNs could use and address memory by position I wouldn't believe it worked. Very nice, but it's a shame that it's just a toy example.

[203]End-to-End Memory Networks

S. Sukhbaatar, A. Szlam, Jason Weston, Rob Fergus - 2015

18 papers in library cite

★★★★★

Google Scholar

This was so surprising! This is very similar to transformers and RAG. Who knew?!

[204]Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond

R. Nallapati, B. Zhou, C. N. D. Santos, C. G. Gulcehre, Bing Xiang - 2016

10 papers in library cite

★★★☆☆

Google Scholar

Cool that they use pointer switching, and they introduce the CNN/daily mail dataset for summarization, which is a nice insight.

[205]Model Compression

C. Bucilua, Rich Caruana, Alexandru Niculescu Mizil - 2006

2 papers in library cite

★★★★★

Google Scholar

Very cool idea, and very early as well!

[206]Skip-Thought Vectors

R. Kiros, Yuxuan Zhu, Ruslan Salakhutdinov, Richard S. Zemel, R. Urtasun, Antonio Torralba, Sanja Fidler - 2015

23 papers in library cite

★★★★☆

Google Scholar

Nice to see an alternative to Word2Vec to sentences, but I don't really like the approach. Good nonetheless.

[207]What Is the Best Multi-Stage Architecture for Object Recognition?

K. Jarrett, Koray Kavukcuoglu, Marc'aurelio Ranzato, Yann Lecun - 2009

20 papers in library cite

★★☆☆☆

Google Scholar

So boring and convoluted... They try to make things more abstract by saying "filter banks" and "2 stages", but in the end it's CNNs. It seems like they are trying to tie NNs to the rest of ML, but it doesn't work.

[208]Deeply-Supervised Nets

Chen Yu Lee, Saining Xie, Patrick Gallagher, Zhengyou Zhang, Zhuowen Tu - 2014

8 papers in library cite

★★★☆☆

Google Scholar

Amazing potential, very nice idea, but bad writing.

[209]Maxout Networks

Yoshua Bengio - 2013

17 papers in library cite

★★★★☆

Google Scholar

A bit hard to understand, but very nice idea.

[210]Building High-Level Features Using Large Scale Unsupervised Learning

Quoc V. Le, M. A. Ranzato, R. Monga, M. Devin, K. Chen, G. S. Corrado, Jeffrey Dean, Andrew Y. Ng - 2012

10 papers in library cite

★★★★☆

Google Scholar

Very nice and very early work - seems very simple but very insightful to use an autoencoder to detect objects. Also, very similar to the neocognitron :)

[211]Caltech-256 Object Category Dataset

Greg Griffin, Alex Holub, Pietro Perona - 2007

9 papers in library cite

★★★☆☆

Google Scholar

Boring read, but at least they were somewhat impactful.

[212]Labeling Images With a Computer Game

Luis Von Ahn, Laura Dabbish - 2004

5 papers in library cite

★★★☆☆

Google Scholar

It's alright. TBH I expected a bit more. It's a good idea but I feel that they were very optimist around people playing - and it didn't work

[213]Biographies, Bollywood, Boom-Boxes and Blenders: Domain Adaptation for Sentiment Classification

John Blitzer, Mark Dredze, Fernando Pereira - 2007

4 papers in library cite

★★★☆☆

Google Scholar

It's alright. I think they focus a lot on methodology of prediction but most citations are actually because of the dataset.

[214]SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems

A. Wang, Y. Pruksachatkun, Nikita Nangia, A. Singh, J. Michael, F. Hill, Omer Levy, Samuel R. Bowman - 2019

15 papers in library cite

★★★★☆

Google Scholar

Nothing too surprising, just getting a bunch of stuff from different places and putting it all together. At least they do a good analysis of the benchmark vs. existing methodologies.

[215]Increased Rates of Convergence Through Learning Rate Adaptation

Robert A. Jacobs - 1988

4 papers in library cite

★★★★☆

Google Scholar

Very early work on learning rate adaptation. I like that it describes the concepts very thoroughly (as they are all new).

[216]Word Representations: A Simple and General Method for Semi-Supervised Learning

J. Turian, L. Ratinov, Yoshua Bengio - 2010

17 papers in library cite

★★★★★

Google Scholar

This basically introduced the concept of embeddings. Very nice.

[217]A Structured Self-Attentive Sentence Embedding

Zongyu Lin, M. Feng, C. D. Santos, M. Yu, Bing Xiang, B. Zhou, Yoshua Bengio - 2017

2 papers in library cite

★★★☆☆

Google Scholar

It's a bit poorly written and I don't like their 2D embeddings, but the visualizations are nice :)

[218]LSTM neural Networks for Language Modeling

M. Sundermeyer, R. Schluter, Hermann Ney - 2010

7 papers in library cite

★★☆☆☆

Google Scholar

It reads like an undergrad project - doesn't add anything new.

[219]Gradient Flow in Recurrent Nets: The Difficulty of Learning Long-Term Dependencies

Sepp Hochreiter, Yoshua Bengio, Paolo Frasconi, Jürgen Schmidhuber - 2001

16 papers in library cite

★★★★★

Google Scholar

Wow, this is so much better than the other paper - I should have read it sooner. It's concise and not too abstract, and also gives very good context on RNN problems and how to solve them.

[220]The PASCAL Recognising Textual Entailment Challenge

Ido Dagan, O. Glickman, Bernardo Magnini - 2005

19 papers in library cite

★★★★☆

Google Scholar

It's very nice how they had the foresight to create a challenge that became relevant like 10 years later.

[221]Learning and Development in Neural Networks: The Importance of Starting Small

Jeffrey L. Elman - 1993

5 papers in library cite

★★★★★

Google Scholar

This is such a nice paper! Maybe because it's written for a specific public, but it's such an easy read and ties back a lot with neural/biological concepts!

[222]Connectionist Learning Procedures

Geoffrey E. Hinton - 1987

11 papers in library cite

★★★★☆

Google Scholar

It's a very good overview of everything that was happening in 1987! A bit too long though, but a good start nonetheless.

[223]Supervised Learning of Universal Sentence Representations From Natural Language Inference Data

Alexis Conneau, Douwe Kiela, Holger Schwenk, L. Barrault, Antoine Bordes - 2017

11 papers in library cite

★★★★☆

Google Scholar

It's nice. Maybe the first to do NLI right (after the SNLI paper tried but failed miserably). It is simple and effective. After that people started "performance maxxxing"

[224]Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

M. Shoeybi, M. Patwary, Raul Puri, P. Legresley, J. Casper, Bryan Catanzaro - 2019

3 papers in library cite

★★★★★

Google Scholar

Simply amazing! very well explained and easy to follow + nice results, and shows that scaling is good for transformers

[225]Learning to Learn Using Gradient Descent

Sepp Hochreiter, A. Steven Younger, Peter R. Conwell - 2001

4 papers in library cite

★★☆☆☆

Google Scholar

I don't see how this is meta-learning.

[226]Beyond the Imitation Game: Quantifying and Extrapolating the Capabilities of Language Models

Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aman Gupta, Adria Garriga Alonso - 2022

4 papers in library cite

★★★☆☆

Benchmark

Google Scholar

Nice initiative but too much focus on models and performance rather than the bench itself. Also, saturated right after

[227]SpanBERT: Improving Pre-Training by Representing and Predicting Spans

M. Joshi, Deli Chen, Yibo Liu, D. Weld, Luke Zettlemoyer, Omer Levy - 2019

5 papers in library cite

★★★☆☆

Google Scholar

Yet another BERT, and it's not surprising since I've seen this in other papers. But is good nonetheless.

[228]DRAW: A Recurrent Neural Network for Image Generation

K. Gregor, Ivo Danihelka, Alex Graves, D. J. Rezende, Daan Wierstra - 2015

5 papers in library cite

★★★★★

Google Scholar

This is SO cool! So interesting that it was biologically inspired. First actual image generation that seems to work, also a jump from the RBM/Autoencoder stuff, and the incremental drawing is amazing!

[229]Quantized Neural Networks: Training Neural Networks With Low Precision Weights and Activations

Yoshua Bengio - 2016

★★★☆☆

Google Scholar

Wow, I expected so much more from this. It is a good paper in the sense that the method makes sense, but at the same time it is so hard to read and full of jargon - and it is not even short!!!

[230]80 Million Tiny Images: A Large Dataset for Non-Parametric Object and Scene Recognition

Antonio Torralba, Rob Fergus, W. Freeman - 2008

8 papers in library cite

★★★☆☆

Google Scholar

The initial part about data collection and dataset description was nice, but the part of classifying was a bit overkill

[231]Deconvolutional Networks

Matthew D. Zeiler, Dilip Krishnan, Graham W. Taylor, Rob Fergus - 2010

3 papers in library cite

★★★☆☆

Google Scholar

It's a very nice idea but I feel that there is a much more intuitive way of explaining the idea.

[232]Memory Networks

Jason Weston, S. Chopra, Antoine Bordes - 2015

18 papers in library cite

★★★★☆

Google Scholar

The first half of the paper (when they discuss the concept in a very abstract way) is amazing. However, the actual methodology was very convoluted - I did not like it. I thought that Neural Turing Machines were inspired in this, but actually they are contemporary... So anyway, the concept is nice, execution is not.

[233]Bidirectional Attention Flow for Machine Comprehension

M. Seo, A. Kembhavi, Ali Farhadi, Hananneh Hajishirzi - 2017

13 papers in library cite

★★★☆☆

Google Scholar

It's alright but the method seems absurdly complex. Maybe I am a bit biased because it's like the 20th paper that I read with attention + LSTMs...

[234]Meteor Universal: Language Specific Translation Evaluation for Any Target Language

Alon Lavie - 2014

2 papers in library cite

★☆☆☆☆

Google Scholar

So boring! Maybe the original Meteor paper is better though

[235]Exact Solutions to the Nonlinear Dynamics of Learning in Deep Linear Neural Networks

Surya Ganguli - 2014

9 papers in library cite

★★★☆☆

Google Scholar

TBH it's been almost 2 months since I read this paper (shame on me for forgetting to add it)... Anyway, as I recall it I liked it, but TBH it's a bit underwhelming because it solved only for linear networks

[236]Programs With Common Sense

Mccarthy - 1959

2 papers in library cite

★★☆☆☆

Google Scholar

It's very early work on artificial intelligence but this focuses much more in putting knowledge in a computer and making it reason vs. AI.

[237]A Neural Conversational Model

Oriol Vinyals, Quoc V. Le - 2015

7 papers in library cite

★★☆☆☆

Google Scholar

No new methodology, no measurements, no results. It should be like 4 pages long if they didn't fill in with the conversation logs.

[238]Domain Adaptation for Large-Scale Sentiment Classification: A Deep Learning Approach

Xavier Glorot, Antoine Bordes, Yoshua Bengio - 2011

3 papers in library cite

★★★☆☆

Google Scholar

Nothing surprising really, just domain adaptation using NNs.

[239]Hybrid Speech Recognition With Deep Bidirectional LSTM

Alex Graves, Navdeep Jaitly, Abdel Rahman Mohamed - 2013

2 papers in library cite

★★★☆☆

Google Scholar

I think this is one of the first Deep Bidirectional LSTMs usages. Good but not too relevant right now.

[240]Sequence Transduction With Recurrent Neural Networks

Alex Graves - 2012

7 papers in library cite

★★★☆☆

Google Scholar

Good contribution. Discusses transducing (converting one sequence to the other) without pre-defined alignment. I didn't really like it as it is too mathy and a bit hard to understand, and I think it was not too impactful.

[241]Improved Backing-Off for M-Gram language Modeling

R. Kneser, Hermann Ney - 1995

11 papers in library cite

★★★★☆

Google Scholar

It's nice, it's simple... But not NNs and seems very incremental on top of existing backoff

[242]Green AI

Oren Etzioni - 2019

4 papers in library cite

★★★★☆

Google Scholar

I like the push a lot! Specially when it comes to make research more democratic, which I personally benefit from. Also, AllenAI seems like they do good, honest work!

[243]CAPTCHA: Using Hard AI Problems for Security

John Langford - 2000

2 papers in library cite

★★☆☆☆

Google Scholar

The idea for captcha is great, but I feel that the paper is too long (18 pages), and they try to theorize too much in having proofs and etc. Overall a bad experience reading it, and I didn't end up knowing more about captcha.

[244]Self-Taught Learning: Transfer Learning From Unlabeled Data

Rajat Raina, Alexis Battle, Honglak Lee, Benjamin Packer, A. Ng - 2007

7 papers in library cite

★★★☆☆

Google Scholar

It's nice to make the distinction of self-taught learning vs. semi-supervised learning. However, it doesn't talk about NNs. Overall, LeCun's CNN unsupervised learning was better.

[245]Automatically Constructing a Corpus of Sentential Paraphrases

W. Dolan, Chris Brockett - 2005

9 papers in library cite

★★☆☆☆

Google Scholar

Small dataset, questionable methodology, not useful for training models

[246]A Theoretically Grounded Application of Dropout in Recurrent Neural Networks

Yarin Gal - 2015

9 papers in library cite

★★★★☆

Google Scholar

The methodology is very simple and effective, but maybe it was too simple for a paper, so they put A TON of math on top of it... Unnecessary really.

[247]The Pyramid Match kernel: Discriminative Classification With Sets of Image Features

Kristen Grauman, Trevor Darrell - 2005

4 papers in library cite

★★★★★

Google Scholar

Very simple and elegant solution to set matching. At first I didn't understand, but then it clicked. I think it could be used for other stuff as well!

[248]Generating Text With Recurrent Neural Networks

Ilya Sutskever, James Martens, Geoffrey E. Hinton - 2011

13 papers in library cite

★★★☆☆

Google Scholar

Pleasant paper but results are underwhelming. They use RNNs for character-level modeling, which is different. They also use the hessian-free method proposed by Martens, but don't go too deep into how it works, which is nice because otherwise it would be very mathy. Other papers cite this more as an example of usage rather than an actual milestone.

[249]Hierarchical Neural Story Generation

A. Fan, Martha Lewis, Yann Dauphin - 2018

4 papers in library cite

★★★☆☆

Google Scholar

I had higher hopes for this paper. The methodology is nice but at this stage things start to get waaay to complicated

[250]Distributed Representations

Geoffrey E. Hinton, J. L. Mcclelland, D. E. Rumelhart - 1986

5 papers in library cite

★★★☆☆

Google Scholar

Very relevant but also very abstract. Not really an awesome read, but interesting nonetheless.

[251]WebGPT: Browser-Assisted Question-Answering With Human Feedback

Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeffrey Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, John Schulman - 2021

7 papers in library cite

★★★★☆

Vetto StudyRLHF

Google Scholar

TBH the nicest thing about this is the idea of using a text-based web browser. Other than that, just another application of RLHF + PPO.

[252]Deep Convolutional Neural Networks for LVCSR

T. Sainath, Abdel Rahman Mohamed, Brian Kingsbury, Bhuvana Ramabhadran - 2013

2 papers in library cite

★★★☆☆

Google Scholar

Nice to see CNNs for other things other than images, but I am biased against speech recognition and CNNs. Very good that the paper is short though.

[253]Hidden Technical Debt in Machine Learning Systems

Dan Dennison - 2014

1 paper in library cites

★★★★☆

Google Scholar

It's very good for raising the flag, and several things they describe I have personally experienced. However, as I have already plenty of experience with this, I think it's not too surprising.

[254]Autoencoders, Minimum Description Length, and Helmholtz Free Energy

Geoffrey E. Hinton, Richard S. Zemel - 1994

3 papers in library cite

★★☆☆☆

Google Scholar

VERY MATHY. Doesn't explain anything. Overall very hard to understand.

[255]A Deep Reinforced Model for Abstractive Summarization

R. Paulus, Caiming Xiong, Richard Socher - 2017

7 papers in library cite

★★★★☆

Google Scholar

It's nice that they introduce intra-attention and RL, but at this point I think a lot of the work in attention is very derivative.

[256]Fine-Tuning Language Models From Human Preferences

Geoffrey Irving - 2020

7 papers in library cite

★★★★★

RLHFVetto Study

Google Scholar

It's so simple how they do it, plus I absolutely LOVED the "challenges" section and how honest they were about it. This is true research!

[257]Realtoxicityprompts: Evaluating Neural Toxic Degeneration in Language Models

Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, Noah A. Smith - 2020

1 paper in library cites

★★★☆☆

Google Scholar

Meh. Good push and good analysis but nothing exciting

[258]Torch7: A Matlab-Like Environment for Machine Learning

Clement Farabet - 2011

5 papers in library cite

★★★★★

Google Scholar

Very nice paper! I am impressed that they used Lua, and the API sounds very simple. Plus, it sounds like they didn't even try that hard to get good performance (vs. Theano which was very tryhard)

[259]Scaling Learning Algorithms Towards AI

Yoshua Bengio, Yann Lecun - 2007

15 papers in library cite

★★★★★

Google Scholar

I should have read this sooner! Such a good explanation of why deep learning > other stuff! Also, better than Bengio's 2006 Learning Deep Archs for AI

[260]Recurrent Continuous Translation Models

N. Kalchbrenner, Phil Blunsom - 2013

27 papers in library cite

★★★★☆

Google Scholar

Good paper, probably the first that used an encoder-decoder. But they used a conv. NN instead of a tradicional decoder, which I don't really like.

[261]Visualizing Higher-Layer Features of a Deep Network

Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pascal Vincent - 2009

4 papers in library cite

★★★★☆

Google Scholar

Very nice the way that they tackled it as an optimization problem using gradient descent. I think this is a similar approach to adversarial examples (not sure if this is what inspired them, I don't remember)

[262]Learning Methods for Generic Object Recognition With Invariance to Pose and Lighting

Yann Lecun, Fu Jie Huang, Leon Bottou - 2004

18 papers in library cite

★★★★☆

Google Scholar

Good paper, nice methodology for creating different images. However, I think that this was not too impactful... I don't see this being used a lot.

[263]Exploiting Similarities Among Languages for Machine Translation

Tomas Mikolov, Quoc V. Le, Ilya Sutskever - 2013

6 papers in library cite

★★★★☆

Google Scholar

It's just word2vec + a projection, but results are very nice and it's surprisingly simple!

[264]Adversarial Examples for Evaluating Reading Comprehension Systems

R. Jia, Percy Liang - 2017

11 papers in library cite

★★★★★

Google Scholar

I liked it a lot! It's good to see people testing things rather than just trying to beat SotA!

[265]Probabilistic Interpretation of Feedforward Classification Network Outputs, With Relationship to Statistical Pattern Recognition

John S. Bridle - 1990

12 papers in library cite

★★★★☆

Google Scholar

Explains using softmax as output. Nice read, but I am not sure if too relevant.

[266]The Winograd Schema Challenge

Hector J. Levesque, E. Davis, Leora Morgenstern - 2011

13 papers in library cite

★★★★★

Google Scholar

It's amazing to see a paper that is very easy to read, opinionated, but also introducing an important contribution to the field. I love that they introduced this challenge as an alternative to the Imitation Game.

[267]Efficient Learning of Sparse Representations With an Energy-Based Model

Marc'aurelio Ranzato, C. Poultney, S. Chopra, Yann Lecun - 2006

20 papers in library cite

★★★☆☆

Google Scholar

It's ok. Not really good, but alright.

[268]A Framework for Learning Predictive Structures From Multiple Tasks and Unlabeled Data

Rie Kubota Ando, Tong Zhang - 2005

10 papers in library cite

★★★★☆

Google Scholar

Very nice and clever way of solving the problem of semi-supervised learning, and makes a lot of sense. I give them more credit for formalizing the concept. The methodology is a bit boring.

[269]Parsing Natural Scenes and Natural Language With Recursive Neural Networks

Richard Socher, C. C. Lin, C. Manning, Andrew Y. Ng - 2011

10 papers in library cite

★★★☆☆

Google Scholar

Good idea and nice results, but not my thing right now.

[270]A Decomposable Attention Model for Natural Language Inference

A. P. Parikh, O. Tackstrom, Dipanjan Das, Jakob Uszkoreit - 2016

11 papers in library cite

★★★★★

Google Scholar

Very nice alternative to the common LSTM encoder-decoder architecture! Seems similar o the Transformers arch in the sense that they don't use RNNs. Nice that they analyze computational complexity as well.

[271]Generalization and Network Design Strategies

Yann Lecun - 1989

5 papers in library cite

★★★★☆

Google Scholar

It's a nice intro to LeCun's early work and how he got to the Conv Net

[272]Cross-Lingual Language Model Pretraining

G. Lample, Alexis Conneau - 2019

5 papers in library cite

★★★☆☆

Google Scholar

I like how they learn cross-lingual stuff via vocabulary sharing. it's probably the start of the multi-language LMs.

[273]Unifying Visual-Semantic Embeddings With Multimodal Neural Language Models

Richard S. Zemel - 2014

5 papers in library cite

★★★★★

Google Scholar

I think what I like the most about this paper is that they can use regularities, but I like that they put the images and text in the same embedding space by increasing similarity of positive samples and reducing similarity of negative ones - neat!

[274]Semi-Supervised Sequence Learning

A. M. Dai, Quoc V. Le - 2015

27 papers in library cite

★★★★★

Google Scholar

Very good paper that was probably the first to introduce pre-training in NLP!

[275]Semi-Supervised Recursive Autoencoders for Predicting Sentiment Distributions

Richard Socher, Jeffrey Pennington, Eric H. Huang, Andrew Y. Ng, Christopher D. Manning - 2011

10 papers in library cite

★★★★☆

Google Scholar

Very nice and very simple method. I like this waaay better than the other Stanford recursive autoencoder papers.

[276]Long Short-Term Memory-Networks for Machine Reading

Mirella Lapata - 2016

8 papers in library cite

★★★☆☆

Google Scholar

I read this more as an example of intra-attention, but this is not the main focus of the paper. I think visualization/explanation is a bit bad, and it doesn't seem too impactful. I kept thinking that this is starting to get too complicated, and indeed it was surpassed by transformers right after that.

[277]RACE: Large-Scale Reading Comprehension Dataset From Examinations

Guokun Lai, Q. Xie, Haozhe Liu, Yining Yang, Eduard Hovy - 2017

11 papers in library cite

★★★★☆

Google Scholar

I really like the idea of using human tests for testing AI. Also, very nice insige for using chinese tests!

[278]Adaptive Deconvolutional Networks for Mid and High Level Feature Learning

M. Zeiler, G. Taylor, Rob Fergus - 2011

2 papers in library cite

★★☆☆☆

Google Scholar

The idea is very nice but I think this is soooo poorly explained. I had to ask AI to explain it to me - isn't that crazy? It isn't even a hard concept!

[279]Generalization of Back-Propagation to Recurrent Neural Networks

Fernando J. Pineda - 1987

5 papers in library cite

★☆☆☆☆

Google Scholar

Too abstract and mathy. Exactly the kind of paper I don't like. It would be waaaay better if it had some images or some concrete concepts to ground this. Also, I was out of patience for this type of stuff when I read this.

[280]One Weird Trick for Parallelizing Convolutional Neural Networks

Alex Krizhevsky - 2014

3 papers in library cite

★★★★☆

Google Scholar

Very nice paper. Good new approach to training and explains the idea and rationale well. Not too many citations but used by SotA CNNs at the time.

[281]Extensions of Recurrent Neural Network Language Model

Tomas Mikolov, S. Kombrink, Lukas Burget, Jan Cernocky, Sanjeev Khudanpur - 2011

16 papers in library cite

★★☆☆☆

Google Scholar

Doesn't add much.

[282]Theano: New Features and Speed Improvements

F. Bastien, P. Lamblin, Razvan Pascanu, James Bergstra, I. Goodfellow, A. Bergeron, A. Bouchard, N. Nicolas, Yoshua Bengio - 2012

13 papers in library cite

★★★☆☆

Google Scholar

The paper itself is ok, but it seems like they just wanted to make a paper to say "hey, ours is so much better now after these improvements, we crushed Torch7". A bit tryhard tbh.

[283]Learning Distributed Representations of Concepts

Geoffrey E. Hinton - 1986

13 papers in library cite

★★★☆☆

Google Scholar

 Probably seminal, but a bit boring overall

[284]Unsupervised Learning of Invariant Feature Hierarchies With Applications to Object Recognition

Marc'aurelio Ranzato, F. Huang, Y. Boureau, Yann Lecun - 2007

8 papers in library cite

★★★★★

Google Scholar

I thought that I had learned pretty much everything before 2015, but this paper proved me wrong. Amazing read and very nice methodology. Very simple as well!

[285]Improving Word Representations via Global Context and Multiple Word Prototypes

Eric H. Huang, Richard Socher, C. Manning, Andrew Y. Ng - 2012

7 papers in library cite

★★★★☆

Google Scholar

Good paper. I would say not too relevant, but has some nice concepts like a "global context vector" (document embedding) that comes from averaging word embeddings, and a joint training objective.

[286]Offline Handwriting Recognition With Multidimensional Recurrent Neural Networks

Alex Graves, Jürgen Schmidhuber - 2009

5 papers in library cite

★★★☆☆

Google Scholar

It's okay. The method is nice but a bit too convoluted.

[287]The Wake-Sleep Algorithm for Unsupervised Neural Networks

Geoffrey Hinton, Peter Dayan, B. Frey, R. Neal - 1995

9 papers in library cite

★★★☆☆

Google Scholar

It's okay... I get the feeling that this is early autoencoders work, but the term still didn't exist. I don't think it adds nothing new though.

[288]Generalization of Backpropagation With Application to a Recurrent Gas Market Model

Paul J. Werbos - 1988

11 papers in library cite

★★★★☆

Google Scholar

The first two sections are amazing. Very nice concepts, well explained. From section 3 on it gets more abstract and too formalized, and it's mostly for Hopfield recurrent nets (so not very relevant now). Section 4 is just the application to the gas market, which I don't care for at all.

[289]Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them

Mirac Suzgun, Nathan Scales, Nathanael Scharli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, Jason Wei - 2022

4 papers in library cite

★★★★☆

Benchmark

Google Scholar

Good read and good analysis but no interesting research. They just distilled the benchmark and applied CoT. Good for showing that it works but not exciting.

[290]Deep & Cross Network for Ad Click Predictions

Mingliang Wang - 2017

★★★★☆

Google Scholar

Very nice idea but I just feel that it's so unexplored! So many opportunities for improvement (e.g. attention!) maybe in v2?

[291]Visualizing and Understanding Recurrent Networks

Li Fei Fei - 2015

3 papers in library cite

★★★★☆

Google Scholar

Very nice! The results were a bit underwhelming though. Nice visualizations but few explanations. Good nonetheless.

[292]A Sick Cure for the Evaluation of Compositional Distributional Semantic Models

Marco Marelli, S. Menini, M. Baroni, L. Bentivogli, R. Bernardi, R. Z. Elli - 2014

7 papers in library cite

★★★☆☆

Google Scholar

Just a basic dataset paper. Nothing much.

[293]Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference

R. T. Mccoy, Ellie Pavlick, Tal Linzen - 2019

5 papers in library cite

★★★☆☆

Google Scholar

TBH I expected more. It's a very shallow analysis of the heuristics (but ok, props to them for making it simple). I think that ultimately they don't solve the problem.

[294]Exploring the Limits of Language Modeling

R. Jozefowicz, Oriol Vinyals, M. Schuster, Noam Shazeer, Yonghui Wu - 2016

20 papers in library cite

★★★★☆

Google Scholar

It's funny because at first I did not like it, but then it clicked and I really liked it - they are trying to come around the large dictionary and the rare word problem. In the end it's SotA, but I think it's too convoluted and was replaced by Transformers.

[295]Practical Issues in Temporal Difference Learning

Gerald Tesauro - 1992

3 papers in library cite

★★★★☆

Google Scholar

(There's a larger paper with the same title) It's a fun read and I learned a bit about early reinforcement learning with NNs. However, I read this because Bengio cited for "problems related to learning deep archs", and that's not it.

[296]CoQA: A Conversational Question Answering Challenge

Siva Reddy, Deli Chen, Christopher D. Manning - 2018

6 papers in library cite

★★★☆☆

Google Scholar

It's a fine paper and a solid addition to QA data + NLU.

[297]LSTM Can Solve Hard Long Time Lag Problems

Sepp Hochreiter, Jürgen Schmidhuber - 1997

5 papers in library cite

★★★☆☆

Google Scholar

Doesn't add much on top of what we already know. Good paper nonetheless.

[298]Deep Learning Made Easier by Linear Transformations in Perceptrons

Tapani Raiko, Harri Valpola, Yann Lecun - 2012

7 papers in library cite

★★★☆☆

Google Scholar

Kudos for introducing shortcut connections (which would become important in the future), but to me it seems a bit mid.

[299]Passage Re-Ranking With BERT

Rodrigo Nogueira, Kyunghyun Cho - 2019

1 paper in library cites

★★★★☆

Google Scholar

It's good but very very short. I think it's just good because it's SotA and straightforward, but lacks content.

[300]An Empirical Evaluation of Deep Architectures on Problems With Many Factors of Variation

Hugo Larochelle, Dumitru Erhan, Aaron Courville, James Bergstra, Yoshua Bengio - 2007

13 papers in library cite

★★★☆☆

Google Scholar

Good paper showing promising results for Deep Learning. Nothing amazing but good nonetheless

[301]How to Construct Deep Recurrent Neural Networks

Razvan Pascanu, C. G. Gulcehre, Kyunghyun Cho, Yoshua Bengio - 2013

7 papers in library cite

★★★★☆

Google Scholar

Very interesting despite not being too relevant. Good read and a new way of thinking about RNNs.

[302]Are Sixteen Heads Really Better Than One?

P. Michel, Omer Levy, Graham Neubig - 2019

1 paper in library cites

★★★☆☆

Google Scholar

I expected more. They don't answer the main question of the paper. I thought they would explain what the heads are for, but they only compare results with/without. It's more of an empirical analysis.

[303]Word Translation Without Parallel Data

Alexis Conneau, G. Lample, Marc'aurelio Ranzato, L. Denoyer, Hervé Jégou - 2018

3 papers in library cite

★★★☆☆

Google Scholar

I really really liked the methodology, but it is a bit hard to read given the math jargon they use. Also, the task (word translation) is very boring and I think lacks practical usages

[304]Hierarchical Probabilistic Neural Network Language Model

F. Morin, Yoshua Bengio - 2005

19 papers in library cite

★★★☆☆

Google Scholar

Nice paper overall. Seems very impactful despite not being as relevant right now.

[305]Towards AI-complete Question Answering: A Set of Prerequisite Toy Tasks

Jason Weston, Antoine Bordes, S. Chopra, Tomas Mikolov - 2015

11 papers in library cite

★★★★☆

Google Scholar

It's a good idea and a nice read but the bad part is that most of the tasks are already easy.

[306]The Need for Biases in Learning Generalizations

Tom M. Mitchell - 1980

2 papers in library cite

★★★★☆

Google Scholar

Somewhat abstract but the main idea is amazing!

[307]Annotation Artifacts in Natural Language Inference Data

Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Richard Schwartz, S. Bowman, Noah A. Smith - 2018

6 papers in library cite

★★★★★

Google Scholar

I love papers that show flaws in current methodology/datasets. Very nice for pointing it out!

[308]Cognitron: A Self-Organizing Multilayered Neural Network

Kunihiko Fukushima - 1975

4 papers in library cite

★★☆☆☆

Google Scholar

Boring overall, and not as impactful as the Neocognitron.

[309]Deep Learning via Hessian-Free Optimization

James Martens - 2010

12 papers in library cite

★★★★☆

Google Scholar

This paper is surprisingly good! When I first read the Hessian-Free optimization part, I thought "ugh, this is going to be full of math", but in the end it was very very enjoyable. I think I just wouldn't give it a 5 because it doesn't seem to have had that much impact.

[310]One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling

C. Chelba, Tomas Mikolov, M. Schuster, Q. Ge, T. Brants, P. Koehn, Tony Robinson - 2013

13 papers in library cite

★★★☆☆

Google Scholar

It's somewhat shallow, but I can see the importance of this paper.

[311]Large Batch Optimization for Deep Learning: Training BERT in 76 Minutes

Cho Jui Hsieh - 2020

3 papers in library cite

★★★☆☆

Google Scholar

The changes are very simple and very derivative, but they make it so complicated! Also, they don't discuss drawbacks, which I am sure there are many!

[312]Learned in Translation: Contextualized Word Vectors

B. Mccann, J. Bradbury, Caiming Xiong, Richard Socher - 2017

14 papers in library cite

★★★☆☆

Google Scholar

Doesn't seem too revolutionary tbh. It's a nice methodology though, and it's nice that they do it in a supervised way

[313]Deep Learning via Semi-Supervised Embedding

Jason Weston, F. Ratle, Ronan Collobert - 2008

10 papers in library cite

★★★★☆

Google Scholar

It's a good paper and nice idea, but seems overly complicated and I don't think it's very used... (PS: this was republished in 2012)

[314]A Scalable Hierarchical Distributed Language Model

A. Mnih, Geoffrey E. Hinton - 2009

16 papers in library cite

★★★★☆

Google Scholar

Good paper that introduces hierarchical trees as an alternative to the expensive softmax output. I think this is not really relevant anymore, but good read.

[315]Japanese and Korean Voice Search

M. Schuster, Kaisuke Nakajima - 2012

3 papers in library cite

★☆☆☆☆

Google Scholar

This article sucks. Very poorly written and the only nice thing (wordpiece) is very poorly explained

[316]On Using Very Large Target Vocabulary for Neural Machine Translation

Yoshua Bengio - 2014

12 papers in library cite

★★★☆☆

Google Scholar

It's nice, but it starts getting a bit into the realm of "yeah, that seems like a minor improvement". It's nice that they use the importance sampling stuff from the previous paper though - I thought it had completely vanished :)

[317]Dynamic Pooling and Unfolding Recursive Autoencoders for Paraphrase Detection

Richard Socher, Eric H. Huang, J. Pennin, C. Manning, A. Ng - 2011

10 papers in library cite

★★★☆☆

Google Scholar

Good paper overall, but seems very simple ideas and complex implementation. Overall not very impactful

[318]Grammar as a Foreign Language

Geoffrey Hinton - 2015

9 papers in library cite

★★★★☆

Google Scholar

It's a nice paper showing that attention can be used for parsing. However, parsing is boring and is very derivative. Good paper nonetheless.

[319]Generating Wikipedia by Summarizing Long Sequences

P. J. Liu, M. Saleh, E. Pot, B. Goodrich, R. Sepassi, Lukasz Kaiser, Noam Shazeer - 2018

7 papers in library cite

★★★★★

Google Scholar

Very nice, the first ones that realized that they could use decoder only from Transformers. Also nice that they expanded the context window, and got nice results. Very inventive to use citations from wikipedia and the first paragraph as target.

[320]Wizard of Wikipedia: Knowledge-Powered Conversational Agents

E. Dinan, S. Roller, K. Shuster, A. Fan, Michael Auli, Jason Weston - 2019

4 papers in library cite

★★★★☆

Google Scholar

This could be the original RAG. It's soooo close to the original thing!

[321]How Much Knowledge Can You Pack Into the Parameters of a Language Model?

Noam Shazeer - 2020

2 papers in library cite

★★★★☆

Google Scholar

Very nice, probably the first to notice that LLMs can answer closed-book questions. However, they don't really answer the question in the title (and TBH there is not a ton of contribution)

[322]Improving the Speed of Neural Networks on Cpus

Vincent Vanhoucke, A. Senior, Mark Z. Mao - 2011

4 papers in library cite

★★★☆☆

Google Scholar

It's good and it's interesting, but I don't think it adds a ton. Good read though. I read it because I thought it discussed quantization (which it does, but not in the sense of making NNs smaller)

[323]Recurrent Nets That Time and Count

Felix A. Gers, Jürgen Schmidhuber - 2000

1 paper in library cites

★★★☆☆

Google Scholar

Not a whole lot different from regular LSTMs. A bit hard to read, and only had toy tasks. Low 3.

[324]Scaling Laws for Reward Model Overoptimization

Leo Gao, John Schulman, Jacob Hilton - 2022

3 papers in library cite

★★★☆☆

Vetto StudyRLHF

Google Scholar

Very technical and a ton of practical insights. I understand the relevance, but I didn't find it interesting.

[325]Unsupervised Neural Machine Translation

M. Artetxe, G. Labaka, E. Agirre, Kyunghyun Cho - 2017

4 papers in library cite

★★★★☆

Google Scholar

Very nice methodology! I am impressed that this is even possible. However, results are somewhat underwhelming...

[326]High Performance Convolutional Neural Networks for Document Processing

K. Chellapilla, S. Puri, Patrice Y. Simard - 2006

3 papers in library cite

★★☆☆☆

Google Scholar

It's ok. It's not bad, it's just a bit boring, and I don't think it's bringing anything new. People already implemented unrolled CNN and GPUs, they just brought it all together.

[327]Addressing the Rare Word Problem in Neural Machine Translation

T. Luong, Ilya Sutskever, Quoc V. Le, Oriol Vinyals, Wojciech Zaremba - 2014

14 papers in library cite

★★☆☆☆

Google Scholar

The method was very poorly explained. It was also worse than a paper released sooner, and more complicated. Overall not that good.

[328]Google street View: Capturing the World at Street Level

Dragomir Anguelov, Carole Dulong, Daniel Filip, Christian Frueh, Stephane Lafon, Richard Lyon, Abhijit Ogale, Luc Vincent, Josh Weaver - 2010

1 paper in library cites

★★★★☆

Google Scholar

Interesting to know how it works, but it's a bit out of topic.

[329]Comparing Biases for Minimal Network Construction With Back-Propagation

S. J. Hanson, Lorien Y. Pratt - 1988

3 papers in library cite

★★★★☆

Google Scholar

I like it! Early work on weight decay, which seemed to be somewhat common sense at the time, but they actually put it in paper.

[330]Is Learning the N-Th Thing Any Easier Than Learning the First?

Sebastian Thrun - 1996

3 papers in library cite

★★☆☆☆

Google Scholar

It's one of those early NN papers that only have toy examples and some very non-standard terminology. TBH doesn't add much.

[331]Comparison of Learning Algorithms for Handwritten Digit Recognition

V. Vapnik - 1995

2 papers in library cite

★★★★☆

Google Scholar

Good paper that gives an overview of what was SotA on digit recognition in the mid-90s. It's not so well written but it's easy to understand. I think that this is even better than LeCun's other paper to understand what's going on

[332]Gradient-Based Learning Algorithms for Recurrent Networks and Their Computational Complexity

Ronald J. Williams, David Zipser - 1992

8 papers in library cite

★★☆☆☆

Google Scholar

Oof, this was a hard read. I had to force myself to finish. I think this is too theoretical and deals with RNNs that are not really used nowadays. I skimmed through the second half.

[333]Multi-Task Sequence to Sequence Learning

M. T. Luong, Quoc V. Le, Ilya Sutskever, Oriol Vinyals, Lukasz Kaiser - 2015

4 papers in library cite

★★★★☆

Google Scholar

Very nice paper, but all of the things that I read about S2S sound very derivative at this point - nonetheless it's nice to see MTL again - seems a bit uncharted territory at this point, despite being very promising

[334]Learning Object Categories From Google's Image Search

Rob Fergus, Li Fei Fei, Pietro Perona, Andrew Zisserman - 2005

3 papers in library cite

★★☆☆☆

Google Scholar

Wow, they focus totally on the LSA stuff, and not on the Google Image Search stuff. Anticlimatic.

[335]Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription

Pascal Vincent - 2012

8 papers in library cite

★★☆☆☆

Google Scholar

I think the main contribution is using HF optimization very early on, but very boring use case and they mix RNNs with RBMs which makes this soooo complicated.

[336]Three New Graphical Models for Statistical Language Modelling

A. Mnih, Geoffrey Hinton - 2007

12 papers in library cite

★★☆☆☆

Google Scholar

I don't know why this is so impactful. I didn't like it and I think this was overly complex.

[337]Semi-Supervised Sequence Tagging With Bidirectional Language Models

M. E. Peters, W. Ammar, C. Bhagavatula, Russell Power - 2017

5 papers in library cite

★★★☆☆

Google Scholar

Nothing too different from other LSTM + Attention papers

[338]SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference

Yejin Choi - 2018

5 papers in library cite

★★★★☆

Google Scholar

Very nice methodology with the adversarial stuff! I think not too relevant though.

[339]Reasoning About Entailment With Neural Attention

Tim Rocktaschel, Edward Grefenstette, K. Hermann, T. Kocisky, Phil Blunsom - 2016

5 papers in library cite

★★★☆☆

Google Scholar

It's nice that they are SotA on top of SNLI, but they just apply existing methodologies.

[340]Math-Shepherd: Verify and Reinforce LLMS Step-by-Step Without Human Annotations

Peng Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yiwei Li, Deli Chen, Yonghui Wu, Zhifang Sui - 2023

1 paper in library cites

★★★★☆

Google Scholar

Low 4. Nice idea but boring paper. Would be nice if it had been used by more people though

[341]MCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text

M. Richardson, C. J. C. Burges, Erin Renshaw - 2013

16 papers in library cite

★★★★☆

Google Scholar

Maybe the best dataset paper I have ever read. So well explained, thoroughly thought! It's a shame it's a very small dataset...

[342]Framewise Phoneme Classification With Bidirectional LSTM Networks

Jürgen Schmidhuber - 2005

14 papers in library cite

★★★★★

Google Scholar

Very nice paper! Simple, no bullshit. Just "hey, we have LSTM and we have BRNN, let's try to join it"

[343]Large Language Models in Machine Translation

Jeffrey Dean - 2007

3 papers in library cite

★★★★☆

Google Scholar

Very nice paper on how google solved their language translation using bigrams. Interesting, despite not NN related.

[344]Theano: A CPU and GPU Math Compiler in Python

Yoshua Bengio - 2010

22 papers in library cite

★★★★☆

Google Scholar

Very nice framework. Symbolic programming is very nice. However, I think that this had very little impact and was mostly used by Bengio's lab.

[345]A Fast and Simple Algorithm for Training Neural Probabilistic Language Models

A. Mnih, Yee Whye Teh - 2012

5 papers in library cite

★★★☆☆

Google Scholar

Too mathy and really did not have a lot of impact TBH

[346]Logical Basis for Information Theory and Probability Theory

Andrei N. Kolmogorov - 1968

1 paper in library cites

★★☆☆☆

Google Scholar

I have no idea what this is about and I have no time nor patience to understand this. It's out of scope.

[347]The LAMBADA dataset: Word Prediction Requiring a Broad Discourse Context

D. Paperno, German Kruszewski, A. Lazaridou, N. Q. Pham, R. Bernardi, S. Pezzelle, M. Baroni, G. Boleda, Raquel Fernandez - 2016

12 papers in library cite

★★★★☆

Google Scholar

Very nice paper - very interesting methodology to building it and very good when they bring a dataset that is meant to make machines to fail

[348]Utility Data Annotation With Amazon Mechanical Turk

Alexander Sorokin, David Forsyth - 2008

2 papers in library cite

★★★☆☆

Google Scholar

It's an alright guide to using Amazon MT. It's good that they lay the foundations to the methodology, but nothing too amazing

[349]Learning Recurrent Neural Networks With Hessian-Free Optimization

James Martens, Ilya Sutskever - 2011

14 papers in library cite

★★★☆☆

Google Scholar

Meh, very very mathy and seems like a minor improvement on LSTMs. They give very impressive results but I don't think it's too much of a fair comparison (vs. a 1999 architecture of LSTMs). I think they did well in showing that exploding/vanishing gradients can be overcome, but their method is not "it". As said by Bengio, they are partially responsible for the revival of RNNs (alongside Mikolov)

[350]Using the Output Embedding to Improve Language Models

O. Press, Lior Wolf - 2017

7 papers in library cite

★★☆☆☆

Google Scholar

I did not like this paper at all - The paper is not bad, it's just that I expected *way* more. Good results but uninteresting

[351]Convolutional neural Networks Applied to House Numbers Digit Classification

P. Sermanet, S. Chintala, Yann Lecun - 2012

6 papers in library cite

★★☆☆☆

Google Scholar

This reads like an undergrad's research paper or a recent master's paper. Only applying stuff to a new dataset.

[352]Context Dependent Recurrent Neural Network Language Model

Tomas Mikolov, Geoffrey Zweig - 2012

12 papers in library cite

★★☆☆☆

Google Scholar

Nothing too interesting, just using the context of the RNN.

[353]Generalized Denoising Auto-Encoders as Generative Models

Pascal Vincent - 2013

2 papers in library cite

★☆☆☆☆

Google Scholar

The paper is not bad itself, but it's full of jargon and math that makes it very hard to understand. It seems like it was written for a maths venue. It seems that the most impressive results (creating good images) are just an afterthought.

[354]On Rectified Linear Units for Speech Processing

Matthew D. Zeiler, M. A. Ranzato, R. Monga, M. Mao, K. Yang, Quoc Le, P. Nguyen, A. Senior, Vincent Vanhoucke, Jeffrey Dean - 2013

3 papers in library cite

★☆☆☆☆

Google Scholar

It's boring and the only thing it brings to the table is usage of ReLU in speech recog (which is unsurprising since it was already used everywhere else)

[355]The Natural Language Decathlon: Multitask Learning as Question Answering

Richard Socher - 2018

9 papers in library cite

★★★★☆

Google Scholar

Very nice results, but TBH the architecture is waaaay to complicated

[356]Learning Distributed Representations of Sentences From Unlabelled Data

F. Hill, Kyunghyun Cho, Anna Korhonen - 2016

12 papers in library cite

★★★☆☆

Google Scholar

I liked it overall (almost gave it 4 starts), but I am a bit tired of embeddings. It's better than the other ones though. FastSent is nice.

[357]Strategies for Training Large Scale Neural Network Language Models

Tomas Mikolov, A. Deoras, D. Povey, Lukas Burget, Jan Cernocky - 2011

9 papers in library cite

★★☆☆☆

Google Scholar

Just builds on other things. Very minor suff in my opinion.

[358]The Goldilocks Principle: Reading Children's Books With Explicit Memory Representations

F. Hill, Antoine Bordes, S. Chopra, Jason Weston - 2015

14 papers in library cite

★★★☆☆

Google Scholar

Cool use of memory networks.

[359]Massive Exploration of Neural Machine Translation Architectures

D. Britz, Anna Goldie, M. Luong, Quoc Le - 2017

1 paper in library cites

★★★★★

Google Scholar

Very good review - these sorts of papers are needed! Good to see how HPs impact results

[360]context2vec: Learning Generic Context Embedding With Bidirectional LSTM

O. Melamud, J. Goldberger, Ido Dagan - 2016

5 papers in library cite

★★★☆☆

Google Scholar

I liked the methodology a lot but the paper reads like an MSc course project.

[361]Solving Math Word Problems With Process- And Outcome-Based Feedback

Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, Irina Higgins - 2022

4 papers in library cite

★★★☆☆

Google Scholar

Very low 3. Almost 2. Boring, too big, and nothing surprising. Results were proved (partially) wrong later. I think they just formalized ORM vs. PRM

[362]Advances in Optimizing Recurrent Networks

Yoshua Bengio, N. B. Lewandowski, Razvan Pascanu - 2013

4 papers in library cite

★★★★★

Google Scholar

Amazing paper that summarizes basically everything that was happening up to 2012 and explains why there was a boom in RNN in 2013 onwards.

[363]An Application to Recurrent Nets to Phone Probability Estimation

A. Robinson - 1994

9 papers in library cite

★★★☆☆

Google Scholar

Very early work. Seems like a good overview of how to apply NNs to speech recog, but TBH there's nothing impressive here.

[364]A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task

Deli Chen, J. Bolton, Christopher D. Manning - 2016

9 papers in library cite

★★★★☆

Google Scholar

Very solid work that shows that things are not always what they seem - very nice!

[365]Fast and Robust Neural Network Joint Models for Statistical Machine Translation

Jacob Devlin, Rabih Zbib, Zhongqiang Huang, Thomas Lamar, Richard Schwartz, John Makhoul - 2014

9 papers in library cite

★★★★☆

Google Scholar

Simple and very nice that it showed that NNs can stand on their own for NMT.

[366]GPU Implementation of Neural Networks

K. S. Oh, Keechul Jung - 2004

2 papers in library cite

★☆☆☆☆

Google Scholar

Very poorly written and poorly explained. The only nice thing is early use of GPUs, and the fact that they do it before CUDA.

[367]Continuous Space Language Models

Holger Schwenk - 2007

12 papers in library cite

★★☆☆☆

Google Scholar

One more paper about speech recog. Nothing special really.

[368]Scalable Agent Alignment via Reward Modeling: A Research Direction

Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, Shane Legg - 2018

5 papers in library cite

★★★★☆

RLHFVetto Study

Google Scholar

Low 4. Good push and good direction, but nothing groundbreaking - other research was already around about reward modeling. Good for them that they pushed for it.

[369]Improving the Convergence of Back-Propagation Learning With Second-Order Methods

S. Becker, Yann Lecun - 1988

9 papers in library cite

★★★★☆

Google Scholar

Surprisingly good. I thought this would be very math-heavy but in the end it's very easy to understand. Sadly results are underwhelming

[370]Improving the Convergence of the Back-Propagation Algorithm

B. Nienhuis - 1992

★★★★☆

Google Scholar

Good paper but I read it wrong lol - it should have been LeCun's paper

[371]The Difficulty of Training Deep Architectures and the Effect of Unsupervised Pre-Training

Pascal Vincent - 2009

5 papers in library cite

★★★★★

Google Scholar

Very nice analysis of why supervised pretraining works!

[372]Learning to Generate Reviews and Discovering Sentiment

Alec Radford, R. Jozefowicz, Ilya Sutskever - 2017

8 papers in library cite

★★★★☆

Google Scholar

Very impressive results. We can see that they were onto something (later became GPT2)

[373]Measuring Invariances in Deep Networks

I. Goodfellow, Quoc Le, A. Saxe, A. Ng - 2009

7 papers in library cite

★★★★☆

Google Scholar

Very nice concept and methodology, but the results in the end are underwhelming

[374]Restoring an Image Taken Through a Window Covered With Dirt or Rain

Rob Fergus - 2013

1 paper in library cites

★★★☆☆

Google Scholar

Good paper, nice to see how image denoising is done. Nothing "wow" but good contribution.

[375]Large Automatic Learning, Rule Extraction and Generalization

John Denker, Daniel Schwartz, Ben Wittner, Sara A. Solla, Richard Howard, Lawrence Jackel, John Hopfield - 1987

4 papers in library cite

★★★★☆

Google Scholar

Good paper overall, but things get a bit boring 2/3 of the way through. However, I feel that this is a good piece of early work into NNs.

[376]Large Scale Image Annotation: Learning to Rank With Joint Word-Image Embeddings

Jason Weston, Samy Bengio, Nicolas Usunier - 2010

3 papers in library cite

★★★★☆

Google Scholar

Very simple approach! Seems like an early word2vec. I like the joint embeddings as well. However, results are underwhelming.

[377]Do CIFAR-10 Classifiers Generalize to CIFAR-10?

Vaishaal Shankar - 2018

2 papers in library cite

★★★★☆

Google Scholar

It's a very nice analysis and it's interesting it took so long for people to see these problems.

[378]Simple and Effective Multi-Paragraph Reading Comprehension

C. Clark, Matt Gardner - 2017

7 papers in library cite

★★★★☆

Google Scholar

Very nice paper! I think it's a stretch to call it "simple", but the paper is very well written and easy to follow.

[379]Products of Experts

Geoffrey E. Hinton - 1999

4 papers in library cite

★★★★☆

Google Scholar

Very interesting concept. This is what paved the way to RBMs and DBNs in the future.

[380]Character-Level Language Modeling With Deeper Self-Attention

R. A. Rfou, D. Choe, Noah Constant, M. Guo, Llion Jones - 2018

6 papers in library cite

★★★☆☆

Google Scholar

It's good, and almost a 4, but I think it is a bit boring. Plus character LMs are not meta.

[381]To Recognize Shapes, First Learn to Generate Images

Geoffrey Hinton - 2006

5 papers in library cite

★★★★☆

Google Scholar

Maybe the best explanation of deep belief nets and RBMs by Hinton.

[382]Two Problems With Backpropagation and Other Steepest-Descent Learning Procedures for Networks

Richard S. Sutton - 1986

2 papers in library cite

★★★★★

Google Scholar

Such a good paper! I think this is the first one to describe the problems with backprop and propose adaptive learning rates. Also, very well explained and not mathy at all!

[383]Adaptive Input Representations for Neural Language Modeling

A. Baevski, Michael Auli - 2018

3 papers in library cite

★★★★☆

Google Scholar

I like the idea and it's a nice paper. Maybe it is a bit outdated now that everything is wordpiece, but I like that they reduce the number of parameters.

[384]Learning a Synaptic Learning Rule

Yoshua Bengio, Samy Bengio, Jocelyn Cloutier - 1990

1 paper in library cites

★☆☆☆☆

Google Scholar

WTF is this Bengio? An incomplete draft? Terrible.

[385]Inferring Algorithmic Patterns With Stack-Augmented Recurrent Nets

Armand Joulin, Tomas Mikolov - 2015

9 papers in library cite

★★★☆☆

Google Scholar

Very underwhelming TBH. I expected more after reading the Neural Turing Machine paper. This reads like "yeah, we lost the race, here's what we were doing before they did something better"

[386]Using GPUs for Machine Learning Algorithms

D. Steinkraus, I. Buck, Patrice Simard - 2005

3 papers in library cite

★★☆☆☆

Google Scholar

It's not bad, it's just not worth the read.

[387]Multi-Dimensional Recurrent Neural Networks

Alex Graves, Santiago Fernandez, Jürgen Schmidhuber - 2007

2 papers in library cite

★★★★☆

Google Scholar

Fun read and it's a good and simple idea. Good that they formalized it.

[388]When and Why Are Pre-Trained Word Embeddings Useful for Neural Machine Translation?

Graham Neubig - 2018

1 paper in library cites

★★☆☆☆

Google Scholar

It's not bad, it's just that it severely lacks deeper analysis on their results. A few of their results are contradictory, and they don't analyze why.

[389]Neural GPUs Learn Algorithms

Lukasz Kaiser, Ilya Sutskever - 2016

5 papers in library cite

★★★★☆

Google Scholar

Results and architecture are nice but TBH it is very poorly written... Doesn't seem like a lot of impact either.

[390]Troubling Trends in Machine Learning Scholarship

Jacob Steinhardt - 2018

1 paper in library cites

★★★★★

Google Scholar

Amazing push and describes much of what I've read!

[391]Learning Process in an Asymmetric Threshold Network

Yann Lecun - 1986

17 papers in library cite

★★☆☆☆

Google Scholar

Meh. Really did not bring much to the table. Describes back-prop but mentions that it comes from Hinton.

[392]One Model to Learn Them All

Lukasz Kaiser, Aidan N. Gomez, Noam Shazeer, Ashish Vaswani, Niki Parmar, Llion Jones, Jakob Uszkoreit - 2017

2 papers in library cite

★★★☆☆

Google Scholar

The idea is amazing, but it gets waaaay too complex, and in the end it feels like they tested it, it didn't give good performance, and they said "just publish it as it is"

[393]High-Performance Neural Networks for Visual Object Classification

Dan C. Ciresan, Ueli Meier, Jonathan Masci, Luca M. Gambardella, Jürgen Schmidhuber - 2011

5 papers in library cite

★★★☆☆

Google Scholar

OK, I can see why this is important (implementing CNNs in GPUs before it was cool), but nothing too surprising. BTW, I read the 6 page version but this one has color so I kept this one.

[394]Empirical Evaluation and Combination of Advanced Language Modeling Techniques

Tomas Mikolov, A. Deoras, S. Kombrink, Lukas Burget, Jan Cernocky - 2011

13 papers in library cite

★★★☆☆

Google Scholar

Early work proving that NNs can be good. But very uninteresting overall.

[395]Strike (With) a Pose: Neural Networks Are Easily Fooled by Strange Poses of Familiar Objects

M. A. Alcorn, Q. Li, Z. Gong, Caitlin Wang, L. Mai, W. S. Ku, Anh Nguyen - 2018

1 paper in library cites

★★★★★

Google Scholar

Very cool the way that they propagate the parameters back to the renderer and it's nice that this is way more applicable than just plain adversarial examples

[396]Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context

U. Khandelwal, He He, P. Qi, Dan Jurafsky - 2018

2 papers in library cite

★★★★☆

Google Scholar

Very nice paper and an analysis that makes sense! People thought that LSTMs had "infinite memory", but this paper shows that it is not so!

[397]Learning Many Related Tasks at the Same Time With Backpropagation

Rich Caruana - 1995

3 papers in library cite

★★★★☆

Google Scholar

Very nice read, fast and simple. Very well and intuitive explanation as well. However, it doesn't seem too impactful overall (the other multitask learning paper seems more relevant, but also more boring)

[398]Learning From Hints in Neural Networks

Yaser S. Abu Mostafa - 1990

3 papers in library cite

★★★☆☆

Google Scholar

Very nice read, but the results are very underwhelming. I hoped for some more concrete concepts/results.

[399]Large-Scale Privacy Protection in Google street view

Andrea Frome, German Cheung, Ahmad Abdulkader, Marco Zennaro, Bo Wu, Alessandro Bissacco, Hartwig Adam, Hartmut Neven, Luc Vincent - 2009

2 papers in library cite

★★★☆☆

Google Scholar

It's ok, nice to know how it works, but it's nothing really amazing really. Very down to earth stuff.

[400]Performance and Scalability of gpu-based Convolutional Neural Networks

D. Strigl, K. Kofler, Stefan Podlipnig - 2010

1 paper in library cites

★★★★★

Google Scholar

Very good paper, and now it seems that I finally understand how CUDA works and how it is used to run NNs!

[401]Theano: Deep Learning on GPUs With Python

Yoshua Bengio - 2011

1 paper in library cites

★☆☆☆☆

Google Scholar

I am convinced this was only written to get some citations off their work. The other papers are infinitely better.

[402]Unsupervised Pretraining for Sequence to Sequence Learning

P. Ramachandran, P. J. Liu, Quoc V. Le - 2017

9 papers in library cite

★★★☆☆

Google Scholar

It's alright, but it's the same Seq2Seq thing with pretraining

[403]Application of Pretrained Deep Neural Networks to Large Vocabulary Speech Recognition

Navdeep Jaitly, P. Nguyen, A. Senior, Vincent Vanhoucke - 2012

6 papers in library cite

★★☆☆☆

Google Scholar

It's not bad, it's just nothing new really. They just get existing methods and apply to very large datasets. I see the contribution, but boring read - just experiment methodology and results.

[404]A Survey of Reinforcement Learning Informed by Natural Language

Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefenstette, Shimon Whiteson, Tim Rocktaschel - 2019

3 papers in library cite

★★★☆☆

Google Scholar

It's a good overview and I like that it gave me context on what's happening, but a bit boring to read.

[405]Pylearn2: A Machine Learning Research Library

Yoshua Bengio - 2013

5 papers in library cite

★★★☆☆

Google Scholar

The paper itself is ok, but I think it had little impact. The idea of having an extensible and easy to use framework is nice (and is something that I also try to do in my projects), but seems like it didn't catch on and was only used by Bengio's lab.

[406]Improving Neural Language Models With a Continuous Cache

E. Grave, Armand Joulin, Nicolas Usunier - 2016

7 papers in library cite

★★★★☆

Google Scholar

This was a surprise to me - I expected this to suck. However, they can provide a simple and intuitive way of improving LMs - nice

[407]Efficient Softmax Approximation for GPUs

E. Grave, Armand Joulin, M. Cisse, D. Grangier, Hervé Jégou - 2017

4 papers in library cite

★★★★☆

Google Scholar

I loved the methodology and the idea behind it. It's good to see some practical improvements (rather than methodological). I just think it's a bit tough to read. I had to ask AI to help explain some stuff.

[408]Making Believers Out of Computers

Hector J. Levesque - 1986

1 paper in library cites

★★★☆☆

Google Scholar

This was tough to read. I like the paper and the flow, but it's mainly about knowledge bases and not about AI. I should give this a 2 but I don't think it's fair. If I was focused on KBs it would be a 4.

[409]Training Recurrent Networks by EVOLINO

Jürgen Schmidhuber, Daan Wierstra, Matteo Gagliolo, F. J. Gomez - 2007

1 paper in library cites

★★★☆☆

Google Scholar

Very mid. They use toy tasks and it's really unclear if Evolino is better than just SGD. Nice variation on learning networks though

[410]Training Millions of Personalized Dialogue Agents

Antoine Bordes - 2018

1 paper in library cites

★★★☆☆

Google Scholar

I think this is very derivative from other papers (which I haven't read yet). But overall is a good read.

[411]Direct Transfer of Learned Information Among Neural Networks

Lorien Y. Pratt, Jack Mostow, Candace A. Kamm - 1991

3 papers in library cite

★★☆☆☆

Google Scholar

Boring read and uninteresting results. I think they stumbled onto transfer learning but didn't know it yet, so they explored it in a bad way.

[412]Neural Network Recognizer for Hand-Written Zip Code Digits

I. Guyon - 1989

2 papers in library cite

★★★★★

Google Scholar

This is probably the best explanation of convolutions I've seen in a paper.

[413]Adaptive Importance Sampling to Accelerate Training of a Neural Probabilistic Language Model

Yoshua Bengio, Jean Sebastien Senecal - 2008

6 papers in library cite

★★☆☆☆

Google Scholar

Just a rerun of a previous paper. This is ok, but really not too different.

[414]How GPUs Work

D. Luebke, Greg Humphreys - 2007

1 paper in library cites

★★★★★

Google Scholar

Wow, I just love it! I now have a working knowledge on how GPUs work, why CUDA is used, shaders... Very very nice!

[415]Flexible Shaping: How Learning in Small Steps Helps

K. A. Krueger, Peter Dayan - 2009

1 paper in library cites

★★★★☆

Google Scholar

Good paper overall, fun to read, but it's just a toy example so results are very meh.

[416]What Will it Take to Fix Benchmarking in Natural Language Understanding?

Samuel R. Bowman, George E. Dahl - 2021

1 paper in library cites

★★★★☆

Benchmark

Google Scholar

It is a nice push but really there are no clear outcomes. The proposal is very shallow.

[417]Audio Chord Recognition With Recurrent Neural Networks

Pascal Vincent - 2013

2 papers in library cite

★★☆☆☆

Google Scholar

Just an application of an RNN. Good read, but not worth it.

[418]Multilingual Language Processing From Bytes

Amarnag Subramanya - 2016

2 papers in library cite

★★★☆☆

Google Scholar

Nothing impressive... Their main innovation was the byte stuff. Plus the old-school NLP stuff makes it boring.

[419]Utility Is in the Eye of the User: A Critique of NLP Leaderboards

Kawin Ethayarajh, Dan Jurafsky - 2020

3 papers in library cite

★★★☆☆

Google Scholar

Ok, agreed. But no news and they don't provide a good solution to the problem.

[420]Subword Language Modeling With Neural Networks

Tomas Mikolov, Ilya Sutskever, A. Deoras, H. S. Le, S. Kombrink, Jan Cernocky - 2012

7 papers in library cite

★★★☆☆

Google Scholar

Doesn't add a ton of contribution other than saying that subword models can perform better than char-level models. Probably the most important thing here is early use of subwords.

[421]Turing Test Considered Harmful

Ford, Hayes - 1995

2 papers in library cite

★★☆☆☆

Google Scholar

It's a good read but it's terrible arguments!

[422]Multi-Digit Recognition Using a Space Displacement Neural Network

O. Matan, C. J. C. Burges, Yann Lecun, John S. Denker - 1992

3 papers in library cite

★★★☆☆

Google Scholar

Meh. They just use a CNN to read more digits, but doesn't seem to have a ton of impact.

[423]The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics

Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Aremu Anuoluwapo, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna Clinciu, Dipanjan Das, Kaustubh D. Dhole, Wanyu Du, Esin Durmus, Ondřej Dušek, Chris Emezue, Varun Gangal, Cristina Garbacea, Tatsunori Hashimoto, Yufang Hou, Yacine Jernite, Harsh Jhamtani, Yangfeng Ji, Shailza Jolly, Mihir Kale, Dhruv Kumar, Faisal Ladhak, Aman Madaan, Mounica Maddela, Khyati Mahajan, Saad Mahamood, Bodhisattwa Prasad Majumder, Pedro Henrique Martins, Angelina McMillan-Major, Simon Mille, Emiel van Miltenburg, Moin Nadeem, Shashi Narayan, Vitaly Nikolaev, Rubungo Andre Niyongabo, Salomey Osei, Ankur Parikh, Laura Perez-Beltrachini, Niranjan Ramesh Rao, Vikas Raunak, Juan Diego Rodriguez, Sashank Santhanam, João Sedoc, Thibault Sellam, Samira Shaikh, Anastasia Shimorina, Marco Antonio Sobrevilla Cabezudo, Hendrik Strobelt, Nishant Subramani, Wei Xu, Diyi Yang, Akhila Yerukola, Jiawei Zhou - 2021

★★☆☆☆

Benchmark

Google Scholar

More of a position piece than a paper. Too long for only describing the bench + it flopped...

[424]Quick Training of Probabilistic Neural Nets by Importance Sampling

Yoshua Bengio, Jean Sebastien Senecal - 2003

11 papers in library cite

★★★★☆

Google Scholar

Good idea to overcome softmax computation cost. Not sure if too relevant today, but definitely better than the 2008 paper that is the same stuff.

[425]exBERT: A Visual Analysis Tool to Explore Learned Representations in Transformers Models

Ben Hoover, Hendrik Strobelt, Sebastian Gehrmann - 2019

1 paper in library cites

★★★☆☆

Google Scholar

Ok, it seems like a nice tool, but TBH doesn't seem too useful since it's very exploratory and the phenomena can be a complex mix of heads.

[426]Machine Reading

Etzioni, Banko, Cafarella - 2006

1 paper in library cites

★★☆☆☆

Google Scholar

It's not bad, it's just uninteresting and doesn't really bring anything new.

[427]GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning

Lakshya A. Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl Ong, Arnav Singhvi, Herumb Shandilya, Michael J. Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alexandros G. Dimakis, Ion Stoica, Dan Klein, Matei Zaharia, Omar Khattab - 2026

★★★★☆

ICLR2026

Google Scholar

Very simple, but not really sure how much it can be applied in practice.

[428]Winner's Curse? On Pace, Progress, and Empirical Rigor

Alex Wiltschko - 2018

1 paper in library cites

★★★☆☆

Google Scholar

Well, it's okay-ish. It's a good push but as an opinion piece is very mid.

[429]Continuous Space Translation Models for Phrase-Based Statistical Machine Translation

Holger Schwenk - 2012

5 papers in library cite

★★☆☆☆

Google Scholar

Nothing special.

[430]Connectionist Language Modeling for Large Vocabulary Continuous Speech Recognition

Holger Schwenk, Jean Luc Gauvain - 2002

14 papers in library cite

★★☆☆☆

Google Scholar

Only real relevance is being early. Otherwise not much to see.

[431]Training Neural Network Language Models on Very Large Corpora

Holger Schwenk, Jean Luc Gauvain - 2005

7 papers in library cite

★★☆☆☆

Google Scholar

Seems very derivative of Schwenk's early work. It's also very focused on speech recognition, and "very large corpora" seems very relative.

[432]Supervising Strong Learners by Amplifying Weak Experts

Paul Christiano, Buck Shlegeris, Dario Amodei - 2018

7 papers in library cite

★★★☆☆

Google Scholar

Nice idea, but doesn't have any concrete implementations or proof that it works. Sounds too aspirational.

[433]Can Artificial Neural Networks Learn Language Models

Weixin Xu, Alex Rudnicky - 2000

5 papers in library cite

★☆☆☆☆

Google Scholar

Shitty paper and I hate that it was the first.

[434]On Our Best Behaviour

Hector J. Levesque - 2013

2 papers in library cite

★★★★★

Google Scholar

I loved this paper. So relevant despite being 10+ years old! Everyone that works with AI should read this.

[435]Dialog-Based Language Learning

J. E. Weston - 2016

1 paper in library cites

★★★☆☆

Google Scholar

It's a nice concept but very abstract and results are very very underwhelming

[436]Continuous Space Language Models for Statistical Machine Translation

Holger Schwenk, D. Dchelotte, Jean Luc Gauvain - 2006

5 papers in library cite

★★★☆☆

Google Scholar

Beginning of translation with NNs. Nothing special but good nonetheless.

[437]Sequential Neural Text Compression

Jürgen Schmidhuber - 1996

3 papers in library cite

★★★★★

Google Scholar

I really liked this paper. Maybe not useful, but the idea is very nice!

[438]A BERT Baseline for the Natural Questions

C. Alberti, K. Lee, Michael Collins - 2019

2 papers in library cite

★★★☆☆

Google Scholar

It's very simple and short, but it's nice that it set a baseline.

[439]A Learning algorithm for Continually Running Fully Recurrent Neural Networks

R. Williams, David Zipser - 1989

8 papers in library cite

★★★★☆

Google Scholar

Quick read and very simple concept. I wish they explained things using a more visual approach instead of maths, but it's ok to follow nonetheless.

[440]A 'Self-Referential' Weight Matrix

Jürgen Schmidhuber - 1993

1 paper in library cites

★☆☆☆☆

Google Scholar

The bad part is that this is purely theoretical. The worst part is that it's very mathy. The terrible part is that the PDF is very badly formatted!!!!

[441]Learning in Random Nets

M. Minsky, Oliver G. Selfridge - 1961

1 paper in library cites

★☆☆☆☆

Google Scholar

Minsky is a hater.

[442]No Training Required: Exploring Random Encoders for Sentence Classification

J. Wieting, Douwe Kiela - 2019

1 paper in library cites

★★★★☆

Google Scholar

It's very amazing what they can do with random projections. Plus, it's a good push for better baselines.

[443]Why the Turing Test Is AI's Biggest Blind Alley

Blay Whitby - 1997

2 papers in library cite

★★★★☆

Google Scholar

Very honest and a good discussion however, I think he makes wrong conclusions.

[444]Do We Train on Test Data? Purging CIFAR of Near-Duplicates

Joachim Denzler - 2020

1 paper in library cites

★★★☆☆

Google Scholar

It's a bit derivative from the other paper regarding CIFAR. It's good though, solid addition.

[445]Large-Scale Object Recognition With CUDA-accelerated Hierarchical Neural Networks

R. Uetz, Sven Behnke - 2009

2 papers in library cite

★★★☆☆

Google Scholar

It's not a bad paper, but it seems like they tried to put too much stuff in it - the CNN part is very forgettable and the CUDA part is not so well explained as other papers.

[446]Recent Advances in Finding Best Operating Conditions

R. L. Anderson - 1953

1 paper in library cites

★★☆☆☆

Google Scholar

The paper is not bad, it's just very abstract and no experiments. It's more documenting existing methods without giving much context, explanation, or proof. I also hated that it has nothing to do with Random Search (the reason I read it) - there's only a small mention about it in the end.

[447]Learning Generative Models With Visual Attention

Ruslan Salakhutdinov - 2014

2 papers in library cite

★★☆☆☆

Google Scholar

Waaaay to mathy and TBH I think there aren't a lot of good results. But also I don't have a ton of knowledge on generative models, so I may be biased.

[448]Recognizing Handwritten Digits Using Hierarchical Products of Experts

G. Mayraz, Geoffrey E. Hinton - 2001

1 paper in library cites

★★☆☆☆

Google Scholar

I don't like it because it uses Restricted Boltzmann Machines and I don't want to understand this. Also very mathy.

[449]If Not Turing's Test, Then What?

Paul Cohen - 2005

2 papers in library cite

★★★☆☆

Google Scholar

It's a very lucid paper, but no hot takes either. It's vanilla.

[450]Training MLPs Layer by Layer Using an Objective Function for Internal Representations

Thierry Denoeux - 1996

2 papers in library cite

★★☆☆☆

Google Scholar

There's nothing really wrong with this paper, I just found it VERY uninteresting.

[451]Can Active Memory Replace Attention?

Lukasz Kaiser, Samy Bengio - 2016

2 papers in library cite

★★★★★

Google Scholar

So nice to see an alternative that works as well as attention!

[452]Learning and Evaluating General Linguistic Intelligence

D. Yogatama, C. D. M. D'autume, J. Connor, T. Kocisky, M. Chrzanowski, L. Kong, A. Lazaridou, W. Ling, Longhui Yu, C. Dyer - 2019

2 papers in library cite

★★★☆☆

Google Scholar

It's a nice contribution and good analysis on generalization, but TBH I expected a bit more given the title.

[453]The Chronicles of Rag: The Retriever, the Chunk and the Generator

Celio Larcher - 2024

★★★☆☆

Google Scholar

Rodrigo's paper - it's good, but I think it's more of a tutorial rather than an actual breakthrough. These sorts of papers are very important. Writing could improve a bit, though.

[454]Embracing Data Abundance: Booktest Dataset for Reading Comprehension

O. Bajgar, R. Kadlec, Jan Kleindienst - 2016

2 papers in library cite

★★☆☆☆

Google Scholar

A bit poorly written, throws some shade in other papers, and very marginal contributions (they just perform the same data extraction as others on more data)

[455]UCF101: A Dataset of 101 Human Actions Classes From Videos in the Wild

Khurram Soomro, Amir Roshan Zamir, Mubarak Shah - 2012

1 paper in library cites

★★☆☆☆

Google Scholar

Very low effort paper.

[456]Efficient Training of Large Neural Networks for Language Modeling

Holger Schwenk - 2004

6 papers in library cite

★★☆☆☆

Google Scholar

Very early paper, maybe that's why it's relevant. But very uninteresting despite the nice name ("large NNs" had a very different meaning back then)

[457]Hybrid Word-Subword Decoding for Spoken Term Detection

Lukas Burget - 2008

1 paper in library cites

★☆☆☆☆

Google Scholar

The writing is bad and it's uninteresting. Also has nothing to do with NNs. I think Mikolov only cited because it's from his alma mater. It doesn't explain anything about the approach (or if it does, it's very hard to follow)

[458]Learning Phonetic Features Using Connectionist Networks

Lokendra Shastri - 1986

4 papers in library cite

★★☆☆☆

Google Scholar

Really nothing interesting to see here. Maybe the only merit is early usage of backprop.

[459]Using a Connectionist Model in a Syntactical Based Language Model

Frederick Jelinek - 2003

6 papers in library cite

★☆☆☆☆

Google Scholar

A copy of the other paper. I still don't get what the "Structured Language Model" is and this seems like trying to fit a cube in a circular hole. I don't like it.

[460]High Performance Pattern Recognition on GPU

S. Lahabar, P. Agrawal, P. J. Narayanan - 2008

1 paper in library cites

★☆☆☆☆

Google Scholar

Poorly written (bad grammar). Figures are blurry. Explanation is bad.

[461]Hybrid Language Models Using Mixed Types of Sub-Lexical Units for Open Vocabulary German LVCSR

M. Shaik, A. Mousa, R. Schluter, Hermann Ney - 2011

1 paper in library cites

★★☆☆☆

Google Scholar

Very uninteresting, and nothing related to NNs. TBH I am not sure why Mikolov citred this specifically, as there are several other good examples. The main contribution here is using a mix of different techniques, but doesn't bring anything original.

[462]Evaluating Benchmark Problems by Random Guessing

Jürgen Schmidhuber, Sepp Hochreiter, Yoshua Bengio - 2001

1 paper in library cites

★★★☆☆

Google Scholar

This could be 3 pages long. They just randomly select the weights of the network. The rest is just description of the experiments. Should be a 2 but I am giving them credit for showing that RG can work.

[463]Training Connectionist Models for the Structured Language Model

P. Xu, A. Emami, Frederick Jelinek - 2003

3 papers in library cite

★☆☆☆☆

Google Scholar

A copy of the other paper. Same comment: I dont know what SLM is and it just sounds like a bad idea.

[464]Mamba-3: Improved Sequence Modeling Using State Space Principles

Aakash Lahoti, Kevin Y. Li, Berlin Chen, Caitlin Wang, Aviv Bick, J. Zico Kolter, Tri Dao, Albert Gu - 2026

★★☆☆☆

ICLR2026

Google Scholar

Very hard to understand. I need a ton of context before I can fully grasp this. Need to revisit.

[465]Is It Enough to Get the Behaviour Right?

Levesque - 2009

2 papers in library cite

★★★★★

Google Scholar

Amazing once again. Levesque is amazingly lucid and understands the real arguments here.

[466]Why Language Models Hallucinate

Edwin Zhang - 2025

★★★★★

Google Scholar

Very nice explanation on why LLMs hallucinate (but the problem isn't solved yet), but more still, a VERY nice push to change evaluation methods.

[467]Deep Big Simple Neural Nets Excel on Handwritten Digit Recognition

Dan C. Ciresan, Ueli Meier, Luca M. Gambardella, Jürgen Schmidhuber - 2010

10 papers in library cite

★★★★★

Google Scholar

It's short, simple and straight to the point (as the network in the paper). It's refreshing to read something less "academic" and more "in your face"

[468]Pre-Training Under Infinite compute

Konwoo Kim, Suhas Kotha, Percy Liang, Tatsunori Hashimoto - 2025

★★★★★

ICLR2026

Google Scholar

Simple ideas, simple method, but very nice results!

[469]Agent Data Protocol: Unifying Datasets for Diverse, Effective Fine-Tuning of LLM Agents

Yueqi Song, Ketan Ramaneti, Zaid Sheikh, Ziru Chen, Boyu Gou, Tianbao Xie, Yiheng Xu, Danyang Zhang, Apurva Gandhi, Fan Yang, Joseph Liu, Tianyue Ou, Zhihao Yuan, Frank Xu, Shuyan Zhou, Xiang Yue, Tao Yu, Huan Sun, Yu Su, Graham Neubig - 2026

★★★☆☆

ICLR2026

Google Scholar

Nice results and good idea. I like pushes for standardization. But TBH a bit uninspiring. Seems like something that will either quietly exist or entirely vanish.

[470]Is It Thinking or Cheating? Detecting Implicit Reward Hacking by Measuring Reasoning Effort

Xinpeng Wang, Nitish Joshi, Barbara Plank, Rico Angell, He He - 2026

★★★★☆

ICLR2026

Google Scholar

Very nice idea, but seems like something that will be surpassed/not important quite soon (given that reasoning is something that might vanish, or change completely).

[471]OverFeat: Integrated Recognition, Localization and Detection Using Convolutional Networks

P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, Rob Fergus, Yann Lecun - 2014

16 papers in library cite

★★☆☆☆

Google Scholar

Very convoluted method, was SotA for only a bit of time, and the paper is very boring.

[472]Chain of Thought Monitorability: A New and Fragile Opportunity for Al Safety

Ilya Sutskever - 2025

★★★☆☆

Google Scholar

Hmm... It's a good push but really nothing new in terms of ideas. It's good to push horizons and it's very needed.

[473]How Reliable Is Language Model Micro-Benchmarking?

Gregory Yauney, Shahzaib Saqib Warraich, Swabha Swayamdipta - 2026

★★★★☆

ICLR2026

Google Scholar

I had higher expectations on practical use cases - they seem to only focus on the theory/academic use. Yet, practical implications are nice.

[474]Less Is More: Recursive Reasoning With Tiny Networks

Alexia Jolicoeur Martineau - 2025

★★★★☆

Google Scholar

Very nice alternative to LLMs - but still in early steps, I don't know if this is going to be something important in the future or not. Still seems like a small toy puzzle solver. However, it's good to see that there is still research into small models/datasets.

[475]Your Spending Needs Attention: Modeling Financial Habits With Transformers

Aman Gupta - 2025

★★★☆☆

Google Scholar

It's nothing groundbreaking, but it's nice because it's Nubank :) They should increase the font size and make it more concise though