Papperoni

2012

Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription

Pascal Vincent

Open PDF Google Scholar

citations

Cite Score

35

AI summary

This paper introduces a recurrent neural network (RNN-RBM) model for polyphonic music generation and transcription, outperforming traditional models on realistic datasets. The model combines recurrent neural networks with restricted Boltzmann machines to discover temporal dependencies in high-dimensional symbolic music sequences, serving as a symbolic prior to improve transcription accuracy.

Main Contributions

Introduces a novel RNN-RBM architecture for modeling temporal dependencies in high-dimensional sequences.
Demonstrates the effectiveness of the model on polyphonic music generation and transcription tasks.
Achieves improved accuracy in polyphonic transcription by using the model as a symbolic prior.
Shows that pretraining the RNN layer of an RNN-RBM via Hessian-free optimization significantly improves performance.
Demonstrates that RNN-NADE is a robust distribution estimator.

Abstract

We investigate the problem of modeling symbolic sequences of polyphonic music in a completely general piano-roll representation. We introduce a probabilistic model based on distribution estimators conditioned on a recurrent neural network that is able to discover temporal dependencies in high-dimensional sequences. Our approach outperforms many traditional models of polyphonic music on a variety of realistic datasets. We show how our musical language model can serve as a symbolic prior to improve the accuracy of polyphonic transcription.

Citation Graph

Loading graph...

References [26]

Sort:

Filter:

[1]Learning Internal Representations by Error Propagation

D. E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams - 1986

46 papers in library cite

I expected very little of this, but was so good in explaining concepts! Very good read. It gets a bit boring when it starts explaining things by the end of the chapter, but good nonetheless.

[2]Learning Long-Term Dependencies With Gradient Descent Is Difficult

Yoshua Bengio, Patrice Simard, Paolo Frasconi - 1994

31 papers in library cite

The first ones to notice that there is a problem with gradient descent, but way too mathy for me.

[3]Learning Deep Architectures for AI

Yoshua Bengio - 2009

25 papers in library cite

It's a nice overview. Some sections get very theoretical, but the first half is very good and I feel that it does a waaaay better job of explaining RBMs and DBNs than other papers. This feels like Bengio is taking your hand and saying "if you don't know what's going on, here you go, everything you need to know to jump into the deep nets train"

[4]Training Products of Experts by Minimizing Contrastive Divergence

Geoffrey Hinton - 2002

23 papers in library cite

Good read, but I think I need to revisit it after I understand RBMs better.

[5]Learning Recurrent Neural Networks With Hessian-Free Optimization

James Martens, Ilya Sutskever - 2011

13 papers in library cite

Meh, very very mathy and seems like a minor improvement on LSTMs. They give very impressive results but I don't think it's too much of a fair comparison (vs. a 1999 architecture of LSTMs). I think they did well in showing that exploding/vanishing gradients can be overcome, but their method is not "it". As said by Bengio, they are partially responsible for the revival of RNNs (alongside Mikolov)

[6]Information Processing in Dynamical Systems: Foundations of Harmony Theory

P. Smolensky - 1986

11 papers in library cite

88 pages; Introduced RBMs

[7]Exponential Family Harmoniums With an Application to Information Retrieval

M. Welling, M. R. Zvi, Geoffrey Hinton - 2005

8 papers in library cite

[8]The Neural Autoregressive Distribution Estimator

Hugo Larochelle, I. Murray - 2011

5 papers in library cite

[9]The Recurrent Temporal Restricted Boltzmann Machine

Ilya Sutskever, Geoffrey Hinton, G. Taylor - 2008

5 papers in library cite

[10]On the Quantitative Analysis of Deep Belief Networks

Ruslan Salakhutdinov, I. Murray - 2008

4 papers in library cite

[11]Learning Multilevel Distributed Representations for High-Dimensional Sequences

Ilya Sutskever, Geoffrey E. Hinton - 2007

3 papers in library cite

[12]Modeling High-Dimensional Discrete Data With Multi-Layer Neural Networks

Yoshua Bengio, Samy Bengio - 2000

3 papers in library cite

[13]Modeling Human Motion Using Binary Latent Variables

Graham W. Taylor, Geoffrey E. Hinton, S. T. Roweis - 2007

3 papers in library cite

[14]A Classification-Based Polyphonic Piano Transcription Approach Using Learned Feature Representations

J. Nam, J. Ngiam, Honglak Lee, M. Slaney - 2011

2 papers in library cite

[15]Finding Temporal Structure in Music: Blues Improvisation With LSTM Recurrent Networks

D. Eck, Jürgen Schmidhuber - 2002

2 papers in library cite

[16]Harmonising Chorales by Probabilistic Inference

M. Allan, Christopher K. I. Williams - 2005

2 papers in library cite

[17]Neural Network Music Composition by Prediction

M. C. Mozer - 1994

2 papers in library cite

[18]A Discriminative Model for Polyphonic Piano Transcription

G. E. Poliner, D. P. W. Ellis - 2007

1 paper in library cites

[19]A Hierarchy of Recurrent Networks for Speech Recognition

B. Schrauwen, L. Buesing - 2009

1 paper in library cites

[20]Bayesian Music Transcription

A. T. Cemgil - 2004

1 paper in library cites

[21]Conditional Restricted Boltzmann Machines for Structured Output Prediction

V. Mnih, Hugo Larochelle, Geoffrey Hinton - 2011

1 paper in library cites

[22]Evaluation of Multiple-F0 Estimation and Tracking Systems

M. Bay, A. F. Ehmann, J. S. Downie - 2009

1 paper in library cites

[23]Pitch Detection in Polyphonic Music Using Instrument Tone Models

Yiwei Li, D. L. Wang - 2007

1 paper in library cites

[24]Polyphonic Music Modeling With Random Fields

V. Lavrenko, J. Pickens - 2003

1 paper in library cites

[25]Probabilistic Models for Melodic Prediction

J. F. Paiement, Samy Bengio, D. Eck - 2009

1 paper in library cites

[26]Style Translation for Human Motion

E. Hsu, K. Pulli, J. Popovic - 2005

1 paper in library cites

Cited by

8

papers in your library

Cites

6

papers in your library

Read

on July 12, 2025

I think the main contribution is using HF optimization very early on, but very boring use case and they mix RNNs with RBMs which makes this soooo complicated.

Tags

Paper Aliases

No aliases