Papperoni

2025

Your Spending Needs Attention: Modeling Financial Habits With Transformers

Aman Gupta

Open PDF Google Scholar

citations

AI summary

This paper introduces nuFormer, a transformer-based representation learning model for financial transaction data, which leverages SSL with both textual and structured attributes, integrates user embeddings with tabular features, and achieves improvements on large-scale recommendation problems at Nubank.

Main Contributions

Propose a new method enabling the use of SSL with transaction data by adapting transformer-based models to handle both textual and structured attributes.
Introduce nuFormer, an end-to-end fine-tuning method that integrates user embeddings with existing tabular features.
Demonstrate improvements for large-scale recommendation problems at Nubank.
Achieve gains solely through enhanced representation learning rather than incorporating new data sources.
Achieve a 1.25% relative improvement in test set AUC.

Abstract

Predictive models play a crucial role in the financial industry, enabling risk prediction, fraud detection, and personalized recommendations, where slight changes in core model performance can result in billions of dollars in revenue or losses. While financial institutions have access to enormous amounts of user data (e.g., bank transactions, in-app events, and customer support logs), leveraging this data effectively remains challenging due to its complexity and scale. Thus, in many financial institutions, most production models follow traditional machine learning (ML) approaches by converting unstructured data into manually engineered tabular features. Conversely, other domains (e.g., natural language processing) have effectively utilized self-supervised learning (SSL) to learn rich representations from raw data, removing the need for manual feature extraction. In this paper, we investigate using transformer-based representation learning models for transaction data, hypothesizing that these models, trained on massive data, can provide a novel and powerful approach to understanding customer behavior. We propose a new method enabling the use of SSL with transaction data by adapting transformer-based models to handle both textual and structured attributes. Our approach, denoted nuFormer, includes an end-to-end fine-tuning method that integrates user embeddings with existing tabular features. Our experiments demonstrate improvements for large-scale recommendation problems at Nubank. Notably, these gains are achieved solely through enhanced representation learning rather than incorporating new data sources.

Citation Graph

Loading graph...

References [43]

Sort:

Filter:

[1]Attention Is All You Need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin - 2017

47 papers in library cite

I mean... it introduced Transformers!

[2]Very Deep Convolutional Networks for Large-Scale Image Recognition

K. Simonyan, Andrew Zisserman - 2014

20 papers in library cite

This is very good! The great thing here is small filters and depth analysis, but truly they do some other stuff as well: SotA, generalization for other tasks, and open source their models. Very nice.

[3]BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding

Jacob Devlin, M. W. Chang, K. Lee, Kristina Toutanova - 2018

39 papers in library cite

Simply amazing. It's very impressive how they make a leap vs. existing stuff (you can see from the references, pretty much no one is doing what they are doing, other than GPT)

[4]ImageNet Classification With Deep Convolutional Neural Networks

Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton - 2012

71 papers in library cite

I'm giving this a 5 just because of the impact, but this is VEEERY derivative of earlier work. Kudos for them for putting it all together, but really there's nothing revolutionary here.

[5]Language Models Are Few-Shot Learners

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, Dario Amodei - 2020

21 papers in library cite

It's just training the GPT arch with more data and more params. Nothing too surprising, but kudos for identifying and formalizing few-shot learning.

[6]Learning Internal Representations by Error Propagation

D. E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams - 1986

46 papers in library cite

I expected very little of this, but was so good in explaining concepts! Very good read. It gets a bit boring when it starts explaining things by the end of the chapter, but good nonetheless.

[7]Neural Machine Translation of Rare Words with Subword Units

R. Sennrich, B. Haddow, Alexandra Birch - 2016

22 papers in library cite

Very good! Simple, explains quite a lot and good results. Forms the basis for a lot of stuff now!

[8]Xgboost: A Scalable Tree Boosting System

T. Chen, C. Guestrin - 2016

1 paper in library cites

XGBoost

[9]Learning Transferable Visual Models From Natural Language Supervision

Alec Radford, J. W. Kim, C. Hallacy, Aditya Ramesh, G. Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark - 2021

2 papers in library cite

CLIP (computer vision)

[10]Llama: Open and Efficient Foundation Language Models

Hugo Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Roziere, N. Goyal, Eric Hambro, F. Azhar - 2023

2 papers in library cite

LLama

[11]Lightgbm: A Highly Efficient Gradient Boosting Decision Tree

G. Ke, Q. Meng, T. Finley, Tianle Wang, Weizhu Chen, W. Ma, Q. Ye, T. Liu - 2017

1 paper in library cites

LightGBM

[12]Lora: Low-Rank Adaptation of Large Language Models

E. J. Hu, Y. Shen, P. Wallis, Z. A. Zhu, Yiwei Li, Shijie Wang, Lisa Wang, Weizhu Chen - 2021

2 papers in library cite

Lora (Canon)

[13]GPT-4 Technical Report

Openai - 2023

6 papers in library cite

GPT 4

[14]Deep Neural Networks for Youtube Recommendations

P. Covington, J. Adams, E. Sargin - 2016

2 papers in library cite

DNN usage in youtube

[15]Self-Attentive Sequential Recommendation

W. Kang, J. Mcauley - 2018

1 paper in library cites

Sequence rec

[16]Flashattention: Fast and Memory-Efficient Exact Attention With Io-Awareness

Tri Dao, D. Fu, Stefano Ermon, A. Rudra, C. Re - 2022

1 paper in library cites

Flash attention! Important

[17]Flashattention-2: Faster Attention With Better Parallelism and Work Partitioning

Tri Dao - 2023

1 paper in library cites

Flash attention-2, less important

[18]Deep Neural Networks and Tabular Data: A Survey

V. Borisov, T. Leemann, K. Sessler, J. Haug, M. Pawelczyk, G. Kasneci - 2022

1 paper in library cites

Survey of DNN for tabular data

[19]Dcn v2: Improved Deep & Cross Network and Practical Lessons for Web-Scale Learning to Rank Systems

R. Wang, R. Shivanna, D. Cheng, Shantanu Jain, D. Lin, L. Hong, E. Chi - 2021

1 paper in library cites

NNs for tabular data

[20]Text Is All You Need: Learning Language Representations for Sequential Recommendation

Jeffrey Li, Mingliang Wang, Jeffrey Li, J. Fu, Xudong Shen, J. Shang, J. Mcauley - 2023

1 paper in library cites

Text is all you need approach

[21]On Embeddings for Numerical Features in Tabular Deep Learning

Y. Gorishniy, I. Rubachev, A. Babenko - 2022

1 paper in library cites

Embeddings for tabular features

[22]The Impact of Positional Encoding on Length Generalization in Transformers

A. Kazemnejad, I. Padhi, K. N. Ramamurthy, P. Das, Siva Reddy - 2023

1 paper in library cites

I think they argue that pos encoding is not required

[23]Flashattention-3: Fast and Accurate Attention With Asynchrony and Low-Precision

J. Shah, G. Bikshandi, Y. Z. Zhang, V. Thakkar, P. Ramani, Tri Dao - 2024

1 paper in library cites

Flash attention-3, probably the least important

[24]Leveraging Large Language Models for Sequential Recommendation

J. Harte, W. Zorgdrager, P. Louridas, A. Katsifodimos, D. Jannach, M. Fragkoulis - 2023

1 paper in library cites

LLMs for Sequence recommendation

[25]Pinnerformer: Sequence Modeling for User Representation at Pinterest

N. Pancha, A. Zhai, J. Leskovec, C. R. Rosenberg - 2022

1 paper in library cites

Sequence recommendation at pinterest

[26]Nxtpost: User to Post Recommendations in Facebook Groups

K. Rangadurai, Yibo Liu, S. Malreddy, Xiaodong Liu, P. Maheshwari, V. Sangale, F. Borisyuk - 2022

1 paper in library cites

User post recommendation in facebook

[27]Tabular Data: Is Attention All You Need?

G. Zabergja, A. Kadra, J. Grabocka - 2024

1 paper in library cites

Comparison of transformers for tabular data

[28]A New Algorithm for Data Compression

P. Gage - 1994

3 papers in library cite

[29]Serial Order: A Parallel Distributed Processing Approach

Michael I. Jordan - 1986

3 papers in library cite

[30]BERT4Rec: Sequential Recommendation With Bidirectional Encoder Representations From Transformer

F. Sun, Joseph Liu, Jeffrey Wu, C. Pei, Xiangning Lin, W. Ou, P. Jiang - 2019

1 paper in library cites

[31]Coles: Contrastive Learning for Event Sequences With Self-Supervision

D. Babaev, N. Ovsov, I. Kireev, M. Ivanova, G. Gusev, I. Nazarov, A. Tuzhilin - 2022

1 paper in library cites

[32]Foundational Models Defining a New Era in Vision: A Survey and Outlook

M. Awais, M. Naseer, S. Khan, R. M. Anwer, H. Cholakkal, Mubarak Shah, Michael Yang, F. S. Khan - 2023

1 paper in library cites

[33]Multi-Interest Network With Dynamic Routing for Recommendation at Tmall

Chun-Liang Li, Ze Liu, M. Wu, Yiheng Xu, H. Zhao, P. Huang, G. Kang, Qinlang Chen, Wentao Li, D. L. Lee - 2019

1 paper in library cites

[34]Practice on Long Sequential User Behavior Modeling for Click-Through Rate Prediction

Q. Pi, W. Bian, G. Zhou, X. Zhu, K. Gai - 2019

1 paper in library cites

[35]Robust Speech Recognition via Large-Scale Weak Supervision

Alec Radford, J. W. Kim, T. Xu, Greg Brockman, C. Mcleavey, Ilya Sutskever - 2023

1 paper in library cites

[36]Speechgpt: Empowering Large Language Models With Intrinsic Cross-Modal Conversational Abilities

Danyang Zhang, Shanda Li, X. Zhang, J. Zhan, Peng Wang, Y. Zhou, X. Qiu - 2023

1 paper in library cites

[37]Speecht5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing

J. Ao, R. Wang, L. Zhou, Caitlin Wang, S. Ren, Yonghui Wu, Shuming Liu, T. Ko, Q. Li, Y. Z. Zhang - 2021

1 paper in library cites

[38]Speechverse: A Large-Scale Generalizable Audio Language Model

N. Das, S. Dingliwal, S. Ronanki, R. Paturi, Zhongqiang Huang, P. Mathur, J. Yuan, D. Bekal, X. Niu, S. M. Jayanthi - 2024

1 paper in library cites

[39]Towards a Foundation Purchasing Model: Pretrained Generative Autoregression on Transaction Sequences

P. Skalski, D. Sutton, S. Burrell, I. Perez, J. Wong - 2023

1 paper in library cites

[40]Towards Universal Sequence Representation Learning for Recommender Systems

Yufang Hou, S. Mu, W. X. Zhao, Yiwei Li, B. Ding, J. Wen - 2022

1 paper in library cites

[41]Uniaudio: An Audio Foundation Model Toward Universal Audio Generation

Diyi Yang, J. Tian, X. Tan, R. Huang, Shuming Liu, X. Chang, J. Shi, Siheng Zhao, J. Bian, Xiaobao Wu - 2023

1 paper in library cites

[42]When Do Neural Nets Outperform Boosted Trees on Tabular Data?

D. Mcelfresh, S. Khandagale, J. Valverde, V. C. Prasad, G. Ramakrishnan, M. Goldblum, C. White - 2023

1 paper in library cites

[43]Zero-Shot Recommender Systems

H. Ding, Yi Ma, A. Deoras, Yuzhi Wang, Haiming Wang - 2021

1 paper in library cites

Cited by

0

papers in your library

Cites

27

papers in your library

Read

on August 17, 2025

It's nothing groundbreaking, but it's nice because it's Nubank :) They should increase the font size and make it more concise though

Tags

Paper Aliases

No aliases