Relational Transformer: Toward Zero-Shot Foundation Models for Relational Data by Rishabh Ranjan

Venue: Bharti501

Abstract: Pretrained transformers readily adapt to new sequence modeling tasks via zero-shot prompting, but relational domains still lack architectures that transfer across datasets and tasks. The core challenge is the diversity of relational data, with varying heterogeneous schemas, graph structures and functional dependencies. In this talk, I will present the Relational Transformer (RT) architecture, which can be pretrained on diverse relational databases and directly applied to unseen datasets and tasks without task- or dataset-specific fine-tuning, or retrieval of in-context examples. RT (i) tokenizes cells with table/column metadata, (ii) is pretrained via masked token prediction, and (iii) utilizes a novel Relational Attention mechanism over columns, rows, and primary-foreign key links. Pretrained on RelBench datasets spanning tasks such as churn and sales forecasting, RT attains strong zero-shot performance, averaging 93% of fully supervised AUROC on binary classification tasks with a single forward pass of a 22M parameter model, as opposed to 84% for a 27B LLM. Fine-tuning yields state-of-the-art results with high sample efficiency. Our experiments show that RT’s zero-shot transfer harnesses task-table context, relational attention patterns and schema semantics. Overall, RT provides a practical path toward foundation models for relational data. https://arxiv.org/abs/2510.06377

Bio: Rishabh Ranjan is a 3rd year PhD student at Stanford University co-advised by Jure Leskovec and Carlos Guestrin and supported by the Amazon Core AI Fellowship. His research is on building foundation models for relational data, which includes databases, tables, time series and graphs. Before Stanford, he has spent time at CMU and IIT Delhi, where he was the President’s Gold Medalist for 2022. https://rishabh-ranjan.github.io