TABERTA: A Language Model for Dataset Discovery

Dataset discovery requires retrieving relevant tables from large collections of relational data given a natural-language input, yet most language models are designed for unstructured text rather than structured databases. As a result, existing retrieval approaches underutilize table-specific signals such as schema organization, value distributions, and relational context. We present TABERTA, a unified retrieval framework for dataset discovery that adapts a bi-encoder language model to relational data by fine-tuning it on serialized representations of tables. The framework combines three structure-aware table serialization views, multiple retrieval-oriented fine-tuning objectives, and an offline-to-online approximate nearest-neighbor retrieval pipeline. Using WikiDBs, a large corpus of multi-table relational schemas, we fine-tune the language encoder to diverse structural patterns using optimized strategies that serialize schema and table content. Because both natural-language queries and serialized tables are encoded by the same fine-tuned model, they share a common embedding space, enabling effective semantic matching between queries and tables. Across benchmarks, TABERTA improves table retrieval by up to 60% while producing compact 768-dimensional embeddings and reducing offline indexing costs in terms of pipeline time and index size.