Exploring Underexplored Limitations of Cross-Domain Text-to-SQL Generalization

Recently, there has been significant progress in studying neural networks for translating text descriptions into SQL queries under the zero-shot cross-___domain setting. Despite achieving good performance on some public benchmarks, we observe that existing text-to-SQL models do not generalize when facing ___domain knowledge that does not frequently appear in the training data, which may render the worse prediction performance for unseen domains. In this work, we investigate the robustness of text-to-SQL models when the questions require rarely observed ___domain knowledge. In particular, we define five types of ___domain knowledge and introduce Spider-DK (DK is the abbreviation of ___domain knowledge), a human-curated dataset based on the Spider benchmark for text-to-SQL translation. NL questions in Spider-DK are selected from Spider, and we modify some samples by adding ___domain knowledge that reflects real-world question paraphrases. We demonstrate that the prediction accuracy dramatically drops on samples that require such ___domain knowledge, even if the ___domain knowledge appears in the training set, and the model provides the correct predictions for related training samples.