Help diagnosing 100% accuracy (Data Leakage) on DeBERTa & 0% (Label Flip) on a Portuguese DeBERTa model
Reddit r/LanguageTechnology1w4 min read
I’m classifying scientific abstracts written in Portuguese into two temporal categories: "Old" vs "Recent". I tested several models, but two of them are giving me massive red flags: DeBERTa (base): Getting exactly 100% accuracy on the test set. Albertina (a Portuguese DeBERTa-based model): Getting exactly 0% accuracy on "Old" and 100% on "Recent". Note: Other models like mBERT, XLM, and local Portuguese BERTs (Bertimbau) are getting realistic results around 75-85%. What I've already tried (Data Cleaning): Knowing that 100% accuracy screams data leakage, I went aggressive on the preprocessing:
