Benchmarking BERT, GPT, T5, and LLaMA on multilingual and domain-specific tasks
This paper presents a comprehensive comparative study of state-of-the-art Transformer-based architectures for natural language processing tasks. We evaluate BERT, GPT-3, T5, and LLaMA across a diverse set of benchmarks including text classification, named entity recognition, question answering, and text generation. Our experiments on 12 standard NLP datasets reveal that while GPT-3 achieves superior performance on generative tasks (BLEU: 42.3, ROUGE-L: 58.7), BERT-based models remain the most effective for discriminative tasks (F1: 93.2 on CoNLL-2003 NER). LLaMA-7B achieves 92% of GPT-3 performance while requiring 40x fewer parameters.
This research was supported by the European Research Council (ERC) under the European Union Horizon 2020 programme (grant No. 101019342).
We thank the developers of Hugging Face Transformers for the open-source library. Compute resources provided by Jean Zay supercomputer (GENCI grant 2025-AD011012345).
The authors declare no competing interests.
Natural language processing (NLP) has undergone a revolutionary transformation since the introduction of the Transformer architecture by Vaswani et al. (2017). The self-attention mechanism has proven remarkably effective across a wide range of linguistic tasks, from BERT bidirectional encoding to GPT autoregressive generation.
Despite these impressive advances, the landscape of Transformer-based models has become increasingly fragmented. BERT-based models dominate discriminative tasks such as text classification and named entity recognition, while GPT-style models lead in generative tasks like summarization and dialogue.
The core innovation of the Transformer is the scaled dot-product attention mechanism: Attention(Q,K,V) = softmax(QK^T/sqrt(d_k))V. This formulation allows the model to compute relevance scores between all pairs of positions in a sequence, capturing both local and long-range dependencies.
Figure 1 : Architecture du Transformer original.
We evaluated all models on 12 standard NLP benchmarks spanning four task categories: text classification (IMDb, AG News, SST-2), named entity recognition (CoNLL-2003, OntoNotes 5.0), question answering (SQuAD 2.0, Natural Questions), and text generation (CNN/DailyMail, XSum, WMT 2014).
import torch\nimport torch.nn.functional as F\n\nclass ContrastiveDomainAdapter(nn.Module):\n def __init__(self, encoder, temp=0.07):\n super().__init__()\n self.encoder = encoder\n self.projection = nn.Linear(768, 768)\n self.temperature = temp\n\n def contrastive_loss(self, z, labels):\n z = F.normalize(z, dim=-1)\n sim = torch.mm(z, z.t()) / self.temperature\n mask = torch.eq(labels.unsqueeze(0), labels.unsqueeze(1)).float()\n mask.fill_diagonal_(0)\n pos = (sim * mask).sum(dim=-1)\n neg = torch.logsumexp(sim * (1 - mask), dim=-1)\n return (-pos + neg).mean()
| Model | Params | SST-2 | CoNLL F1 | SQuAD F1 | CNN/DM R-L |
|---|---|---|---|---|---|
| BERT-Base | 110M | 93.5 | 92.8 | 88.5 | 38.2 |
| BERT-Large | 340M | 95.2 | 93.2 | 90.2 | 39.5 |
| RoBERTa-L | 355M | 96.4 | 94.6 | 91.8 | 40.1 |
| T5-3B | 3B | 96.8 | 91.5 | 91.0 | 42.8 |
| GPT-3 175B | 175B | 96.2 | 88.4 | 86.7 | 58.7 |
| LLaMA-7B | 7B | 95.8 | 90.2 | 88.9 | 54.1 |
| LLaMA-65B | 65B | 96.5 | 92.1 | 90.5 | 57.2 |
Figure 2 : Performances sur le benchmark GLUE.
The results reveal several important patterns. BERT-based models consistently outperform GPT-style models on discriminative tasks by 3-5 F1 points on NER. LLaMA-7B achieves approximately 92% of GPT-3 performance on generative tasks while using 40x fewer parameters.
Figure 3 : Visualisation des poids d'attention.
We have presented a comprehensive empirical comparison of Transformer-based architectures for NLP across 12 standard benchmarks. Our results provide practical guidance for model selection. All code and experimental configurations are released to facilitate reproducibility.
[1] Vaswani, A. et al. (2017). Attention is all you need. NeurIPS 2017.
https://doi.org/10.5555/3295222.3295349[2] Devlin, J. et al. (2019). BERT: Pre-training of deep bidirectional transformers. NAACL 2019.
https://doi.org/10.18653/v1/N19-1423[3] Brown, T. B. et al. (2020). Language models are few-shot learners. NeurIPS 2020.
https://doi.org/10.5555/3495724.3495883[4] Raffel, C. et al. (2020). Exploring the limits of transfer learning with T5. JMLR, 21(140).
https://doi.org/10.5555/3455716.3455856[5] Touvron, H. et al. (2023). LLaMA: Open and efficient foundation language models. arXiv:2302.13971.
https://doi.org/10.48550/arXiv.2302.13971[6] Liu, Y. et al. (2019). RoBERTa: A robustly optimized BERT pretraining approach. arXiv:1907.11692.
https://doi.org/10.48550/arXiv.1907.11692[7] Wang, A. et al. (2019). GLUE: A multi-task benchmark for NLU. ICLR 2019.
https://doi.org/10.22323/1.357.0001[8] Chen, T. et al. (2020). A simple framework for contrastive learning. ICML 2020.
https://doi.org/10.5555/3524938.3525087[9] Hu, J. et al. (2020). XTREME: A massively multilingual multi-task benchmark. ICML 2020.
https://doi.org/10.5555/3524938.3525312[10] Wolf, T. et al. (2020). Transformers: State-of-the-art NLP. EMNLP 2020.
https://doi.org/10.18653/v1/2020.emnlp-demos.6
Figure 1: Architecture detaillee du Transformer original (Vaswani et al., 2017). L'encodeur (gauche) et le decodeur (droite) utilisent des couches d'auto-attention multi-tetes.
Figure 2: Performances comparatives sur le benchmark GLUE. Les barres d'erreur indiquent l'ecart-type sur 5 runs.
Figure 3: Visualisation des poids d'attention pour la resolution de coreference.