Why Turkish NLP Needs a Better Embedder?

Introduction

Modern enterprise AI systems rely on one foundational layer: embeddings.If embeddings fail to accurately represent meaning, especially in noisy, domain-specific business language; every downstream system weakens: retrieval, similarity, and interpretation.For English, this foundational layer is mature; however, for Turkish, it is still catching up.Although multilingual models have improved substantially, high-quality Turkish sentence embeddings tailored for enterprise use cases remain limited. Research consistently shows that Turkish’s rich agglutinative morphology and highly productive derivational structure pose challenges for current NLP architectures [1, 2]. Errors in tokenization and semantic drift frequently occur when models trained mostly on English or multilingual corpora attempt to encode Turkish business text.In our benchmark of 15 state-of-the-art embedding models on real Turkish enterprise data, multilingual-e5-large delivered the most stable and production-ready performance. At the same time, the results reveal a structural gap: Turkish NLP still lacks a linguistically robust, enterprise-grade monolingual embedding model.Given production data, we asked: Which existing embedding model performs best on real Turkish enterprise data?To answer this, we benchmarked 15 state-of-the-art embedding models on a small but highly representative dataset for semantic textual similarity.This experiment produced a clear internal winner. But more importantly, it revealed a deeper insight:The Turkish NLP ecosystem urgently needs a strong, general-purpose embedding model that is linguistically robust, domain-aware, and production-ready.

1. The Models We Evaluated

We selected a wide variety of embedding architectures: multilingual, Turkish-specific, compact, large, dense, and hybrid.

Dense Embedders

• Qwen-0.6-Embedder
• all-mpnet-v2
• muvera
• multilingual-e5-large [3]
• gemma-300m
• bge-m3
• gte-multilingual-base [4]
• LaBSE
• jinaai-embeddings-v4
• paraphrase-multilingual-MiniLM-L12-v2 [5]
• bert-base-turkish-cased-mean-nli-stsb-tr
• multilingual-e5-large-instruct [3]

Hybrid (Dense + BM25)

• muvera + BM25
• Qwen-0.6-Embedder + BM25
• multilingual-e5-large + BM25

2. Evaluation Setup

We avoided generic STS datasets and used text that directly reflects MDP’s production workloads.This dataset captures the messy linguistic reality of Turkish enterprise environments, precisely where multilingual models struggle most.We measured:

• Precision
• Recall
• F1-score
• False positives
• Threshold stability (0.89 - 0.92)
• Top-3 nearest-neighbor performance

Our goal was not just accuracy, but also:“How predictable and reliable is this model in real production thresholds?” 

3. Internal Winner: Multilingual E5-Large

Across nearly all tasks, multilingual-e5-large demonstrated the most stable and reliable performance.Its strengths aligned closely with TR-MTEB findings, where E5 variants also led the benchmark with the highest Mean (Task) and Mean (Type) scores [6].

E5 Top 3 Retrieval Results

Key Observations

• 90 is the operational “sweet spot”, balanced precision & recall
• False positives drop sharply after 0.90
• 92 yields perfect precision but too low recall
• Competing models exhibited unstable similarity scores or failed on morphology-heavy text
• E5 was the only model consistently giving >0.90 similarity only when texts were truly related

4. The State of Turkish Embedding Research (TR-MTEB)

Our internal results align with the broader academic landscape.The TR-MTEB benchmark, introduced in 2025, is the first large-scale, task-diverse evaluation suite for Turkish embeddings [8]. It includes:

• Retrieval
• STS
• Classification
• Clustering
• Pair Classification
• Bitext Mining

What TR-MTEB Shows

1. Multilingual E5 models dominate overall performance.
2. Turkish-specific models are improving but not yet surpassing multilingual E5.
3. The ecosystem recognizes that Turkish needs dedicated, high-quality embedding models.

Notable Emerging Models

• turkish-e5-large: E5 finetuned on Turkish corpora [7]
• TR-MTEB’s contrastive-trained Turkish model [6]
• TurkEmbed: matryoshka-trained Turkish embedder [9]
• TurkColBERT: late-interaction IR for Turkish retrieval [10]

These models represent important progress but remain research-grade; plus, not yet validated across noisy enterprise domains like finance, logistics, customer support, or procurement.

5. Why Multilingual and Current Turkish Models Still Fall Short and a General-Purpose Turkish Embedder Matters

Research on Turkish morphology shows persistent issues around:

• Agglutinative structures
• Sparse tokenization
• Morphological segmentation errors
• Semantic drift in long compounds

Combined with real enterprise text issues:

• Typos, abbreviations
• Mixed Turkish-English technical phrases
• Domain-heavy vocabulary

Even strong multilingual models lose nuance and mis-rank similarities.In practice, this leads to:

• Excessive false positives
• Low recall in critical business categories
• Missed duplicate tickets
• Retrieval drift in RAG
• Decreased agent trust in AI output

Thus, even though E5-large is a strong baseline, there is still room and need for:

• better tokenization tailored to Turkish
• contrastive learning on enterprise corpora
• morphology-aware pretraining
• lightweight monolingual architectures (smaller and faster than multilingual E5)

A monolingual Turkish embedder is not just a linguistic improvement. It’s also lighter, faster, and cheaper to deploy on a scale.

Conclusion

Our benchmark confirmed an important truth:

• multilingual-e5-large is currently the strongest practical choice for enterprise Turkish text.
• TR-MTEB research reinforces this result across large-scale evaluations.
• Turkish-specific models are improving, but still not surpassing E5 in robustness.
• There is a clear gap between research prototypes and production-grade Turkish embeddings.

As MDP continues to expand its AI product ecosystem, building a high-quality Turkish embedder becomes a strategic foundation; not just for our own systems, but for any serious enterprise operating in Türkiye’s data landscape.

References

[1] Çöltekin, Ç. (2014, May). A set of open-source tools for Turkish natural language processing. In LREC (pp. 1079-1086). [2] Tohma, K., & Kutlu, Y. (2020). Challenges encountered in Turkish natural language processing studies. Natural and Engineering Sciences, 5(3), 204-211. [3] Wang, L., Yang, N., Huang, X., Yang, L., Majumder, R., & Wei, F. (2024). Multilingual E5 Text Embeddings: A Technical Report. arXiv Preprint arXiv:2402. 05672. [4] Zhang, X., Zhang, Y., Long, D., Xie, W., Dai, Z., Tang, J., … Others. (2024). mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, 1393–1412. [5] Reimers, N., & Gurevych, I. (2019). Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084. [6] Baysan, M. S., & Gungor, T. (2025, November). TR-MTEB: A Comprehensive Benchmark and Embedding Model Suite for Turkish Sentence Representations. In C. Christodoulopoulos, T. Chakraborty, C. Rose, & V. Peng (Eds), Findings of the Association for Computational Linguistics: EMNLP 2025 (pp. 8867–8887). doi: 10.18653/v1/2025.findings-emnlp.471 [7] Kesgin, H. T., Yuce, M. K., & Amasyali, M. F. (2023). Developing and Evaluating Tiny to Medium-Sized Turkish BERT Models. arXiv Preprint arXiv:2307. 14134. [8] https://huggingface.co/trmteb/turkish-embedding-model [9] Ezerceli, Ö., Gümüşçekiçci, G., Erkoç, T., & Özenç, B. (2025, September). TurkEmbed: Turkish Embedding Model on Natural Language Inference & Sentence Text Similarity Tasks. 2025 Innovations in Intelligent Systems and Applications Conference (ASYU), 1–6. doi:10.1109/asyu67174.2025.11208511 [10] Ezerceli, Ö., Hussieni, M. E., Taş, S., Bayraktar, R., Terzioğlu, F. B., Çelebi, Y., & Asker, Y. (2025). TurkColBERT: A Benchmark of Dense and Late-Interaction Models for Turkish Information Retrieval. arXiv [Cs.CL]. Retrieved from http://arxiv.org/abs/2511.16528