ديوان — محرك البحث الدلالي في الشعر العربي

Deewan is a semantic search engine for Arabic poetry. Instead of matching letters, it understands meaning — search for an idea or feeling like "longing for home" and it surfaces verses that express it, even when those exact words never appear.

The Corpus

• 6.57 million verses indexed in total
• 6.44M classical (fusha) verses from the open ashaar dataset
• 125K Nabati verses from the tarab dataset
• Spanning twelve literary eras, from pre-Islamic to modern

How It Works

• Every verse is encoded into a 384-dimensional vector using the Arabic-Triplet-Matryoshka-V2 embedding model
• Verses close in meaning end up close in vector space
• An HNSW (Hierarchical Navigable Small World) index searches all 6.57M verses in a few seconds on a single server
• Classical and Nabati live in two separate indexes so the traditions never cross-contaminate
• Era-aware re-ranking gives classical poets a fair share alongside modern ones

Academic Benchmark:
Deewan was evaluated on Fann-or-Flop (EMNLP 2025), a peer-reviewed benchmark of 53,047 scholar-written verse explanations. Given only an explanation, it must retrieve the correct poet out of thousands. Deewan places the correct poet in the top-10 for 3.13% of queries — about 22 times better than random (~0.14%). To our knowledge, this is the first time an Arabic poetry search engine has been measured against a published academic benchmark.

The Stack

• FastAPI backend running as a systemd service
• Next.js 15 (App Router) frontend, Arabic-first, with next-intl
• hnswlib for approximate-nearest-neighbor search
• Self-hosted on a single dedicated server with no external API dependency
• Spans about 10,600 lines of code across frontend, backend, and data pipelines