The reliability
study.
Before a single agent was built, one question had to be answered empirically: can a large language model be trusted to recall Dubai’s permitting facts, or must every fact be retrieved from a live authority source? Twelve questions, three GPT models, no internet — scored against ground truth researched independently from primary sources. The answer set the whole architecture.
The method
GPT vs retrieval-grounded controlThree GPT models — gpt-5.5, gpt-5 and gpt-5-mini, all at maximum reasoning effort — answered the same twelve closed-book questions about the Dubai permitting landscape from parametric knowledge alone. Each answer was scored 0–5 by an independent judge on four axes against a ground-truth dossier assembled from live primary sources. The questions deliberately mix stable facts (which authority governs what), volatile facts (fees, platform names) and recent facts (2024–2026 regulatory change) — because that last band is where parametric recall decays.
The evidence
four figuresPer-model results
mean of 12 questions| Model | Accuracy | Hallucination | Currency | Usability | Composite | Build-ready |
|---|---|---|---|---|---|---|
| gpt-5.5 | 3.08 | 2.92 | 2.75 | 2.33 | 2.77 | 1 / 12 |
| gpt-5 | 3.00 | 2.50 | 2.50 | 2.25 | 2.56 | 0 / 12 |
| gpt-5-mini | 2.75 | 3.58 | 2.17 | 1.75 | 2.56 | 0 / 12 |
A telling inversion: gpt-5-mini scored best on hallucination control and worst on usability — it hedges more, so it invents less, but it also commits to less. The larger, more confident models are the more dangerous: gpt-5 produced 26 critical or major hallucinations to gpt-5-mini’s 12.
Knowledge decays where it matters most
composite by band| Band | gpt-5.5 | gpt-5 | gpt-5-mini |
|---|---|---|---|
| Stable — who governs what | 3.00 | 2.94 | 2.75 |
| Volatile — fees, platforms | 2.83 | 2.67 | 2.75 |
| Recent — 2024–2026 change | 2.55 | 2.20 | 2.30 |
The decay is the diagnostic finding. A permit pipeline lives in the recent band — the 2026 code edition, the current fee basis, the platform that launched last year — and that is precisely where recall is weakest.
Representative critical fabrications
confident, and wrong| Model | Q | Fabricated claim | Reality |
|---|---|---|---|
| gpt-5.5 | Q07 | “DM fee is AED 10 per m² of BUA”, with a worked AED 45,000 example | Fee is AED 1.00 per square foot of BUA — min AED 200, cap AED 250,000 |
| gpt-5.5 | Q08 | “Bronze Sa'fa is the mandatory minimum tier”, stated at High confidence | Silver Sa'fa is the mandatory minimum; Gold and Platinum are voluntary |
| gpt-5.5 | Q02 | DIFC Authority issues building permits inside DIFC | DDA issues the permit; DIFC's planning department issues only NOCs |
| gpt-5.5 | Q02 | DMCC / Concordia is the permitting authority for JLT | Trakhees issues building permits for JLT despite DMCC's free-zone status |
| gpt-5 | Q11 | DDA established by Decree No. 1 of 2018 / Decree No. 13 of 2014 | Law No. 1 of 2000, Law No. 15 of 2014, Law No. 10 of 2018 |
| gpt-5.5 | Q06 | Permit time “reduced from ~30 days to ~5 days (~83%)” | Not a published DM figure — an invented statistic |
Each of these would propagate directly into a defective submission: a wrong fee mis-invoices the client, a wrong Al Sa’fat floor produces a non-compliant design, a wrong authority routes the whole application to the wrong regulator.
The architectural conclusion
reasoning engine, not system of recordThe study does not show GPT to be a weak model. It shows that parametric recall is the wrong retrieval mechanism for statutory data — the jurisdiction is fragmented, the parameters move, and confidence is uncorrelated with correctness. So GPT belongs in the pipeline as a reasoning and drafting engine, and never as the system of record for regulatory facts. That role is reserved for a retrieval layer bound to live authority sources, where every statutory assertion carries a source, a retrieval timestamp and a freshness expiry. That layer is the Knowledge Spine.
How the Spine enforces it →