SavedAdd your journal

Journals / Turkish Journal of Engineering / 2026 / Volume: 9 Issue: 3

Improving LLM Reliability with RAG in Religious Question-Answering: MufassirQAS

Ahmet Yusuf Alan(MUĞLA SITKI KOÇMAN ÜNİVERSİTESİ, MÜHENDİSLİK FAKÜLTESİ, BİLGİSAYAR MÜHENDİSLİĞİ BÖLÜMÜ)0009-0005-3631-8111
Enis Karaarslan(MUĞLA SITKI KOÇMAN ÜNİVERSİTESİ, MÜHENDİSLİK FAKÜLTESİ, BİLGİSAYAR MÜHENDİSLİĞİ BÖLÜMÜ, BİLGİSAYAR DONANIMI ANABİLİM DALI)0000-0002-3595-8783
Ömer Aydın(MANISA CELAL BAYAR UNIVERSITY, FACULTY OF ENGINEERING, DEPARTMENT OF ELECTRICAL AND ELECTRONIC ENGINEERING)0000-0002-7137-4881
Pages
—
Publication Date
2025-07-01
🏛️Bibliographic metadata and full text access for this article are provided via DergiPark.
DergiPark Record ↗
✨

AI Summary & Key Takeaways

AI Insight
⚡ TL;DR(Too Long; Didn't Read)

Future work will focus on improving accuracy and refining prompt engineering techniques to further minimize biases and ensure even more reliable responses.

📌 3 Key Takeaways

🎯
Objective & Hypothesis:Religious teachings can sometimes be complex and challenging to grasp, but chatbots can serve as effective assistants in this domain.
📊
Key Findings & Data:However, LLMs are prone to hallucinations as they can generate inaccurate or irrelevant information, and these can include sensitive content that could be offensive, inappropriate, or controversial.
💡
Conclusion & Impact:Future work will focus on improving accuracy and refining prompt engineering techniques to further minimize biases and ensure even more reliable responses.
🔬Design:Modeling & Simulation
📍Scope:Comprehensive Sample
⚙️Variables:Question Answering Systems · Chatbots · Generative Artificial Intelligence

Abstract

Religious teachings can sometimes be complex and challenging to grasp, but chatbots can serve as effective assistants in this domain. Large Language Model (LLM) based chatbots, powered by Natural Language Processing (NLP), can connect related topics and provide well-supported responses to intricate questions, making them valuable tools for religious education. However, LLMs are prone to hallucinations as they can generate inaccurate or irrelevant information, and these can include sensitive content that could be offensive, inappropriate, or controversial. Addressing such topics without inadvertently promoting hate speech or disrespecting certain beliefs remains a significant challenge. As a solution to these issues, we introduce MufassirQAS, a system that enhances LLM accuracy and transparency using a vector database-driven Retrieval-Augmented Generation (RAG) approach. We built a dataset comprising fundamental books containing Turkish translations and interpretations of Islamic texts. This database is leveraged to answer religious inquiries while ensuring that responses remain reliable and contextually grounded. Our system also presents the relevant dataset sections alongside the LLM-generated answers, reinforcing transparency. We carefully designed system prompts to prevent harmful, offensive, or disrespectful outputs, ensuring that responses align with ethical and respectful discourse. Moreover, MufassirQAS provides supplementary details, such as source page numbers and referenced articles, to enhance credibility. To evaluate its effectiveness, we tested MufassirQAS against ChatGPT with sensitive questions, and our system demonstrated superior performance in maintaining accuracy and reliability. Future work will focus on improving accuracy and refining prompt engineering techniques to further minimize biases and ensure even more reliable responses.

🔖

Cite This Article

Alan, A. Y., Karaarslan, E., Aydın, Ö. (2025). Improving LLM Reliability with RAG in Religious Question-Answering: MufassirQAS. Turkish Journal of Engineering, Volume: 9 Issue: 3. https://doi.org/10.31127/tuje.1624773

📄 Full Text PDF Viewer

📄

PDF görüntüleyici hazırlanıyor...

References (42)

  1. Öztürk, A., Allahverdı, N., & Saday, F. (2022). Application of artificial intelligence methods for bovine gender prediction. Turkish Journal of Engineering, 6(1), 54-62. https://doi.org/10.31127/tuje.807019
  2. Sesugh, T., Onyia, M., & Fidelis, O. (2024). Predicting the compressive strength of self-compacting concrete using artificial intelligence techniques: A review. Turkish Journal of Engineering, 8(3), 537-550. https://doi.org/10.31127/tuje.1422225
  3. Zela, K., & Saliaj, L. (2023). Forecasting through neural networks: Bitcoin price prediction. Engineering Applications, 2(3), 218-224.
  4. Uzer, A. U. (2024). Efficient prediction of compressive strength in geotechnical engineering using artificial neural networks. Turkish Journal of Engineering, 8(3), 457-468. https://doi.org/10.31127/tuje.1415931
  5. Çubukçu, E. A., Demir, V., & Sevimli, M. F. (2022). Digital elevation modeling using artificial neural networks, deterministic and geostatistical interpolation methods. Turkish Journal of Engineering, 6(3), 199-205. https://doi.org/10.31127/tuje.889570
  6. Kayıran, H. F. (2022). The function of artificial intelligence and its sub-branches in the field of health. Engineering Applications, 1(2), 99-107.
  7. Kayıran, H. F., & Şahmeran, U. (2022). Development of individualized education system with artificial intelligence fuzzy logic method. Engineering Applications, 1(2), 137-144.
  8. Ansay, S., & Köse, B. (2023). Precipitation Forecast with Artificial Neural Networks Method. Journal of AI, 7(1), 15-31. https://doi.org/10.61969/jai.1310918
  9. Mema, B., Basholli, F., & Hyka, D. (2024). Learning transformation and virtual interaction through ChatGPT in Albanian higher education. Advanced Engineering Science, 4, 130-140.
  10. Nyaaba, M., Kyeremeh, P., Majialuwe, E. K., Owusu-fordjour, C., et al. (2024). Generative AI in Academic Research: A Descriptive Study on Awareness, Gender Usage, and Views among Pre-Service Teachers. Journal of AI, 8(1), 45-60. https://doi.org/10.61969/jai.1400867
  11. Baruwa, I. (2024). A Conversation with ChatGPT: Philosophy, Critical Thinking, and Higher Education. Journal of AI, 9(1), 1-12. https://doi.org/10.61969/jai.1498257
  12. Zaazaa, O., & El Bakkali, H. (2024). SmartLLMSentry: A Comprehensive LLM Based Smart Contract Vulnerability Detection Framework. Journal of Metaverse, 4(2), 126-137. https://doi.org/10.57019/jmv.1489060
  13. Aydin, O., Karaarslan, E., Erenay, F. S., & Bacanin, N. (2025). Generative AI in Academic Writing: A Comparison of DeepSeek, Qwen, ChatGPT, Gemini, Llama, Mistral, and Gemma. arXiv preprint arXiv:2503.04765. https://doi.org/10.48550/arXiv.2503.04765
  14. Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N.,. & Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems, 33, 9459-9474.
  15. Temel, N. (2018). Kıraat ve Tecvid Istılahları. (4.b.) İstanbul, M. Ü. İlahiyat Fakültesi Vakfı Yayınları.
  16. Çelik,Ö. (2013). Tefsir Usûlü ve Tarihi. İstanbul, Türkiye: Kampanya Kitapları.
  17. Danacı, S. (2020). Kur’ân-ı Kerim’in Metinleşme Süreci ve Sûrelerin Tertibine Dair Görüşlerin Değerlendirilmesi. Çankırı Karatekin Üniversitesi Karatekin Edebiyat Fakültesi Dergisi, 8(1), 138-166.
  18. Keskioğlu, O. (1987). Nüzûlünden Günümüze Kur’an-ı Kerim Bilgileri. Türkiye Diyanet Vakfı Yayınları, Ankara.
  19. Öge, A. (2015). 18. yüzyıl Osmanlı âlimlerinden Yusuf Efendizâde'nin kıraat ilmindeki yeri. Hüner Yayınevi.
  20. Çalışkan, M. (2021). Meal ve Tefsir ne demektir?. Diyanet Haber.
  21. Ebu’l-Bekâ elKefevî (1939). el-Külliyyât. Algeria, General Franco Ensititusu.
  22. Borgeaud, S., Mensch, A., Hoffmann, J., Cai, T., Rutherford, E., Millican, K.,. & Sifre, L. (2022, June). Improving language models by retrieving from trillions of tokens. In International conference on machine learning (pp. 2206-2240). PMLR.
  23. Al Ghadban, Y., Lu, H., Adavi, U., Sharma, A., Gara, S., Das, N.,. & Hirst, J. E. (2023). Transforming healthcare education: Harnessing large language models for frontline health worker capacity building using retrieval-augmented generation. medRxiv, 2023-12. https://doi.org/10.1101/2023.12.15.23300009
  24. Mallen, A., Asai, A., Zhong, V., Das, R., Khashabi, D., & Hajishirzi, H. (2022). When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. arXiv preprint arXiv:2212.10511. https://doi.org/10.48550/arXiv.2212.10511
  25. Ram, O., Levine, Y., Dalmedigos, I., Muhlgay, D., Shashua, A., Leyton-Brown, K., & Shoham, Y. (2023). In-context retrieval-augmented language models. Transactions of the Association for Computational Linguistics, 11, 1316-1331. https://doi.org/10.1162/tacl_a_00605
  26. Pal, A., Saha, S., & Anita, R. (2020). Musenet: Music generation using abstractive and generative methods. International Journal of Innovative Technology and Exploring Engineering, 9(6), 784-788. https://doi.org/10.35940/ijitee.F3580.049620
  27. Kulkarni, M., Tangarajan, P., Kim, K., & Trivedi, A. (2024). Reinforcement learning for optimizing rag for domain chatbots. arXiv preprint arXiv:2401.06800. https://doi.org/10.48550/arXiv.2401.06800
  28. Mutawa, A. M., & Al-Terkait, S. M. (2011). Al Usouly: An expert system in the origins of Islamic jurisprudence domain. Kuwait Journal of Science and Engineering, 38, 143-166.
  29. Alnefaie, S., Atwell, E., & Alsalka, M. A. (2023, October). Is gpt-4 a good islamic expert for answering quran questions?. In Proceedings of the 35th Conference on Computational Linguistics and Speech Processing (ROCLING 2023) (pp. 124-133).
  30. Sembok, T.M.T., Wani, S. (2024). Is ChatGPT not Appropriate for Religious Use?. In: Badioze Zaman, H., et al. Advances in Visual Informatics. IVIC 2023. Lecture Notes in Computer Science, vol 14322. Springer, Singapore. https://doi.org/10.1007/978-981-99-7339-2_48.
  31. Khan, R. (2023). QuranGPT - Seek guidance from The Almighty, and he shall not refuse!.
  32. Altammami, S., & Atwell, E. (2022). Challenging the transformer-based models with a classical arabic dataset: Quran and hadith. In Proceedings of the Thirteenth Language Resources and Evaluation Conference (pp. 1462-1471). European Language Resources Association.
  33. Yosry, H. (2023). Quran GPT and the dangers of propagating a homogenized version of Islam. Alarabiya News.
  34. Ovadia, O., Brief, M., Mishaeli, M., & Elisha, O. (2023). Fine-tuning or retrieval? comparing knowledge injection in llms. arXiv preprint arXiv:2312.05934. https://doi.org/10.48550/arXiv.2312.05934
  35. Kandemir, M.Y. (2003). Kütüb-i Sitte. TDV İslâm Ansiklopedisi.
  36. Yazıcı, S., Şentürk. L. (2019). İslam İlmihali | yayin.diyanet.gov.tr. Diyanet.gov.tr.
  37. Karaman, H., Dönmez, İ.K., Çağrıcı, M., Gümüş. S. (2019). Kuran Yolu Türkçe Meal Ve Tefsir – 1. Diyanet İşleri Başkanlığı Yayınları.
  38. LangChain (2024). Applications that can reason. Powered by LangChain.
  39. Flowise (2023) - Build LLM Apps Easily. GitHub Web Site.
  40. Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N.,. & Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems, 33, 9459-9474.
  41. Borgeaud, S., Mensch, A., Hoffmann, J., Cai, T., Rutherford, E., Millican, K.,. & Sifre, L. (2022, June). Improving language models by retrieving from trillions of tokens. In International conference on machine learning (pp. 2206-2240). PMLR.
  42. Al Ghadban, Y., Lu, H., Adavi, U., Sharma, A., Gara, S., Das, N.,. & Hirst, J. E. (2023). Transforming healthcare education: Harnessing large language models for frontline health worker capacity building using retrieval-augmented generation. medRxiv, 2023-12. https://doi.org/10.1101/2023.12.15.23300009