Journals / Bilişim Teknolojileri Dergisi / 2026 / Cilt: 19 Sayı: 4

Twitter Bot Detection Using Text-Based Feature Extraction Methods and Machine Learning Algorithms: A Comprehensive Comparative Study

Metin Tabanlı Özellik Çıkarma Yöntemleri ve Makine Öğrenmesi Algoritmaları ile Twitter Bot Tespiti: Kapsamlı Bir Karşılaştırma Çalışması

Duygu Delice(GAZİ ÜNİVERSİTESİ)0009-0005-9677-6237
İsmail Atacak(GAZİ ÜNİVERSİTESİ)0000-0002-6357-0073
Kazım Kılıç(SAMSUN ÜNİVERSİTESİ)0000-0003-2168-1338
Pages
543–556
Publication Date
2026-10-31
🏛️Bibliographic metadata and full text access for this article are provided via DergiPark.
DergiPark Record ↗
✨

AI Summary & Key Takeaways

AI Insight
⚡ TL;DR(Too Long; Didn't Read)

Experimental results show that bot detection models based on TF-IDF and Count N-gram representations outperform those using Word2Vec and FastText across almost all metrics.

📌 3 Key Takeaways

🎯
Objective & Hypothesis:This study presents a comprehensive analysis of bot detection approaches that integrate text representation methods with machine learning (ML) algorithms using only tweet text without relying on profile metadata.
📊
Key Findings & Data:Within this scope, 28 experimental models, formed by combinations of four different text representation methods (Term Frequency-Inverse Document Frequency (TF-IDF), Count N-gram, Word2Vec, and FastText) and seven different classical ML algorithms (Linear Support Vector Machine (SVM), Logistic Regression (LR), Decision Tree (DT), Random Forest (RF), Gradient Boosting (GB), Light Gradient Boosting Machine (LightGBM), and eXtreme Gradient Boosting (XGBoost)), were systematically evaluated in the study.
💡
Conclusion & Impact:Experimental results show that bot detection models based on TF-IDF and Count N-gram representations outperform those using Word2Vec and FastText across almost all metrics.
🔬Design:Modeling & Simulation
📍Scope:Comprehensive Sample
⚙️Variables:twitter bot tespiti · özellik çıkarımı · metin tabanlı özellik çıkarımı

Abstract

This study presents a comprehensive analysis of bot detection approaches that integrate text representation methods with machine learning (ML) algorithms using only tweet text without relying on profile metadata. Within this scope, 28 experimental models, formed by combinations of four different text representation methods (Term Frequency-Inverse Document Frequency (TF-IDF), Count N-gram, Word2Vec, and FastText) and seven different classical ML algorithms (Linear Support Vector Machine (SVM), Logistic Regression (LR), Decision Tree (DT), Random Forest (RF), Gradient Boosting (GB), Light Gradient Boosting Machine (LightGBM), and eXtreme Gradient Boosting (XGBoost)), were systematically evaluated in the study. Experimental results show that bot detection models based on TF-IDF and Count N-gram representations outperform those using Word2Vec and FastText across almost all metrics. Among them, TF-IDF+LightGBM achieved the highest scores on most metrics: accuracy of 0.9393, precision of 0.9384, recall of 0.9404, F1-score of 0.9394, ROC-AUC of 0.9854, and PR-AUC of 0.9849. Quartile-zone analyses further indicate that TF-IDF+LR offers a favorable trade-off between performance and computational cost, combining a competitive F1-score (0.9270) with very low latency (0.000593 ms/sample), making it a promising candidate for real-time applications. The study also provides a systematic comparative framework for future research in this field.

Özet

Bu çalışma, profil meta-verilerinden bağımsız olarak yalnızca tweet metinlerinin kullanıldığı metin temelli özellik çıkarım yöntemleri ile makine öğrenimi (ML) yaklaşımlarını entegre eden bot tespit yaklaşımları üzerine yapılmış kapsamlı analizleri sunar. Bu kapsamda çalışmada dört farklı metin temsil yöntemi (Terim Sıklığı - Ters Belge Sıklığı (TF-IDF), Count N-gram, Word2Vec ve FastText) ile yedi farklı klasik ML algoritmasının (Linear SVM, LR, DT, RF, GB, LightGBM ve XGBoost) kombinasyonlarından oluşan toplam 28 deneysel model sistematik olarak değerlendirilmiştir. Deneysel sonuçlar, TF-IDF ve Count N-grama dayalı bot tespit modellerinin, Word2Vec ve FastText tabanlı modellere kıyasla neredeyse tüm metriklerde daha iyi performans gösterdiğini ortaya koymuştur. Bunlar arasında TF-IDF+LightGBM modeli, çoğu metrikte en yüksek skorlara ulaşmıştır. Model doğrulukta 0.9393, kesinlikte 0.9384, duyarlılıkta 0.9404, F1-skorunda 0.9394, ROC-AUC’ ta 0.9854 ve PR-AUC’ ta 0.9849 skorlarını elde etmiştir. Çeyrek bölge analizleri ayrıca TF-IDF+LR modelinin, rekabetçi bir F1-skorunu (0.9270) çok düşük gecikme süresiyle (0.000593 ms/örnek) birleştirerek sınıflandırma performansı ile hesaplama maliyeti arasında elverişli bir denge sunduğunu ve gerçek zamanlı uygulamalar için umut vadeden bir aday olduğunu göstermektedir. Çalışma bu anlamda ilgili alandaki gelecek araştırmalar için sistematik bir karşılaştırmalı çerçeve sağlamaktadır.

🔖

Cite This Article

Delice, D., Atacak, İ., Kılıç, K. (2026). Twitter Bot Detection Using Text-Based Feature Extraction Methods and Machine Learning Algorithms: A Comprehensive Comparative Study. Bilişim Teknolojileri Dergisi, Cilt: 19 Sayı: 4, 543–556. https://doi.org/10.17671/gazibtd.2034345

📄 Full Text PDF Viewer

📄

PDF görüntüleyici hazırlanıyor...

References (51)

  1. E. Ferrara, O. Varol, C. Davis, F. Menczer, and A. Flammini, “The rise of social bots,” Commun. ACM, vol. 59, no. 7, pp. 96–104.
  2. A. M. Kaplan and M. Haenlein, “Users of the world, unite! The challenges and opportunities of Social Media,” Bus. Horiz., vol. 53, no. 1, pp. 59–68, Jan. 2010, doi: 10.1016/J.BUSHOR.2009.09.003.
  3. M. Aljabri, R. Zagrouba, A. Shaahid, F. Alnasser, A. Saleh, and D. M. Alomari, “Machine learning-based social media bot detection: a comprehensive literature review,” Soc. Netw. Anal. Min., vol. 13, no. 1, pp. 20-, Jan. 2023, doi: 10.1007/S13278-022-01020-5.
  4. C. Shao, G. L. Ciampaglia, O. Varol, K. C. Yang, A. Flammini, and F. Menczer, “The spread of low-credibility content by social bots,” Nat. Commun., vol. 9, no. 1, pp. 4787:1–9, Nov. 2018, doi: 10.1038/s41467-018-06930-7.
  5. S. Feng, H. Wan, N. Wang, Z. Tan, M. Luo, and Y. Tsvetkov, “What Does the Bot Say? Opportunities and Risks of Large Language Models in Social Media Bot Detection,” in In Proceedings of the Annual Meeting of the Association for Computational Linguistics, Bangkok, Thailand: Association for Computational Linguistics (ACL), Aug. 2024, pp. 3580–3601. doi: 10.18653/V1/2024.ACL-LONG.196.
  6. A. Garcia-Silva, C. Berrio, and J. M. Gomez-Perez, “Understanding Transformers for Bot Detection in Twitter,” Apr. 2021. Accessed: Aug. 02, 2026. [Online]. Available: https://arxiv.org/pdf/2104.06182
  7. A. Sallah et al., “Fine-Tuned Understanding: Enhancing Social Bot Detection with Transformer-Based Classification,” IEEE Access, vol. 12, pp. 118250–118269, 2024, doi: 10.1109/ACCESS.2024.3440657.
  8. H. D. Nguyen et al., “Supervised learning models for social bot detection: Literature review and benchmark,” Expert Syst. Appl., vol. 238, p. 122217, Mar. 2024, doi: 10.1016/J.ESWA.2023.122217.
  9. Z. E. H. Al-Khersan and N. A. Flayh, “Enhanced Twitter Bot Detection via Static and Temporal Feature Integration,” Journal of Al-Qadisiyah for Computer Science and Mathematics, vol. 17, no. 4, pp. 349-359, Dec. 2025, doi: 10.29304/JQCSM.2025.17.42574.
  10. E. Alothali, K. Hayawi, and H. Alashwal, “Hybrid feature selection approach to identify optimal features of profile metadata to detect social bots in Twitter,” Social Network Analysis and Mining, vol. 11, no. 1, pp. 84-, Sep. 2021, doi: 10.1007/S13278-021-00786-4.
  11. A. A. Darem et al., “Cybersecurity in social networks: An ensemble model for Twitter bot detection | International Journal of ADVANCED AND APPLIED SCIENCES,” International Journal of Advanced and Applied Sciences, vol. 11, no. 11, pp. 130–141, 2024, Accessed: Aug. 01, 2026. [Online]. Available: https://www.science-gate.com/IJAAS/2024/V11I11/1021833ijaas202411014.html
  12. S. Kudugunta and E. Ferrara, “Deep neural networks for bot detection,” Inf. Sci. (N. Y)., vol. 467, pp. 312–322, Oct. 2018, doi: 10.1016/J.INS.2018.08.019.
  13. K. J. Krishna and V. M. S. Babu, “Machine learning based social bot detection with transformer-based classification,” Journal of Nonlinear Analysis and Optimization, vol. 16, no. 1, pp. 2254–2260, 2025.
  14. D. Martin-Gutierrez, G. Hernandez-Penaloza, A. B. Hernandez, A. Lozano-Diez, and F. Alvarez, “A Deep Learning Approach for Robust Detection of Bots in Twitter Using Transformers,” IEEE Access, vol. 9, pp. 54591–54601, 2021, doi: 10.1109/ACCESS.2021.3068659.
  15. D. Li et al., “BotLGT: Social bot detection based on LLM and graph transformer,” Neurocomputing, vol. 658, p. 131453, Dec. 2025, doi: 10.1016/J.NEUCOM.2025.131453.
  16. S. Shi et al., “SStackGNN: Graph Data Augmentation Simplified Stacking Graph Neural Network for Twitter Bot Detection,” International Journal of Computational Intelligence Systems, vol. 17, no. 1, pp. 106-, Apr. 2024, doi: 10.1007/S44196-024-00496-7.
  17. X. Tian, Y. He, Y. Wu, and X. Lu, “SeBot-MLS: Multi-level structural feature learning for graph-based social bot detection,” Journal of Safety Science and Resilience, vol. 7, no. 2, p. 100310, Jun. 2026, doi: 10.1016/J.JNLSSR.2026.100310.
  18. A. Shevtsov, D. Antonakaki, I. Lamprou, P. Pratikakis, and S. Ioannidis, “BotArtist: Generic Approach for Bot Detection in Twitter via Semi-automatic Machine Learning Pipeline,” Lecture Notes in Computer Science, vol. 16323 LNCS, pp. 19–27, 2026, doi: 10.1007/978-3-032-13821-7_2/SAVE-RESEARCH.
  19. K. Hayawi, S. Saha, M. M. Masud, S. S. Mathew, and M. Kaosar, “Social media bot detection with deep learning methods: a systematic review,” Neural Computing and Applications, vol. 35, no. 12, pp. 8903–8918, Mar. 2023, doi: 10.1007/S00521-023-08352-Z.
  20. A. Ramalingaiah, S. Hussaini, and S. Chaudhari, “Twitter bot detection using supervised machine learning,” J. Phys. Conf. Ser., vol. 1950, no. 1, p. 012006, Aug. 2021, doi: 10.1088/1742-6596/1950/1/012006.
  21. A. Sankaranarayanan, K. U., G. T.P., and S. T., “Detection of Spam Bots on Twitter using Machine Learning,” International Journal of Innovative Technology and Exploring Engineering, vol. 9, no. 6, pp. 249–252, Apr. 2020, doi: 10.35940/IJITEE.F3691.049620.
  22. A. Hannousse and Z. Talha, “A Hybrid Ensemble Learning Approach for Detecting Bots on Twitter,” International Journal of Performability Engineering, vol. 20, no. 10, p. 610, Oct. 2024, doi: 10.23940/IJPE.24.10.P3.610620.
  23. B. A. Ayogu, G. O. Ogunleye, L. B. Adewole, M. Olagunju, and W. A. Oyatoyinbo, “Comparative Analysis of Machine Learning Algorithms for the Classification of Twitter Bots,” University of Ibadan Journal of Science and Logics in ICT Research, vol. 11, no. 1, pp. 2714–3627, Apr. 2024, Accessed: Sep. 02, 2026. [Online]. Available: https://journals.ui.edu.ng/index.php/uijslictr/article/view/1364
  24. D. P. Kavadi, C. S. Sanaboina, R. Patan, and A. Gandomi, “N-Gram-Based Machine Learning Approach for Bot or Human Detection from Text Messages,” in Proceedings of the 2022 6th International Conference on Intelligent Systems, Metaheuristics & Swarm Intelligence, Association for Computing Machinery, Apr. 2022, pp. 80–85. doi: 10.1145/3533050.3533063;CSUBTYPE:STRING:CONFERENCE.
  25. F. K. Alarfaj, H. Ahmad, H. U. Khan, A. M. Alomair, N. Almusallam, and M. Ahmed, “Twitter Bot Detection Using Diverse Content Features and Applying Machine Learning Algorithms,” Sustainability, vol. 15, no. 8, p. 6662, Apr. 2023, doi: 10.3390/SU15086662.
  26. F. Wei and U. T. Nguyen, “Twitter bot detection using bidirectional long short-term memory neural networks and word embeddings,” in Proceedings - 1st IEEE International Conference on Trust, Privacy and 2019 First IEEE International Conference on Trust, Privacy and Security in Intelligent Systems and Applications (TPS-ISA), Los Angeles, CA, USA: Institute of Electrical and Electronics Engineers Inc., Dec. 2019, pp. 101–109. doi: 10.1109/TPS-ISA48467.2019.00021.
  27. E. Arin and M. Kutlu, “Deep Learning Based Social Bot Detection on Twitter,” IEEE Transactions on Information Forensics and Security, vol. 18, pp. 1763–1772, 2023, doi: 10.1109/TIFS.2023.3254429.
  28. L. Ilias and I. Roussaki, “Detecting malicious activity in Twitter using deep learning techniques,” Appl. Soft Comput., vol. 107, p. 107360, Aug. 2021, doi: 10.1016/J.ASOC.2021.107360.
  29. Z. Ellaky, F. Benabbou, Y. Matrane, and S. Qaqa, “A Hybrid Deep Learning Architecture for Social Media Bots Detection Based on BiGRU-LSTM and GloVe Word Embedding,” IEEE Access, vol. 12, pp. 100278–100294, 2024, doi: 10.1109/ACCESS.2024.3430859.
  30. H. G. Shah and H. Joshi, “Spam bot detection on Twitter platform using positional attention based dense convolutional neural network,” Appl. Soft Comput., vol. 184, p. 113725, Dec. 2025, doi: 10.1016/J.ASOC.2025.113725.
  31. S. Dhar and I. Bose, “An ensemble deep learning model for fast classification of Twitter spam,” Information & Management, vol. 61, no. 8, p. 104052, Dec. 2024, doi: 10.1016/J.IM.2024.104052.
  32. D. Javed, N. Z. Zaman, N. A. Khan, S. K. Ray, A. Al-Dhaqm, and V. R. Kebande, “Identification of Spambots and Fake Followers on Social Network via Interpretable AI-Based Machine Learning,” IEEE Access, vol. 13, pp. 52246–52259, 2025, doi: 10.1109/ACCESS.2025.3551993.
  33. L. Ilias, I. Michail Kazelidis, and D. Askounis, “Multimodal Detection of Bots on X (Twitter) Using Transformers,” IEEE Transactions on Information Forensics and Security, vol. 19, pp. 7320–7334, 2024, doi: 10.1109/TIFS.2024.3435138.
  34. X. Wang, K. Wang, K. Chen, Z. Wang, and K. Zheng, “Unsupervised twitter social bot detection using deep contrastive graph clustering,” Knowl. Based. Syst., vol. 293, p. 111690, Jun. 2024, doi: 10.1016/J.KNOSYS.2024.111690.
  35. rrsr28, “GitHub,” Twitter-Bot-Detection. Accessed: Jul. 12, 2026. [Online]. Available: https://github.com/rrsr28/Twitter-Bot-Detection
  36. G. Salton, A. Wong, and C. S. Yang, “A Vector Space Model for Automatic Indexing,” Commun. ACM, vol. 18, no. 11, pp. 613–620, Nov. 1975, doi: 10.1145/361219.361220;PAGE:STRING:ARTICLE/CHAPTER.
  37. D. Jurafsky and J. H. Martin, Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition. Upper Saddle River, New Jersey: Prentice Hall, 2000. Accessed: Sep. 03, 2026. [Online]. Available: http://www.mitpressjournals.org/doi/pdf/10.1162/089120100750105975
  38. T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient Estimation of Word Representations in Vector Space,” 1st International Conference on Learning Representations, ICLR 2013 - Workshop Track Proceedings, Jan. 2013, Accessed: Sep. 03, 2026. [Online]. Available: https://arxiv.org/pdf/1301.3781
  39. P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov, “Enriching Word Vectors with Subword Information,” Trans. Assoc. Comput. Linguist., vol. 5, pp. 135–146, Dec. 2017, doi: 10.1162/TACL_A_00051/43387.
  40. C. Cortes and V. Vapnik, “Support-vector networks,” Mach. Learn., vol. 20, no. 3, pp. 273–297, Sep. 1995, doi: 10.1007/BF00994018.
  41. D. R. Cox, “The Regression Analysis of Binary Sequences,” J. R. Stat. Soc. Series B Stat. Methodol., vol. 20, no. 2, pp. 215–232, Jul. 1958, doi: 10.1111/J.2517-6161.1958.TB00292.X.
  42. J. R. Quinlan, “Induction of decision trees,” Machine Learning, vol. 1, no. 1, pp. 81–106, Mar. 1986, doi: 10.1007/BF00116251.
  43. L. Breiman, “Random forests,” Mach. Learn., vol. 45, no. 1, pp. 5–32, Oct. 2001, doi: 10.1023/A:1010933404324/METRICS.
  44. J. H. Friedman, “Greedy function approximation: A gradient boosting machine.,” The Annals of Statistics, vol. 29, no. 5, pp. 1189–1232, Oct. 2001, doi: 10.1214/AOS/1013203451.
  45. G. Ke et al., “LightGBM: A Highly Efficient Gradient Boosting Decision Tree,” Adv. Neural Inf. Process. Syst., vol. 30, 2017, Accessed: Sep. 03, 2026. [Online]. Available: https://github.com/Microsoft/LightGBM.
  46. T. Chen and C. Guestrin, “XGBoost: A scalable tree boosting system,” Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, vol. 13-17-August-2016, pp. 785–794, Aug. 2016, doi: 10.1145/2939672.2939785;CSUBTYPE:STRING:CONFERENCE.
  47. V. Jayaswal, “Performance Metrics: Confusion matrix, Precision, Recall, and F1 Score,” Towards Data Science. Accessed: Jan. 12, 2026. [Online]. Available: https://towardsdatascience.com/performance-metrics-confusion-matrix-precision-recall-and-f1-score-a8fe076a2262/
  48. E. Alothali, K. Hayawi, and H. Alashwal, “Hybrid feature selection approach to identify optimal features of profile metadata to detect social bots in Twitter,” Social Network Analysis and Mining, vol. 11, no. 1, pp. 84-, Sep. 2021, doi: 10.1007/S13278-021-00786-4.
  49. S. Shi et al., “SStackGNN: Graph Data Augmentation Simplified Stacking Graph Neural Network for Twitter Bot Detection,” International Journal of Computational Intelligence Systems, vol. 17, no. 1, pp. 106-, Apr. 2024, doi: 10.1007/S44196-024-00496-7.
  50. K. Hayawi, S. Saha, M. M. Masud, S. S. Mathew, and M. Kaosar, “Social media bot detection with deep learning methods: a systematic review,” Neural Computing and Applications, vol. 35, no. 12, pp. 8903–8918, Mar. 2023, doi: 10.1007/S00521-023-08352-Z.
  51. J. R. Quinlan, “Induction of decision trees,” Machine Learning, vol. 1, no. 1, pp. 81–106, Mar. 1986, doi: 10.1007/BF00116251.