Evaluation of an Ai-Powered Chatbot for University Student Support: A Framework-Based Analysis

Che Wan Shamsul Bahri Che Wan Ahmad; Syed Arbaz Ahmed; Khirulnizam Abd Rahman; Syarbaini Ahmad; Mokmin Basri; Syahrul Nizam Junaini; Mohd Shahrul Nizam Mohd Danuri

doi:10.58915/amci.v14i4.1384

Authors

Che Wan Shamsul Bahri Che Wan Ahmad Universiti Islam Selangor, 43000, Kajang, Selangor, Malaysia https://orcid.org/0009-0006-2632-1410
Syed Arbaz Ahmed Universiti Islam Selangor, 43000, Kajang, Selangor, Malaysia https://orcid.org/0009-0005-6756-6556
Khirulnizam Abd Rahman Universiti Islam Selangor, 43000, Kajang, Selangor, Malaysia
Syarbaini Ahmad Universiti Islam Selangor, 43000, Kajang, Selangor, Malaysia https://orcid.org/0000-0003-0776-3708
Mokmin Basri Universiti Islam Selangor, 43000, Kajang, Selangor, Malaysia https://orcid.org/0009-0009-7222-9369
Syahrul Nizam Junaini Universiti Malaysia Sarawak, 94300 Kota Samarahan, Sarawak, Malaysia https://orcid.org/0000-0001-7193-8862
Mohd Shahrul Nizam Mohd Danuri Universiti Malaya, 50603 Kuala Lumpur, Malaysia

DOI:

https://doi.org/10.58915/amci.v14i4.1384

Keywords:

Artificial Intelligence, Chatbot, Chatbots, AI Assistant, Conversational Agent, Automated Chat System, Conversational AI, Intelligent Agent

Abstract

This study presents a comprehensive evaluation of an AI-powered chatbot designed to assist university students with their inquiries. By employing a structured and rigorous framework, the chatbot's performance is assessed based on diverse feedback from participants with both IT and non-IT backgrounds. The evaluation hones in on four critical dimensions: response relevance, understandability, conversation outcomes, and interaction efficiency. Participants were asked ten questions—nine rated on a scale of 1 to 5 and one open-ended—to offer detailed insights into the chatbot’s functionality. Their feedback provided valuable information on the chatbot’s effectiveness in addressing student queries, ease of use, and its potential to enhance the traditional student support system. The findings reveal key strengths, such as improved response accuracy, alongside areas needing further refinement. This study offers practical insights for developers and decision-makers aiming to optimize AI-powered chatbot solutions for enhanced student support, ultimately contributing to a more streamlined and responsive university environment.