Assessing the capabilities of ChatGPT in recognising customer intent in a small training data scenario.

Ezenkwu, Chinedu Pascal; Ibeke, Ebuka; Iwendi, Celestine

Abstract

This study addresses the issue of recognising customer intent when only limited training data is available. The performance of ChatGPT was evaluated in this scenario, and it was found to be better than traditional machine learning algorithms and the Bidirectional Encoder Representations from Transformers (BERT) model, which performed the worst in this case. While Random Forest with PCA was objectively the best among traditional models when the training examples were randomly selected, a qualitative evaluation showed that ChatGPT had better generalisation ability and could produce contextually correct outputs. Our research found that, to improve ChatGPT's performance on small data classification tasks, it is essential to utilise stratified sampling to select representative examples for few-shot learning. This research provides valuable insights into using ChatGPT in customer-facing applications with limited training data. Knowing the strengths and limitations of ChatGPT can enhance response accuracy, and customer satisfaction and loyalty.

Citation

EZENKWU, C.P., IBEKE, E. and IWENDI, C. 2024. Assessing the capabilities of ChatGPT in recognising customer intent in a small training data scenario. To be presented at the 3rd International conference on advanced communication and intelligent systems (ICACIS 2024), 16-17 May 2024, New Delhi, India.

• Presentation Conference Type: Lecture
• Conference Name: 3rd International conference on advanced communication and intelligent systems (ICACIS 2024)
• Conference Location: New Delhi, India
• Start Date: May 16, 2024
• End Date: May 17, 2024
• Deposit Date: Apr 2, 2024
• Keywords: ChatGPT; Artificial intelligence (AI); Chatbots; Machine learning; Few-shot learning; Human computer interaction (HCI)
• Public URL: https://rgu-repository.worktribe.com/output/2293496