Introduction: As artificial intelligence systems like ChatGPT-4 become integral information sources, scrutinizing their reliability is crucial. This study seeks to assess the authenticity of references generated by ChatGPT-4 on primary immunodeficiencies.
Method: 10 distinct subjects related to primary immunodeficiencies were selected and the prompt “Act as a researcher, your task will be to write about [subject related to primary immunodeficiencies] for an immunology literature review paper, make it as extensive and comprehensive as possible, use proper citations and references.” was submitted to ChatGPT-4. For validation, the extracted references were first cross-checked with the PubMed database. Those not located on PubMed underwent further scrutiny via the Google search engine.
Results: A total of 88 references were extracted from ChatGPT-4 relating to 10 topics on primary immunodeficiencies. Out of the 88 references, 55 (62.5%) were real and had no mistakes, 21 (23.8%) were found when searched but presented mistakes that did not correlate with the original article and 12 (13.6%) were fabricated. From the articles that were mistaken, a total of 56 errors were identified, 6 mistakes were related to the authors, 14 mistakes were related to the date of publication, 2 mistakes were related to the title of the article, 4 mistakes were related to the name of the journal, 15 mistakes were related to the volume of the article and 15 mistakes were related to the pages of the article.
Conclusions: With only 13.6% being completely fabricated, there is a big increase in the authenticity of references provided by ChatGPT-4 when compared with previous models and a step in the right direction. Improvements are still needed for it to become a trustworthy tool for academic purposes.