Research Article - Journal of Psychology and Cognition (2018) Volume 3, Issue 3
Limitations of integrating technology in cognitive testing of non-verbal memoryPorrselvi AP*, Shankar V
Department of Neurology, Sri Ramachandra University, Chennai, India.
- Corresponding Author:
- Porrselvi AP
UGC Senior Research Fellow
A2, Department of Neurology, Sri Ramachandra University
Tamil Nadu, India
E-mail: [email protected]
Accepted on August 06 2018
Citation: Porrselvi AP, Shankar V. Limitations of integrating technology in cognitive testing of non-verbal memory J Psychol Cognition 2018;3(3):50-53.
Objective: With the rapid progress of computer technology, there has been increased integration of the technology in cognitive assessment and rehabilitation. This study aims to bring to light some of the limitations posed by the integration of technology in cognitive testing especially for an older cohort.
Method: In this study the performance of older people in a test of non-verbal memory using digital capture method and paper and pencil method has been compared.
Results: Significant differences were found between the test performances on both the versions of the same test with the testees performing better on the paper and pencil version. Gender or level of education did not affect the test performance while age affected test performance more on the digital version than on the paper and pencil version.
Conclusion: The tester must be trained and must take necessary precaution for the purpose of accurate test findings and in order to avoid diagnostic errors.
Test construction, Computerized testing, Technology limitations, Cognitive assessment.
Since the 1980s, the possibilities of computer-automated neuropsychological assessments have been discussed and compared with the paper and pencil forms of testing . The field of cognitive testing has since then been enhanced by the recent innovations in the use of computer applications and technology. Digital measures have reportedly offered better standardization in the administration of tests, test subjectspecific presentation of stimuli, collection of accurate and precise data, ease of scoring and data management as well as reporting . Wider applications of cognitive testing are now possible due to the ability of such computerized assessments to accurately measure data points that were not possible before the advent of technology like response latency . There is no doubt that this integration of technology has gotten professionals working in the field of cognition excited about the possibilities it offers but this research study tries to throw light on the need to exercise caution and take any necessary measures before integrating technology into neurocognitive assessment.
Historically, neuropsychological assessments have been paper and pencil-based tests to assess cognitive abilities but now, many traditional neuropsychological assessments have been adapted to the computerized format and numerous computerized screening measures and comprehensive test batteries have been developed. Virtual reality technology is now being studied to be incorporated into ecologically valid assessments as well as rehabilitation . Remote assessments have become common place.
The recent invasion of technological advances has affected the way people- babies, children and adults carry out their everyday activities and it remains to be seen if this has affected the way our brains functions. Hence, integrating technology in neuropsychological assessment must consider the testees’ degree of exposure to technology and the comfort they have in using the computer devices, especially the kind of device that is being used to test. Special care must be taken when the target patient population is an older cohort who may find technology intimidating as was reported by testees during the current study. Studies have found that greater subjects with greater exposure to computers showed better performance on computer-based assessments than those with less computer experience , which suggests that familiarity with technology influences computer-based testing. Results from computerized versus examiner-administered testing have been shown to be different in computer-competent versus computer-naive populations . Positive affect, intrinsic motivation and engagement in the technology  have shown to influence performance on computerized assessments. Presumably because these factors influence in learning the use of technology. Therefore, normative properties for integrating technology in neuropsychological assessment should consider individual variability . Such validity (including construct and concurrent validity with traditional paper and pencil tests) and reliability, normative data and administration are of particular importance when the assessment aims to detect late-life neurocognitive disorders to avoid technology specific effects on the results of the assessment.
The National Academy of Neuropsychology and the American Academy of Clinical Neuropsychology in their joint position paper on CNADs have discussed several key issues in developing computerized assessments . They note that some patients with cognitive, motor, or sensory disabilities may have difficulty completing a computerized test in the standardized manner. They also note that individual differences in use and familiarity with the computer device can affect how the testees interact with the devices, utilize response modalities, and respond to stimuli. They recommend that such factors should be accounted for during test standardization and validation and the developers should provide guidance to the clinicians and researchers about target patient populations to whom the test can be administered, under what conditions and who the tests cannot be administered to based on their abilities and limitations.
The current research and its methods were approved by the Institutional Ethics Committee of Sri Ramachandra University. The reported findings in the study were observed by chance during the pilot study for a comprehensive computer assisted cognitive tests battery for testing cognition in the elderly Tamil population called “TAM Battery”, results of which are yet to be published. The Tamil Arithiran Mathipeedu Battery (TAM Battery) is a comprehensive computer-assisted battery of neuropsychological tests that is available in English and Tamil languages. It includes a quick cognitive screen module in addition to tests of attention, working memory, memory, language, fluency, calculation, visuo spatial skills, executive functions and information processing speed. The test tablet computer device used for the normative data collection to run the computer-assisted TAM Battery is the 2013 Samsung Galaxy Tab 3 Model Number SM-T311 which runs on Android OS v4.4.2 (KitKat). The Display size is 8.0 inches (20.31 cm) with multi-touch capability and resolution of 1280x800. The chipset is Exynos 4214 Dual with a Dual Core 1.5 GHz Cortex A9 CPU. The device has a 5MP primary camera with autofocus capability. The device has loudspeakers with stereo quality.
Only subjects aged above 55 years were included in the study. They were included only if the Clinical Dementia Rating Scale  score was 0 and they had no previous psychiatric or neurological history. They were screened for other metabolic comorbidities and were included in the study only when a physician deemed that any such comorbidities had no effect on their mood or cognition. The included subjects had to selfreport if they have previous exposure to technology, specifically if they use smartphones or worked on computers and if yes, in what frequency.
The TAM battery was initially built to have a digital capture format for the Figural Memory Test which was included in the battery as a test of non-verbal memory. The test has four abstract line drawings (Figures 1 & 2) which are shown sequentially to the testee and he or she must copy the figure (Copy). Following which, the testee must draw the copied figure from memory immediately (Immediate Recall). After 20 min, the testee is asked to draw the figures from memory (Delayed Recall). Care was taken that there were no non-verbal memory tests administered in the intervening duration. There is a recognition trial in which the testee must choose the figure they saw from a group of four similar figures which had subtle differences from the original figure. This trial has four items. The copy subtest has no score. Care was taken that the testees had an errorless learning experience in which the testee was asked to specifically see certain parts of the figures which were prone to be missed or learnt wrong e.g. extension of lines, the side to which a figure is facing etc. For the Immediate Recall and the Delayed Recall subtests there is a set of criteria by which individual figures were scored and then the score is summed to give the subtest score. A Figural Memory retention score is also calculated by subtracting the Delayed Recall Score from the Immediate Recall Score. In the recognition subtest, each of the correct responses carries a score of 1. In the first version of the TAM Battery the testees had to draw all the figures for each of the subtests including Copy on the touchscreen. It was noted that the scores of the testees showed high discrepancy from their everyday cognitive functioning and when compared against other test scores, that is, their test performance implied a lower level of cognitive functioning from their everyday cognitive functioning. Hence, in the second version of the TAM Battery, the testees were required to draw on A4 sheets of blank white paper instead of on the touchscreen display using a pen or a pencil. Care was taken so that once drawn, the figures will not be visible to the testee again. The same testees in the pilot group were tested after a gap of 6 months to avoid any practice effects on the scores. For both versions of the test, the figures were shown to the testees on the same computer tablet device.
16 subjects were included in the study, 8 of whom were male. The mean age of the subjects was 69 years, the oldest was 83 years old and the youngest was 56 years old. The mean number of years of formal education was 13 years, with maximum number of years being 22 and the minimum was 5 years. Only 4 of the subjects had no or minimal exposure to technology while the others used smartphones or computers. Statistical analyses of the data were done. Test of significance was done using paired two tailed t-test at p<0.05 and Pearson’s correlation coefficient was calculated. There were no significant gender differences found in the test performance in both versions of the test. It was found that the number of years of formal education did not significantly affect test performance on the Figural Memory Test Digital as well as the Paper and Pencil version.
Test performance on the Figural Memory Test Digital as well as paper and pencil version was significantly affected by the age of the testee. In the immediate recall subtest and the delayed recall subtest it is noted that age significantly affected the score for each of the test items in the digital version while only the total score for the paper and pencil version. In delayed recall, for figure alone, age significantly affected the score in both digital and paper and pencil versions. Age did not affect the testee performance on the Recognition subtest.
Pearson’s correlation coefficient when performance on immediate recall was compared against exposure to technology was 0.5523 showing a moderate positive correlation. Pearson’s correlation coefficient when performance on Delayed recall was compared against exposure to technology was 0.503 showing a moderate positive correlation. When the test performance in the immediate recall (Figure 1) and delayed recall (Figure 2) subtests in the digital capture version was plotted against the testees’ previous exposure to technology in a box and whisker plot those testees who were previously exposed to technology were found to have performed better on both subtests in the digital capture version.
When the performance on the figural memory test digital capture version was compared with the testees’ performance on the paper and pencil version it was found that except for the retention score and the figure 1 immediate recall score, all other scores were significantly better on the paper and pencil version including the recognition trial scores. Thus, it can be confirmed that the testees performed significantly better on the paper and pencil version than on the digital capture version of the test.
The findings and observations made by the authors in the describe study were made by chance. It must be noted that if the tester is not well-trained, does not have adequate information on the background of the testee and their performance on the other tests, such observations can be easily missed. This is particularly true in settings with high patient load who rely on trainees and inexperienced psychometrists and when multiple testers administer and score the tests. It is also of significance since remote assessment of patients is on the rise in the field of cognitive testing in which the testers, clinicians and scorers have minimal information on the qualitative aspects of test performance.
Care was taken on both the versions for errorless learning in the first subtest but the ability to learn, retain and recall the figures was affected. It is hypothesized that the novelty of using the digital capture method to draw might have affected the learning process of the testees which in turn affected the score on immediate recall and subsequently the delayed recall. There was no such significant effect on Figure 1 which was the easiest of the four figures and hence may have been immune to such effects of novelty on learning. It is of special note that even the scores on the recognition trials were significantly affected which maybe because learning did not happen. Retention scores did not show any drastic effect which shows that the memory of the testees was not impaired. Thus, though these results are from a small cohort of testees who participated in the pilot of the TAM Battery normative data collection, from these findings it can be concluded that the method of test administration influenced the performance of the test subjects on the non-verbal memory test. There are test items which are immune to the effects of such differences in test administration and if necessary the test must be developed or adapted with care by including only test items that are immune to such effects.
There is undoubtedly a pressing need for innovative and ecologically valid cognitive tests, but these requirements do not outweigh the necessity for measures that have sound psychometric properties. Therefore, to avoid the risk of diagnostic errors which may have significant impact on the management of the diseases, it is recommended that clinicians and researchers make informed decisions about using Computerized Neuropsychological Assessment Devices based on their suitability for their clients and their individual requirements.
Sample size could be larger to study the performance of older people in a test of non-verbal memory using digital capture method and paper and pencil method.
- Bauer RM, Iverson GL, Cernich AN, et al. Computerized neuropsychological assessment devices: Joint position paper of the American Academy of Clinical Neuropsychology and the National Academy of Neuropsychology. Arch Clin Neuropsychol. 2012;27:362-373.
- Feldstein SN, Keller FR, Protman RE, et al. A comparison of computerized and standard version of the Wisconsin Card Sorting Test. Clin Neuropsychol. 1999;13:303-313.
- Gaggioli A (2012) Quality of experience in real and virtual environments: Some suggestions for the development of positive technologies. Stud Health Technol Inform. 2012;181:177-181.
- Morris JC. The clinical dementia rating (CDR): Current version and scoring rules. Neurology. 1993;43:2412-2414.
- Parsey CM, Schmitter-Edgecombe M. Applications of technology in neuropsychological assessment. The Clinical Neuropsychologist. 2013;27:1328-1361.
- Parsons TD. Clinical neuropsychology and technology. Switzerland: Springer International Publishing. 2016.
- Tun PA, Lachman ME. The association between computer use and cognition across adulthood: Use it so you won’t lose it? Psychol Aging. 2010;25:560-568.