Subjects: Psychology >> Social Psychology submitted time 2023-03-28 Cooperative journals: 《心理科学进展》
Abstract： Computerized classification testing (CCT) can adaptively classify test-takers into two or more different categories, and it has been widely used in qualifying tests and clinical psychology or medical diagnosis. As an essential part of CCT, the termination rule determines when the test is to be stopped and to which category the test-taker is ultimately classified into, directly affecting the test efficiency and classification accuracy. According to the theoretical basis of the termination rules, existing rules can be roughly divided into the likelihood ratio, Bayesian decision theory, and confidence interval rules. And their core ideas are constructing hypothesis tests, designing loss functions, and comparing the relative positions of confidence intervals, respectively. At the same time, when constructing specific termination rules, the requirement of different test scenarios (e.g., the number of categories and the number of tests’ dimensions) should also be considered. There are advantages and disadvantages to each of the three types of termination rules. Specifically, the likelihood ratio rule is based on the likelihood ratio test, with better theoretical properties. However, the method requires prior determination of the indifference interval and the type I and II error rates, introducing the impact of subjective factors. Also, it is more challenging to extend the method in complex test situations, such as multidimensional and multicategory CCT. Bayesian decision theory rules make classification decisions based on the loss function. It can dynamically optimize the decision from a more global perspective since it works backward from the final stage of the test. In addition, the variety of loss functions makes the method very flexible in form and makes it easy to be applied to different test situations. However, in practice, the flexibility will inevitably result in the uncertainty of the choice of loss function, and the inappropriate loss function may be biased. The confidence interval method is the most straightforward because of its relatively simple principle and low computational effort. However, this method is less robust and has a relatively low test efficiency. Currently, CCT is mainly applied in eligibility tests and clinical medicine questionnaires. In eligibility tests, all three types of termination rules have the potential to be widely applied. However, in practice, the principles of the likelihood ratio rule and the Bayesian decision theory rule are not easily understood by the general public, and these methods are also accompanied by the problem of over-exposure of items for their preference of cut-point based item selection methods. Therefore, the confidence interval rule, which is relatively simple in principle and has alleviated item exposure, has been widely used in existing qualifying tests. Bayesian decision theory rules are more applicable in clinical questionnaires because of their finer control over various classification losses. The following can be considered for future research on CCT termination rules. First, Bayesian decision theory rules can be improved by considering non-statistical constraints with the help of the flexibility of its loss function. Second, termination rules can be developed for multidimensional and multicategory CCT to meet more practical needs. Third, termination rules that integrate response time can be developed to improve test efficiency and classification accuracy. Fourth, it is possible to construct termination rules under the framework of machine learning.
Subjects: Psychology >> Social Psychology submitted time 2023-03-27 Cooperative journals: 《心理学报》
Abstract： Computerized classification testing (CCT) is a subset of computerized adaptive testing (CAT), and it aims to classify examinees into one of at least two possible categories that denote results such as pass/fail or non-mastery/partial mastery/mastery. Therefore, CCTs focus on increasing the accuracy of classification which is different from CATs designed for precise measurement. The termination rule is one of the key components of CCT. However, as pointed out by Nydick (2013), most CCTs (i.e., UCCTs) were designed under unidimensional item response theory (IRT), in which the unidimensionality assumption is easily violated in practice. Thus, researchers then began to construct multidimensional CCT termination rules (i.e., MCCT) based on multidimensional IRT. To date, however, these rules still have some deficiencies in terms of classification accuracy or test efficiency. Most current studies on termination rules of MCCT are based on termination rules of UCCT. In UCCTs, termination rules require setting a cut point, θ0θ0{{\theta }_{0}}, of the latent trait to calculate the statistics; and when they are extended from UCCT to MCCT, the cut point will become a classification bound curve or even a surface (i.e., g(θ)=0g(θ)=0g(\theta )=0). At this time, a question is how to convert the curve or surface into θ0θ0{{\theta }_{0}}. To this end, the projected sequential probability ratio test (P-SPRT), constrained SPRT (C-SPRT; Nydick, 2013), and multidimensional generalized likelihood ratio (M-GLR) were respectively proposed to solve the problem in different ways. Among them, P-SPRT and C-SPRT choose specific points on g(θ) as the approximate cut point, θ^0θ^0{{\hat{\theta }}_{0}}, by projecting into Euclidean space or constraining on g(θ) respectively; as for M-GLR, because the generalized likelihood ratio statistic can be calculated without a cut point, it can be directly employed in MCCT. To overcome the limitation that P-SPRT may lead to unstable results at the beginning of the test, this study proposed the Mahalanobis distance-based SPRT (Mahalanobis-SPRT). In addition, stochastic curtailment is a technique for shortening the test length by predicting whether the classification of participants will change as the test continues. This article also combined M-GLR with the stochastic curtailment and proposed M-GLR with stochastic curtailment (M-SCGLR). A full-scale simulation study was conducted to (1) compare both the Mahalanobis-SPRT and M-SCGLR with the P-SPRT, C-SPRT, M-GLR, and multidimensional stochastically curtailed SPRT (M-SCSPRT) under varying conditions; (2) compare the classification performance of the above six termination rules for participants with specific abilities to explore whether there is a significant difference in the sensitivity of various rules to classify specific participants. To achieve the first research objective, three levels of correlation between dimensions (ρ=0, 0.5, and 0.8), two item bank structures (within-item multidimensionality and between-item multidimensionality), and two kinds of classification boundary (compensatory boundary and non-compensatory boundary) were considered; to achieve the second objective, 36 specific ability points (θ1,θ2)(θ1,θ2)({{\theta }_{1}},{{\theta }_{2}}) were generated where θ1,θ2∈{−0.5,−0.3,−0.1,0.1,0.3,0.5}θ1,θ2∈{−0.5,−0.3,−0.1,0.1,0.3,0.5}{{\theta }_{1}},{{\theta }_{2}}\in \{-0.5,-0.3,-0.1,0.1,0.3,0.5\}. The results showed that: (1) when the compensatory classification function was used, the Mahalanobis-SPRT led to higher classification accuracy and similar test length to the rules without stochastic curtailment; (2) under almost all conditions, the M-SCGLR not only possessed higher precision but also maintained the short test length, compared to M-SCSPRT that also uses stochastic curtailment; (3) the six termination rules showed a consistent change in the sensitivity of the precision and test length to specific participants. To sum up, two new MCCT termination rules (Mahalanobis-SPRT and M-SCGLR) are put forward in this article. Although the simulation results are very promising, several research directions merit further investigation, such as the development of MCCT termination rules for more than two categories, and the construction of MCCT termination rules by incorporating process data like the response time.
Subjects: Psychology >> Psychological Measurement submitted time 2021-11-16
Abstract： Computerized classification testing (CCT) has been widely used in eligibility testing and clinical psychology since it can efficiently classify participants. As an essential part of CCT, the termination rule determines when the test is to be stopped and what category the participants are ultimately classified into, directly affecting the test efficiency and classification accuracy. The existing termination rules can be roughly divided into the likelihood ratio, Bayesian decision theory, and confidence interval rules. And their core ideas are constructing hypothesis tests, designing loss functions, and comparing the relative positions of confidence intervals, respectively. Based on these ideas, in different test situations, CCT termination rules have various specific forms. Future research can further extend Bayesian rules, construct rules for multicategory MCCT, integrate process data into termination rules, and build rules under the framework of machine learning. In addition, from the perspective of practical requirement, all three types of rules have the potential to be applied in the eligibility test, while the clinical questionnaire tends to choose Bayesian rules.
Peer Review Status:
Awaiting Review
Subjects: Psychology >> Psychological Measurement submitted time 2021-04-14
Abstract： "
Peer Review Status:Awaiting Review
Hits
Hits
Downloads
Comment