Subjects: Psychology >> Psychological Measurement submitted time 2019-09-16
Abstract： Cognitive diagnostic assessments (CDAs) can provide fine-grained diagnostic information about students' knowledge states, so as to help to teach in accordance with the students’ aptitude. The development of cognitive diagnosis model for polytomous response data expands the application scope of cognitive diagnostic assessment. As the basis of CDAs, Q-matrix has aroused more and more attention for the subjective tendency in Q-matrix construction that is typically performed by domain experts. Due to the subjective process of Q-matrix construction, there inevitably have some misspecifications in the Q-matrix, if left unchecked, can result in a serious negative impact on CDAs. To avoid the subjective tendency from experts and to improve the correctness of the Q-matrix, several objective Q-matrix validation methods have been proposed. Many Q-matrix validation methods have been proposed in dichotomous CDMs, however, the research of the Q-matrix validation method under polytomous CDMs is stalling lacking. To address this concern, several relative fit statistics (i.e., -2LL, AIC, BIC) were applied to the Q-matrix validation for polytomous cognitive diagnosis model in this research. The process of Q-matrix validation is as follows: First, the reduced Q-matrix is represented by , which represents a set of potential q-vectors and contains possible q-vectors when attributes are independent. When validating the q-vector of the first category of item j, all possible q-vectors in can be used as the q-vector of the first category of item j, and the Q-matrix of remaining items remains intact. From this, the item parameters and the attribute patterns of students can be estimated, and the -2LL, AIC, and BIC can be calculated accordingly. The q-vector with the largest likelihood (or smallest AIC/BIC) is regarded as the q-vector of the first category of item j. The q-vector of the next category of the item j can also be obtained in the same way. The algorithm stops when the validated Q-matrix is same as the previous Q-matrix, or every item has been reached. In order to improve the efficiency of the method, a sequential search algorithm was proposed. Several simulation studies were conducted to evaluate the effectiveness and practicality of these methods, and the performance of the methods in this paper was compared with the stepwise method (Ma & de la Torre, 2019). Three experimental factors were considered in simulation studies, including sample size, Q-matrix error types and CDMs. The results show that (1) BIC method can be used for Q-matrix validation under polytomous response CDMs, and the performance of the BIC method is better than the stepwise method. (2) In general, the performance of the three methods from good to bad is the BIC method, AIC method, and -2LL method. (3) The performance of Q-matrix validation methods is affected by the sample size, and increasing the number of sample size can improve the accuracy of the Q-matrix validation. In this study, Q-matrix validation methods for polytomous response CDMs were studied. It was found that the BIC method can be used for the Q-matrix validation under polytomous response CDMs. The method proposed in this paper can not only improve the accuracy of Q-matrix specification but also increase the model-data fit level. Besides, the data-based Q-matrix validation method can also reduce the workload of experts in Q-matrix construction and improve the classification accuracy of cognitive diagnosis. " " " " "
Subjects: Psychology >> Social Psychology submitted time 2023-03-27 Cooperative journals: 《心理学报》
Abstract： Currently, a large number of cognitive diagnosis models (CDMs) have been proposed to satisfy the demands of the cognitively diagnostic assessment. However, most existing CDMs are only suitable for dichotomously scored items. In practice, there are lager polytomously-score items/data in educational and psychological tests. Therefore, it is very necessary to develop CDMs for polytomous data. Under the item response theory (IRT) framework, the polytomous models can be divided into three categories: (i) the cumulative probability (or graded-response) models, (ii) continuation ratios (or sequential) models, and (iii) the adjacent-category (or partial-credit) models. At present, several efforts have been made to develop polytomous partial-credit CDMs, including the general diagnostic model (GDM; von Davier, 2008) and the partial credit DINA (PC-DINA; de la Torre, 2012) model. However, the existing polytomous partial-credit CDMs need to be improved in the following aspects: (1) These CDMs do not consider the relationship between attributes and response categories by assuming that all response categories of an item measure the same attributes. This may result in loss of diagnostic information, because different response categories could measure different attributes; (2) More importantly, the PC-DINA is based on reduced DINA model. Therefore, the current polytomous CDMs are established under strong assumptions and do not have the advantages of general cognitive diagnosis model.The current article proposes a general partial credit diagnostic model (GPCDM) for polytomous responses with less restrictive assumptions. Item parameters of the proposed models can be estimated using the marginal maximum likelihood estimation approach via Expectation Maximization (MMLE/EM) algorithm.Study 1 aims to examine (1) whether the EM algorithm can accurately estimate the parameters of the proposed models, and (2) whether using item level Q-matrix (referred to as the Item-Q) to analyze data generated by category level Q-matrix (referred to as the Cat-Q) will reduce the accuracy of parameter estimation. Results showed that when using Cat-Q fitting data, the maximum RMSE was less than 0.05. When the number of attributes was equal to 5 or 7, the minimum pattern match rate (PMR) was 0.9 and 0.8, respectively. These results indicated that item and person parameters could be recovered accurately based on the proposed estimation algorithm. In addition, the results also showed that when Item-Q is used to fit the data generated by Cat-Q, the estimation accuracy of both the item and person parameters could be reduced. Therefore, it is suggested that when constructing the polytomously-scored items for cognitively diagnostic assessment, the item writer should try to identify the association between attributes and categories. In the process, more diagnostic information may be extracted, which in turn helps improve the diagnostic accuracy.The purpose of Study 2 is to apply the proposed model to the TIMSS (2007) fourth-grade mathematics assessment test to demonstrate its application and feasibility and compare with the exiting GDM and PC-DINA model. The results showed that compared with GDM and PC-DINA models, the new model had a better model fit of test-level, higher attribute reliability and better diagnostic effect.
Hits
Hits
Downloads
Comment