Subjects: Psychology >> Psychological Measurement submitted time 2024-04-27
Abstract： Process data encompasses the human-computer interaction data captured in computer-based learning and assessment systems, reflecting participants’ problem-solving processes. Among various types of process data, action sequences stand out as a quintessential type, delineating participants’ step-by-step problem-solving processes. However, the non-standardized format of action sequences, characterized by varying data lengths among participants, presents challenges for the direct application of traditional psychometric models like diagnostic classification models (DCM). Extending psychometric models applicable to standardized structured data to process data analysis often necessitates key-action encoding – determining if each participant’s data contains essential problem-solving actions and encoding them (e.g., “1” for contains and “0” for does not contain ). Zhan and Qiao (2022) proposed a key-action encoding method facilitating the application of DCM to process data analysis for identifying participants’ mastery of problem-solving skills. Nevertheless, their approach overlooks the adverse impact of misconceptions on problem-solving. To this end, this study introduces a key-action encoding approach incorporating misconceptions and explores its utility in diagnostic classification analysis of process data. This new encoding method integrates both problem-solving skills and misconceptions, extending Zhan and Qiao’s (2022) approach.
An illustrative example is provided to compare the performance of the proposed encoding approach with Zhan and Qiao’s (2022) approach using a real-world interactive assessment item, “Tickets,” from PISA 2012. For the proposed approach, eight attributes (four problem-solving skills and four misconceptions) and 28 phantom items (i.e., key actions) were defined based on the scoring rule and assessment framework of the interactive assessment item. In contrast, Zhan and Qiao’s approach defined four attributes (problem-solving skills) and 10 phantom items. Four DCMs – DINA, DINO, ACDM, and GDINA models – were employed for data analysis. The relative fit metrics for model-data comparison were selected from AIC, BIC, CAIC, and SABIC. Additionally, a chi-square test was employed to evaluate whether there existed a significant difference in the fit to the data between GDINA and each of the constrained models. For assessing absolute fit between the model and the data, the SRMSR metric was utilized. Moreover, item quality was evaluated using the item differentiation index (IDI), while classification reliability was determined by calculating the classification accuracy index.
The findings reveal that: (1) considering both problem-solving skills and misconceptions enables more nuanced participant classification, facilitating identification of specific factors influencing problem-solving success and failure and offering targeted remedial suggestions for personalized instruction; (2) the introduction of misconceptions slightly enhances diagnostic classification reliability; (3) a moderate-to-high negative correlation exists between participants’ mastery of misconceptions and raw scores, indicating misconceptions diminish students’ overall problem-solving performance.
In summary, this study proposes a key-action encoding approach incorporating misconceptions and explores its application in diagnostic classification analysis of process data, specifically action sequences. The proposed approach aids researchers in pinpointing specific factors influencing problem-solving outcomes and provides methodological support for targeted interventions. To enhance participants’ problem-solving performance, beyond improving their skills, addressing misconceptions’ adverse effects merits consideration.
Peer Review Status:
Awaiting Review
Subjects: Other Disciplines >> Synthetic discipline submitted time 2023-10-09 Cooperative journals: 《心理学报》
Abstract： Process data refers to the human-computer or human-human interaction data recorded in computerized learning and assessment systems that reflect respondents’ problem-solving processes. Among the process data, action sequences are the most typical data because they reflect how respondents solve the problem step by step. However, the non-standardized format of action sequences (i.e., different data lengths for different participants) also poses difficulties for the direct application of traditional psychometric models. Han et al. (2021) proposed the SRM by combining dynamic Bayesian networks with the nominal response model (NRM) to address the shortcomings of existing methods. Similar to the NRM, the SRM uses multinomial logistic modeling, which in turn assigns different parameters to each possible action or state transition in the task, leading to high model complexity. Given that actions or state transitions in problem-solving tasks have correct and incorrect outcomes rather than equivalence relations without quantitative order, this paper proposes two action sequence models based on binary logistic modeling with relatively low model complexity: the one- and two-parameter action sequence models (1P and 2P-ASM). Unlike the SRM, which applies the NRM migration to action sequence analysis, the 1P-ASM and 2P-ASM migrate the simpler one- and two-parameter IRT models to action sequence analysis, respectively. An illustrated example was provided to compare the performance of SRM and two ASMs with a real-world interactive assessment item, “Tickets,” in the PISA 2012. The results mainly showed that: (1) the latent ability estimates of two ASMs and the SRM had high correlation; (2) ASMs took less computing time than that of SRM; (3) participants who are solving the problem correctly tend to continue to present the correct actions, and vice versa; and (4) compared with the fixed discrimination parameter of the SRM, the free estimated discrimination parameter of the 2P-ASM helped us to better understand the task. A simulation study was further designed to explore the psychometric performance of the proposed model in different test scenarios. Two factors were manipulated: sample size (including 100, 200, and 500) and average problem state transition sequence length (including short and long). The SRM was used to generate the state transition sequences in the simulation study. The problem-solving task structure from the empirical study was used. The results showed that: (1) two ASMs could provide accurate parameter estimates even if they were not the data-generation model; (2) the computation time of both ASMs was lower than that of SRM, especially under the condition of a small sample size; (3) the problem-solving ability estimates of both ASMs were in high agreement with the problem-solving ability estimate of the SRM, and the agreement between 2P-ASM and SRM is relatively higher; and (4) the longer the problem state transition sequence, the better the recovery of problem-solving ability parameter for both ASMs and SRM. Overall, the two ASMs proposed in this paper based on binary logistic modeling can achieve effective analysis of action sequences and provide almost identical estimates of participants' problem-solving ability to SRM while significantly reducing the computational time. Meanwhile, combining the results of simulation and empirical studies, we believe that the 2P-ASM has better overall performance than the 1P-ASM; however, the more parsimonious 1P-ASM is recommended when the sample size is small (e.g., 100 participants) or the task is simple (fewer operations are required to solve the problem).
Subjects: Psychology >> Social Psychology submitted time 2023-03-28 Cooperative journals: 《心理科学进展》
Abstract： Problem-solving competence is an individual’s capacity to engage in cognitive processing to understand and resolve problem situations where a method of solution is not immediately obvious. The measurement of problem-solving competence requires the use of relatively more complex and real problem situations to induce the presentation of problem-solving behaviors. This brings challenges to both the measurement methods of problem-solving competence and the corresponding data analysis methods. Using virtual assessments to capture the process data in problem-solving and mining the potential information contained therein is a new trend in measuring problem-solving competence in psychometrics. Firstly, this paper reviews the development of measurement methods from pen-and-paper tests to virtual assessments. Compared with the traditional paper-and-pencil test, modern virtual assessments are not only conducive to simulating real problem situations, improving the ecological validity of the test, but also can record the process data generated by individuals in the process of problem-solving. Process data refers to man-machine or man-human interaction data with timestamps that can reflect the process of individual problem-solving. It records the detailed steps of individual problem solving and reflects the strategy and cognitive process of individual problem-solving. However, it is not easy to adopt effective methods to analyze process data. Secondly, two methods of analyzing process data are summarized and compared: data mining methods and statistical modeling methods. Data mining is the process of using algorithms to uncover new relationships, trends, and patterns from big data. It is a bottom-up, data-driven research method that focuses on describing and summarizing data. Its advantage is that it can use existing algorithms to analyze a variety of process data at the same time, screen out variables related to individual problem-solving competence, and realize the classification of individual problem-solving competence. But sometimes, different algorithms could get different conclusions based on the same data, which leads to part of the results can not be explained. This method can not construct variables that can reflect the individual's latent trait, either. Statistical modeling method mainly refers to the method of analyzing data by using the idea of artificial modeling. It is a top-down, theory-driven approach. In statistical modeling, function models are generally constructed based on theoretical assumptions, and the observed variables are assumed to be randomly generated by the probability law expressed by the model. For the data recorded by virtual assessments, the existing modeling methods can be divided into three categories: psychometric joint modeling, hidden Markov modeling, and multi-level modeling. The main advantage of statistical modeling is that its results are easy to interpret and conform to the general process of psychological and educational research. Its limitation lies in that the modeling logic has not been unified yet because different types of process data need to be modeled separately. However, by giving full play to the advantages of the two data analysis methods, different problems in psychological and educational assessments can be dealt with. The interpretability of the results is very important in psychological and educational measurements, which determines the dominant role of statistical modeling in process data analysis. Finally, the possible future research directions are proposed from five aspects: the influence of non-cognitive factors, the use of multimodal data, the measurements of the development of problem-solving competence, the measurements of other higher-order thinking competence, and the definition of the concept and structure of problem-solving competence.
Subjects: Psychology >> Psychological Measurement submitted time 2023-01-05
Abstract： Process data refers to the human-computer or human-human interaction data recorded in computerized learning and assessment systems that reflect respondents’ problem-solving processes. Among the process data, action sequences are the most typical data because they reflect how respondents solve the problem step by step. However, the non-standardized format of action sequences (i.e., different data lengths for different participants) also poses difficulties for the direct application of traditional psychometric models. Han et al. (2021) proposed the SRM by combining dynamic Bayesian networks with the nominal response model (NRM) to address the shortcomings of existing methods. Similar to the NRM, the SRM uses multinomial logistic modeling, which in turn assigns different parameters to each possible action sequence in the task, leading to high model complexity. Given that action sequences in problem-solving tasks have correct and incorrect outcomes rather than equivalence relations without quantitative order, this paper proposes two action sequence models based on binary logistic modeling with relatively low model complexity: the one- and two-parameter action sequence models (1P and 2P-ASM). Unlike the SRM, which applies the NRM migration to action sequence analysis, the 1P-ASM and 2P-ASM migrate the simpler one- and two-parameter IRT models to action sequence analysis, respectively. An illustrated example was provided to compare the performance of SRM and two ASMs with a real-world interactive assessment item, “Tickets,” in the PISA 2012. The results mainly showed that: (1) the latent ability estimates of two ASMs and the SRM had high correlation; (2) ASMs took less computing time than that of SRM; (3) participants who are solving the problem correctly tend to continue to present the correct action sequences, and vice versa; and (4) compared with the fixed discrimination parameter of the SRM, the free estimated discrimination parameter of the 2P-ASM helped us to better understand the task. A simulation study was further designed to explore the psychometric performance of the proposed model in different test scenarios. Two factors were manipulated: sample size (including 100, 200, and 500) and average problem state transition sequence length (including short and long). The SRM was used to generate the state transition sequences in the simulation study. The problem-solving task structure from the empirical study was used. The results showed that: (1) two ASMs could provide accurate parameter estimates even if they were not the data-generation model; (2) the computation time of both ASMs was lower than that of SRM, especially under the condition of a small sample size; (3) the problem-solving ability estimates of both ASMs were in high agreement with the problem-solving ability estimate of the SRM, and the agreement between 2P-ASM and SRM is relatively higher; and (4) the longer the problem state transition sequence, the better the recovery of problem solving ability parameter for both ASMs and SRM. Overall, the two ASMs proposed in this paper based on binary logistic modeling can achieve effective 6 analysis of action sequences and provide almost identical estimates of participants' problem-solving ability to SRM while significantly reducing the computational time. Meanwhile, combining the results of simulation and empirical studies, we believe that the 2P-ASM has better overall performance than the 1P-ASM; however, the more parsimonious 1P-ASM is recommended when the sample size is small (e.g., 100 participants) or the task is simple (fewer operations are required to solve the problem).
Peer Review Status:Awaiting Review
Subjects: Psychology >> Psychological Measurement submitted time 2022-10-06
Abstract： Longitudinal cognitive diagnostics can assess students' strengths and weaknesses over time, profile students' developmental trajectories, and can be used to evaluate the effectiveness of teaching methods and optimize the teaching process.Existing researchers have proposed different longitudinal diagnostic classification models, which provide methodological support for the analysis of longitudinal cognitive diagnostic data. Although these parametric longitudinal cognitive diagnostic models can effectively assess students' growth trajectories, their requirements for coding ability and sample size hinder their application among frontline educators, and they are time-consuming and not conducive to providing timely feedback. On the one hand, the nonparametric approach is easy to calculate, efficient to apply, and provides timely feedback; on the other hand, it is free from the dependence on sample size and is particularly suitable for analyzing assessment data at the classroom or school level. Therefore, this paper proposed a longitudinal nonparametric approach to track changes in student attribute mastery. This study extended the longitudinal Hamming distance discriminant (Long-HDD) based on the Hamming distance discriminant (HDD), which uses the Hamming distance to represent the dependence between attribute mastery patterns of the same student at adjacent time points. To explore the performance of Long-HDD in longitudinal cognitive diagnostic data, we conducted a simulation study and an empirical study and compared the classification accuracy of the HDD, Long-HDD, and Long-DINA models. In the simulation study, five independent variables were manipulated, including (1) sample sizes N = 25, 50, 100, and 300; (2) number of items I = 25 and 50; (3) number of time points T = 2 and 3; (4) number of attributes measured at each time point K = 3 and 5, and (5) data analysis methods M = HDD, Long-HDD, and Long-DINA. The student’s real attribute mastery patterns were randomly selected with equal probability from all possible attribute patterns, and the transfer probabilities among attributes between adjacent time points were set to be equal (e.g., p(0→0) = 0.8, p(0→1) = 0.2, p(1→0) = 0.05, p(1→1) = 0.95), while the first K items constituting the unit matrix in the Q-matrix at each time point were set to be anchor items, and the item parameters were set to be moderately negative correlation, generated by a ?bivariate normal distribution. For the empirical study, the results of three parallel tests with 18 questions each, measuring six attributes, were used for 90 7th graders. The Q-matrix for each test was equal. The results of the simulation study showed that (1) Long-HDD had higher classification accuracy in longitudinal diagnostic data analysis; (2) Long-HDD performed almost independently of sample size and performed better with a smaller sample size compared to Long-DINA; and (3) Long-HDD consumed much less computational time than Long-DINA. In addition, the results of the empirical data also showed that there was good consistency between the results of the Long-HDD and the Long-DINA model?in tracking changes in attribute development. The percentage of mastery of each attribute increased with the increase of time points. In summary, the long-HDD proposed in this study extends the application of nonparametric methods to longitudinal cognitive diagnostic data and can provide high classification accuracy. Compared with parameterized longitudinal DCM (e.g., Long-DINA), it can provide timely diagnostic feedback due to the fact that it is not affected by sample size, simple calculation, and less time-consuming. It is more suitable for small-scale longitudinal assessments such as class and school level. " "
Subjects: Psychology >> Psychological Measurement submitted time 2022-05-03
Abstract：
In psychological, educational, and behavioral studies, measuring change over time is essential to developmental study. These changes can sometimes be captured by longitudinal latent variable models, such as longitudinal item response theory models and latent growth curve models. With the spread of computerized (or web-based) assessments, it has become common to collect process data such as item response time (RT) in addition to traditional item response accuracy (RA) data. RT data is used as a complement to RA data, describes the total time taken by individuals to solve problems and can be used to analyze the latent processing speed of individuals. However, a review of the existing studies reveals that existing longitudinal models focus on longitudinal RA data and lack attention to longitudinal RT data; Moreover, most of the existing RT models are limited to analyzing cross-sectional RT data and cannot track the development of students' latent processing speed over time. To this end, four longitudinal RT models based on two commonly used longitudinal modeling methods (i.e., multivariate normal distribution modeling and latent growth curve modeling) were proposed to achieve objective tracking of individual potential processing speed development and enrich the analysis methods of longitudinal RT data.
Based on the most commonly used cross-sectional RT model, the lognormal RT model (LRTM), four longitudinal RT models were proposed, including the multivariate normal distribution-based LRTM (denoted as MVN-LRTM) and its constraint model with the Markov property (denoted as MVN-LRTM-M), the linear latent growth curve-based LRTM (denoted as LGC-LRTM-L), and the nonlinear latent growth curve-based LRTM (denoted as LGC-LRTM-N). The measurement models are consistent across the four models, with differences mainly in the structural model describing how the latent processing speed changes over time. First, an adaptive learning/assessment dataset about spatial rotation ability was used as an empirical example to show the practical applicability of the proposed models. Second, two simulation studies were conducted further to explore the psychometric performance of the proposed models. The purpose of simulation study 1 was to explore the recovery of parameter estimation under different simulated conditions. The purpose of simulation study 2 was to explore the tolerance of the proposed models to different proportions of missing RT data.
The results of the empirical study mainly indicated that all four longitudinal RT models are practically applicable and have high consistency in the analysis results for the same cohort of data. The results of simulation study 1 showed that the parameters of the proposed models can be well recovered under various simulated conditions. The results of simulation study 2 mainly indicated that the proposed models are tolerant to different proportions of missing RT data, and it was suggested that the proportion of missing RT data should be controlled below 60% in practical applications.
Overall, the four longitudinal RT models proposed in this paper have practical applicability and good psychometric performance, which enriches the analysis of longitudinal RT data in psychological and educational assessments.
Peer Review Status:Awaiting Review
Subjects: Psychology >> Psychological Measurement submitted time 2021-10-04
Abstract： Problem-solving competence is an individual’s capacity to engage in cognitive processing to understand and resolve problem situations where a method of solution is not immediately obvious. The measurement of problem-solving competence requires the use of relatively more complex and real problem situations to induce the presentation of problem-solving behaviors. This brings challenges to both the measurement methods of problem-solving competence and the corresponding data analysis methods. Using virtual assessments to capture the process data in problem-solving and mining the potential information contained therein is a new trend in measuring problem-solving competence in psychometrics. To begin with, we reviewed the development of the measurement methods of problem-solving competence: from paper-and-pencil tests to virtual assessments. In addition, we summarized two types of process data analysis methods: data mining and statistical modeling. Finally, we look forward to possible future research directions from five perspectives: the influence of non-cognitive factors on problem-solving competence, the use of multimodal data to measure problem-solving competence, the measurement of the development of problem-solving competence, the measurement of other higher-order thinking competencies, and the definition of concept and structure of problem-solving competence.
Peer Review Status:Awaiting Review
Hits
Hits
Downloads
Comment
YES