Publikationen
The Potential of Anchoring Vignettes to Increase Intercultural Comparability of Non-Cognitive Factors
Marksteiner, Tamara/Kuger, Susanne/Klieme, Eckhard (2018):
The Potential of Anchoring Vignettes to Increase Intercultural Comparability of Non-Cognitive Factors.
In: Assessment in Education: Principles, Policy & Practice
Jahrg.: 26, H. 4, S. 516-536
The aim of the present study was to investigate whether Anchoring Vignettes (AV) actually improve intercultural comparability of non-cognitive student-directed factors like procrastination or planning and organization. In the past, international large-scale studies have incorporated AV in order to reduce culture-specific response biases. So far, correlation analyses for anchored and non-anchored scores with a criterion have been used to demonstrate the effectiveness of AV in improving data quality. However, correlation analyses are often used to investigate external validity of a scale. Nonetheless, before testing for validity, the reliability of the measurement of a construct should be examined, especially in cross-cultural research. Reliability in international data can be evaluated by confirming scale dimensionality and internal consistency as well as invariance of measurement in different groups or languages of assessment. In the present study, we tested for measurement invariance across countries and languages and compared anchored and non-anchored student-directed self-reports that are highly relevant for the students’ self and their behavior and performance. In addition, we apply further criteria for testing reliability. The results show inconclusive results across different scales. Results of exploratory and confirmatory factor analyses indicate that the data quality for some of the constructs can—in fact—be improved slightly by anchoring, whereas for other self-reports, anchoring is less successful than was hoped. The results are discussed with regard to possible consequences for research methodology.
The Potential of Anchoring Vignettes to Increase Intercultural Comparability of Non-Cognitive Factors.
In: Assessment in Education: Principles, Policy & Practice
Jahrg.: 26, H. 4, S. 516-536
The aim of the present study was to investigate whether Anchoring Vignettes (AV) actually improve intercultural comparability of non-cognitive student-directed factors like procrastination or planning and organization. In the past, international large-scale studies have incorporated AV in order to reduce culture-specific response biases. So far, correlation analyses for anchored and non-anchored scores with a criterion have been used to demonstrate the effectiveness of AV in improving data quality. However, correlation analyses are often used to investigate external validity of a scale. Nonetheless, before testing for validity, the reliability of the measurement of a construct should be examined, especially in cross-cultural research. Reliability in international data can be evaluated by confirming scale dimensionality and internal consistency as well as invariance of measurement in different groups or languages of assessment. In the present study, we tested for measurement invariance across countries and languages and compared anchored and non-anchored student-directed self-reports that are highly relevant for the students’ self and their behavior and performance. In addition, we apply further criteria for testing reliability. The results show inconclusive results across different scales. Results of exploratory and confirmatory factor analyses indicate that the data quality for some of the constructs can—in fact—be improved slightly by anchoring, whereas for other self-reports, anchoring is less successful than was hoped. The results are discussed with regard to possible consequences for research methodology.
