The Transfer Project https://transfer.cpc.unc.edu Fri, 22 Mar 2024 17:56:56 +0000 en-US hourly 1 https://transfer.cpc.unc.edu/wp-content/uploads/2021/04/cropped-transfer_logo_transparent-32x32.png The Transfer Project https://transfer.cpc.unc.edu 32 32 Wrong answers are useful too! Using all multiple choices to uncover hidden variation to estimate test scores from large field surveys https://transfer.cpc.unc.edu/wrong-answers-are-useful-too-using-all-multiple-choices-to-uncover-hidden-variation-to-estimate-test-scores-from-large-field-surveys/?utm_source=rss&utm_medium=rss&utm_campaign=wrong-answers-are-useful-too-using-all-multiple-choices-to-uncover-hidden-variation-to-estimate-test-scores-from-large-field-surveys Thu, 21 Mar 2024 20:18:35 +0000 https://transfer.cpc.unc.edu/?p=4053 Continued]]> by Hannah Silverstein

Tests and assessments measure a variety of outcomes including skills, ability, performance, and knowledge. Cognitive-type tests have become quite common in many large scale surveys. However, these multi-topic surveys often have limited space, and can only include a short number of items. In our new paper, “Large Field Surveys and Short Items Sets: Estimating Reasoning Skills among Distinct Samples in Malawi and Zambia” recently published in Journal of Methods and Measurement in the Social Sciences, we investigate how to uncover hidden variation in response patterns of correct and incorrect answers to multiple choice questions measuring logical reasoning.

 

Motivation:

Logical reasoning captures a set of non-academic cognitive skills and can be measured via tests that involve series of incomplete images, called Ravens Progressive Matrices (RPMs).  For each RPM item, respondents are given multiple choices from which they choose the option that best completes the image. The original RPM sets include dozens of items, however many surveys such as the Mexican Family Life Survey; Indonesia Family Life Survey; and HEalth, Ageing and Retirement Transitions study in Sweden use short sets of RPMs. Transfer Project evaluations of the Malawi Social Cash Transfer Program and the Zambia Child Grant both included eight RPMs in the questionnaires. Using evaluation data from both of these studies, we analyzed responses from youth in Malawi and caregivers in Zambia to RPM questions.

Example RPM from Malawi and Zambia short sets

 

The problem:

We wanted to generate scores for each individual respondent. Initially, we summed the total number of items answered correctly. We quickly realized that this approach had limitations  because the RPMs selected for the surveys were too difficult. In both studies about 50% of the sample correctly answered one or zero questions–in other words, our summed score distribution was highly skewed.

Distribution of summed scores among youth in Malawi

After reviewing the items and multiple choices, we also realized that some incorrect options resembled the correct choice, while others choices were very different. For instance, in the example above, D is the correct choice while choice B, though incorrect, is just a rotated image of choice D. On the other hand, answer choice C does not share visual properties with the correct answer. Therefore, some wrong choices appear to be more ‘right’ than others—picking one of these ‘almost’ correct choices ought to be considered when estimating overall test scores.

And finally, questions were not equally difficult. In general, the first RPMs in the series were much easier than the final items, suggesting that respondents correctly answering difficult items should probably have higher scores than those answering the same number of easy items.

So, the bottom line was, categorizing RPM responses into ‘correct’ and ‘incorrect’ removed a large amount of information and we wanted to find a way to use this information to estimate more accurate scores.

 

The solution:

Researchers in psychology frequently use item response theory (IRT) to estimate test scores from multiple choices and questions of different difficulties.  There are several different types of IRT, but we explored two types.[i] The first IRT method (IRT1) we used still categorized responses as either ‘correct’ or ‘incorrect’, but allowed for more difficult questions to have a higher weight in the overall score. The majority of scores estimated using IRT1 were still concentrated at the low end of the spectrum.

IRT1: Distribution of scores among youth in Malawi

[i] IRT1 scores were estimated using two-parameter logistic IRT. IRT2 scores were estimated using nominal response model IRT.

wanted to see if the distribution of the scores improved even more when using information in all possible multiple choices. The second IRT method (IRT2) we used considered all multiple choice options, while still giving more difficult questions a higher weight in the overall score. We used IRT2 to identify the best ordering for multiple choice options and to calculate overall test scores. As a result, the distribution of test scores from IRT2 were much improved—the majority of respondents had scores falling near the middle of the range of possible scores.

IRT2: Distribution of scores among youth in Malawi

Testing validity:

At this point, we had three different types of logical reasoning scores: summed, IRT1, and IRT2 scores. We needed to know if the improvements in the distribution of IRT2 scores were meaningful. So we next used all three types of scores in regression models, as outcomes and as predictors. We found our improved IRT2 scores could capture more nuance when relationships were less strong compared to the other score types. As an outcome, these scores were predicted by lower levels of education than in both countries. Among youth in Malawi, IRT2 scores were more sensitive to risk preferences. These improved scores also had more consistent relationships with future preferences among caregivers in Zambia as well.

Applicability and considerations:

As mentioned, this approach can be used to estimate scores for large datasets using RPMs. However, IRT can be used to estimate scores for many other types of multiple choice assessments types too. There are several strengths to using the 2nd IRT method to estimate scores:

  • Sample response patterns inform the ordering of the multiple choice options. Roughly speaking, the most frequent incorrect response is automatically given more weight relative to a less frequently selected wrong answer. This is preferred to having the researcher decide which incorrect response is the closest to the correct response (though we tried this approach as well!).
  • IRT2 scores are standardized to the sample and population under investigation not an external population different from the one being studied. Even with full RPM sets, researchers often use British norms for estimating scores. This method reduces the cultural bias from which the test was created without fundamentally changing the test itself—this is important for research in low- and middle-income countries where population level standardized norms usually do not exist. .
  • Estimation in this way can be applied to challenging datasets, such as highly skewed summed scores or incomplete sets of items.

There are several conditions for this 2nd IRT approach to successfully uncover meaningful information among the incorrect responses.:

  • The shorter the item set, the larger the sample size needed to analyze response patterns. In our two studies, we administered 8-item sets and had sample sizes of 2,513 and 1,173 in Malawi and Zambia respectively. These large sample sizes were crucial for having enough data to detect patterns in responses within and across questions.
  • The ordering and estimation procedure is most beneficial for short item sets. IRT1 and summed scores might be sufficient approaches for estimating scores from large item sets.
  • Tests that are too easy are particularly challenging. There are limits to potential improvements in measurement for tests that are too easy because there are fewer incorrect responses to work with. Individuals with only correct answer choices all have the same response pattern. For tests that are too easy, there will be little information to be gained among high performing respondents. In this sense then, it is better to err on the side of a slightly harder set than an easier set.
[1] IRT1 scores were estimated using two-parameter logistic IRT. IRT2 scores were estimated using nominal response model IRT.

 

For more information on our process and findings, please see the full, open access article.You can find the full paper here: https://journals.librarypublishing.arizona.edu/jmmss/article/id/6047/

 

]]>
Reflections on the “Reimagining Social Protection in a Time of Global Uncertainty” Conference https://transfer.cpc.unc.edu/reflections-on-the-reimagining-social-protection-in-a-time-of-global-uncertainty-conference/?utm_source=rss&utm_medium=rss&utm_campaign=reflections-on-the-reimagining-social-protection-in-a-time-of-global-uncertainty-conference Fri, 22 Sep 2023 13:20:33 +0000 https://transfer.cpc.unc.edu/?p=3994 Patience Masi and Tia Palermo reflect on the recent “Reimagining Social Protection in a Time of Global Uncertainty” Conference hosted by the Institute for Development Studies in Brighton, England.

]]>
Using machine learning to understand who benefits the most from an intervention https://transfer.cpc.unc.edu/using-machine-learning-to-understand-who-benefits-the-most-from-an-intervention/?utm_source=rss&utm_medium=rss&utm_campaign=using-machine-learning-to-understand-who-benefits-the-most-from-an-intervention Wed, 21 Sep 2022 17:56:04 +0000 https://transfer.cpc.unc.edu/?p=3762 Continued]]> Sudhanshu Handa and Zhiyuan Liu discuss how a machine learning technique can be applied to study the impact of unconditional cash transfers on different programme participants and contribute to the development of a middle range theory on graduation out of poverty.

]]>
Ten lessons from the largest scale up of cash transfers in history https://transfer.cpc.unc.edu/ten-lessons-from-the-largest-scale-up-of-cash-transfers-in-history/?utm_source=rss&utm_medium=rss&utm_campaign=ten-lessons-from-the-largest-scale-up-of-cash-transfers-in-history Fri, 05 Aug 2022 14:06:45 +0000 https://transfer.cpc.unc.edu/?p=3735 Over the past 30 months we have been carefully tracking countries’ unprecedented social protection responses to Covid-19. But what are we learning from such wealth of experiences?

]]>
Can social protection simultaneously reduce violence against children and violence against women? https://transfer.cpc.unc.edu/can-social-protection-simultaneously-reduce-violence-against-children-and-violence-against-women/?utm_source=rss&utm_medium=rss&utm_campaign=can-social-protection-simultaneously-reduce-violence-against-children-and-violence-against-women Wed, 25 May 2022 14:54:40 +0000 https://transfer.cpc.unc.edu/?p=3710 This blog entry summarizes five studies completed in the last two years that examine impacts of cash, cash-for-work and cash plus programs on both violent discipline of children and male intimate partner violence against women (IPV).

]]>
Do Cash Grants Increase Pregnancies? Evidence from Asia and the Pacific says “No” https://transfer.cpc.unc.edu/do-cash-grants-increase-pregnancies-evidence-from-asia-and-the-pacific-says-no/?utm_source=rss&utm_medium=rss&utm_campaign=do-cash-grants-increase-pregnancies-evidence-from-asia-and-the-pacific-says-no Wed, 25 May 2022 14:53:00 +0000 https://transfer.cpc.unc.edu/?p=3707 Read about whether cash transfers increase incentives for women to have children in this blog by Transfer Project researcher Amber Peterman.

]]>
Tips for linking social protection and gender-based violence prevention and response during COVID-19 https://transfer.cpc.unc.edu/tips-for-linking-social-protection-and-gender-based-violence-prevention-and-response-during-covid-19/?utm_source=rss&utm_medium=rss&utm_campaign=tips-for-linking-social-protection-and-gender-based-violence-prevention-and-response-during-covid-19 Wed, 25 May 2022 14:50:23 +0000 https://transfer.cpc.unc.edu/?p=3704 With COVID-19 throwing a spotlight on both social protection and gender-based violence, experts explore some of the ways these topics intersect.

]]>
Beyond targeting: Making social protection work for women in Ghana https://transfer.cpc.unc.edu/beyond-targeting-making-social-protection-work-for-women-in-ghana/?utm_source=rss&utm_medium=rss&utm_campaign=beyond-targeting-making-social-protection-work-for-women-in-ghana Wed, 11 May 2022 15:29:55 +0000 https://transfer.cpc.unc.edu/?p=3700 This blog post discusses some of the evidence from Ghana’s LEAP1000 program on gender and the policy takeaways going forward.

]]>
Social protection, agri-food systems and tackling inequalities: vital for an inclusive recovery from COVID-19 https://transfer.cpc.unc.edu/social-protection-agri-food-systems-and-tackling-inequalities-vital-for-an-inclusive-recovery-from-covid-19/?utm_source=rss&utm_medium=rss&utm_campaign=social-protection-agri-food-systems-and-tackling-inequalities-vital-for-an-inclusive-recovery-from-covid-19 Tue, 12 Apr 2022 14:55:54 +0000 https://transfer.cpc.unc.edu/?p=3694 Read about the importance of maintaining the social protection measures put in place during COVID-19.

]]>
COVID-19 crisis: extending social protection to rural populations is key to recovery https://transfer.cpc.unc.edu/covid-19-crisis-extending-social-protection-to-rural-populations-is-key-to-recovery/?utm_source=rss&utm_medium=rss&utm_campaign=covid-19-crisis-extending-social-protection-to-rural-populations-is-key-to-recovery Tue, 12 Apr 2022 14:40:02 +0000 https://transfer.cpc.unc.edu/?p=3691 The FAO Director-General addresses the International Labour Organization about the importance of human-centered recovery to COVID-19.

]]>